Recovering from out of disk space corruption

Core Blockchain
#1

So your machine running blockstack-server ran out of disk space and won’t start?

It happened to me! Here’s how I recovered (sharing in case others find this useful!):
(see https://github.com/blockstack/blockstack-explorer/issues/28).

blockstack-server wouldn’t start because it saw the pid file of another instance.

So, I tried running blockstack-server stop to make sure it wasn’t running.

There was an error relating to the fact that the pid file existed but was empty. (I don’t have the exact error - it scrolled off my terminal buffer - apologies dear reader, but it was trying to unsuccessfully cast an empty string to int when reading the pid file.)

I deleted the pid file and tried to start the server again:

(blockstack-venv) larry@blockstack-explorer:~/.blockstack-server$ rm blockstack-server.pid 
(blockstack-venv) larry@blockstack-explorer:~/.blockstack-server$ /data/blockstack-venv/bin/python2.7 /data/blockstack-venv/bin/blockstack-server start --foreground
  File "/data/blockstack-venv/bin/blockstack-server", line 42, in <module>
    run_blockstackd()
  File "/data/blockstack-venv/local/lib/python2.7/site-packages/blockstack/blockstackd.py", line 2223, in run_blockstackd
    exit_status = run_server( foreground=args.foreground, expected_snapshots=expected_snapshots, port=int(args.port) )
  File "/data/blockstack-venv/local/lib/python2.7/site-packages/blockstack/blockstackd.py", line 1776, in run_server
    running = index_blockchain(expected_snapshots=expected_snapshots)
  File "/data/blockstack-venv/local/lib/python2.7/site-packages/blockstack/blockstackd.py", line 1642, in index_blockchain
    rc = virtualchain_hooks.sync_blockchain( bt_opts, current_block, expected_snapshots=expected_snapshots, tx_filter=blockstack_tx_filter )
  File "/data/blockstack-venv/local/lib/python2.7/site-packages/blockstack/lib/nameset/virtualchain_hooks.py", line 465, in sync_blockchain
    new_db = BlockstackDB.borrow_readwrite_instance( db_filename, last_block, expected_snapshots=expected_snapshots )
  File "/data/blockstack-venv/local/lib/python2.7/site-packages/blockstack/lib/nameset/namedb.py", line 159, in borrow_readwrite_instance
    blockstack_db = BlockstackDB( db_path, DISPOSITION_RW, expected_snapshots=expected_snapshots )
  File "/data/blockstack-venv/local/lib/python2.7/site-packages/blockstack/lib/nameset/namedb.py", line 128, in __init__
    read_only=read_only )
  File "/data/blockstack-venv/local/lib/python2.7/site-packages/virtualchain/lib/indexer.py", line 179, in __init__
    traceback.print_stack()
Aborted (core dumped)
(blockstack-venv) larry@blockstack-explorer:~/.blockstack-server$

At this point, concerned that this might a bigger problem, I created a tarball backup of the .blockstack-server directory so that we could replicate the problem in the future. (Happy to provide it to you @jude at your convenience, if you’d like to see what happened!)

Next, I tried to run blockstack-server restore - hoping it would restore the database to the last known consistent state.

And it worked!

#2

Just fixed that in my local tree. It’s odd that the PID file would be unreadable, but I just noticed that we don’t fsync() the PID file (so an out-of-disk error wouldn’t get caught immediately).

At this point, concerned that this might a bigger problem, I created a tarball backup of the .blockstack-server directory so that we could replicate the problem in the future. (Happy to provide it to you @jude at your convenience, if you’d like to see what happened!)

If you have the logs, I’d like to take a look :slight_smile:

Next, I tried to run blockstack-server restore - hoping it would restore the database to the last known consistent state.

And it worked!

Woo!