Where is immutable data stored?

I stored some immutable data in the name devname0.id with the data_id “test”.

I see that the hash of the immutable data is stored in the name’s zonefile:

$ORIGIN devname0.id $TTL 3600 test IN TXT "#9724c1e20e6e3e4d7f57ed25f9d4efb006e508590d528c90da597f6a775c13e5" _http._tcp URI 10 1 "https://www.larrysalibra.com/blockstack-data/devname0.id/profile.json"

When I retrieve the data with the CLI, I’m given the value:

blockstack get_immutable devname0.id test

{
    "data": {
        "key": "value"
    }, 
    "hash": "9724c1e20e6e3e4d7f57ed25f9d4efb006e508590d528c90da597f6a775c13e5"
}

Looking at the code for the CLI:

it appears that it’s loading the data from the local user database. If it’s not found in the user database, it looks for an immutable data url. I don’t recall setting that url or uploading the data anywhere. Where is immutable data stored?

I tried to lookup up the same immutable data test on name devname0.id using the blockstack explorer server’s cli instead of the machine with which I ran put_immutable on, and it is unable to retrieve the immutable data:

(blockstack-venv) larry@blockstack-explorer:~$ blockstack get_immutable devname0.id test
[2016-11-28 09:19:31,073] [DEBUG] [spv:103] (20783.139762018248448) Using mainnet
[2016-11-28 09:19:31,281] [DEBUG] [cli:210] (20783.139762018248448) Enabling advanced methods
[2016-11-28 09:19:31,298] [DEBUG] [client:100] (20783.139762018248448) Connect to localhost:6264
[2016-11-28 09:19:31,314] [DEBUG] [client:135] (20783.139762018248448) Loaded storage driver 'disk'
[2016-11-28 09:19:31,314] [DEBUG] [client:135] (20783.139762018248448) Loaded storage driver 'blockstack_resolver'
[2016-11-28 09:19:31,315] [DEBUG] [client:135] (20783.139762018248448) Loaded storage driver 'blockstack_server'
[2016-11-28 09:19:31,315] [DEBUG] [client:135] (20783.139762018248448) Loaded storage driver 'http'
[2016-11-28 09:19:31,315] [DEBUG] [client:135] (20783.139762018248448) Loaded storage driver 'dht'
[2016-11-28 09:19:31,384] [DEBUG] [profile:155] (20783.139762018248448) Fetched 60ef539a835ec02c5ad87892345e4f559cf97420 from Atlas peer localhost:6264
[2016-11-28 09:19:31,392] [DEBUG] [storage:303] (20783.139762018248448) get_immutable 9724c1e20e6e3e4d7f57ed25f9d4efb006e508590d528c90da597f6a775c13e5
[2016-11-28 09:19:31,392] [DEBUG] [storage:331] (20783.139762018248448) Try disk (9724c1e20e6e3e4d7f57ed25f9d4efb006e508590d528c90da597f6a775c13e5)
[2016-11-28 09:19:31,392] [DEBUG] [disk:104] (20783) No such file or directory: '/home/larry/.blockstack/storage-disk/immutable/9724c1e20e6e3e4d7f57ed25f9d4efb006e508590d528c90da597f6a775c13e5'
[2016-11-28 09:19:31,392] [DEBUG] [storage:340] (20783.139762018248448) No data: disk.get_immutable_handler(9724c1e20e6e3e4d7f57ed25f9d4efb006e508590d528c90da597f6a775c13e5)
[2016-11-28 09:19:31,392] [DEBUG] [storage:331] (20783.139762018248448) Try blockstack_resolver (9724c1e20e6e3e4d7f57ed25f9d4efb006e508590d528c90da597f6a775c13e5)
[2016-11-28 09:19:31,392] [DEBUG] [storage:340] (20783.139762018248448) No data: blockstack_resolver.get_immutable_handler(9724c1e20e6e3e4d7f57ed25f9d4efb006e508590d528c90da597f6a775c13e5)
[2016-11-28 09:19:31,392] [DEBUG] [storage:331] (20783.139762018248448) Try blockstack_server (9724c1e20e6e3e4d7f57ed25f9d4efb006e508590d528c90da597f6a775c13e5)
[2016-11-28 09:19:31,392] [DEBUG] [blockstack_server:106] (20783) Get profile for devname0.id
[2016-11-28 09:19:32,242] [ERROR] [storage:350] (20783.139762018248448) Invalid data hash from blockstack_server.get_immutable_handler
[2016-11-28 09:19:32,242] [DEBUG] [storage:331] (20783.139762018248448) Try http (9724c1e20e6e3e4d7f57ed25f9d4efb006e508590d528c90da597f6a775c13e5)
[2016-11-28 09:19:32,242] [DEBUG] [storage:340] (20783.139762018248448) No data: http.get_immutable_handler(9724c1e20e6e3e4d7f57ed25f9d4efb006e508590d528c90da597f6a775c13e5)
[2016-11-28 09:19:32,242] [DEBUG] [storage:331] (20783.139762018248448) Try dht (9724c1e20e6e3e4d7f57ed25f9d4efb006e508590d528c90da597f6a775c13e5)
[2016-11-28 09:19:37,267] [ERROR] [storage:335] (20783.139762018248448) timed out
Traceback (most recent call last):
  File "/data/blockstack-venv/local/lib/python2.7/site-packages/blockstack_client/storage.py", line 333, in get_immutable_data
    data = handler.get_immutable_handler( data_hash, data_id=data_id, zonefile=zonefile, fqu=fqu )
  File "/data/blockstack-venv/local/lib/python2.7/site-packages/blockstack_client/backend/drivers/dht.py", line 173, in get_immutable_handler
    return dht_get_key(key)
  File "/data/blockstack-venv/local/lib/python2.7/site-packages/blockstack_client/backend/drivers/dht.py", line 112, in dht_get_key
    ret = dht_client.get(data_key)
  File "/data/blockstack-venv/local/lib/python2.7/site-packages/basicrpc/basicrpc.py", line 46, in default
    return self.request()
  File "/data/blockstack-venv/local/lib/python2.7/site-packages/basicrpc/basicrpc.py", line 82, in request
    c = self.sock.recv(1)
timeout: timed out
[2016-11-28 09:19:37,268] [DEBUG] [storage:336] (20783.139762018248448) Method failed: <module 'dht' from '/data/blockstack-venv/local/lib/python2.7/site-packages/blockstack_client/backend/drivers/dht.pyc'>.get_immutable_handler(9724c1e20e6e3e4d7f57ed25f9d4efb006e508590d528c90da597f6a775c13e5)
No immutable data returned
(blockstack-venv) larry@blockstack-explorer:~$