How to Run Your Own Gaia Hub and Use It with the CLI
Several people have asked me how to use their own Gaia hub instead of the one hosted by Blockstack PBC. While this currently is disabled in the Browser for safety reasons (see Disclaimers below), the recent release and deployment of Gaia 2.2 has made it possible from the CLI’s authenticator. This tutorial provides a brief overview on how to:
- Set up your own Gaia hub
- Migrate your app data out of Blockstack PBC’s Gaia hub and upload it to yours
- Use your Gaia hub to store your app data from now on
Before you run this tutorial, you should install the experimental Blockstack CLI.
Disclaimers
The processes described in this document are experimental and are not yet officially supported. Moreover, this tutorial is meant for power users and developers. Think of this as a tech preview instead of a release announcement.
Before proceeding, you should have strong familiarity with the following concepts:
- How to use the UNIX command-line
- How to compile Node.js code from source
- How to file a meaningful bug report
If you don’t follow this tutorial correctly, the following bad outcome(s) may occur:
- You could lose your app data
- You could lose your profile
- You could lose your Blockstack ID
If any of the above made you uncomfortable, this tutorial is not for you. Even if you’re totally fine with the above, I recommend that you create and use a separate Blockstack ID for trying this feature set out.
This tutorial should work, but there may be bugs. If you try it out and encounter problems, please reply to this post with sufficient detail to fix it, and I will do my best to keep this tutorial updated.
Step 1: Set up a Gaia Hub
All Blockstack IDs have profiles that get stored in a Gaia hub somewhere. The profile’s URL is stored in a zone file, whose contents are stored in Blockstack’s peer network (Atlas), and its hash is anchored to the blockchain. The URL itself points to your profile in your Gaia hub. So, the first thing that you need to do is set up your Gaia hub, and you’ll need a storage account on some storage provider Gaia supports. This tutorial uses Amazon S3.
First, Go get the Gaia hub and install it on the host or VM that will be running it. Note that this host will need to be publicly routable–it needs a public IP address.
# grab the source
$ git clone https://github.com/blockstack/gaia blockstack_gaia_hub
# install it
$ cd blockstack_gaia_hub/hub
$ npm install
$ npm run build
$ sudo npm install -g
# verify that the Gaia hub binary exists
$ which blockstack-gaia-hub
/usr/bin/blockstack-gaia-hub
Second, we’ll configure the Gaia hub. Recall that a Gaia hub does not store any state of its own, but instead acts as a write-only proxy for another storage system. Gaia hubs do not serve read requests.
If you deploy your Gaia hub publicly, you usually do so by running it behind a nginx proxy. The nginx proxy acts an an SSL terminus and rate-limiter. Calls to putFile() in blockstack.js will POST to your Gaia hub through the nginx proxy, and the Gaia hub itself will write the data to your storage system of choice (e.g. Amazon S3). If you deploy your Gaia hub privately (such as on localhost), you do not need to worry about setting up nginx or SSL.
Let’s assume you’ve already set up your S3 bucket and have your S3 access key and secret key. To do so, you’d create a Gaia config file like this:
# your config file.
# Fill in the following fields:
# * SERVER_NAME -- this is the IP address or DNS name of your Gaia hub's host
# * S3_BUCKET_NAME -- this is your S3 bucket name
# * S3_ACCESS_KEY
# * S3_SECRET_KEY
$ cat /etc/gaia.conf
{
"servername": "SERVER_NAME",
"port": 4000,
"driver": "aws",
"readURL": "https://S3_BUCKET_NAME.s3.amazonaws.com/",
"proofsConfig": {
"proofsRequired" : 0
},
"pageSize": 20,
"bucket": "S3_BUCKET_NAME",
"awsCredentials": {
"accessKeyID": "S3_ACCESS_KEY",
"secretAccessKey": "S3_SECRET_KEY"
},
"whitelist": [],
"argsTransport": {
"level": "debug",
"handleExceptions": true,
"stringify": true,
"timestamp": true,
"colorize": false,
"json": true
}
}
(Note that the readURL value must end in /).
In this configuration, anyone can write to your Gaia hub. This is probably not what you want 
. We’ll address this in the next section – you’ll fill in the list of authorized writer addresses in the whitelist section of the above config. Stay tuned!
In the above config, the Gaia hub will listen for POST requests on port 4000. It will forward them to an S3 bucket whose files can be read via https://S3_BUCKET_NAME.s3.amazonaws.com.
The third thing we’ll do is set up an nginx process to proxy SSL requests to port 4000. I recommend using Let’s Encrypt to set up nginx as an SSL terminus, described in this tutorial and many others like it. Note that if you’re running your Gaia hub privately, you can skip this step.
Regardless of how you set up SSL on your nginx process, you’ll need to make sure all POST requests get sent to 4000. You can do this as follows:
$ cat /etc/nginx.conf
server {
# the proxy_pass is the important part:
location / {
proxy_pass http://localhost:4000
}
}
When this is all set up, you should be able to do the following:
$ curl -sL https://YOUR_GAIA_HUB/hub_info
{"challenge_text":"[\"gaiahub\",\"2018\",\"storage.blockstack.org\",\"blockstack_storage_please_sign\"]","latest_auth_version":"v1","read_url_prefix":"https://S3_BUCKET_NAME.s3.amazonaws.com"}
(Note that if you’re running without SSL, such as on localhost, you should use http).
Make absolutely sure the read_url_prefix field matches your S3 bucket, since this is the URL prefix that all calls to getFile() will use.
Setting up your S3 bucket
To use the S3 bucket in this way, you’ll want to do two things: make it publicly readable and listable, and enable permissive CORS headers. To do this, use the following policies:
The general bucket policy I use is this:
{
"Id": "YOUR_S3_BUCKET_POLICY_NAME",
"Statement": [
{
"Action": "s3:GetObject",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Resource": "arn:aws:s3:::YOUR_S3_BUCKET_NAME/*",
"Sid": "AllowPublicRead"
},
{
"Action": "s3:ListBucket",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Resource": "arn:aws:s3:::YOUR_S3_BUCKET_NAME",
"Sid": "AllowPublicList"
}
],
"Version": "2008-10-17"
}
The CORS policy I use is as follows:
{
"CORSRules": [
{
"AllowedOrigins": ["*"],
"AllowedMethods": ["GET", "HEAD"]
}
]
}
You can set them with the AWS console, or with the aws CLI tool. To do the latter, I did this:
$ aws s3api put-bucket-policy --bucket YOUR_S3_BUCKET_NAME --policy file:///path/to/policy.json
$ aws s3api put-bucket-cors --bucket YOUR_S3_BUCKET_NAME --cors-configuration file:///path/to/cors.json
Step 2: Make a Blockstack ID and Point it to your Gaia Hub
I highly recommend making a new Blockstack ID for this tutorial, so if something breaks or goes wrong, you don’t lose everything.
To make a Blockstack ID from the command-line and store its profile to your Gaia hub, you would do the following:
First, you’d make a new 12-word phrase
# create a 12-word backup phrase
$ blockstack-cli make_keychain
{
"mnemonic": "identify wrist tongue leopard asset bind silly debate public cannon scheme kingdom",
"ownerKeyInfo": {
"privateKey": "e0020b02c0aeb7835922fa8d1404be10d806a277d399629f12bf6c26b09725ad01",
"version": "v0.10-current",
"index": 0,
"idAddress": "ID-199S9kwtxVryPWj5vqzdXcKDi2iUu6eNL2"
},
"paymentKeyInfo": {
"privateKey": "875cb95e01df38bbecbd36f996172533f546b87765e618d2f17fb0afbfd39ffd01",
"address": {
"BTC": "1tJs7qUH7BDZPUkwLMyfz93sjiRrcavaf",
"STACKS": "SP4V882J15B4Y0BA0QRJ0WMG6GB065RDBESBN7K8"
},
"index": 0
}
}
Second, you’d save some environment variables from the keychain (to make things easier).
# save it so we don't have to keep re-typing it
$ export BACKUP_PHRASE="identify wrist tongue leopard asset bind silly debate public cannon scheme kingdom"
$ export OWNER_KEY="e0020b02c0aeb7835922fa8d1404be10d806a277d399629f12bf6c26b09725ad01"
$ export OWNER_ADDRESS="199S9kwtxVryPWj5vqzdXcKDi2iUu6eNL2"
Third, you’ll want to edit your gaia.conf file so that only this key can write to it. You’ll want to add your $OWNER_ADDRESS to the whitelist array, as follows:
"whitelist": [
"199S9kwtxVryPWj5vqzdXcKDi2iUu6eNL2"
],
Fourth, you’d register a Blockstack ID. I recommend using a subdomain, since they’re free and can be used immediately. You’ll want to do this after you set up your whitelist above, so you can store your profile to your Gaia hub upon registering.
# register a subdomain
$ blockstack-cli register_subdomain test.id.blockstack "$OWNER_KEY" "https://YOUR_GAIA_HUB" "https://registrar.blockstack.org"
{
"txInfo": {
"status": true,
"message": "Your subdomain registration was received, and will be included in the blockchain soon."
},
"profileUrls": null
}
If all went well, you should be able to directly fetch your Blockstack ID’s profile using your owner key’s address:
# verify that your profile is uploaded
$ curl -sL "https://S3_BUCKET_NAME.s3.amazonaws.com/$OWNER_ADDRESS/profile.json"
[
{
"token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzI1NksifQ.eyJqdGkiOiIyZTllMDEyZC03MTI5LTRhMzEtYjAyNC04ZjdiMDkwM2UyMWIiLCJpYXQiOiIyMDE4LTA5LTA0VDA0OjA0OjUwLjQ2N1oiLCJleHAiOiIyMDE5LTA5LTA0VDA0OjA0OjUwLjQ2N1oiLCJzdWJqZWN0Ijp7InB1YmxpY0tleSI6IjAzYTcwYTdkYjg4NmQ2MjIyYjUyYzE0MjhhZDM5ZTU1NTFjNzdlNTJhZjM2YmFjYjY2NTA1NmRjMTU0ZDc1NTdjNyJ9LCJpc3N1ZXIiOnsicHVibGljS2V5IjoiMDNhNzBhN2RiODg2ZDYyMjJiNTJjMTQyOGFkMzllNTU1MWM3N2U1MmFmMzZiYWNiNjY1MDU2ZGMxNTRkNzU1N2M3In0sImNsYWltIjp7InR5cGUiOiJAUGVyc29uIiwiYWNjb3VudCI6W119fQ.iUvWDtrb1Uyyhj6c-XygbvEcNsjXDl4ahz7wXMT1bAwo_E8mDL1jJ2DGNFeL411nfv1KDpIAwvNuTi0IpZ7J8Q",
"decodedToken": {
"header": {
"typ": "JWT",
"alg": "ES256K"
},
"payload": {
"jti": "2e9e012d-7129-4a31-b024-8f7b0903e21b",
"iat": "2018-09-04T04:04:50.467Z",
"exp": "2019-09-04T04:04:50.467Z",
"subject": {
"publicKey": "03a70a7db886d6222b52c1428ad39e5551c77e52af36bacb665056dc154d7557c7"
},
"issuer": {
"publicKey": "03a70a7db886d6222b52c1428ad39e5551c77e52af36bacb665056dc154d7557c7"
},
"claim": {
"type": "@Person",
"account": []
}
},
"signature": "iUvWDtrb1Uyyhj6c-XygbvEcNsjXDl4ahz7wXMT1bAwo_E8mDL1jJ2DGNFeL411nfv1KDpIAwvNuTi0IpZ7J8Q"
}
}
]
You should also be able to check your profile with the CLI:
$ blockstack-cli lookup hello.id.blockstack
{
"zonefile": "$ORIGIN hello.id.blockstack\n$TTL 3600\n_http._tcp\tIN\tURI\t10\t1\t\"https://S3_BUCKET_NAME.s3.amazonaws.com/199S9kwtxVryPWj5vqzdXcKDi2iUu6eNL2/profile.json\"\n\n",
"profile": {
"type": "@Person",
"account": []
},
"profileUrl": "https://S3_BUCKET_NAME.s3.amazonaws.com/199S9kwtxVryPWj5vqzdXcKDi2iUu6eNL2/profile.json"
}
If you made it this far, then congratulations! You have your own Gaia hub, and it stores your profile! You should now be able to go to the Blockstack Explorer and see your profile there as well by typing in your Blockstack ID.
Step 3: Authenticate to Applications with your Blockstack ID
Now that you have your Gaia hub set up and can successfully store and read your Blockstack ID’s profile to it, you can proceed to use it to sign into applications. To do so, you would use the CLI’s authenticator command with your backup phrase:
$ blockstack-cli authenticator "https://YOUR_GAIA_HUB" "$BACKUP_PHRASE"
This launches a minimalist blockstack: protocol handler on port 8888, just like the Browser. You will need to stop your Browser before running the above command.
Now, when you click “Log in with Blockstack” in a Blockstack app, you’ll be redirected to a minimalistic web page from the CLI that will prompt you to sign in. When you sign in, your profile will be updated and saved to your Gaia hub with the new app-specific bucket URL, and your writes will be sent to your Gaia hub instead of https://hub.blockstack.org.
Using an Existing Blockstack ID with a New Gaia Hub
If you already registered a Blockstack ID with the Browser, you can still use it with your own Gaia hub. Your profile would continue to be stored on https://hub.blockstack.org, but your app-specific data would be stored on your Gaia hub.
WARNINGS
- In case the CLI screws up somehow (recall that it is not yet stable), you should make a backup of your profile first. You can do this by directly downloading your
profile.jsonfile from the URL in your zone file (obtained through thelookupcommand) - Your profile will continue to be stored on https://hub.blockstack.org. If this is a problem, you can use the
updatecommand in the CLI to make your Blockstack ID point to your Gaia hub instead. More on that later. - Be careful with your 12-word backup phrase. If possible, use the encrypted backup phrase instead.
To do this, you will need either your 12-word phrase for your Browser keychain, or the encrypted 12-word phrase that was emailed to you when you signed up for your Blockstack ID in the Browser.
# BROWSER_KEYCHAIN will be either your 12-word phrase or the base64 ciphertext
$ export BROWSER_KEYCHAIN="..."
# You will be prompted for your decryption password
$ blockstack-cli authenticator "https://YOUR_GAIA_HUB" "$BROWSER_KEYCHAIN" "https://hub.blockstack.org"
Enter password:
Migrating Your Data to Your Gaia Hub
If you have used an existing Blockstack ID extensively with one or more Blockstack apps and want to migrate your data to your new Gaia hub, you can do so with the CLI.
Recall that each application gets its own unique private key for storing data in Gaia. You’ll need either your 12-word backup phrase or your encrypted 12-word backup phrase (the base64-encoded string that was emailed to you) to proceed.
# can be your 12-word phrase or ciphertext
$ export BROWSER_KEYCHAIN="..."
$ blockstack-cli gaia_dump_bucket hello.id.blockstack "https://YOUR_APP" "https://hub.blockstack.org" "$BROWSER_KEYCHAIN" "./gaia-dump/"
This should download all of your app-specific files from the Blockstack Gaia hub and store them to ./gaia-dump/
Then, you can upload all of the dumped files to your Gaia hub as follows:
$ blockstack-cli gaia_restore_bucket hello.id.blockstack "https://YOUR_APP" "https://YOUR_GAIA_HUB" "$BROWSER_KEYCHAIN" "./gaia-dump/"
This should upload all your app-specific files from ./gaia-dump/ to your new Gaia hub. Then, you should be able to sign into the app with the CLI’s authenticator and proceed to use the app like before (note that you will need to log out and log back with the CLI).
Migrating your Profile to Your Gaia Hub
If you want to stop using https://hub.blockstack.org as your Gaia hub completely, and you registered your Blockstack ID through the Browser, you’ll need to issue a NAME_UPDATE transaction to make your Blockstack ID’s profile URL point to your new Gaia hub.
WARNINGS
- This requires a Bitcoin transaction. It is not free.
- If you mess this up, your Blockstack ID will become unusable.
- This does NOT work on subdomains yet. Only on-chain names.
To do so, first make a new zone file that contains a link to your profile in your Gaia hub:
$ export BLOCKSTACK_ID="hello.id" # put your .id name here
$ export GAIA_URL_PREFIX="https://S3_BUCKET_NAME.s3.amazonaws.com"
$ export ID_ADDRESS="ID-$OWNER_ADDRESS"
$ blockstack-cli make_zonefile "$BLOCKSTACK_ID" "$ID_ADDRESS" "$GAIA_URL_PREFIX" > /tmp/your_zonefile.txt
# check out the zone file
$ cat /tmp/your_zonefile.txt
$ORIGIN hello.id
$TTL 3600
_http._tcp IN URI 10 1 "https://S3_BUCKET_NAME.s3.amazonaws.com/199S9kwtxVryPWj5vqzdXcKDi2iUu6eNL2/profile.json"
Second, send a NAME_UPDATE transaction with this zone file. You’ll need to create and fund a Bitcoin payment key to do so.
$ export PAYMENT_KEY="...",
$ export BLOCKSTACK_ID="hello.id" # replace with your Blockstack ID
$ blockstack-cli update "$BLOCKSTACK_ID" /tmp/your_zonefile.txt" "$OWNER_KEY" "$PAYMENT_KEY"
84a9363f885e156d080033e1a74c0f828491b7a90465e63cf7bdd806800a7d69
Finally, when the transaction gets 7+ confirmations (about 70 minutes), you’ll need to replicate your zone file as follows:
$ blockstack-cli zonefile_push /tmp/your_zonefile.txt
This command sends your zone file to Blockstack’s peer network, and is required in order to ensure your new profile resolves.