Gaia nginx container not starting

Apps Storage (Gaia)
#1

I’m trying to get a Gaia storage hub working on an t2.medium AWS EC2 instance using these instructions. I’m using the blockstack-gaia_hub-ebs-2.5.3-hvm AMI.

I have an elastic IP 54.173.170.105 linked to registered domain www.tcphub.ga.

/etc/environment:

API_KEY=tcphub
DOMAIN=www.tcphub.ga
STAGING=1

docker ps:

CONTAINER ID        IMAGE                                   COMMAND                  CREATED             STATUS              PORTS                    NAMES
bff17d85599f        quay.io/blockstack/gaia-reader:v2.5.3   "node lib/index.js"      30 minutes ago      Up 29 minutes       0.0.0.0:8008->3000/tcp   gaia-reader
85fad1c8bb2f        quay.io/blockstack/gaia-admin:v2.5.3    "docker-entrypoint.s…"   30 minutes ago      Up 29 minutes       0.0.0.0:8009->3000/tcp   gaia-admin
1f4325cd65fd        quay.io/blockstack/gaia-hub:v2.5.3      "docker-entrypoint.s…"   30 minutes ago      Up 29 minutes       0.0.0.0:3000->3000/tcp   gaia-hub

I tried running sudo systemctl restart reset-ssl-certs.service, but it hangs and never returns.

I tried journalctl -xe but get -- No entries --

I looked for logs, but /var/log/nginx did not even exist. I created it and restarted the instance, but still no logs. I get the impression from watching processes that something is hanging in certbot, but without any logs or output I’m not getting any traction troubleshooting the issue.

Any help is much appreciated!

#2

Hello, I am facing a similar issue, where I am not able to get the nginx.service up and running. I’m using t2.micro with the blockstack-gaia_hub-ephemeral-2.5.3-hvm AMI, without the use of EIP. I have enabled journald logging to get visibility of the error messages against the services. Also I have edited the /etc/resolv.conf by including the google public DNS (8.8.8.8) nameserver. From the logs I have confirm that by DOMAIN is pointing to the publiv IPv4 address associated with the gaia-hub instance running on EC2. However, I am observing the nginx service keeps on restarting. On checking the logs, it is throwing the following warnings/error:

Mar 31 12:27:46 ip-172-31-22-198 docker[5962]: 2021/03/31 12:27:46 [warn] 1#1: the “http2_max_field_size” directive is obsolete, use the “large_client_header_buffers” directive instead in /etc/nginx/ngi>
Mar 31 12:27:46 ip-172-31-22-198 docker[5962]: nginx: [warn] the “http2_max_field_size” directive is obsolete, use the “large_client_header_buffers” directive instead in /etc/nginx/nginx.conf:30
Mar 31 12:27:46 ip-172-31-22-198 docker[5962]: 2021/03/31 12:27:46 [warn] 1#1: the “http2_max_header_size” directive is obsolete, use the “large_client_header_buffers” directive instead in /etc/nginx/ng>
Mar 31 12:27:46 ip-172-31-22-198 docker[5962]: nginx: [warn] the “http2_max_header_size” directive is obsolete, use the “large_client_header_buffers” directive instead in /etc/nginx/nginx.conf:31
Mar 31 12:27:46 ip-172-31-22-198 docker[5962]: 2021/03/31 12:27:46 [warn] 1#1: the “http2_max_requests” directive is obsolete, use the “keepalive_requests” directive instead in /etc/nginx/nginx.conf:32
Mar 31 12:27:46 ip-172-31-22-198 docker[5962]: nginx: [warn] the “http2_max_requests” directive is obsolete, use the “keepalive_requests” directive instead in /etc/nginx/nginx.conf:32
Mar 31 12:27:46 ip-172-31-22-198 docker[5962]: 2021/03/31 12:27:46 [emerg] 1#1: unexpected end of file, expecting “;” or “}” in /etc/letsencrypt/options-ssl-nginx.conf:1
Mar 31 12:27:46 ip-172-31-22-198 docker[5962]: nginx: [emerg] unexpected end of file, expecting “;” or “}” in /etc/letsencrypt/options-ssl-nginx.conf:1

I’m guessing this config file is residing inside the nginx container. When I checked this file on the host instance, it was found on the location /gaia/nginx/certbot/conf/options-ssl-nginx.conf with its contents 404: Not Found.

Besides, all 4 gaia services are in active state. I have tried the workaround mentioned in the “Deploy on Amazon EC2 | Stacks” link, still was not able to get the nginx service up.

Any pointers will be much appreaciated.

1 Like
#3

Hi, @mustafaalam958 Have you found a solution to this? I’m also attempting to set up Gaia storage hub on an AWS EC2 instance using the documentation. I’m having the same problem as you.
The nginx service could not be started.Greetings, @Gina and @markmhendrickson .Please assist us.Any leads on this will be extremely helpful.Thank you so much.

#4

Hello, I didn’t find a solution to this issue. I tried with various other flavors like t2.medium, but it doesn’t seem like a resource issue. It’s more like a misconfiguration/syntax issue.

Anywho, I switched to a different approach. I have an Amazon EC2 instance running Ubuntu 20 (kernel: 5.4.0-1041-aws) and followed the “Deploy the Hub with Docker” guide given in the gaia/hub/README.md. I’m pasting the procedure below for reference:

Deploy the Hub with Docker, Nginx

First have docker,nginx and certbot installed on a server with a FQDN pointed to it. The following guides should help you get this setup.

Then do the following:

  • Make a folder $HOME/hub and copy the configuration file config.sample.json to $HOME/hub/config.json and add in your desired configuration.
  • Copy nginx.conf.sample to $HOME/hub/nginx.conf and replace hub.example.com with your FQDN.
  • sudo rm /etc/nginx/sites-enabled/default && sudo ln $HOME/hub/nginx.conf /etc/nginx/sites-enabled/default
  • Finish certbot setup and cert generation
  • Pull the docker image and start an instance of the container:
$ docker pull quay.io/blockstack/gaia-hub:latest
$ docker run -d --restart=always -v $HOME/hub/config.json:/src/hub/config.json -p 3000:3000 -e CONFIG_PATH=/src/hub/config.json quay.io/blockstack/gaia-hub:latest
# Now you can test the hub! The exact output will depend on your configuration
$ curl https://hub.example.com/hub_info | jq
{
  "challenge_text": "[\"gaiahub\",\"2017-09-19\",\"{{ .serverName }}\",\"blockstack_storage_please_sign\"]",
  "read_url_prefix": "https://{{ .bucketName }}.{{ .storageProviderUrl }}/"
}
1 Like
#5

This is most likely due to the AMI not being updated in a long time - I will note though that for the AMI to work you will need an EIP and a domain setup to point to that IP. Gaia requires an SSL connection, and there’s no way for letsencrypt to verify a domain without an external IP over HTTP.

The specific method that builds this AM is here: gaia-docker/packer at master · hirosystems/gaia-docker · GitHub

PR’s are welcome

1 Like
#6

Thank you @mustafaalam958 and @jwiley for your inputs! I will try out the suggested steps. If I face any issues, will get back to you. Appreciate your assistance on the same!! Thank you so much.

1 Like
#7

Sure! i would say to use the idea from @mustafaalam958 vs trying to fix the AMI directly (but i would welcome a PR if you have the time!).

One thing to note about the AMI is that it’s built off of a Container OS, so typical linux file structures etc are very different (i.e., there is no /var/log for nginx). We did it this way to reduce resource consumption by the host OS, so there was more available for gaia/nginx. But again, a PR to change this would definitely be something we could look at.

Because of that, most of the work is done through systemd unit-files.

1 Like
#8

Hi @mustafaalam958, I have tried the Docker setup suggested by you. It didn’t work out for me, unfortunately. The last command "docker run -d --restart=always -v $HOME/hub/config.json:/src/hub/config.json -p 3000:3000 -e CONFIG_PATH=/src/hub/config.json quay.io/blockstack/gaia-hub:latest" is returning a container id but when I tried accessing the hub_info, always getting 404: Not found.

Were you successful in setting up the same? If so, requesting your help to debug the same. Let me know your thoughts. Thank you.

#9

Hi, @jwiley I have gone through the Image building instructions. Unfortunately, I am not much familiar with image building and I couldn’t progress much on the same. Requesting your help in building a new AMI for AWS EC2 Gaia installation. I am sure that will be helpful for the community members like me to help get started with setting up Gaia. Please let me know your thoughts on the same. Thank you so much.

#10

well, i can’t promise anything here…but there may be sometime next week that we can devote here to fix this up and simplify things a bit.

Once i know for sure i’ll let you know

1 Like
#11

Yes, that sounds fantastic! Updated AMI would undoubtedly be helpful. Thank you so much for your support. Looking forward to hearing from you.

#12

Can you share your hub URL? My hub is working on https://lumstx.tk/hub_info. Did you find any issues in the nginx logs (/var/log/nginx/)?

#13

Hi @mustafaalam958! I haven’t configured the domain name and was not successful with Nginx, I was trying to get Gaia to work on HTTP first. Something like YOUR_IP/hub_info. Were you able to achieve the same? Is SSL or FQDN mandatory for Gaia to work? Please let me know your thoughts. Thank you.

#14

Hello, My guess is that it should work without having to register a domain name, but since I haven’t tried that in practice, so I’m not sure. Can you elaborate on what issues did you get with nginx?

#15

Thank you @mustafaalam958. Regarding Nginx, I guess my config is incorrect. I am seeing the below logs. It will be of great help if you can share your nginx.config and config.json for reference. Thank you.
image

#16

ok. That may be the reason why docker is not able to start. the config.son i use for AWS is below:

[email protected]:~/hub$ cat config.json
{
“port”: 3000,
“driver”: “aws”,
“bucket”: “s3-stacks”,
“readURL”: “https://s3-stacks.s3.amazonaws.com/”,
“maxFileUploadSize”: “1000”,
“argsTransport”: {
“level”: “debug”,
“handleExceptions”: true,
“timestamp”: true,
“colorize”: false,
“json”: true
},
“awsCredentials”: {
“accessKeyId”: “XXXXXXXXXXXXXXXXXXXX”,
“secretAccessKey”: “YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY”
}
}

and nginx.conf is:

[email protected]:~/hub$ cat nginx.conf
limit_req_zone $binary_remote_addr zone=std_rl:10m rate=10r/s;
limit_req zone=std_rl burst=10 nodelay;

upstream gaia_hub {
server localhost:3000;
}

server {
listen 80;
server_name lumstx.tk;
root /var/www/lumstx.tk/html;
index index.html index.html index.nginx-debian.html;

location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://gaia_hub;
break;
}
}

One observation I have, is that the gaia-hub docker is not yet started, the nginx server fails when you restart its service. Once the gaia-hub is started, nginx finds that gaia-hub (upstream) running on localhost:3000, and the service is now in active state, after you give it a restart. I hope this makes sense.

1 Like
#17

Please do note that I used a separate server block (virtual host) for my nginx configuration. If you are working with default, the “server: root” directive will hold a different value.

1 Like
#18

Sure, thank you @mustafaalam958! I will retry the setup referencing your config files. Mine is at root, so Nginx has a different value. I guess, I just have to update the server value. Anyways, I will try once again and will keep you posted on how it went. Thank you so much for your assistance.

#19

Hi, @mustafaalam958 Thank you so much for your assistance! I successfully configured my Gaia hub following the docker setup. I was able to make my Gaia work with HTTP. I really appreciate your support on the same.

1 Like
#20

Hi @mustafaalam958 In continuation to this setup, have you tried pointing your app-level storage to this hub? If so, requesting you to kindly share the steps. Thank you so much.