Yesterday, the default Gaia hub experienced significant downtime due to an expired SSL certificate from September 5, 23:03:09 GMT to September 5, 01:08:00 GMT. This led to a service disruption for blockstack-browser and any applications users attempted to use during that period.
Our monitoring processes failed to detect this issue, which led to much more downtime than we view as acceptable. This monitoring issue was corrected by linking a runscope policy (polled every 5 minutes) to this instance, and so issues like this should be dealt with much more rapidly in the future. Furthermore, Blockstack is in the process of re-architecting our infrastructure and monitoring practices to align it more closely with the very high reliability standards that our community and developers deserve. We already have a dedicated DevOps team that is in the process of upgrading the deployment and monitoring processes.
However, it is important to note that while we operate the majority of the infrastructure in the Blockstack ecosystem today, a key benefit of decentralized ecosystems is that there are no such vitally important entities or infrastructure. That’s the world we’re moving towards. Decentralizing Gaia hubs in particular is a major focus for the Blockstack team in the coming months.