Last December, Blockstack users experienced outages due to a bug in the newly released Blockstack Browser version at the time. The issue was addressed via a patch one week after the initial discovery. We felt that this kind of impact was unacceptable to users, developers and the broader community. In the first quarter of 2019, Blockstack PBC made it a top organizational goal to improve the engineering process and raise the bar on software quality. As we near the end of this quarter, we wanted to update the community on the improvements we’ve made to our process.
Automated End-to-End Testing
We implemented a full suite of automated end-to-end tests using Selenium and BrowserStack. These tests mimic real user input on a web browser and runs through the Blockstack account sign-up and app sign-in flows. They target many combinations of platforms and web browsers including Windows, macOS, iOS, Android, Chrome, Safari, Firefox and Edge. Additionally, the tests are configured as a requirement for every pull request merge on the Blockstack browser GitHub repository. This will help ensure that any major outage-causing bug will be detected and fixed long before it has a chance of impacting users. To see how the tests work, visit our GitHub repo: https://github.com/blockstack/blockstack-browser/tree/develop/test-e2e
Pre-release Testing and Engagement with Developers
We’ve also added public beta testing to our release process for the Blockstack Browser and blockstack.js. For every release, we will now announce the availability of beta versions on the #pre-releases Slack channel. App developers will have at least one week to test the new versions of Blockstack software with their apps. We’re also reaching out to top app developers to create a more streamlined beta testing process that incorporates their apps.
In the next Blockstack Browser release, we’re adding a “beta mode” that can be toggled by option+clicking the Blockstack status tray icon on macOS. By turning on beta mode, users will now authenticate into apps using the latest pre-release versions of the browser. Similarly, we’ve added the ability to test with beta versions of the Blockstack Browser web app on our iOS and Android SDKs. Please refer to the respective SDK documentation on how to activate beta mode as a developer.
Engineering On-call Schedule
The Blockstack PBC engineering team is also scheduling team members to be on-call internally to triage and repair reported outages on a more timely basis. This is in addition to our on-going efforts to reduce our GitHub issue response time to under 24 hours.
Next Steps
We see software quality assurance as an on-going goal and hope to continue to make improvements. Some of the next steps we are considering include adding automated exception tracking to Blockstack Browser, staged roll-out of updates and converting the blockstack.js library to Typescript.
We’re also looking for community input on our QA process, so any comments or feedback is highly appreciated.