On Friday, May 5th, approx 8:47pm ET a read-only contract call caused a follower node in Hiro’s infrastructure to crash. Upon investigation with community and core developers via Github, it was determined this was a latent Clarity bug that has existed since at least 2.05 or earlier and is unrelated to recent upgrades (and thus, not a regression).
There was a potential Denial-of-Service vector, so core developers and contributors quickly prepared release 2.3.0.0.1 to address the issue before providing further details widely. The timing and information in this post are also taking the risk with this particular bug into consideration. The hope was to have node operators upgrade before someone accidentally triggered the bug. Unfortunately, the bug was triggered on mainnet before the upgrade was rolled out and caused miners to crash. As a result, blocks were not produced for approximately 3 hrs.
The hotfix release is now circulating and block production has resumed. All node operators are advised to upgrade. This release is compatible with previous 2.3 chainstates.
Changelog: stacks-blockchain/CHANGELOG.md at master · stacks-network/stacks-blockchain · GitHub
Release: Release Release 2.3.0.0.1 · stacks-network/stacks-blockchain · GitHub
Reminder: The Stacks Foundation provides a bounty program to help protect the Stacks network. We recommend everyone use that path when observing a potential network bug so that they are a) rewarded and b) bugs are disclosed responsibly, affording core developers and contributors the maximum amount of time to assess and react.
Emails: If you’d like emails when there are new releases, you can join the mailing list by sending an email request to [email protected].
Note on communications: I’ve noted some feedback about communication around this issue so I wanted to point out a few things and invite anyone with strong opinions to get involved:
- There was a DoS vector here, advertising an ongoing issue where it doesn’t absolutely need to be only increases the chance someone will zero in on and exploit the vulnerability. Not going out of the way to publicly advertise security holes of this nature until a fix is released or imminent is pretty standard in software development. That was the approach taken here.
- Folks that want truly real-time information can hop into Discord and Github where work happens. These are generally safe places to openly discuss things and have back and forth as issues are corralled. Once that happens, comms that go wider can be responsibly crafted with better information. The fact is, most issues never rise to the level of needing to be pushed to the general public and they generally don’t care to have this pushed to them so long as its available to find when they want it.
- The Stacks Foundation posts to @StacksStatus and posts forum posts on bigger issues as a service to the ecosystem and takes a certain approach in doing so. We encourage others to take a different approach if desired, the more people helping in these types of situations, the better the coverage! My hope is anyone doing that does their best to take into account the bigger picture as they share information, including attention to responsible disclosures, technical accuracy, and timing/prioritization of information flow that best protects the chain and Stacks project.
With our own approach, we try to strike a balance between quick information and accuracy while prioritizing reaching key groups without creating confusion or unnecessary panic. For example, with this latest issue, we felt it was most important that miners were receiving information so they could implement the fix to get blocks flowing again. This was done before pushing out more public comms on Twitter to reduce the danger of that information getting to a bad actor before miners could upgrade. Unfortunately, to further complicate this one, someone tried a contract just before advised, at which point we were able to/had to go fully public. Again, if anyone wants to see this done differently, there is definitely no gate on observing the blockchain/network and posting about it/raising alarms in the way you’d like to see.
Last, a reminder to join the community call at the end of the month to discuss improvements to testing and upgrades: AddEvent