In the past few days, there has been a sharp spike in the number of pending transactions on the Stacks blockchain (> 3000 as I’m writing this). A few different factors came together to create a perfect storm of mempool congestion. In this post we share some observations and specific next steps as far as Hiro’s products are concerned.
How we got here
There’s a known issue with how the current miner software walks the mempool, that does not properly factor in transaction fees and only considers the origin account nonce: this ends up penalizing origin accounts with high nonce values. A fix is under review in this PR.
Block limits and various cost-related constants were set with conservative initial values in Stacks 2.0. There are several known improvements related to cost-tracking in Clarity — most of them are slated for Stacks 2.1 and a lot of work has been underway. However, with the basic cost-estimates set in Stacks 2.0 and presumably most miners running the default miner software (which undoubtedly has a lot of room for improvement and sophisticated strategies), miners end up over-estimating the compute budgets — this acutely manifests itself in contracts like Stacks Punks. In the public blockchain meeting earlier today, it was reported that roughly two thirds of recent transactions exceeded the runtime compute budget, while the remainder exceeded the read_count budget.
Most of these transactions originate from the Hiro web wallet, which currently does not support Replace By Fee (RBF) — a mechanism to rebroadcast transactions with a higher fee. This means once users sent a transaction, and if the network was already congested, they did not have any recourse to try and get their transaction processed.
These factors put together make it impossible for an efficient “fee market” to exist for transactions on the Stacks blockchain right now: it’s not easy for users to set or choose higher transaction fees, the default miner software doesn’t properly take fees into account and some Clarity contracts gaining in popularity are pushing against the conservative default cost estimates in Stacks 2.0.
What happens next?
Miner improvements: as mentioned above, there is a PR in review. This was also discussed at length in the public blockchain call today (Monday 8/30, 11am ET) and on public Discord channels. Miners are welcome to try this PR, while it goes through the standard release process.
Hiro web wallet: We are working to release a version of the Hiro Web Wallet in the next 1-2 days that allows users to set manual nonce and fee values upon signing of any transaction.
The build will give the users a prominent “Replace by fee” option for all pending transactions listed under activity.
This will take them to a form to submit the same transaction with the same nonce but different fee, with guidance about increasing it.
Note that Chrome and Firefox will need to approve this build before it’s live for most users. Meanwhile power users can install from source.
There are many pieces that must come together to create a robust fee market. In the meantime, the current default transaction fees don’t reflect or capture network activity adequately. To improve the experience for our users the default fee for transactions made through the Hiro Wallet will be changed to 0.25 STX, with optionality for users to change that fee.
Clarity cost tracking: Hiro has made a proposal to introduce a cost-estimator abstraction in the miner software that would allow for incrementally improving the cost-estimates without having to wait for Stacks 2.1. This proposal was shared with Stacks core-developers last week and is being reviewed.
In addition to an immediate fix to the Hiro Wallet, the Hiro team is working on a modification to stacks.js/connect that would allow developers to pass a default fee amount upon hand-off to transaction signing.
That way developers like StacksPunks can choose for their users to pay higher fees by default (e.g. 2 STX if they deem that appropriate to get their transactions settled reliably).
Of course, users can then edit this developer-set amount as they please subsequently before broadcasting.
This seems like an important change to prevent lots of users from still broadcasting transactions with too low fees before realizing there’s even a problem with fees, and having to go back and replace by fee (or depend on the app to inform them first of the fee to set manually).
Observations on operating in a decentralized ecosystem
The Hiro team will always be there to respond to Hiro-related products. This is an open source ecosystem, like Bitcoin, and there is no one party that can unilaterally make changes nor implement new features to the Stacks blockchain. Around the core decentralized base of the Stacks blockchain, individual companies can update their products however they want and as frequently as they want.
Decentralization is one of core values of the Stacks ecosystem and there is sometimes a natural trade-off between “doing it fast” and “doing it right”. Hiro understands these tradeoffs of the underlying technology. The Hiro wallet fix should be complete soon, however any changes to the underlying network may take more time to be more broadly socialized given it requires independent Stacks miners to opt into the new mining software that we anticipate will be ready for deployment, following the release process, later this week. It is in Stacks miners’ economic interest to run the new mining software, as they can collect higher fees, so we expect that the economic self-interest will result in faster adoption. Miners may implement their own versions of the open-source software to further optimize revenue from gas fees as well.
Longer term, we also expect a healthy network fee market to emerge where users can pay higher transaction fees for important transactions or do replace-by-fee to bump up the priority of their transactions. A robust fee market is a sign of a healthy blockchain network and Hiro plans to update Hiro products as the underlying network fee market evolves.
Overall, it’s wonderful to see the Stacks ecosystem thriving, developers building apps that are gaining traction, resulting in increased network usage. This is a good problem to have, and one I’m confident that we – different ecosystem entities and stakeholders, developers and users, miners etc – can solve together!
Thanks, Diwaker for the post! I’m writing a post giving an overview of the fee market that I’d expect to emerge around Stacks and it’s sort of linked to this. So I’ll link it here. Great to see these issues and potential fixes and improvements surfacing before the network starts seeing a lot more traffic!
This really great insight, thank you all! The Stacks Foundation is game to help here and are generally excited about app chains as well as creating better resources for builders that can help us avoid unnecessary congestion from the jump.
A new version of the Hiro Wallet extension (v2.16.0) is now available that supports manual fee configuration both for new transactions and existing ones that remain pending via “replace by fee” (RBF).
This version has been approved by the Firefox store already, so if you’ve installed it from there, it should update automatically soon if it hasn’t already. The version is pending approval for the Chrome store and should update automatically there soon as well.
If you are a Chrome or Brave user who needs this version before store approval, you can install it from source.
If you are a developer with contracts that have been failing to confirm reliably on the network, we encourage you to upgrade to the latest version of the Connect library (v6.1.0 or v12.0.0 if you’re using connect-react).
This version will allow you to pass a custom “fee” value during hand-off for transaction signing, which will be used to overwrite the default fee for users to execute your contract. We encourage you to pass a fee value that you believe will help these transactions settle on the network reliably given recent network usage trends.
This will save your users from having to realize themselves that a different fee is needed and to take steps to modify the fee manually, ensuring that a higher percentage of them broadcast transactions that settle quickly upon first attempt.
We will be working in the coming weeks to improve the UX around fee configuration even further. This is just a first set of steps to alleviate the most urgent issues with fees and transaction settlement. For example, we hope to provide users with useful fee recommendations (slow, medium and fast values) based on current network trends and the computational needs of their transactions.
I wanted to follow up here with some more details on what Hiro is focusing on with respect to some of the issues mentioned in this thread.
First, quick summary of the work items mentioned earlier in this thread:
For miners: 2.0.11.3.0-rc2 was tagged 6 days ago. A release build is under review, I’m optimistic that we’ll have a 2.0.11.3.0 release available shortly. Worth repeating though that it’s ultimately up to the miners to upgrade their software.
RBF support in Hiro Wallet / Connect etc: Covered in ample detail by Mark earlier in this thread
For exchanges: Or anyone else using the send-many contract + CLI, v1.3.0 of the CLI now supports a fee multiplier. It already supported specifying a nonce, and together this enables one to RBF their existing send-many transactions.
Obviously performance and reliability at the blockchain layer directly affects developer satisfaction and so I want to take a moment to talk about what Hiro would be focusing on. We can think of possible improvements / changes / new capabilities on the Stacks blockchain in the short (4-6 weeks), medium (1-2 quarters) and long-term (6-12 months).
In the short-term, Hiro is focusing our efforts on three workstreams:
Facilitate a healthy fee market: As I mentioned earlier, the lack of a robust and dynamic fee market on Stacks makes the chain less resilient. An expensive but popular contract call can make the entire network crawl. We have already started implementing our proposal for improving cost estimation and mempool processing. While this won’t fundamentally change the underlying blockchain capacity, it will allow for more natural throttling and traffic control to emerge via transaction fees.
Profile, Benchmark, Optimize: We’d like to dramatically improve the visibility into the current performance of the Stacks blockchain – exactly where time is being spent (e.g. block assembly, block propagation, block assembly, MARF I/O etc). We’re also going to run some rigorous benchmarks to understand the limits of the current implementation under different workloads (e.g. how many contract-calls for a specific contract/call can be packaged in a block). This profiling and benchmarking will shed light on the biggest bottlenecks in the current codebase which we can then work on improving. We’re also working on integrating some of this benchmarking within Clarinet, so smart-contract developers have more visibility and insight into expected costs and performance of their Clarity contracts.
Explore feasibility of Clarity cost-voting as a non consensus-breaking solution: Changing block limits and runtime costs would normally be a consensus breaking change (necessitating a hard-fork). Clarity supports in-situ upgrades, which allows changing runtime costs through an on-chain vote. Hiro is currently validating this on the testnet and exploring the general feasibility of this approach.
There’s also amazing ongoing discussion around topics like nonces, block capacity, performance, microblocks etc on the community Discord server. Join us there!