A bug in stacks-increase call is impacting Stacking rewards this cycle

Core Blockchain
#1

Hey all, I’m capturing the latest information from an ongoing discussion in Discord among community members about a discrepancy with Stacking rewards this cycle. I wanted to surface this here for better visibility and to provide a place for folks if they have questions. First, a big shout out to @friedger as well as Joe Crypto and yoyo liber for discovering this and jumping in with other core developers to get to the bottom of it.

What we know

  • There is a bug in the stack-increase function in the pox-2 contract. The bug affects the contract’s internal state of stacked amount, for the address that calls stacks-increase.
  • The bug is allowing an address (currently aware of just one) to earn more than its share of Stacking rewards associated with its reward slots.
  • This will reduce the overall payouts to other Stackers for this cycle. The current best estimate is about a 50% reduction in rewards for Stackers this cycle, but we can’t be certain of the total impact until the cycle ends.
  • This does not impact callers of delegated-stacks-increase (e.g. stacking pools), other than in terms of the reduced rewards.
  • Data returned from the /v2/pox endpoint in the current_cycle section will be inaccurate for the remainder of the cycle.
  • The current data suggests this bug was activated unintentionally and was not a deliberate attack.

What’s next
Core developers and other Stacking pool experts in the community are working through the best possible solutions. It seems likely that a fork/upgrade will take place before the start of the next stacking cycle to address the bug. Different solutions offer different diagnoses for effects on upcoming rewards, however, the leading proposal would fix the issue while maintaining correct rewards next cycle!

If you know this Stacker (Reward address: bc1qpyjutel6d4gj50dscphjrqcp29ljtfjel7ccap), please recommend they get in touch via [email protected] for a white hat bounty offer for returning the rewards to fellow Stackers.

Updates will be posted to this thread as they are available, thanks!

15 Likes
Stacks 2.4 is here! 🚀 (Stacking to be re-enabled)
Xverse has been failing staking for 2 months No staking or rewards, funds locked
Breaking down the recent Blockworks report
#2

Thank you. We should take this opportunity to also look at testing coverage, procedures, etc - from the outside looking in this looks like a simple test case.

7 Likes
#3

Thanks for the update! Good to catch the issue. The potential fix looks simple enough and getting that done before the next cycle can be great to minimize impact on stacking.

+1 on increased testing and more bug bounties. As the ecosystem grows, the best way to harden the code is by discovering issues and fixing them and keep iterating that process rigorously; with more real-world use the systems will harden over time. Appreciate all the hard work that various open-source devs put in and appreciate people like @friedger who helped discovered the issue :pray:

11 Likes
#4

Here’s a SIP for the emergency upgrade: SIP-022: Emergency Fix to PoX Stacking Increase by jcnelson · Pull Request #127 · stacksgov/sips · GitHub

5 Likes
#5

We’re grateful that this bug was caught early, and we owe a big thanks to Friedger for bringing it up. Thank you for providing such a concise and clear summary, Mitchel.

4 Likes
#6

Where can we learn about the progress regarding this? I have not seen much on the Stacks Discord. Is there a Console channel?

2 Likes
#7

I think I answered my own question … FYI for those asking … Enable PoX-3 · Issue #3681 · stacks-network/stacks-blockchain · GitHub

2 Likes
#8

Hey again everyone, as you may have seen on Github and Discord, the community and various developers in the ecosystem have been working hard to assess potential solutions to the bug related to the stacks-increase function. Again, I’m offering an attempt to summarize these efforts and make it fairly easy to digest in the interest of transparency and community awareness.

It seems discussions around potential solutions are converging on the idea that resetting the PoX state and letting everyone Stack again once the issue is resolved, is the cleanest path forward. This approach also has the added benefit of allowing core developers and the community more time to test solutions during the upcoming cycle.

Broadly speaking, the options that were recently evaluated can be put into one of two buckets:

a) Implement a fork that attempts to fix the incorrect PoX state caused by the bug.
b) Implement a fork that resets the PoX state and allows users to stack again.

Fixing the incorrect PoX state (Option A) caused by the bug is much more complex and doesn’t appear technically feasible to pursue within the short time frame before the beginning of the next cycle. A hasty solution could have unintended consequences and the Stacks ecosystem prides itself on security and a methodical approach to everything from upgrades to bug triage.

This leaves Option B.

What Option B would mean:

  • The network would move forward by reverting to Proof-of-burn instead of PoX while a fix is further tested. This means no Stacker will receive BTC rewards for at least the upcoming cycle. It is possible that depending on technical progress and testing, a solution takes longer than one cycle. In that case, the network will continue running Proof-of-burn until a solution is implemented.
  • Pending the fork, all Stacker’s tokens will be unlocked regardless of how long they had delegated or pooled for. This enables a clean state so people can lock again later.
  • Core developers will continue working on and testing the pox3 contract with community to replace pox2, ideally during the next cycle.

Why Proof-of-burn?

To offer my own imperfect analogy, Proof-of-burn is effectively a ‘safe mode’. Further, it’s already ‘built-in’, and is a proven, established part of normal Stacks consensus. Reverting to Proof-of-burn allows the network to serve users somewhat as normal (sans the Stacking reward element for Stackers) while developers write a new version of pox3.

As a reminder, PoX on the network is activated by Stackers when more than 5% of liquid supply lock their STX in consensus. Otherwise, the protocol automatically reverts to Proof-of-burn under which no STX holder gets the BTC rewards (and mining works by sending miner bids to a burn Bitcoin address). There is even some BTC burned during normal protocol operations as well e.g., during the switch from one reward cycle to the next.

Given the bug is resulting in an incorrect PoX state where some participants can earn more rewards than STX they locked, the fairest and most decentralized option is to reset the PoX state, push the version that fixes the bug, and let everyone lock again later. The established rule of thumb from a protocol fairness perspective is that whenever there are any safety concerns over which addresses the BTC rewards from Stacking should go to, BTC rewards default to being burned so no party can have a claim on them, and the protocol can remain neutral and decentralized.

Next steps

  • An emergency SIP (SIP-022) has been drafted and is being reviewed by CABs. Final approval is expected by the end of day 4/26/2023. This will approve two forks. The first will reset the PoX network state and revert the network to Proof-of-burn consensus while a fix is further developed. The second fork would introduce the new PoX (pox-3) contract, fixing the function causing the issue and reenabling PoX/Stacking.
  • Core developers will continue working on an updated PoX contract and address the underlying issues with the stacks-increase function. This will be deployed via the second fork noted above.
4 Likes
PoX-3 (SIP-022 aka Release 2.4) progress update - testing will go into Cycle 59
#9

Should we revoke delegation now?

1 Like
#10

The pox-2 bug stems from the stack-increase function intertwining side effects, such as database modifications, with the logic determining the state changes. This issue is further exacerbated by the (mis)use of reward-cycle-total-stacked as a global variable that retains state across successive iterations.

There is an opportunity to leverage that Clarity is at its core a functional, expression-oriented language to improve contract testing and develop more robust and maintainable smart contracts. The functional and expression-oriented nature makes Clarity a suitable candidate for static analysis and formal verification techniques. Functional programming encourages the use of pure functions, which have no side effects and always produce the same output for the same input. In contrast to statements with side effects, pure expressions evaluate to predictable values and types without affecting global state.

By separating the side effects from pure functions with the logic computing the updated values, it becomes easier to reason about the contract’s behavior and write comprehensive unit tests.

1 Like
#11

interesting idea - there won’t be time to consider this for the current work in progress (unless i’m misunderstanding what you’re proposing), but it would be a good idea to look into for a future upgrade to the clarity VM.
could you create an issue so it can be considered after the current work is complete?
https://github.com/stacks-network/stacks-blockchain/issues

2 Likes
#12

Hey all, a release that begins to address the ongoing bug affecting Stacking is now available!

Core developers have recommended, and a SIP has been approved, for disabling PoX for one or two cycles while a bug in the stacks-increase function in the pox-2 boot contract is addressed. This is an established route for addressing bugs like this and will ensure adequate time for further testing of the next release.

Conversations between developers, contributors, & community converged on an option that disables PoX as a first step in addressing the bug. SIP-022 approves this fork and introduces a rule to temporarily disable PoX and switch fully to Proof-of-burn as previously outlined. Post-upgrade, the network will ignore any non-upgraded nodes.

This option has been recommended because it is technically sound and follows an established route for addressing bugs like this. It will also ensure adequate time for further testing of solutions. Various community reviewers, CABs, and the SIP process have been followed/included.

Notably, all STX will automatically be unlocked from the network, affording users the most flexibility during this time. However, Stacking rewards will not be available for these 1-2 cycles while work is underway. Exchanges and Stacking Pools have been notified about the upgrade. Normal trading and deposits/withdrawals should not be impacted as long as exchanges upgrade.

This upgrade should be simple since the chainstate is compatible with 2.1.0.0.0. There will be another upgrade in 2-4 weeks after the bug has been resolved and Stackers will reactivate PoX by Stacking as normal.

Key Resources:

1 Like
PoX-3 (SIP-022 aka Release 2.4) progress update - testing will go into Cycle 59
#13
#14

Countdown tool for tomorrow’sstrong text 2.2 hard fork:

https://stacks-network.github.io/when-rewards/2.2/

1 Like
#15

I couldn’t find concrete information about this bug regarding when certain steps will take place regarding unlocking previously locked STX.

Has previously locked STX been unlocked (my STX is still locked from 9 days ago when I locked it successfully)?
Has the Proof of Burn protocol been implemented?

I currently have my STX locked, with no way of unlocking (registration open page)

My STX address is SP25DP4A9QDM42KC40EXTYQPMQCT1P0R5243GWEGS,
The successful contract lock (inability to unlock right now): https://explorer.hiro.so/txid/0x32da46931655c1b7b6b753c4cd313c508857829f6fa2d00bf418eaa07adac787?chain=mainnet

No shade, any thoughts/clarification?

2 Likes
#16

@highrollerBTC
First, the hard fork to unlock tokens was activated roughly 6 days ago, and was announced via the mailing list and many other places when it went live after going through the SIP process to approve of the change.

Second, the tx you shared was confirmed before the activation of the first hard fork defined in SIP-022, 2.2.0.0.1, so it would have been correct at the time as though you had successfully stacked tokens.

However, per SIP-022 - all stacking was disabled at the start of cycle 58, or bitcoin block 787651, meaning the tokens you had locked for stacking for the start of cycle 58 were unlocked when the cycle started.

Current chain data also supports this assertion that your tokens are not currently locked:

➭ stx balance SP25DP4A9QDM42KC40EXTYQPMQCT1P0R5243GWEGS
{
  "balance": "3304177047",
  "locked": "0",
  "unlock_height": 0,
  "nonce": 15
}

Work is ongoing for pox-3, and is expected to reactivate stacking rewards for cycle 59, bitcoin block 789751 which arrives in roughly 10 days. Like we did for the 2.2.0.0.1 hard fork, we will notify people far and wide of the second hard fork release. Once 2.4.0.0.0 is activated and cycle 59 starts (again, roughly 10 days from now) stacking will be re-enabled.

https://stacks-network.github.io/when-activation/2.4/

2 Likes
#17

Super helpful, thank you!

Clarification for others, I didn’t see the 0 locked STX when I loaded my Xverse, as it still says locked.

Also thank you for this link, I’ve been waiting for this one! :smiley:

#19

Update posted in a fresh thread for easier visibility: PoX-3 (SIP-022 aka Release 2.4) progress update - testing will go into Cycle 59

#20

if you use the xverse mobile app, you may want to uninstall and reinstall to see your Unlock STX. before you uninstall save your seed phrase and clear data in application manager from your mobile app.

1 Like