Orphan Blocks: Workstream Update Thread

What is the ‘orphan block’ issue?

There have been more orphan blocks and reorgs (more than 2 blocks) on the Stacks network for a few months. Such reorgs happen when miners or sets of miners try to work on different forks potentially because they’re unaware of the presence of the other fork. Many of these reorgs appear to correspond to different mining groups, one of which seems to be a Stacks mining pool or mining service comprising a large share of the mining power on the network recently.
Based on funding patterns, we have reason to believe this mining pool or service is composed of multiple independent miners. Further, from the network behavior it seems that this mining service/pool is connected through a low-bandwidth connection or a remote geographic location (it’s hard to say exactly what region or how low the bandwidth is); this results in the mining service working on forks that other miners might not be aware of.

For the orphan blocks issue, open-source devs participating in working groups are running network diagnostics to determine values for the miner config settings that could help improve this issue in the short-term. There are also software patches under consideration that can help improve/resolve the problem in the long-term.

Impact

The main impact is on exchanges/partners and downstream, that negatively affects users directly. For exchanges, they can see transactions rolled back and possibly dropped - which causes them to increase the number of confirmations required (which in turn affects users where it takes longer to transfer tokens, etc). Secondly, it affects miners directly - they are spending bitcoin to mine a block, and then losing the reward of mining that block if they are orphaned. The orphaning of blocks also negatively impacts the performance of the chain, because as longer forks are mined there may be some transactions that are reverted to the mempool or will generally take longer to be confirmed in an anchor block.

Further information and links

To better understand this issue:

Workstream Information:

Lead: Jesse Wiley

Status: A significant amount of diagnostic work has been done, with more ahead.

What’s next: This is a particularly difficult issue to zero on the underlying cause for so the developers working on this issue are focused on surfacing useful network diagnostics and creating tools to gather the necessary data. The focus now is on increasing visibility into the state of the network, with first steps being to “map” the network peers, in particular, the neighbors of miners. There are also plans to launch more dedicated seed nodes in various geographical regions to see if that helps what so far appears to be a peering issue.

How you can help: Comment on the SIP proposal

:rotating_light: For continuing updates: Please follow this thread, updates will be posted as replies below by core developers and contributors.

2 Likes

Some discussion happened here: Orphan MEV
with a proposal to award rewards for miners regardless whether the mined blocked was orphaned or not.
While this does not fix the chain quality, it helps to keep the mining marketplace open and rewarding for all miners. The solution could be integrated into the bitcoin MEV solution.

1 Like

To add a small update to the work that is being planned here - there are a couple of assumptions and observations that I’ve made that need to be investigated further.

The planned actions here involve firstly collecting data. Thanks to @rafael @zone117x for the repo: GitHub - hirosystems/stacks-scout, which should make this task a bit easier.

The second is to (in)validate some assumptions that I have once I have a better picture of what the entire network looks like from a connected peer level. There are some other metrics I’ve noticed on a small scale that I’ll be looking into (spot checking several recent orphaned blocks all showed that the orphaned stacks miner had an older view of the bitcoin network, essentially they were a block behind the winning miner). It’s too small of a sample to make any inference, but it was interesting and older orphaned blocks need to be checked for similar data.

Lastly, and this had been planned already - we’ll be doing some work to strengthen the bootstrap nodes of the network by both running more of them, and distributing them more globally.

Will keep this thread updated as work progresses and/or anything new is discovered or ruled out.
Thanks to @friedger as well for linking that discussion.

1 Like