Orphan MEV

bowtiedfox · April 8, 2023, 6:55am

Recently there has been a lot of discussion about the “Orphan MEV” issue that currently plagues the Stacks blockchain, primarily in RFC: Minimum-bid PoX (to thwart Bitcoin MEV). There were several suggestions to break the discussion out into a dedicated thread, but I haven’t seen that thread materialize yet. So, here it is.

Summary

There is a coalition of Stacks miners who coordinate to orphan the blocks of miners who are outside of the coalition. Because the coalition controls >51% of mining power (currently very close to 100%), they can successfully do this as often as they want. This makes it unprofitable for new miners to enter the pool, which in turn eliminates competition and allows the coalition to pay an artificially low price for STX rewards, which in turn lowers the return on stacking. Currently the coalition clears upwards of $30k per day at the expense of stackers, other miners, and would-be miners.

This has apparently been a known issue among Stacks leadership since November, though many users are learning about it for the first time. Per anecdotal observations, the problem has been getting noticeably worse in recent weeks/months.

There is also speculation that the coalition is controlled by a single entity, though of course this can’t be proven with 100% certainty. However, @Metastack_BMC has provided cryptographic proofs suggesting that this is the case.

For more context, I’d encourage you to read the other thread if you haven’t done so already. In particular, see the original post from @Metastack_BMC and additional context from @yukan.

Proposed Solutions

I think this thread would be a good place to brainstorm additional solutions. I’ll drop one of my own below and would encourage others to do the same.

bowtiedfox · April 8, 2023, 7:51am

I wanted to keep the OP heavy on facts and light on opinion. So, here are my opinions on this issue:

I believe that the coalition is likely controlled by a single entity. At best it’s a few coordinated entities who are carrying out a deliberate value extraction scheme. It is certainly not the product of an innocent bug or any of the other more benign explanations that were offered in the other thread. When you really look at the data, the behavior is simply way too coordinated and aggressive for those explanations to make sense.
I believe that 1-3 entities controlling a $1B blockchain and censoring others from joining the mining pool is an unmitigated emergency. To the extent that there’s competition for engineering resources, I’d argue that Orphan MEV should clearly be prioritized ahead of Bitcoin MEV. I’d also argue that it should be prioritized ahead of sBTC and any other top priorities.
I believe that the issue is totally fixable.

A Few Assertions

Before I get into my own proposed solution, I want to lay out a few assertions. If you don’t agree w/ these assertions (which is cool), you probably will not agree w/ my proposed solution. So I want to lay them out systematically and make a case for them:

Assertion #1: Forking is an anti-pattern

As I see it, there are only 2 reasons for a miner to fork the chain:

It was an accident
Manipulation

(1) is a catchall for any benign technical reason that a miner may fork the chain (e.g. flash blocks, network too slow, bug in mining code, etc.). (2) is what the coalition does. I am truly curious to see if others can put forth reasons outside of these 2 buckets, b/c in my (admittedly non-expert) mind this is the complete list.

In other words, forking is an anti-pattern. I truly can not think of a scenario where it’s better for the network for a miner to create a fork vs. simply mining from the longest chaintip. Forks create confusion about the canonical state of the chain, reduce throughput, and serve as a vector for manipulation and value extraction. They are an undesirable event, and it is completely legitimate to disincentivize them at the protocol level.

Assertion #2: Stacks current protocol design encourages forks

I think all of us understand the macro benefit that the coalition gets from incessant forking of other miners: they eliminate potential competion.

What I haven’t seen discussed is the micro benefit. And I’ll preface this by saying that I am not 100% sure on this point, so it’s really more of a question than an assertion, which is: How are mining rewards handled in the case of a (successful) fork?

stacks-dump seems to suggest that the miner gets their normal 1k STX reward for the block, but logically that doesn’t seem right to me. It seems more likely that they would get a larger reward similar to what happens with missed blocks. IF that is the case – and it would be great if someone who really knows can confirm/refute – then the coalition’s incentives to fork are about more than squashing competition. The forking operation would be not just a mechanism to deny a potential competitor a block reward, but instead a mechanism to effectively steal the other miner’s block reward.

So micro benefit is a question to me, and if it’s real I think it’s very much worth calling out. That said, I’d stand behind this assertion based on macro benefit alone. Admittedly it does need a qualifier though, so perhaps I should amend to “Stacks current protocol design encourages forks if you control >51% of the mining power”, which unfortunately someone does.

Assertion #3: Designs that explicitly penalize excessive forking are the most direct way to combat Orphan MEV

There have been some interesting proposals on how to combat Orphan MEV, but I’ve not yet seen one that seeks to explicitly measure and penalize excessive forking. To me this is the most direct and likely the most effective way to make the behavior stop. It can and should be combined with other approaches like compensating orphaned miners, but penalizing excessive forking is a natural tool to address Orphan MEV.

Proposed Solution: Slash and Redirect Mining Rewards When Orphan Frequency Is Too High

Here’s the idea:

Every X Bitcoin blocks, calculate an Orphan Frequency O, defined as the number of orphans divided by the total number of Stacks blocks mined. Feed O into a slashing function F(O) and slash mining rewards on any “forking blocks” within the X block interval, where a forking block is defined as any block that creates one or more orphans. If slashing occurs, redirect a percentage P of the slashed funds to orphaned miners, pro rata based on the number of orphaned blocks they mined in the interval.

Attempting to set parameters, I would suggest X=100 and P=0.5. Designing a good F is probably the trickiest part, and I think that would be worthy of discussion if the group wanted to get serious about this proposal. But as a jumping off point I’ll throw out a uniform distribution between O=5 and O=15. In other words, if fewer than 5% of blocks are orphans then there is no slashing, and if >=15% are orphans then forking blocks are fully slashed (i.e. miners of forking blocks receive nothing). For values between O=5 and O=15 the slashing percentage grows linearly from 0 to 100.

Let’s walk through an example with hard numbers. Imagine that the coalition controls 100% of the mining power. A new miner attempts to enter the pool at the beginning of a 100 block interval, and they put up enough capital to claim 15% mining power. In the subsequent 100 blocks, then new miner wins 15 sortitions while the coalition wins 85 and orphans all 15 of the new miner’s blocks. In other words, O=15 and the coalition’s forking blocks are fully slashed. Under the assumption in Assertion #2 that forkers effectively steal the reward of orphan miners, the total slashed rewards are 30k STX, and P=.5 implies that the new miner is made made fully whole on his/her orphan blocks with a reward of 15k STX (if the “stolen reward” assumption is not correct, then I’d advocate for P to be higher). The coalition receives 70k STX for their 85 blocks, and 15k STX is burned.

There are a couple things I’d point to here to suggest that this design would quickly break the coalition’s monopoly. First, the new miner has earned fair value on his/her contributions to the pool and should be able to mine profitably and thus remain in the pool. Even if the coalition pursues a different and arguably better strategy and forks the new miner exactly 5 times, the new miner still earns 10k STX, which is still somewhere near breakeven. The new miner also has countermoves (increase capital commitment, fork back) to improve profitability. In the end, the coalition simply does not have a viable strategy to lock even a small-ish competitor out of the pool.

The second point is that not only did the coalition fail to lock the new miner out of the pool, they did worse for themselves by trying. They could’ve had a total reward of 85k if they’d mined honestly, but instead they ended up with 70k. This is stark contrast to the situation today, where they earn money by manipulating the chain rather than lose money.

Put these things together, and I think this change would quickly break the monopoly.

Curious to hear thoughts and additional ideas from others.

jonkho · April 8, 2023, 6:29pm

This reminds me of another project called decred (no association) did a hybrid PoW-PoS where they had separation of concerns between the actors doing the job of block production and actors who performed chain-tip decision-making.

The miner’s job was to produce blocks; they did not get to decide which fork gets to be the canonical fork.

The staker’s job was to receive the block and build the canonical fork (they may have used voting because their staking lingo was “buying tickets”).

The maintainers often spoke of how fork resistant the chain was compared to bitcoin.

bowtiedfox · April 9, 2023, 2:01pm

Interesting. And it does seem like you could do a similar thing w/ stackers, which I believe has been discussed here and there.

And I’m stating the obvious, but it’s probably worth stating anyways: Stacks chain history would be dramatically more difficult to manipulate if stackers determined the chaintip rather than miners. It takes maybe $100k to gain 51% of the mining power. Gaining 51% of the stacked STX would take what like $200M+?

jonkho · April 9, 2023, 5:17pm

Sip-21

Proposes something similar except stacker support comes in only after a 7-block fork. Decred makes it’s a 1-block decision.

bowtiedfox · April 9, 2023, 5:50pm

Interesting. I guess at that point the question becomes, what is the value to the network in continuing to allow miners to create <7 block forks?

I’d argue that there’s little to no value in allowing those forks (Assertion #1 above), while the cost is huge b/c it exposes a vector for manipulation. So perhaps the best and easiest route to address Orphan MEV is to simply amend Sip-21 such that stackers determine the chaintip every block?

jude · April 10, 2023, 3:21am

~~I mentioned in a sibling comment thread that if there are B Bitcoin blocks between two Stacks blocks S and S+1, then the winner of Stacks block S+1 earns B coinbases.~~

This gives me an idea: what if they only receive coinbases for Bitcoin blocks without sortitions? The reason it’s not unprofitable to orphan blocks in Stacks is because the miner doing the orphaning (the “Entity”) is going to receive their coinbases anyway. If that weren’t the case, then the most profitable course of action for the “Entity” would be to confirm the highest chain tip.

EDIT: I was mistaken – it is already the case that the winner of Stacks block S+1 receives coinbases for only the Bitcoin blocks without sortitions. If a Bitcoin block between Stacks blocks S and S+1 had a sortitions on a different fork, then the winner of S+1 does not earn a coinbase for it. So, we’re already at the place I described.

bowtiedfox · April 10, 2023, 1:49pm

I like the direction you’re going but want to make sure I’m clear on what you’re suggesting.

Let’s say Miner X mines Stacks block S at Bitcoin block B. This block builds on Stacks block S-1 at B-1 and is the tip of the longest chain at the time that it’s mined. Miner Y then comes along at Bitcoin block B+1 and builds Stacks block S’ on top of S-1. They later defend the fork such that S’ becomes canonical and S is orphaned.

Under the current system, Miner Y gets 2 coinbases for block S’, while Miner X gets nothing for block S.

Under your suggested change, Miner Y would now get 1 coinbase for block S’, while Miner X would still get 0 for block S? Is that correct?

jude · April 11, 2023, 4:25am

Hey @bowtiedfox, see my edit to my comment

jude · April 11, 2023, 4:39am

Given my above edit, an alternative idea could be that Stacks does not award a coinbase for any sibling blocks. The order in which Stacks blocks at height S are mined is well-defined. If Stacks block S has a sibling block-commit, then the system could just not award S’s miner.

bowtiedfox · April 11, 2023, 1:07pm

@jude I like it, but I would go a step further and say that S-1’s miner still gets their coinbase, even if S-1 does not become canonical. So basically, the decision rule would become: if you mine from the chaintip, you get a coinbase(s). And if not, you don’t.

At first it may seem weird to pay miners for blocks that didn’t end up in the canonical Stacks blockchain, but I think it’s fair and incentive-aligned with what the network should want. Miners can only control what they do when they win sortition. They can’t control what happens to that block afterwards (unless they control >51% of the network). So if they mined off the chaintip, then they did their job and should get compensated. And if they didn’t, then they shouldn’t (as you suggested above).

More importantly though, paying out those orphaned miners slams the door on The Entity’s ability to defend its monopoly. Currently it’s free for The Entity to create/defend their monopoly, so of course they do so. Your change is a great step in that it creates a cost to defending that monopoly; if a new miner tries to enter the pool, the Entity will need to forfeit money in order to lock that miner out. Clearly that’s not short-term optimal for The Entity, but could it still be medium/long-term optimal? That depends on how you assess the value of the monopoly and the expected cost of defending it, but it certainly might be. Who knows how The Entity (or anyone else who gained 51% control) would assess that, but I do know that they would still have the option to lock other miners out of the pool, so long as they’re willing to pay the cost.

If you pay out the orphaned miners, then that option is gone. A controlling miner simply does not have any levers to censor the payments of other miners, and their only rational choice is to mine honestly.

jude · April 11, 2023, 7:35pm

Keep in mind that a malicious node could mine a block and then never publish it. The rest of the system would see the block-commit, but it would be unknown as to whether or not the block exists (after all, someone could have just created a throw-away block-commit transaction).

Any scheme that pays STX to someone for mining atop the canonical chain tip must ensure that publishing that Stacks block is a pre-condition for getting paid. This makes it challenging to pay miners who mine Stacks blocks that are later orphaned – somehow, the fork that orphaned them needs to be forced to acknowledge the existence of that orphaned Stacks block. That’s why I originally proposed the block pre-commit system, which does this.

bowtiedfox · April 11, 2023, 8:28pm

Gotcha. So if “validating” a block-commit is not something that can be done automatically in the code, then returning to the suggestion by @jonko above, is that something that stackers could do? And if so, is there a good reason that they’re only asked to do so (in SIP-21 anyways) after a 7 block fork rather than simply doing it every block?

bowtiedfox · April 11, 2023, 8:36pm

I guess another way to put my question is, could stackers do the job that pre-commit voting is intended to do? My main concern about pre-commit voting is that (if I understand it correctly) it may end up as just another vector for a controlling miner to wield monopoly power, whereas stackers should be more impartial on average and therefore more likely to vote honestly.

jude · April 12, 2023, 2:13am

I think it’s of paramount importance to keep block production separate from ownership of STX. Otherwise, stackers are just stakers and Stacks is a PoS chain – the already-rich decide who (if anyone) gets to mine, and have the power to do things like require that miners grant them a share of their coinbases to ensure that the STX rich get richer.

bowtiedfox · April 12, 2023, 3:01am

I’m open to other solutions that don’t involve stackers, just think it’s important to definitively slam the door on The Entity’s monopoly power. Otherwise we may end up back in the same degenerate equilibrium we’re in today, only w/ The Entity extracting a bit less money.

Admittedly I’m out of my depth when talking about low-level details of the Stacks blockchain, how blocks get published, etc., but I have to say I still don’t really understand this part:

Any scheme that pays STX to someone for mining atop the canonical chain tip must ensure that publishing that Stacks block is a pre-condition for getting paid

I guess I’m unclear on why this is hard. Is it really not possible to write a piece of code that would validate that a miner has published his/her Stacks block? At the very least it seems like you could check for the extreme case that you mentioned, where a miner simply created a throw-away transaction?

jonkho · April 12, 2023, 5:49am

Pertaining to SIP-21: what is the reasoning that selects the fork length of 7 as the length before stackers should intervene?

Assuming stacker’s intervention can fairly referee the correct fork at length 7, what’s stopping stackers from refereeing correctly at lengths 6,5,4,3,2, and 1 ?

If there exists a forking strategy that grants forkers an economic advantage then eventually coalitions of miners will form, orphaning each others block while vying to create their own proprietary 7-block forks for stackers to tie-break. If stackers can decide fairly then what is stopping them from doing so at fork lengths 6, 5, 4, 3, 2, 1?

If stackers decision can be bribed/corrupted when deciding a 1-block fork, then it seems it can also be true for refereeing fork lengths of 2, 3, 4, 5, 6 or 7. Unless there is special property at fork lengths of 7 that prevents this. What is the logic behind choosing 7?

jonkho · April 12, 2023, 6:46am

This is the case where the block producer could be accused of withholding the block, while the block propagator could be accused of orphaning the block. How can you tell the difference in order to determine why the block did not propagate? What the litmus test to determine this?

One general 5000-ft-view approach to solve this is that sortition participants must provably have a block that is referenced that can automatically be dereferenced on demand by anyone. Otherwise that participant cannot be declared the winner of the sortition.

jonkho · April 14, 2023, 2:37am

I’d like to propose the following:

Miners have stricter requirements for accepting the longest fork. Works as follows: suppose miner Alice is building off of the chain tip block S. If miner Bob is the orphaner and proposes a longer fork with block S’ looking to displace block S, it must be required that the amount of btc committed for S’ must already be greater than that of block S. Otherwise bob’s fork is not accepted by Alice. In a sufficiently connected network of honest miners, Bob is forked off into his own chain state.

This may or may not completely solve the issue of monopoly mining but might a step in the right direction:

if the profit margin of block S is already very slim, Bob in his attempt to produce a acceptable S’ may be doing so at an economic loss and therefore there is no economic incentive to produce such a fork.
this will incentivize all miners to produce an aggregate bid close to market bid of Stx/btc. After all, their margin is another miner’s opportunity. This also solves the price dislocation we currently have in the market price of the block reward.
if block S is actually being withheld, other miners, given time, can route around that by slightly out-pricing that block and recover the block reward. Therefore block producers are incentivized to quickly propagate their block.

Has this idea been explored?

friedger · April 24, 2023, 11:40am

@jonkho Jude suggestion Orphan MEV - #10 by jude to not reward any miner that does orphan a block goes in the same direction of decreasing the profit for orphaner.

Stacks Forum