I’m not involved in the day-to-day, but I’m a bit surprised this seems to be the first time quite a few people are learning of it.
It is but if we implement a proposal like Jude suggested without a fix for the centralization issue, the last small miners like SP2P would be forced to stop as well.
@muthu.btc I believe what they’re saying with the above is that if the current MEV miner is only mining Stacks due to the highly profitable MEV available, then removing that profit would also mean that miner stops mining entirely.
Of course, they could still continue mining as a normal miner and make the overall profit margin other (large) miners are making, but assuming they only started mining due to the MEV, and would stop if MEV is effectively removed, all else equal, that would make the current relatively centralized mining even more centralized.
Yes I agree that it returns to status quo on centralization. What I am asking is how does this make it less profitable for small miners than it already is ? Is that because the large miner is able to do more re-orgs?
Them stopping entirely = more centralized mining = the current big miner can continue excluding other miners unless new equally sized miners enter the field
@MattySTX Forgive me if I’m asking a noob question: aren’t stackers supposed to vote on the chain-tip? If a small miner produces a block and appends the chain, stackers are supposed to acknowledge that chain tip? How can a party of miners just arbitrarily orphan a block and decide what the chain-tip is?
Should stackers have their own “full node” software to participate in consensus analogous to btc full-nodes so when blocks propagate the nodes can confirm/witness as well?
Correct, due to the ongoing “Sybil attack” the profit margin of the MEV miner will fall dramatically if they have to raise their commit. Furthermore, small miners like SP2P won´t stay online if they have to raise their commit as well (atm they simply act as an “safety backup” if the entity has some problems with their miners.)
I think you are talking about SIP21 (its´s a draft and not implemented). Furthermore, only forks of at least 7 blocks deep will need Stackers support (Link)
Even if a new honest entity with 51% would enter the field, with the standard 2.1 release they won´t be able to compete against the “bad” entity. Because as soon as the entity builds a longer fork, the honest miner would choose it to build the next block on it.
That’s why we proposed a solution like this-link and appreciate the Judes proposal in RFC: L1 Speed Improvements (pre-commits).
I think aiming for 90th percentile of the bids in the last 1,050 blocks (so approx one week) is one way to do it.
If the trading price of Stx/btc falls below this price floor of 90th percentile over the last 1050 blocks, would Stx block production stop due to misaligned economic incentives? I.e it’s cheaper on average to buy Stx on the market than to mine?
I’ve been following the issue with the single entity miner ignoring honest miner blocks. I think this is way more serious than the MEV issue.
I’ve performed experiments myself and can fully back up @Metastack_BMC’s observations.
No honest miner is able to mine profitably right now. The total miner burn per block in the past month has been way lower than it should be, which lowers stacking yield significantly. The dishonest miner has no incentive to increase BTC burn to match the current price of STX. And no other miners can join because they will lose money.
It was believed that the issue might be miners using old Stacks node versions which had bugs that caused it to ignore other miner blocks. And that the 2.1 update/fork was going to force an update and fix this. It has not.
And if I’m not mistaken it’s critical to get this fixed before the release of sBTC. If the miner burn rate is going to be used as an on-chain oracle for BTC STX price and ultimately determine the maximum sBTC supply, with the current situation, the oracle will be wildly inaccurate.
I’m not sure I agree with this. MEV is possible because miners are able to deduce the true value of a transaction before the transaction is mined (or sufficiently confirmed), and take actions to extract that value at a profit. If they are able to do this, then of course they will try and capture extra value from the transaction – if they don’t, then other miners will, and out-compete them.
This suggest that the defining characteristic of a MEV-free protocol is its ability to ensure that any value-extraction strategy would never be profitable. We don’t have to share MEV among miners; we just need to ensure that any MEV strategy they could take can never make them more money than honest participation. I think that this is achievable in general with careful protocol design, and can be done specifically with Stacks mining.
Recall that Bitcoin miners rely on two means of extracting value. They can:
Decide the order in which transactions are mined in their blocks
Decide which transactions get included in their blocks
Today, Bitcoin MEV miners extract value from Stacks mining by ensuring that only their block-commit transactions get mined in the Bitcoin blocks they produce. Then, they’re guaranteed to win the Stacks block reward for a minimal cost to themselves (namely, the cost of a forfeited Bitcoin transaction fee).
What would a MEV-free mining protocol look like? A MEV-free mining protocol would need to prevent Bitcoin miners from underpaying for STX by means of delaying or reordering Stacks miners’ Bitcoin transactions. I think this could be achieved by lifting all Stacks mining to an open-membership mining pool, and rendering that the de-facto mining mode. Beyond that linked post, I think we would specifically require the following behaviors:
Stacks transaction activity happens mainly within microblocks, not blocks. Microblocks are signed by a majority threshold of miners (e.g. 66% or more by BTC weight) before they confirm, and this all happens off of the Bitcoin chain. Microblock stream budgets grow dynamically as a function of provable time delays (via a VDF), so the chain nominally remains live regardless of what happens with Bitcoin blocks.
Every so often, Stacks miners collectively send a “checkpoint” transaction on Bitcoin. This checkpoint commits to the current tip hash of the microblock stream, and consumes any UTXOs sent by Stacks miners wishing to join the pool. It also re-seeds the VRF and resets the block budget VDF, and pays PoX recipients. Checkpoint transactions are targeted to every Bitcoin block, even though some of them may not get mined right away (e.g. due to flash blocks or attempted MEV behavior). Checkpoint transactions replace sortitions.
PoX payouts are a function of the aggregate BTC held by the pool, and cannot be chosen by miners. If a miner commits X BTC to the pool, then in checkpoint transactions S through S+N, the PoX payouts increase by X/N BTC. A checkpoint transaction is not valid unless this is true. The miner receives a pro-rata share of the block’s STX as a function of what percentage of the BTC they were responsible for committing in each checkpoint S through S+N. Neither Stacks miners nor Bitcoin miners can change the rate of PoX emissions – they can only slow down checkpoints. Below, we’ll make the act of slowing them down unprofitable.
Checkpoints trigger STX disbursement to the miners and PoX payouts to Stackers. Reward cycles are measured in the number of checkpoint transactions successfully mined, not the number of Bitcoin blocks. The only way for miners, including Bitcoin MEV miners, to get STX and PoX payouts is to ensure that checkpoint transactions are mined in a timely manner.
There will be a well-defined protocol for would-be miners to join the pool by sending BTC. Would-be miners create a “pool-join” UTXO, and the Stacks mining pool spends the UTXO to add the miner. These pool-join transactions are sent to both the Stacks chain and to the Bitcoin chain, and Stacks nodes verify that these pool-join transactions are mineable on Bitcoin. By doing this, the consensus rules can require nodes monitor the Bitcoin chain for MEV behavior: if the mineable pool-join UTXO is not spent within a fixed number of Bitcoin blocks (something that Stacks nodes can collectively track), then STX and PoX disbursements are delayed until it is consumed. This means that Bitcoin MEV miners block STX and PoX payment to themselves by preventing other non-Bitcoin-MEV miners from joining.
If we can do all of these things, then the only profitable course of action for a MEV miner would be to participate honestly. Refusing to mine checkpoint transactions or pool-join transactions merely delays disbursement of any STX or PoX rewards they might be entitled to. Moreover, there’s no money to be made by 100% blocking these checkpoints and pool-joins, because then Bitcoin miners wouldn’t get any STX or PoX payouts.
Aside: I don’t think block pre-commits solve this problem (see my L1 improvements post), because if enough Bitcoin miners wanted, they could prevent any block-commit besides their own from getting enough pre-commitments. Importantly, they can do so without spending extra BTC, meaning that they’d be getting the STX at lower cost than honest mining. Block pre-commits solve a different problem: among other things, they ensure that Stacks miners get some STX for trying to mine on the canonical chain tip (even of other Stacks miners orphan it later). If Bitcoin miners are in on the Stacks mining game, then Bitcoin miners with minimal Stacks mining power can still prevent Stacks miners from pre-committing.
Per @jude original suggestions for values of F and B, it might be sufficient to make the floor price 70% of the previous 1 block. The reasoning is that as long as there is at least one honest block producer always active in recent action, that would in theory coalesce MEV miners to put up at least around 70% of the price of the highest priced honest block in “recent history” in the event they actually become the miner of next block. There are some fluctuations based on math around optimal EV based on the % of the total miners are MEV miners in the pool but overall it should work out okay for the time being. More importantly the one-block duration doesn’t lock in a floor price for too long in the event of market price volatility and allows miners to adjust their bid downward quickly in response to down trending market conditions.
Adding a note here to encourage discussion around a timeline for this.
Since this impacts the whole network, any potential change would need to undergo the SIP process. Ideally, we would like the SIP process to unfold in parallel to some of the research + development efforts. I encourage everyone to highlight places where my estimation seems off, bring up any other linked dependencies, or mention other related deadlines.
A group of community contributors and core devs have proposed various solutions (write ups on the forum or coming soon) and we’ve quickly hired a tokenomics expert to model out the ones that involve changing how miners are rewarded. Changes like this need to be done carefully and the data will be useful to everyone during the eventual SIP process. We’re looking forward to sharing that data/report with everyone in the next few weeks as they complete their research.
Possible Engineering Timeline
4/21 - Research complete, proposal selected for implementation.
5/15 - Dev work + testing complete
Preliminary reviews occurring in tandem
Release testing happening ASAP (perhaps even before all unit tests are written)
5/25 - All engineering work wrapped up, release tested and ready
6/1 - All nodes upgrade to newest release
SIP Timeline
[UPDATED]: I’ll be working HeroGamer, Jenny, and the community members that join weekly SIP calls to discuss alignment on a SIP timeline that works for the community.
[NOTE: This is a very preliminary (and possibly optimistic) timeline. The Core devs are comfortable taking the time to ensure we get the solution right!]
Still two things I don’t understand about the Sybil attack miner.
How it can be fixed other than organically having more mining competition
The detailed mechanics of how it’s actually reducing BTC yield to stackers
On point 1…
Isn’t the problem that the bad entity controls a sufficient amount of total mining power that even if other blocks get made, they can win a few blocks in a row and thus create a new longest chain?
If, for example, 4 new honest miners entered with the same size, such that Sybil attack miner had <20% of total mining power, wouldn’t that prevent this behavior by making it effectively impossible? Isn’t that exactly how this is meant to be prevented?
What, other than a lack of mining competition from the aggregate set of honest miners to begin with, is enabling the Sybil attacker to ignore previous blocks and create new longest chains?
On point 2…
Mechanically, how is the Sybil attacker lowering stacking yield?
Let’s say honest miner A wins STX block 100,000. They do so by sending native BTC to mine that block. In doing so, BTC yield has already been paid to the selected stacking addresses.
Now, the Sybil attack miner ignores miner A’s STX block of 100,000 (version A), and wins 2 vrfs in a row, which they use to build on block 99,999 and establish a new longest chain.
They overwrite/ignore honest miner A’s block, and propose Version B of block 100,000 and a new block 100,001. To do so, didn’t the Sybil miner also have to send BTC to stacking addresses?
Thus, I don’t see how this inherently reduces the BTC going to stacking addresses, except for if the Sybil miner can spend less than an average miner - but aren’t they only able to ignore previous blocks specifically because they’re spending more than other participants? Is the hypothesis that they are indirectly reducing stacking yield by stifling new mining competition and aggregate mining activity in general?
The problem here is that when the 4 new honest miners enter, their miner software is set up to naively trust all miners. Therefore it’s not sybil attack miner’s 20% against other miners’s 80%. Because all the honest miners will still confirm sybil attack miner’s blocks.
Because honest miners simply cannot mine and the dishonest miner has no incentive to increase their commit.
Look at this over a longer term. For example, miner A participated in 100 blocks, committing enough BTC for a 25% theoretical win rate. They would be in profit if they had a 25% actual win rate. However, due to their blocks being ignored by dishonest miner B continuously. Their actual win rate is only 5%. This means they cannot profitably mine over a period of time, and will eventually stop mining. When they do that, the total miner commit on the network drops.
At this point, dishonest miner B has no incentive to increase their commit. So the total miner commit on the network is now lower than it should be in a correctly functioning mining system. New miners will join at some point, however they will all run into the same problem and eventually stop mining.
You can go back and look at the mining data for the past few months. In a perfectly competitive scenario, the total BTC commit per block should be very close to the BTC STX exchange rate. But it is much lower than that, most of the time it’s 50%-70% of that.
A bit of background. The behavior referred to by @Metastack_BMC and @yukan is consistent with someone running a white-glove mining service on behalf of multiple individuals. The BTC that funds the miners come from separate accounts, and the STX outflows go to separate accounts. Moreover, the nodes run on public IP addresses, which makes their presence obvious (which makes me think it’s not deliberately malicious – it’s trivial to hide this, but this pool is not doing so). Also, myself and @jwiley have made contact with one of the individuals participating in the service, and can confirm that it is not one person.
The mining pool orphans blocks outside of the pool at a very wide range of frequencies. On some days, they confirm honest miners’ blocks nearly 100% of the time. On other days, it’s more like 40% of the time. There does not appear to be any consistency to the behavior, but I (or someone) should get a spreadsheet of all of the confirm vs. orphan rates to see if there’s any kind of pattern to confirm this.
If they are trying to be malicious, then the fact that it’s not consistent at orphaning blocks is surprising. Again, because the mining pool is not taking any steps to hide their behavior, it makes me suspect that this orphaning behavior could be due to an unforeseen bug in the block relay code. I’ve opened an issue to outline the steps we should take to investigate this further, and steps we can take to make block relay more reliable (which would help us rule out bugs in the networking stack, if indeed that is the root cause).
At a meta level, this thread seems to be discussing two unrelated topics:
Bitcoin MEV miners underpaying for STX
Some Stacks miners’ blocks get orphaned more often than others
These are both problems to be sure, and I think the post @pavitthrap outlined above is a sensible course of action to addressing the first problem (which poses a direct existential risk to sBTC). The latter problem needs further investigation. I’m not one to call it an “attack” without smoking-gun evidence, nor am I inclined to do so until we can rule out the possibility that this is actually due to a bug.
However, the intentions of this mining pool is irrelevant. The more immediate problem its presence reveals is that mining is not fair enough. The property Stacks should have is that if a miner spends X% of the BTC for mining on the canonical chain (even if they’re orphaned later), then they should receive X% of the STX. This is not the case today, and is not the case for most Nakamoto-consensus blockchains – if your block does not fall onto the canonical fork, then you get nothing. But because Stacks can keep track of all forks that ever arise, we can address this in a future hard fork. There are a few different ways we can do this that are outlined here and elsewhere (so I’ll not repeat them), but I think this is solvable and can be addressed in the near future once we know more.
We should not conceal facts but openly discuss existing problems?
1.) We provided a forensic snapshot last November which proved, that one BTC account funded > 80% of the miners at the time. They simply tried to hide their tx-connections. Anyway BTC-Forensic is merciless.
2.) Not anymore (since we talked in public about it)
3.) The fact that they block 100% of the blocks from the f2pool miner confirms that it’s not due to an unforeseen bug.
Why is this problematic? If the honest miners build on the Sybil miner’s blocks, the Sybil miner still has to outpace all of them collectively in order to propose new longest chains right?
If there is a bug with block propagation, certainly that should be fixed, but isn’t the low honest miner participation what is causing this bug to be so problematic?
If there is no bug with block propagation, and a miner is intentionally ignoring blocks, again we come back to the fact that low miner participation is really what makes this a problem right? To my knowledge, you can’t force a miner to build on a recent block (except by them having such little share of mining power that they can’t create a longer alternative chain).
Are you saying that some form of a reputation system (so honest miners would not build on Sybil miner’s blocks) would help the problem? I agree it would help, but it seems like a second-order term.
I am also slightly confused how this attack works. Please correct my understanding:
If the sortition winner is an honest miner (miner A) who provably earns the right to publish the next block (N+1), how can a competing miner (miner B) exclude that block, and when given the chance, publish a longer chain that replaces the (N+1) block with their own version and that all other nodes in the network would accept? Miners are supposed to only be able to build on blocks that have been created by the miner (miner A) that won sortition? If not, and it only depended on fork length wouldn’t that defeat the purpose of the sortition? Wouldn’t the Stx chain state become invalid if there was a block in its history that was not provably produced by its sortition winner?
I hope these two examples (attached .png) help to understand the problem. In both cases the honest miners won ~56% of the sortitions but the bad entity earned more STX. The honest miners lost several block rewards (all orphaned blocks).
A bad entity is able to ignore honest miners blocks as long as they desire (self build release). The honest miners will build their blocks simply on the longest chain (default release). Thats why they are doomed to lose.
We reported this “hostile” takeover last September on Immunefi - but the foundation did not see the problem and persisted in claiming that it´s not a bug or any kind of critical issue.
Apologies but this still doesn’t really address what I’m asking. Yes, I understand the miner ignoring previous blocks has an advantage, and this advantage is made exponentially worse as their share of collective mining power increases.
This isn’t really true though, is it? For example, honest miners are not “doomed to lose” if they have a collective share of 90% of total mining power, right?
It’s still unclear to me what you are advocating for. Please correct me if I’m wrong, but in Bitcoin mining, or any longest-chain consensus mechanism, you can’t force miners to build on previous blocks, right? The reason miners do build on previous blocks is that the incentives guide them to do so when mining is sufficiently decentralized.
So what, specifically, are you proposing? This seems like a symptom, not a cause, of mining centralization.
I suggest it’s about time to move on from the medium of a forum board to a community call with people like @jude who can help address these open questions. At least to me, it feels like we’re going in circles on the last few posts.
How about next Friday, using the SIP call with you Matty, could be a good space for it? Allocate enough time for presenting key findings from reports, and we can get into proper discussion and questions?
