Scaling Blockstack name and token operations with operation batching off of the main chain

[ryan]

Blockstack operations today are limited by blockchain transactions. A single operation can fit into a single transaction. This has massive implications for scalability and we’ve been thinking of ways around this.

One proposal that I’d like for us to explore is the idea of packing many operations into a single transaction using off-chain packing.

This was not possible earlier because we made the decisions to have all consensus-critical data be stored on chain, with flexible availability of off-chain data (including zone files).

There’s an incredible opportunity we have available to us now, though, with the addition of a Blockstack token with it’s own on-chain layer 2 mining system. This layer 2 mining will allow for the establishment of a canonical longest chain.

Further, if this mining process requires the packing and processing of operations in bulk, it will ensure that there is a clearly defined immutable set of off-chain operations. All participants in the mining process will have all of the data available to them.

The layer 2 mining can entail either proof-of-burn or a layer 2 embedded proof-of-stake system. Both are possible.

What does this mean?

Right now, if we want to register 10 million names per year on the blockchain we have to have 10 million transactions. This would take up 10% of all the Bitcoin blocks in a given year.

However, if we can pack multiple name operations into a single on chain “operation batch” operation, then we can conceivably allow 10 million name registrations with many fewer transactions.

In addition, as the Blockstack token comes out we will be severely limited in the volume of token transfers.

However, with operation packing we’ll be able to have a single “operation batch” operation that can carry a huge set of token transfer operations. This would allow us to have much lower fees for token transfers.

In a sense, this is a blockchain that rides on another blockchain. It uses embedded operations on chain in combination with packed operations off chain.

Once again that this is only possible because of embedded mining on a strong proof of work chain.

@jude has mentioned that we can take this to its logical extreme by actually running a separate blockchain and having it sync into the embedded consensus layer on the main chain (currently Bitcoin).

Note that has much better properties than merge mining because it doesn’t share a proof-of-work algorithm with another chain.

I’m starting this thread so we can explore the potential here.

There is enormous potential for scaling here and could actually be the most promising method for the scaling of blockchain name and token operations.

[jude]

@jude has mentioned that we can take this to its logical extreme by actually running a separate blockchain and having it sync into the embedded consensus layer on the main chain (currently Bitcoin).

If we’re going to anchor batches of operations to the blockchain, we might as well call those batches “layer 2 blocks.” To a first approximation, the same kinds of incentives and behaviors would apply in layer 2 as well as layer 1:

• Layer 2 nodes would relay name operations to each other (“layer 2 transactions”) and store them until a block containing them gets announced (akin to how Bitcoin has a mempool). This is much like how Bitcoin works today with transactions.

• Layer 2 nodes would reject a name operation from their mempools if it conflicted with an existing operation (e.g. two conflicting transfers) or was invalid (e.g. not signed, did not consume a UTXO, etc.).

• Layer 2 transactions (name ops) would have a transaction fee in the layer 2 token. This transaction fee rewards layer 2 miners for packaging transactions into layer 2 blocks (operation batches). Nodes could set a lower bound on how high of a transaction fee must be paid in order to propagate them (akin to Bitcoin’s min_relay).

Mining would be done by sending a NAME_BATCH transaction to Bitcoin with the Merkle root of the batch of operations consumed. Other nodes would accept the NAME_BATCH transaction into the virtualchain (e.g. if it had the right consensus hash), but only credit the sender’s layer 2 token account once the actual data arrives. In other words, there’s no incentive for sending gratuitous NAME_BATCH transactions.

Tokens would be distributed to miners every time a layer 1 block gets processed. The block reward would be split among the nodes that sent NAME_BATCH transactions and propagated the operation batch. The coinbase would be encoded within the batch. NAME_BATCH transactions would consume layer 1 tokens in the form of a transaction fee.

This brings out an additional similarity to layer 2: Since blocks are not guaranteed to arrive at all, and are not guaranteed to arrive in order, we’ll need a “valid batch history with the most layer 1 tokens spent” history selection rule. This is akin to the “valid blockchain with the most PoW” rule in layer 1. This lets the system resolve name operation conflicts (i.e. given two or more conflicting operation batches, at most one will be valid) and gives us a way to recover from lost blocks (i.e. re-package the name operations and send more NAME_BATCH transactions until the reconstructed history is also the longest valid history).

Bitcoin miners would eventually get interested in this, since their participation would mean that they can share in both the layer 1 and layer 2 token rewards. However, since tokens are rewarded to those who send valid NAME_BATCH transactions with valid operation batches, we can expect miners to eventually start censoring NAME_BATCH transactions from everyone but themselves (i.e. they’re greedy). This is acceptable, however, since it’s equivalent to merged mining where all miners participate in the merging (i.e. it’s just as secure as layer 1). We’d make it so the layer 2 tokens would only be spendable ~150 blocks (about a day) later, so miners won’t re-org the Bitcoin blockchain to fight over layer 2 rewards.

Anyway, the reason I stand by the logical extreme of running a separate blockchain and anchoring each block to the main blockchain is because the existing blockchain software already does most of what we need (mempool, UTXO set, block validation and processing, transaction propagation, etc.). What we’d need to add/change is replace the PoW logic with code that reads Blockstack’s virtualchain and matches blocks to NAME_BATCH transactions in the underlying chain before updating UTXOs.

We cannot have been the first people to think of this. Does anyone have any links to prior discussions about pinning a blockchain on top of a blockchain? The closest related work I can think of are extension blocks (and this approach doesn’t suffer from the same problems, since it concerns an entirely separate token).