Proposal: Collect Browser API Endpoints into the Core Node

Core Blockchain
#1

There are a few functions that the browser relies on that are outside the Core Node. The most notable ones (that are fixable at least) include:

Having these dependencies make it a bit hard to decentralize the browser via the settings page, as they all have different inputs and outputs. For instance, a user cannot simply put any new networkFeeUrl that they want because the Browser can only accept the data type which is exactly like what bitcoinfees.21.co gives.

I believe it would be beneficial if these features could become part of the core node, so that if a end-user wants to boot up their own core node they can rely simply on that node, and if the node they currently connect to goes down then they can switch only one node instead of 3 different ones that are currently required.

UTXO server should be fairly easy to merge with the core node, though perhaps it should run on a separate thread due to not wanting one to slow the main one down.

Network Fees shouldn’t be entirely hard to calculate since the core node also has a full bitcoin node running. Just take the average TX fee for the past couple hours or so, should be simple enough.

Originally I was also going to ask for the exchange rate server to also be included, but that’s impossible due to how exchange rates are dependent (and thus only available) on the marketplaces themselves. I outlined a solution here if anyone is curious on how to keep it simple for the end-user to configure. This solution also works for the TX fees server, but not for the UTXO server.

#2

Blockstack.js already has code for interfacing with either a Bitcore API or a Bitcoin node, allowing you to select alternative service endpoints. The Blockstack Core node does not ship with either of these endpoints built-in because they take a much larger amount of disk space, CPU, and RAM to run.

I think you can already set endpoint URLs for the UTXO provider and fee service in the API settings page? If not, then that’s a bug in the Browser and should be fixed.

#3

Sorry for the long wait ~

I was under the impression that the blockstack-core node required a full bitcoin node. I suppose it is good it is not, though.

Can you give me more information about the UTXO server? What runs on blockstack’s utxo backend (specifically utxo.blockstack.org)?

I know there are different types of those services for assembling transactions and the getting insight information, so while it is technically settable in the browser settings, I’m quite unsure of how “replaceable” it is due to the lack off common wanted/returned values.

Regardless, does this mean that the browser could, if it implemented as such, use a bitcore node instead of the ambiguous utxo.blockstack.org for transaction broadcasting and insight? That might make it a little easier for users to know how to set up their own “stack” of services that blockstack requires.

#4

I was under the impression that the blockstack-core node required a full bitcoin node. I suppose it is good it is not, though.

A Blockstack Core node does need to talk to a bitcoin node with a full txindex in order to fetch and verify Blockstack transactions. However, it does so by using SPV headers to verify the authenticity of blocks and transactions. It’s possible to fetch the SPV headers from a lightweight bitcoin node running locally (or a node you trust), and then use those SPV headers to authenticate data from an untrusted remote bitcoin node with a full txindex (to be clear, the code does not do this by itself yet, but it could be achieved by running a separate process that syncs the SPV headers to the SPV header file that your Core node uses).

Can you give me more information about the UTXO server? What runs on blockstack’s utxo backend (specifically utxo.blockstack.org)?

Blockstack Core doesn’t use or need a UTXO service; only Blockstack clients like the CLI and Browser need it in order to generate transactions. The current versions don’t use utxo.blockstack.org anymore for names (see below) – they use blockchain.info by default (i.e. per-user request volume is so low that it’s acceptable to direct them to public UTXO servers instead of needing to run our own).

I know there are different types of those services for assembling transactions and the getting insight information, so while it is technically settable in the browser settings, I’m quite unsure of how “replaceable” it is due to the lack off common wanted/returned values.

Your transactions are assembled client-side using UTXOs obtained from a UTXO provider and a fee estimate obtained from 21.co (by default, and you can change this), and broadcasted to the Bitcoin peer network via a remote transaction broadcaster. I don’t think there’s a way to change this URL in the Browser; in the CLI at least, you can use the -T switch to choose your broadcaster.

Looking through the Browser code a bit more, I don’t think the broadcastURL affects name transactions any longer – I think it only affects sending Bitcoin transactions. Same goes for Insight API URL. Generally speaking, the Browser uses whatever blockstack.js uses when it comes to name and subdomain operations. Looks like I stand corrected on what I said before your reply – these settings should be exposed to the user via the API Settings page on the Browser. Could you open an issue for it?

1 Like
#5

In the OP I posted the a link that are still used in the blockstack-browser for insight, using utxo.blockstack.org unlike what you said above:

(I’m deciding to ignore broadcastURL because it’s only used for sendCore which doesn’t seem to be in use at all anymore)

Though due to your comment it sounds good that I can migrate over to a different UTXO server. Going to need to implement an interface that allows the user to make it replaceable somehow, though, whether it is something like I laid out here or simply presets for blockchain.info or a bitcore API… it’s a bit more complicated than exchange rates and tx fees.

Are you saying to open an issue for allowing the editing of the transaction broadcaster? That sounds like a good idea but how would it be replaceable? Different transaction/broadcast servers have different requirements unless all blockstack browsers need to use the one you linked above. Regardless I can open one up if this is what you were specifying.

All in all it would be nice to have a blog post or something on the complete stack of Blockstack, from the blockchain up to the app and everything in between (core node, gaia hub, atlas, fee estimate, transaction broadcaster, etc) so people know how to add other nodes to help decentralize the network.

The goal I was aiming for here was to make it easier to democratize/decentralize by merging features into the core node. I see now that is quite impossible due to storage limitations and things of that sort. But perhaps there could be an effort made to reduce the number of dependencies at some point so that third parties could start to provide their own nodes?

#6

That URL doesn’t get used when creating name transactions. The code path that handles name registration is here: https://github.com/blockstack/blockstack-browser/blob/master/app/js/profiles/store/registration/actions.js#L93. Note that this code path uses objects and methods in blockstack.js, including using the default mainnet network object in blockstack.js for communicating with an upstream UTXO server (blockchain.info) and a default fee server (21.co).

I was mistaken before in saying that utxo.blockstack.org was used for creating Bitcoin spends. It seems that the current code path for doing so also defers to whatever blockstack.js uses: https://github.com/blockstack/blockstack-browser/blob/master/app/js/account/store/account/actions.js#L318

@yukan can confirm the above.

Though due to your comment it sounds good that I can migrate over to a different UTXO server. Going to need to implement an interface that allows the user to make it replaceable somehow, though, whether it is something like I laid out here or simply presets for blockchain.info or a bitcore API… it’s a bit more complicated than exchange rates and tx fees.

However the interface is implemented, it looks like the API endpoints need to be fed into blockstack.js’s default mainnet network object in order to take effect.

Yes please :slight_smile:

They need to use one that conforms to the transaction broadcaster’s API specification, per the link above. The broadcast protocol is specific to Blockstack since the transaction broadcaster needs to do things like relay NAME_PREORDERs and NAME_REGISTRATIONs in the right order (and with sufficient delay between them), and broadcast zone files to the Blockstack peer network. However, it should be possible to run an alternative deployment – we do this in the test framework, for example. The choice of the instance to use just needs to be exposed to the user.

Agreed 100% on the need for a blog post with an architectural diagram (cc @moxiegirl).

1 Like
#7

Issue created:

I suppose it was a little gray about what I was talking about regarding transactions/broadcasting, so thank you for specifying that the transaction broadcaster is used specifically for name acquisition and other utilities are used for bitcoin transactions.

So just to summarize:

  • core nodes can’t be more helpful regarding bitcoin insight etc because of space requirements (only requires a slimmer tx collection or uses someone else’s node)
  • utxo.blockstack.org is not really used anymore – instead, bitcoin transactions and insight should be able to use bitcore or blockstack.info instead
  • there is a transaction broadcaster used for name transactions – this is a middleware that is required so that the name transactions are according to the schema

otherwise,

  • issue has been raised regarding the configuration of the transaction broadcaster url in the browser
  • request has been made for the blog post regarding architectural diagrams/etc for users to be able to help democratize/decentralize the blockstack stack
1 Like