Component Architecture

[jackzampolin]

Problem

Currently the components of the Blockstack ecosystem have confusing naming that does not highlight delineation/borders between the components. This makes getting started with the platform significantly more difficult than it should be. Blockstack has built a working, scalable, feature rich system for Application developers. To drive wider adoption we need to reduce the mental overhead of getting started with the platform. We can accomplish this by bringing our naming closer to the common idioms in blockchain command line tooling.

Components of the system

From a deployment and infrastructure point of view the following components are used in our system:

• blockstack-browser
  • Serves the user interface
  • Communicates with the blockstack api to complete name operations and auth
• blockstack
  • Command line interface for communicating with the blockstack api and blockstack-core.
• blockstack api
  • User friendly interface on blockstack-core
  • Proves a programatic way to interact with Blockstack core
  • Documentation
• blockstack-core
  • Communicates with a bitcoind node and by reading the blockstack virtual chain creates an index of all domain names in the system.
  • Exposes an API that blockstack api interacts with.
  • Blockstack production deployment running at node.blockstack.com
• "the flask app"
  • A wrapper to make blockstack api more performant and safer to expose to the open internet.
  • Enables a light clients to connect to the network in a safe and performant way.

Proposal

Updating the naming to take into account the additional complexity introduced by the addition of the flask app. We could think about the components as follows:

• blockstack api -> blockstackd
• blockstack-core -> blockstack-indexer
• "the flask app" -> blockstack-api

Architectural diagrams describing the current setup are below with the new and old names. I would love some feedback on this.

Running on a Laptop or Desktop

Run your own full Blockstack installation

Support for both the Desktop usecase and the Light Client Usecase

Run your own Light Client Server

CC @aaron @jude @muneeb

[vsund]

First of all, I really like these diagrams. They do a great job in showing how all the different parts interact with each other. Also I totally agree on these name changes, by using common naming conventions we lower the mental barrier for new developers who try to get into this ecosystem. One thing to keep in mind though is that name changes confuse users and therefore must be well-thought, but in this case I think it’s totally worth it.

I also like to propose blockstack-browser -> blockstack-ui. Although “browser” is the right word for this kind of software, users are confused because they think of a web browser (like Firefox, Chrome, etc) when hearing this word.

I think we need to get more clear about all possible setups that can happen. Not sure about this, but the diagrams might lacks a setup (maybe more, depending on how separate Atlas is from Core). We definitely need to find clearer terms for these different setups / use cases.

Before I try to get into all setups I’d like to clear things up with all parts that somehow can interact separately with each other. It’s mainly the same thing as above, but split up in more parts.

• Browser: An easy user interface for managing own identity, app permissions and accessing apps
• CLI: Cli tool for managing identity (and more advanced developer things)
• blockstack.js (and other libraries): Provides easy access to profiles and user storage from Javascript, gets data from the API endpoint and asks Browser for permissions
• Gaia storage: User storage that gets distributed on multiple cloud providers (that the user connected)
• API endpoint: Provides a RESTful API to connect Browser and Core, needs private keys, extremely lightweight
• Core node: Asks a Bitcoin node for relevant transactions and builds a name database with these, provides an RPC interface, needs to be online 24/7, processes all Bitcoin blocks (1mb every 10 minutes)
• Atlas node: Stores a full replica of all zone files (Atlas network), needs ~5gb disk space, runs (by default) on a Core node, fallback for light clients possible
• bitcoind: Bitcoin node that processes standard transactions

Please correct me if there’s something wrong, not precise or missing.

---

Full Slack conversation for reference

vsund [17:19] 
what's the difference between `core.blockstack.org` and `node.blockstack.org`?


jack
[17:22] 
`node.blockstack.org` is the default server your `blockstack` cli connects to. Check out your `~/.blockstack/client.ini`


[17:23] 
`core.blockstack.org` offeres a subset of that functionality and is used in the browser lookups.


vsund [17:26] 
so `core.blockstack.org` uses internally `node.blockstack.org` and provides an interface for browser to get data?


[17:26] 
if yes, why are there two endpoints for it? why don't merge both?


jack
[17:28] 
I think thats a question for @jude and @aaronb but i think it has to do with light client support


vsund [17:43] 
how separated is an atlas node from a core node? is atlas always part of a core node? can i have a core node locally and ask for zone files remotely? (edited)


jack
[17:44] 
Yes it is. I don’t think you can


aaronb [17:49] 
Sorry -- just saw this, we're going through some networking troubles over at Blockstack HQ. core.blockstack.org runs a public facing API that talks to its own core node (and atlas node)


[17:49] 
If you run a core node, by default it runs atlas


[17:49] 
And gossips/synchronizes zonefiles with the rest of the atlas network


vsund [17:50] 
but i can't split up core and atlas into separate nodes, right?


aaronb [17:50] 
You can configure core to not run atlas


vsund [17:51] 
so that my api uses a local core node but uses another node for zone files


aaronb [17:51] 
But I don't think you can run atlas without core


[17:51] 
If you set Atlas=False in your blockstack server's config file, it should run that way, yes


vsund [17:52] 
so my restful api then uses a local name database but fetches zone files from `node.blockstack.org` (by default)?


[17:53] 
how would i configure this? (i only have one config directive for `server`)


aaronb [18:12] 
The local server will get your zonefile requests and then request them from the atlas network


[18:17] 
Err, actually it ends up falling back to the client


[18:17] 
Which then tries to fetch the zonefile from our atlas backup on s3 (edited)


vsund [18:19] 
to which client?


[18:21] 
i can either run a core node with atlas, then it can answer my zone file requests directly; or i can run it without atlas then it asks some atlas-enabled core node for me?


nedupaul [18:54] 
joined #engineering


aaronb [19:20] 
yeah -- if you run a core node without atlas, and point your api server at it, it will fail to return a zonefile to the api server, which will then read the zonefile from our s3 backups


vsund [19:21] 
ah ok :+1::skin-tone-3:


[19:21] 
then the next question :smile: why are core rpc and restful api endpoint two things? couldn't the core node expose a restful api?


[19:22] 
(it could also handle things like zonefile s3 failover)


[19:23] 
different question: why does the api endpoint asks the s3 backup and not a random node from the atlas network?


aaronb [19:25] 
the reason that the core rpc and restful api endpoint are two things is to support "light clients" which don't run a full node (edited)


[19:26] 
the light client handles interactions with the browser


[19:26] 
and issues requests to core nodes running at node.blockstack.org


vsund [19:28] 
but i see no reason for light setups to run an own api endpoint


[19:28] 
couldn't they just use their core node's rest api (if it was integrated there)?


jack
[19:29] 
Core doesn’t have a REST API right?


vsund [19:30] 
it exposes an rpc interfast, but no rest api


[19:30] 
but i don't know why it's not in there :slightly_smiling_face:


jack
[19:33] 
Because the `blockstack api` is the REST API for the rpc interface


vsund [19:35] 
but why not integrating the rest api into core?


[19:35] 
and the browser talks directly with its core node instead with a local api endpoint


jude [19:38] 
The indexer (`blockstack-core/blockstack`) is a heavyweight process that reads the blockchain.  It needs to be online 24/7 and must download and process 1MB blocks every 10 minutes.  The RESTful API server is a separate process specifically to avoid compelling users to run heavyweight processes


[19:39] 
you can run both locally without any problems, however (I do this on my laptop)


[19:39] 
but there are bandwidth and power requirements that not everyone can meet (especially if they're data-constrained, such as on a cellular network)


vsund [19:40] 
i understand this decision, but then why don't integrate the local endpoint into the remote core node?


[19:40] 
what's the benefit from running it locally?


jude [19:40] 
run it locally if you don't want to trust the remote node

vsund [19:41] 
which trust do i gain? i thought it just relays requests to a core node?

[19:42] 
maybe the better question is: what happens in the local api endpoint?

jude [19:42] 
ah, I misunderstood

[19:42] 
you need the local API endpoint since it registers names on behalf of your browser

[19:42] 
it needs your private keys to do so

aaronb [19:43] 
^yes, you send private keys to your local api endpoint, you don't send them to the core node

vsund [19:43] 
ok, and in a really-light setup i can run this on a well-trusted node

[19:44] 
or is the endpoint (we need a better name for this :D) light enough for this?


jude [19:44] 
yes. you would do this for your mobile phone, for example, since it can't run background processes


vsund [19:44] 
ah ok, so it's another problem


jude [19:44] 
yeah :disappointed:  there's no technical reason why a mobile phone can't run a local API daemon (it's super-lightweight).  however, both Android and iOS forbid you from doing this


vsund [19:45] 
i have some apps on my android phone that expose a permanent notification, might be a solution on android


jude [19:46] 
android activities are not guaranteed to be resident forever (edited)


[19:46] 
the OS can stop them whenever it wants


muneeb [19:47] 
@vsund we had a discussion about simplifying these various moving parts. Jack posted it at https://forum.blockstack.org/t/component-architecture/1417 (edited)


vsund [19:47] 
but it doesn't need to run the whole time (like to core node), it could be started if necessary?


muneeb [19:47] 
core.blockstack.org will become api.blockstack.org (just a public node that exposes the API) and anyone can run it


[19:47] 
that’d be less confusing


vsund [19:48] 
yeah, posted there :slightly_smiling_face: currently on a diagram that tries to integrate all components


muneeb [19:48] 
and node.blockstack.org can go away with time (once we ship consensus-breaking changes)


[19:48] 
just saw your comment!


[19:48] 
so api.blockstack.org can be the main fleet as well


jude [19:48] 
> but it doesn't need to run the whole time (like to core node), it could be started if necessary?
a hypothetical local API node activity would need to come online before your browser opens


[19:49] 
the failure mode we need to avoid is the OS deciding to shut down the local API node when you're in the middle of interacting with a blockstack app


vsund [19:49] 
ah ok :+1::skin-tone-3:

[vsund]

To make a great user experience in understanding the ecosystems, we need the different parts to

• have a clear purpose
• be distinguished from each other
• not serve the same or a similar purpose as another part

(To be extended with more points )

[eric.stermer]

I am in agreement that the naming is one way to help developers get over the mental barriers for developing.

I like what @vsund said about the browser being the right term but the confusion from the desktop app that usually constitutes a browser where ours is more like it’s own website that has an interface. I like to think of it as a the blockstack-app-store like Apple. But your not really purchasing the apps. I dont think though that this confusion is really enough to demand a name change.

In terms of the core, I like that naming currently too because it is not really a ‘server’ but it is. It really links everything together from connecting the UI to the nodes and Gaia, so it is the core or cornerstone to blockstack from how I understand it.

I think having the pictures and diagrams to really show how everything interacts is a better start than to renaming things.

[vsund]

I tried to put all different parts in relationships. Not sure if this helps, but I started with it and then…

[jackzampolin]

I really like this. Can’t that RESTful API be either blockstack api or "the flask app" in the above diagram?

[vsund]

Sure, here’s an updated version (I think you can edit it, do what every you want to do with it ).

Used RESTful API though to say clearly which API it actually is

[larry]

Currently the components of the Blockstack ecosystem have confusing naming that does not highlight delineation/borders between the components. This makes getting started with the platform significantly more difficult than it should be.

Yes! Just caught up on this thread. Lots of good stuff here! I’m my phone right but will come back a bit later with some point by point comments. (Quoting text is finicky on mobile)

[jeremyrwelch]

In agreement that the current naming is a little confusing, and the suggested names adequately address the concern.

On the short run the Blockstack Browser is going to feel more like an app store, though I don’t think we should follow that naming. Perhaps “Blockstack Identity” or “Blockstack Apps” is better.

[larry]

Great job on these diagrams, @jackzampolin and @vsund !

@vsund’s diagram is a bit closer to my mental model of the structure of the system.

In my view, “Blockstack” should be the name of the end user package that users install on their computer. “To use this app, you need to install Blockstack”.

We shouldn’t be pushing most users to be installing only the command line version of blockstack. Instead, we should bundle it with the end user packages like Keybase and others do. We can refer to the blockstack CLI as the blockstack command.

I like the idea of calling the portion of blockstack-core that indexes the blockchain and generates the name database the blockstack-indexer.

---

Having said that, here’s another way to think about things:

It seems like a lot of confusion comes from the fact that part of what is now blockstack-core is trusted to handle user specific state, specifically, private keys and part of it is not.

Looking at it from the perspective of how we deploy code, the main functional difference between what is running on core.blockstack.org and what runs in our current macOS distribution when you run blockstack api start with the client.ini configured to use core.blockstack.org is that the latter is trusted to work with your private keys while the former is not.

What if instead we did something like this:

blockstackd is a full core node. It uses virtual chain to index the blockchain and builds a name database. It is resource intensive, the same as a bitcoind node. You only run blockstackd if you want to independently build and maintain a database of names and participate in the Atlas network. We can think of its input as the bitcoin blockchain and output as a database of names and zone file content. It doesn’t know anything about profiles, gaia storage, identity, authentication or, most importantly, state of any clients querying it.

blockstack-api is what we currently get when you run blockstack api start. It provides a rest api that browser and cli (see below) talk to. It is trusted to handle the user’s private keys. It includes identity related logic (resolving profiles, verifications, etc). It talks to blockstackd to get information about names and zone files, utxo.blockstack.org to get information needed to build bitcoin transactions.

blockstack-search is the search service which interfaces with blockstack-api to generate a searchable index of information pointed to by names.

blockstack-cli is the command line interface. It is an alternative interface to the http REST api interface provided by blockstack-api.

blockstack-gui is the single page javascript app component of what is currently called blockstack-browser. Like blockstack-cli, it is also an alternative interface to the http REST interface provided by blockstack-api.

blockstack-macos is the current macOS swift app and packages the components above in an end user distribution for macOS. Users get both the graphical user interface AND the command line interface after installing this. They also get blockstackd so that they can enable a node on their local machine with one click. To users, this is Blockstack. When we talk about installing Blockstack

blockstack-windows is the same thing as blockstack-macos but for windows.

(and so on and so forth for other platforms)

I’m entirely familiar with what @jackzampolin calls “the flask app” above, but my sense is that its functionality should be be included in blockstack-api.

In this world, blockstack-gui, blockstack-cli and blockstack-search are all consumers of the API provided by blockstack-api. If you want a full node, you run blockstackd and point your blockstack-api to blockstackd. Otherwise, you point blockstack-api to a remote blockstackd.

[asllc.james]

@vsund, I get a broken diagram when I click on your updated version - could you include the link and paste in a rendering to the post?

[vsund]

@asllc.james I cannot reproduce this in Firefox or Chrome, are you using another browser?

Here’s the new version as image:

[asllc.james]

[ @vsund I am using Firefox on macOS. ]

I really like this version of the diagram, great job - clear and quick to take in.

I am very interested in the naming, as I believe there is a client/server vernacular (e.g. client, wrapper libraries, api, server) and somehow I would like to create a new vernacular for the distributed computing world (something like: dapps (or just apps), libraries, api, core (as is the current trend - I sometimes thing “engine”), etc.)

Perhaps I need to write something longer form on these thoughts.

[jackzampolin]

@larry love your proposed naming. Especially having a blockstackd that closely mirrors what bitcoind does. Think that is an easy mental model for users. The one question I have is what is the function of the blockstack-search component? Is that to make blockstack DNS lookups performant? Other than that this would make understanding the system significantly easier.

Also it seems like in this arch blockstack.js would be a wrapper around the blockstack-api REST interface?

[larry]

he one question I have is what is the function of the blockstack-search component? Is that to make blockstack DNS lookups performant?

blockstack-search is really an identity specific component that indexes the profile information that’s pointed at by names & zone files so that people can find names based on information contained in the profile. for example, i would search a database generated by blockstack-search to find all users who say their location is “New York”.

In the long term, there will be really be many blockstack-search type packages that index various application specific data pointed at by names and let you search for it. For example, if names pointed to land title information instead of people, you’d want a package that indexes that land title information. This would allow you to quickly and easily find all parcels in a certain county in a certain acreage range. Otherwise, you would have to iterate over the entire set of names and data each time you wanted to find something.

Also it seems like in this arch blockstack.js would be a wrapper around the blockstack-api REST interface?

blockstack.js is a wrapper around blockstack-api and the authentication api provided by the browser.

---

from slack:

Seeing this question (for probably the sixth or seventh time) in conjunction with this discussion made me realize what I think we can improve in our component architecture.

Right now, user state is stored in several places (using macOS as an example):

• Keychain(s) in localStorage of their web browser. If they’re using multiple web browsers, there’s at least one set of state in each web browser.
• Bitcoin wallet stored in ~/Library/Application Support/Blockstack/ used by all browser instances
• Bitcoin & Identity wallet stored in ~/.blockstack for the blockstack command line tool.

This is incredibly confusing to end users and is complicated for developers. They buy a name on their mac and can’t figure out why it exists in Safari, but doesn’t show up with the blockstack command line tool nor does it show up in Chrome.

User state should be stored in a single place per user account. If a user registers a name on the command line it should be automatically available to use to log in to Blockstack apps. If a user registers a name in Chrome, they should be able to operate on it using the blockstack command line tool.

The blockstack-api package should manage and provide access to this state, both providing the APIs currently provided by core and the authentication API currently provided by the browser.

On iOS mobile, we can wrap this same code in Swift and provide access to the user state specific APIs via custom protocol handler and generic public APIs through a public instance of blockstack-api.

This also addresses the problem that @ryan and I have discussed: that we have made a series of small decisions over a period of time to push operations from the browser to core in the interest of optimal use of time and engineering resources. Instead these operations should be put in a blockstack-api type package, written in javascript so that they can be easily used cross platform and possibly used in the browser as well.

[aaron]

Right now, user state is stored in several places (using macOS as an example):

Keychain(s) in localStorage of their web browser. If they’re using multiple web browsers, there’s at least one set of state in each web browser.
Bitcoin wallet stored in ~/Library/Application Support/Blockstack/ used by all browser instances
Bitcoin & Identity wallet stored in ~/.blockstack for the blockstack command line tool.

This is incredibly confusing to end users and is complicated for developers. They buy a name on their mac and can’t figure out why it exists in Safari, but doesn’t show up with the blockstack command line tool nor does it show up in Chrome.

User state should be stored in a single place per user account. If a user registers a name on the command line it should be automatically available to use to log in to Blockstack apps. If a user registers a name in Chrome, they should be able to operate on it using the blockstack command line tool.

The blockstack-api package should manage and provide access to this state, both providing the APIs currently provided by core and the authentication API currently provided by the browser.

I agree that we need a better story about where a user’s “wallet” is stored. Currently, it’s split into 3 places, as you say.

But I’m not sure I’m following the blockstack-api solution. Where is the user’s root key stored?

[larry]

But I’m not sure I’m following the blockstack-api solution. Where is the user’s root key stored?

On macOS, the root key (we call this the identity key in v0.13 and have also called it the backup phrase or bip 39 mnemonic) would be stored either in the macOS Keychain or somewhere ~/Library/Application Support/Blockstack/ encrypted with a secret (possibly a user selected password?) that is also possibly stored in the macOS Keychain so that the user doesn’t have to enter it each time blockstack is started.

On iOS, it would be the same but instead stored in the Blockstack app container.

On Linux it would be stored in ~/.blockstack, again also encrypted with a secret.

[jackzampolin]

@aaron @larry I think making the default blockstack folder on each platform the only place we install things will also make it much easier on users/developers. It will also make it much easier to start up a second install for testing and our messaging around backups can be consistent ("Back up the ENTIRE blockstack folder: /path/on/your/arch"). We could go even a bit further and have blockstack-api, blockstack-core, blockstack-* subfolders. This would keep common configuration together and reduce the overhead of learning the system significantly.

[jackzampolin]

blockstack-search is really an identity specific component that indexes the profile information that’s pointed at by names & zone files so that people can find names based on information contained in the profile. for example, i would search a database generated by blockstack-search to find all users who say their location is “New York”.

So then blockstack-search could be a reference implementation and users who want to build, say, a tweet indexer would have somewhere to start. I like this idea.

[guylepage3]

Great job with this @jackzampolin…

First of all, I really like these diagrams. They do a great job in showing how all the different parts interact with each other. Also I totally agree on these name changes, by using common naming conventions we lower the mental barrier for new developers who try to get into this ecosystem.

Agreed. This is beautifully easy to understand.

I like to think of it as a the blockstack-app-store like Apple. But your not really purchasing the apps.

On the short run the Blockstack Browser is going to feel more like an app store

I was on a call with @larry last night discussing the browser and implementation of the new UI design, one of the things that I feel strongly about is that much like the Apple Home Screen and App Store we should be splitting them apart.

1. Blockstack Home Screen – Use blockstack.com as the Blockstack Home Screen where users can easily navigate to their favorite apps by clicking app icons or by navigating the old fashioned way via their browser’s search bar.
2. Blockstack App Store – A site or “App,” from the home screen, where users can discover new apps in a much more presentable manor that will be user friendly and assist them in finding the apps they want and “installing” them quickly and painlessly.

This method is much clearer to the user as to discoverability of apps vs navigation to apps.

Obviously for the time being we will simply only have “pre-installed” apps until we have 25-40 apps on the Home Screen, at which time we will need to start building out the App Store.