2017-08-08 Storage Meeting

Community Meetings
#1

Attendees

Review of single reader storage

The app specific key is used to sign all the data that you write.

Datastore record includes:

  • drivers to use
  • public key
  • root directory

The signature is used so that you know the data you read was put by you.

Each logical file is stored as a separate header file and a payload file.

There are two separate files so that the payload can returned as an http url and used directly by browser and apps.

Problems

Key Discovery

To read other people’s data I need to know the public key of the person-device that wrote.

Speed/Consistency (Write performance)

Write performance is slow because we need to write 4 files for each file: A header and payload for the file and then a header and payload for the directory. If any of these fails, you end up in and inconsistent state.

Solutions

Key Discovery

Publish a key file that includes:

  • profile
  • a delegation object of public keys of all of your devices
  • app keys

Resolution path: Name -> Zone File -> Key File -> Delegate object -> App keys

There’s a secure method to solve the key discovery

Speed/Consistency (Write performance)

This is more complicated to solve.

Questions?

What is the minimum underlying consistency model we can assume?

We’re trying to avoid a world where everyone has to have their own server.

Proposed solution

Each device maintains a write log, that is logically a history of what it does. Each entry is a put or delete.

What happens globally is a global log in your storage provider.

Eventually, there is one writer. At any time there can be more than one writer, but eventually only one will succeed.
Optimistic concurrency resolution.

We can take the last write wins, but the local logs can see that maybe one of the other devices won.

  • Writes are now synchronously fast.
  • Asynchronously append to the global log.
  • Enables search by regex (aka “a poor man’s directory”)

Also enables use of Bittorrent and IPFS.

Mobile:

Core node (or a lighter Storage node) sitting in the cloud that’s receiving writes from the mobile device.

Where do reads come from

  1. Your file goes to the storage provider
  2. Append to local
  3. In background, your device will try to append it to global log
    I do a write

A write returns the URL to the file it just wrote.

With collections, given a Blockstack name and a type of collection, you’ll be able to logically construct a URL without a Blockstack Core node.

You sign and encrypt data locally, but you’re trusting the remote node to write your data for you.

These storage nodes have to be multi-user and you could belong to multiple user nodes.

Action items

  • Iron out iOS authenication

Future discussion:

  • AWS key distribution service
  • Core node that runs in the cloud
2 Likes
#2

Update on the proposed solution

Instead of constructing a per-device write log (which will need to be squashed periodically), it would be more efficient and just as effective to simply have a per-device file listing. Each of the entries in the listing would have URLs that point to that device’s files, and readers would calculate the user’s listing by fetching each of the users’ device’s listings and merging them in a last-write-wins fashion.

Writing a new file is done by (synchronously) replicating the data and getting back a URL, and then (asynchronously) replicating the device’s listing.

Reading is done by fetching and merging the devices’ listings (which can be done in parallel), caching them, and using them to find files.

We can scale up the size of a listing by splitting it and joining it into K-sized pages (but we won’t worry about this in the initial implementation).

From an engineering perspective, this is a nicer approach since we have most of the code already written.

2 Likes