Blockstack Client-Side ORM and Conflict Resolution

Apps Storage (Gaia)
#1

A community member brought up in today’s engineering meeting that it would be very useful for blockstack apps to have available something like an ORM (Object Relational Mapping) for interacting with Gaia.

Radiks does provide an interface like this — however, it requires hosting a server to perform indexing. Many applications don’t need this, but would still dramatically benefit from the ease of an ORM – especially one that can help resolve conflicting writes, if, for example, a user has an application open in two tabs.

There’s a number of different ways something like this could be implemented (as an additional library that can be included alongside blockstack.js, by adapting an existing ORM library to be backed by Gaia, by extending blockstack.js, etc.), but before we make any decisions on that, we’d need to figure out how to prioritize a project like this.

I think @hank and @zone117x have already had some discussions around this (and @jude mentioned that he has some ideas as well)

3 Likes
#2

Copy & pasting my reply to issue https://github.com/blockstack/blockstack-todos/issues/31#issuecomment-499942217 –

Some thoughts I’ve had regarding difficulties with using gaia’s API for simple CRUD-like operations –
Also pinging @hstove since he has been thinking about this as well (radiks-lite ?).

One of the main problems with apps that require CRUD-like operations (most apps) is how difficult it is to do anything reliably .

The common approach is to maintain a index.json file that could be as simple as an array of uploaded files. Another approach is a single large content.json kind of file that stores everything. Another could be using Radiks, which I think helps here but does not fully eliminate the problem, depending on the app design.

I just tested out graphitedocs, and surprised to see that I can’t even have the same document open in multiple tabs without data loss. That isn’t even an indexing problem, but same underlying issue.

All these approaches can cause race conditions, resulting in an incorrect app data state. Correcting for these race conditions requires complex and expensive conflict resolution code where:

  • Every file update & upload must be followed by a download of the same file, followed by state comparison to ensure the changes were written and the state is correct.
  • Additional code to perform conflict resolution in the event of invalid state.
  • Also, because the app window can closed or connection lost during this process, you’d need to store all current state and pending operations in localStorage before even attempting upload.

Some of these race-condition problems could be solved with something like a webrtc data channel layer. I could have this Notes app open in a couple tabs and on my phone, and the instances could sync state with each other while online. This is relatively complex to implement, and might not make sense to add to blockstack.js. A second layer lib, like radiks-lite, could perhaps do this.

The second layer lib could also implement the localStorage caching for dropped connections or offline apps, and some abstracted or common form of conflict resolution.

1 Like
#3

plugins can also do some of this – i.e. radiks itself could become a gaia plugin, as well as modifiable-document-streaming (push changes with last_hash, new_hash or something).

race conditions are a huge issue though, and one I’m still trying to figure out for an app of mine – and because drivers simply overwrite everything, and changes can only be figured out by polling…

perhaps (for a simple solution) apps could simply have a .app_metadata for themselves, and for every file stored, push a .app_metadata/{path}/filename.extension with the hash inside, so before pushing they can make sure that they have the most recent copy. For example:

Gaia App Bucket Layout:

.myapp_metadata/docs/diary.md.json
docs/diary.md

App Psuedocode:


let diary_hash = '';

async getDiary() {
  const res = await Promise.all([
    fetch('gaia.blockstack.org/hub/asdf/.myapp_metadata/docs/diary.md.json'
    fetch('gaia.blockstack.org/hub/asdf/docs/diary.md')
  ]);
  
  diary_hash = res[0];
  return res[1];
}

async setDiary(doc: Blob) {
  const hash = await fetch('gaia.blockstack.org/hub/asdf/.myapp_metadata/docs/diary.md.json');
  if(hash === diary_hash) {
    return Promise.all([
      uploadFile(doc, 'docs/diary.md'),
      uploadFile(hash(doc), '.myapp_metadata/docs/diary.md.json')
    ]);
  }
}

there’s still a small race condition when uploading/downloading, but it’s minimal at least. perhaps one day we could have a “batch store” and “batch read” or something to remove that as well.

1 Like
#4

I am a fan of the immutable.js theories and the redux pattern to be able to time travel your application state. It’s not quite an ORM but I am also a fan of automerge.js https://github.com/automerge/automerge

2 Likes