Gaia Hub - Data Archival and Migration

Community
#1

We have been migrating Hiro hosted services from stacks.co and the much older blockstack.org to hiro.so and so far we have migrated most of Hiro hosted services and we are in the final phase of our migration with 2 pending migrations left - the Subdomain Registrar and Gaia Hub.

Screenshot 2023-07-06 at 12.45.37 AM

Gaia Hub Migration

Gaia’s domain migration will be handled in 2 phases.

Phase I - Data Archival [Jul 12- Aug 12] :

Context:
- Last year, we shared that Hiro’s operating expenses for Gaia have been prohibitive and ran a detailed survey with our users. Since then the costs have continued to spike higher.
Strategy:
- As part of the migration we have decided to archive data that has been unmodified for the past 1 year and is >500MB. These are the apps that are unaffected by this change and will be fully migrated with data intact, all their respective data meets our non-archival criteria - been in frequent use and total bucket size ≤ 500 MB:

- Hiro Wallet
- BNS
- Sigle
- BlockSurvey

What’s next:
- We are archiving the data that meets our archival criteria by Jul 12, if you think your app is affected by this change and isn’t listed above, please let us know via comments here or email us at [email protected] and we will evaluate the next steps case by case.
- We will hold onto the archived data for the next 30 days [July 12-Aug 12] after which it will be deleted permanently.
- If you know you have old and large data, then now is the time to back it up.

Phase II - Data Migration [ Aug 12 ] :

-Post Phase I and the confirmation that the data Archival process was smooth, we will kick off the Data Migration.
-Please note that at this time, there’s no means for our users to migrate their own data and this will be handled by the Hiro team.
-Once the relevant data is archived , we will copy the remaining data to GCP.
-We will spin up the new GaiaHub in the new hiro.so domain to use the copied GCP Data
-We will thoroughly test this data migration in a Dev env
-This will also warrant an update to stacks.js that the team at Hiro will handle.
-We will continue to retain a transparent DNS redirect to the old domain and will follow up with more details on any actions needed by the apps and clients as we near this phase.

- ⚠️ **Note**:
    You’re safe if you use the challenge *text and read_url_prefix from the* https://hub.hiro.so/hub_info , if you have hardcoded urls anywhere, your app will need changes to use the updated  *challenge_text* and *read_url_prefix*

-After successful tests to ensure that the data in Azure is safe to delete, we will delete the Azure blob and continue to use the GCP backed data.

Update Jul 25 2023:

  • First pass at our archival was a success.
  • We are making a second pass at the Gaia Hub data with the following criteria: data that has been unmodified for the past 6 months and is >250MB.
  • Please note that we haven’t migrated the Gaia Hub domain yet and that is scheduled in ~2 weeks from now.

Update Aug 16 2023:

  • Second pass at our archival was a success where we archived data unmodified for the past 6 months and is >250MB
  • We are making another pass at the Gaia Hub data with the following criteria: any data bucket that is >500MB.
  • Please note that we haven’t migrated the Gaia Hub domain yet and that is scheduled next.
1 Like
#2

Hi,

I’m developer of Brace.to and Justnote. Can I ask some questions?

-How to calculate a total bucket size? Is it a size that one account uses for one app?
-Is unmodified data per bucket? If there is a new file or one modified file in a bucket, that bucket is modified. Am I right?
-Is there a way I can check if the data for Brace.to and Justnote meet the criteria?

Thank you

2 Likes
#3

Hey! Will do my best to answer these questions.

How to calculate a total bucket size? Is it a size that one account uses for one app?

A bucket in this context boils down to the identifier used when accessing files stored in Azure through the Gaia Hub. For example, in the URL https://gaia.blockstack.org/hub/<BUCKET_ID>/profile.json, <BUCKET_ID> is what we’re referring to in this context. I believe this evaluates to a single account. So the total bucket size would be the summed sizes of all files under the https://gaia.blockstack.org/hub/<BUCKET_ID>/ URL.

Is unmodified data per bucket? If there is a new file or one modified file in a bucket, that bucket is modified. Am I right?

Correct, if you have at least one file in a bucket that has been modified within the past year, then that bucket will be excluded from the archival process.

Is there a way I can check if the data for Brace.to and Justnote meet the criteria?

I’m not aware of a way you’re able to check unless you have a record on your end of when you’ve modified files last. However, you could send a small list of bucket IDs to [email protected] to request us to check for you.

2 Likes
#4

Thank you so much for your answers. :pray:

1 Like
#5

For our xck.app Dapp:

  1. At this moment, we are experiencing non accessibility to our user’s account data.
  2. Some doubts about the migration procedure.
    How the user’s zonefile will be update from blockstack.org to stacks.so?
  3. Should we provide a service to update the zonefile?
  4. We are not sure which users accout matches the non activity for the last year >= 500 MB
    Thanks
1 Like
#6

Is there a intermittance in the gaia.blockckstack.org?

1 Like
#7

Update Aug 24 2023

  • Third pass at our archival was a success where we archived buckets over 500MB in size.
  • Next, we’ll be migrating the Gaia Hub domain on Monday August 28th.
    • This will involve a DNS change for the old domain, https://hub.blockstack.org/, and should be completely transparent for clients making use of this URL.
    • This old domain will be officially deprecated and slated to be removed at a later date. Clients should make the switch to the new URL below as soon as possible.
    • This new Gaia Hub URL that should used is https://hub.hiro.so/
1 Like
#8

Update Dec 19 2023

The data we’ve archived between July and August will be deleted over the next 24 hours. We originally stated that we’d hold on to this archived data for 30 days in case someone needed their data restored, but ended up holding on to it longer out of an abundance of caution.

This deletion will not affect any live data being served by the Gaia Hub. This is only affecting data we’ve archived earlier this year as part of the initiative, and thus has not been available since July or August.

1 Like