Disappearing Blobs from gaia Hub

marie · August 5, 2020, 2:56pm

I suddenly get 404s on all blobs in this format:

https://gaia.blockstack.org/hub/<address>//request/9761133.json

While all the other blobs are still accesable, for example this ones:

https://gaia.blockstack.org/hub/<address>/payload/9761133.json

I didn’t create any updates on my dApp
The blobs where there when I checked last time a week ago
I also tried to remove the excessive / I still can’t access this blobs

Did you issue any update which maybe filters for the term request or are you doing some sort of sharding and the node with my data in the request “folder” is down?

This is quite an issue for me, as it also affects my App on production and active users.

When I post a blob I even get a positive 200 OK Response:

{“publicURL”:“https://gaia.blockstack.org/hub/18Kgxa4XmJ7CNqpfSV1Uxh3ETtkvoX7tKx//request/5034226061874442652.json",“etag”:"0x8D83951D806A645”}

But as you can see, the Blob is not available afterwards.

jwiley · August 6, 2020, 12:09am

This domain doesn’t look right - try using hub.blockstack.org
i.e.

08:07:36 ~$ curl -s https://hub.blockstack.org/hub_info | jq
{
  "challenge_text": "[\"gaiahub\",\"0\",\"storage2.blockstack.org\",\"blockstack_storage_please_sign\"]",
  "latest_auth_version": "v1",
  "max_file_upload_size_megabytes": 20,
  "read_url_prefix": "https://gaia.blockstack.org/hub/"
}
0

compared to the domain you’re currently using:

08:05:43 ~$ curl https://gaia.blockstack.org/hub_info
<?xml version="1.0" encoding="utf-8"?><Error><Code>OutOfRangeInput</Code><Message>One of the request inputs is out of range.
RequestId:419cb9ee-201e-00fd-0e85-6b8563000000
Time:2020-08-06T00:05:48.1354249Z</Message></Error

the domain you’ve listed appears to be trying to talk directly to the backend storage and not going through gaia itself.

jwiley · August 6, 2020, 12:11am

to be clear, i’m not sure how this would have worked in the past - but there were recently changes (out of our control) on the storage backend that necessitated some DNS changes on our end several days ago.

marie · August 6, 2020, 10:02am

I fear, I wasn’t clear enough in my Bug report.

I’m actually using blockstack-js to store the blobs to https://hub.blockstack.org, and I also get a 204 response:

The response body I get is this:

But when I try to fetch the blob from the returned publicURL I get a 404. So even If i’m trying to fetch from the wrong domain (I tried various domains, but the file never can’t be read), at least there is a bug in the API which returns me a wrong publicURL

The weird thing is that this problems only occurs on blobs in the //request/ path. All my other blobs I can read. So as I’m using the official blockstack-js library and the publicURL I get back from the official API I still think there is an issue.

I tried to replace the domain in the publicURL with https://gaia.blockstack.org it didn’t help, I still can’t find any of these files.

marie · August 6, 2020, 10:39am

Some additional Info:

1.) when I request the list-file endpoint https://hub.blockstack.org/list-files/12D5MUizN8syujHJFEGua5S4KhwkrcX4RE It still returns me the missing blobs in the response

2.) I upgraded blockstack-js from 19.3.0 to the newest stable version 21.1.0, but I still have the same issue.

3.) I also ran through my code again and checked if I have any of the domains hardcoded somewhere, but it’s clearly not the case. I actually use blockstack-js to store all my files and the returned URLs to fetch them.

marie · August 6, 2020, 11:58am

I think I figured out what causes the issue.

The problem seems to be caused by the leading / in the beginning of the path, like here: https://gaia.blockstack.org/hub/18Kgxa4XmJ7CNqpfSV1Uxh3ETtkvoX7tKx//request/1243032195920652490.slip.json

Until a few days ago this still worked, but now I can’t read files stored with a leading / anymore. Storing my files different from now on doesn’t solve the issue, as I still need to find a solution to recover my existing files.

marie · August 10, 2020, 1:54pm

@jwiley Is there any update on the issue that existing blobs with a leading / are not available anymore?

jwiley · August 10, 2020, 3:42pm

can you share with me a few of the paths you’re having a problem with specifically?
email is fine if you’re not comfortable sharing here.

or DM here, whichever you’d prefer

marie · August 13, 2020, 11:00am

Of course:

https://gaia.blockstack.org/hub//18Kgxa4XmJ7CNqpfSV1Uxh3ETtkvoX7tKx/request/8906780712174450666.json

https://gaia.blockstack.org/hub//18Kgxa4XmJ7CNqpfSV1Uxh3ETtkvoX7tKx/request/8376821717229073656.json

It’s also super easy to reproduce. Just create a blob starting with a ‘/’ and you will get a 201 OK Response when creating it, but then you won’t be able to fetch it.

jwiley · August 17, 2020, 2:41pm

Thanks, will take a look and see if anything is apparent.

edit:
i can confirm the files do exist, but i’m looking into why they’re not accessible.
offhand, the slashes do seem important, as other files i know of have the same issue as here.

edit 2:
I can confirm the files are accessible, although not through https at the moment. looking into why that is, but it’s definitely related to the second “slash” here

marie · August 25, 2020, 12:55pm

This issue is fixed now, thanks a lot for your support

jwiley · September 21, 2020, 3:25pm

To recap for future searchers - this was due to a change in the backend storage CDN, the CDN provider had changed and the new one treated that second / as part of the path.
The provider was manually reverted and the problem mentioned here was resolved.