There was a Blockstack Browser release just before the holidays. This, under normal circumstances, is great. It was a big release (first iteration of user-selected storage is now supported!). However, this release was pushed out right before the entire Blockstack team went away for the holidays. Again, in and of itself, that’s not a problem. The problem comes if and when there are significant problems with a release and no one is around to address them for 2 weeks. That’s what happened here.
Below are a collection of issues that all appear to have stemmed from the new browser release (or the new year not being handled properly):
Many of the above bugs are showstoppers (meaning regular users would not be able to use apps because of them).
As a side note, this issue was not addressed in the browser release and it seems like it should have been before any new release considering its impact:
I appreciate the Blockstack team’s work immensely, but my big ask would be to hold off on releases when there will be no one around to address problems that might come from the release. My smaller ask would be to work on prioritizing bugs in a manner that better identifies what is critical, what is high, what is medium, etc, etc.
Yeah, I noticed issues starting arise back before Christmas, but I was able to deal with them. Now, however, it seems like more and more things have broken. I’m having trouble even getting things like Graphite to load properly. Along with this, my profile information in the local Blockstack browser is just gone. My social links for verification, my profile username and bio, and my profile pic which is now throwing errors when I try to update the picture.
Guys, these issues have completely paralyzed Stealthy in an irrecoverable way. We will have to send out a patch through both App Stores to resolve this issue.
Here’s a video on how every user would currently have to recover to a good state.
Some questions I have are:
What is the testing policy for the apps downstream if any?
How are developers supposed to rely on this auth system if we are let down like this?
I think if app developers are expected to build the amazing apps on this platform, there needs to be best practices in place to ensure these issues don’t keep repeating themselves.
First off, I’d like to take personal responsibility for releasing these changes to the browser right before the holidays and not ensuring that our team was in place over them to handle any problems that could arise, as they apparently have. I’m not sure yet just which of these issues are the fallout of that release, but it’s certainly not a good feeling to think that they could have been avoided or resolved already by timing the release better.
So, my apologies to all the developers – and end-users – affected here by these problems. I’m scrambling to get more information for you all about the specific issues. The team is back to work officially tomorrow, so we’ll have more for you at the very latest. But I hope we can start getting more information today as well.
@prabhaav I’ll get you full answers to your specific answers soon once I’ve had a chance to talk more with the team. You all deserve to know that we have the right practices in place here.
Thanks @markmhendrickson for the update and response. I know that @yukan is already working on a hot fix.
Totally agree.
@prabhaav I 100% agree with your comments and feel responsible for the issues Stealthy users are experiencing.
There should’ve been (a) better QA process in place for the release and (b) buffer time to address any issues that slip through the cracks before the holidays. My apologies for this.
QA and testing is by far the #1 priority for us in Q1 2019. We’re moving from a “startup building features” to a more mature platform with many developers relying on it and we need to massively improve our reliability.
I’ll personally make sure that adequate time/attention/resources are allocated to this in the coming weeks and months. This includes hiring dedicated team members focused on QA and release management. We should include testing the top apps in our release tests as well.
I want to reiterate that the reliability of our developer platform is critically important to us and we fully realize that other companies and developers are dependent on the platform. How this release caused issues, how our QA process didn’t catch the issues, and how long the issues were unaddressed is totally unacceptable and all of this needs to improve.
I’m confident that we’ll get there soon. Thanks for your patience as we iron out the current issue. We’ll update this thread as the hotfix is deployed.
Thanks for posing PBJ and sorry about these issues. We were chatting with Justin and Nick for about 24 hours prior to him posting this so we were aware of the problems, the challenge was diagnosing and getting fixed while everyone was on vacation.
Think we’ll have some internal discussions on what happened this week and share an update here. Again, our apologies. We know QA is a big problem and are focused on improving in Q1.
I think its a great idea if we implemented “gated releases” as part of the new QA process where we test the top apps before each major release! Maybe we can use trymyui for this? @jeffd
Impact on Blockusign
Just to put in my 2 cents about the fallout of this bug: I had a new user unable to use Blockusign, fortunately he reported a bug me here: https://github.com/ntheile/blockusign/issues/78 . Hopefully I will not lose him as a user.
No worries @markmhendrickson, as developers we have all been there before . I really appreciate the troubleshooting over the holidays!
In response to the issues we experienced over the holidays, we’re making major changes to our QA process and will have updates for the community shortly!
. I really appreciate the troubleshooting over the holidays!