App Mining Rewards Is Totally Broken

prabhaav · January 22, 2019, 8:58pm

You might think we’re complaining because our App, Stealthy, went from 2nd place in the last two rewards cycles (Alpha and December) to 13th place in the recent January rewards. However, even the math in the App Mining white paper suggests that a drop this big should be improbable in a single review cycle (see: Step 4: Function F of the App Mining).

There are serious problems with each of the App Mining Rewards program reviewers and the overall numbers. We shine a light on these issues below, hoping to make the App Mining Rewards system more than a gameable popularity contest plagued by incorrect incentive structures.

prabhaav · January 22, 2019, 8:59pm

App Mining Reviewer Democracy Earth

Fraudulent Voting

Votes on Democracy Earth are supposed to be from a select group of individuals as described on p. 11 of the Blockstack Token White Paper (https://blockstack.com/tokenpaper.pdf). Unfortunately that’s not happening.

Despite @nicktee identifying that anybody could log in to cast 1000 votes during the December review cycle, the problem wasn’t fixed. For instance below we see the votes of @benoror, an app developer, illustrating the problem:

Nobody caught unqualified voter, @critic, signing up and giving 1000 votes to the App Hermes for the January rewards:

Let’s see who Critic really is. Here’s Critic’s profile.json: https://gaia.blockstack.org/hub/1KpM4VMhnKTU95MsDrY24FNCtqao1RgCrT/4/profile.json

Here’s the other profile.json’s associated with that keychain:

https://gaia.blockstack.org/hub/1KpM4VMhnKTU95MsDrY24FNCtqao1RgCrT/0/profile.json

https://gaia.blockstack.org/hub/1KpM4VMhnKTU95MsDrY24FNCtqao1RgCrT/1/profile.json

https://gaia.blockstack.org/hub/1KpM4VMhnKTU95MsDrY24FNCtqao1RgCrT/2/profile.json

https://gaia.blockstack.org/hub/1KpM4VMhnKTU95MsDrY24FNCtqao1RgCrT/3/profile.json

All those ID’s belong to Dan Healy, the developer of Hermes Chat.

Invisible App Candidates

One reason why Stealthy might have seen a reduction in the number of Democracy Earth votes (shown below) is that voters could not find it in the list of Apps to vote on.

That’s because Stealthy appeared at the very bottom of a list of 46 apps so you had to scroll down for days if you were interested in casting a vote for it. Worse still, the list is lazy loaded, meaning that if you were smart enough to just search for Stealthy, you wouldn’t find it.

You had to scroll for days in order for it to even appear in the page to be searched, here is a video showing this problem: https://www.youtube.com/watch?v=ljFcAKRskHQ

Democracy Earth has indicated that the page order is randomized, but based on what trigger?

In all of our uses of the page after keeping it open or closing it and reloading, Stealthy continues to be on the bottom. This means that Stealthy and possibly other Apps are effectively invisible, especially given the lazy loading behavior, requiring an inordinate amount of effort by voters.

prabhaav · January 22, 2019, 8:59pm

App Mining Reviewer Product Hunt

Paid Popularity Contest

We’ve learned that there are a number of ways to game your upvotes on Product Hunt beyond just getting every person you know to vote. There is even an official way as this email we received from Product Hunt shows–it says we can pay them to re-feature our product on the homepage and their newsletter:

There’s also the unofficial way that @bodymindarts pointed out during the mining rewards alpha run, where you buy 90 votes for a bargain–$5: Some initial thoughts on the App Mining Program .

Transparency

The Product Hunt “Team Score” is mysterious offering no direct feedback or transparency other than a number. We significantly improved user experience December through January, yet our score dropped from 8.85 to 6.25 during this period. It’s puzzling when we compare it to the scoring of other Chat Apps–one of which is not even functional. The below image shows Hermes after sending a message–it disappeared and all we saw was errors in the Chrome console…

That same non-working App is the highest ranked by Product Hunt in the January scoring below:

Since we have no visibility into our Product Hunt team score, we can only speculate that they ran into technical issues related to the numerous Blockstack issues during the review cycle. For instance:

If they tried the app during last week of December, then they probably faced this issue: Number of Issues After Most Recent Browser Release
If they tried the app during the app review period, they probably faced this issue: Recent Blockstack Browser Vulnerability Discovery and Mitigation

prabhaav · January 22, 2019, 9:00pm

App Mining Reviewer TryMyUI

Your App Does Not Need To Be Functional

A messenger that literally does not work, outperformed our App in the rankings from TryMyUI! Below are TryMyUI results sorted in descending order for January. Stealthy is at the very bottom for communication applications, ranking 39th out of 46 and one of the highlighted Apps literally does not function and throws errors in the Chrome console when attempting to use it:

This data shows that nobody is inspecting the mining results and checking to see if they make sense. It’s very strange that Blockstack would accept this data for an App that won their decentralized Chat Bounty), is featured in every Blockstack presentation, website, blog, fund, app.co, etc., and has been developed for 14 months?

The Mobile Disadvantage

Reviewing Apps is not a trivial task–how for instance do we normalize the comparison between Apples and Oranges? When nearly every App is for desktop and Stealthy is mobile, little details in the review process cause problems–for instance the TryMyUI task description:

Stealthy is mobile first with a minimalist onboarding and login process–the testers were never directed to our website with links to our github, medium, twitter, news articles, FAQ’s and a variety of other information about what the app is. The users in our review videos didn’t understand anything about the app, Blockstack, decentralized technology, or blockchain. They were directed to the Android App store, where they ignored the description we had about our technology and skipped right into the App.

Imagine you had to review Signal without any prior knowledge about it or it’s purpose or having any contacts to message. This is what Signal looks like when you open the app and have no contacts:

Some feedback was platform specific and we were unable to get a balanced review because Blockstack technical issues that delayed our iOS availability:

Stealthy iOS app with the vulnerability fix was delayed by this issue: https://github.com/blockstack/blockstack-browser/issues/1790

Three of our eight testers complained of not being able to upload a file. We were baffled until we realized that the instructions below from TryMyUI had confused them–Stealthy does not have a file upload feature.

The totality of the TryMyUI results calls into question their ability to properly evaluate communication Apps or understand what they are evaluating. None of the other communication apps are on mobile or feature push notifications–very strange for top performing communications tools.

The Numbers Don’t Add Up

Here’s are Stealthy’s TryMyUI results from Blockstack’s App Mining rewards spreadsheet for January:

Unfortunately, you can clearly see that they do not match the numbers provided by TryMyUI to us, as shown in the screenshot below:

Unfortunately, App Rewards Mining has suffered from human error before in the previous round with a ranking problem due to miscalculations: Issuing an app mining ranking correction

Clearly Blockstack needs to improve their process and attention to detail in the simple data entry and computation of the App Mining Rewards.

prabhaav · January 22, 2019, 9:00pm

Fixing App Mining Rewards

Stealthy’s App Review Resolution

The negative score currently assigned to Stealthy for January is averaged into future scores, affecting Stealthy’s ability to compete fairly, reclaim higher rankings, and fund our development. While there are many issues with these App Mining Rewards, at a minimum,we believe Stealthy should be re-reviewed by the Product Hunt and TryMyUI reviewers. These re-reviews should, address the problems we’ve outlined: transparency, technical failures from Blockstack, poor instructions, and tester guidance.

Long Term

To date App Mining Rewards have heavily awarded two Apps that appeared 2 months ago (Zinc, SpringRole) that only use Blockstack Authentication. These Apps do not use the multitude of other Blockstack features, failing to push the platform further. How does this get Blockstack to the 1M users milestone or improve Blockstack’s storage, cryptography, and mobile platform features?

Blockstack’s own core engineers agree that Blockstack Auth PLUS Gaia storage should be required. Additionally, Blockstack, itself, defines a Blockstack app as:

“I. Users own their data

II. Users own their identities
III. Users have free choice of clients”

Instead it’s left in the hands of illegitimate Democracy Earth users, gameable Product Hunt votes, and uninformed TryMyUI testers. Why doesn’t the actual Blockstack community get consideration in the scoring, for instance with their logins or more detailed feedback:

prabhaav · January 22, 2019, 9:02pm

Integrity of App Mining Rewards

The integrity of the Blockstack App Mining Rewards system is compromised. If we don’t address these critical issues and properly align the incentives for app developers, Blockstack will continue to lose great Apps that add value to the platform (like Misthos). Ultimately Blockstack could end up being a graveyard of crummy gambling Apps, like other platforms.

Blockstack as a platform for app developers is only as good as the applications being built. Our goal since Day 1 has been to build a company around our product and help Blockstack towards the Milestone 2 goal. We’ve had to customize a lot of code to make a working, production-ready application. Others have had to do the same. Generally, we’ve been happy to do so because we knew Blockstack was supporting that. Unless things change with App Mining, we don’t see how we can feel supported enough to continue building on Blockstack. We want to work together to figure out a path forward that works for everyone.

@muneeb @jeffd @hank @patrick @larry @jude @aaron @yukan

benoror · January 22, 2019, 9:15pm

For the sake of transparency, I raised the issue in Slack, then went ahead to see if it really was possible, then casted the votes back:

https://blockstack.slack.com/archives/C07QPN6SV/p1546861784081700

https://blockstack.slack.com/archives/C074L9QQG/p1547761008371200?thread_ts=1547736526.349300&cid=C074L9QQG

cc @medied @Gina

hank · January 22, 2019, 9:22pm

It appears that we failed to give fully accurate information behind the numbers used in the ranking process related to TryMyUI.

The numbers used in the rankings are the ‘trimmed mean’, not the actual mean. They eliminated the highest and lowest scores to control for outliers, and then computed the mean from the middle 6 scores. They felt (and I personally agree) that this results in a more accurate figure.

If you uncheck the top and bottom scores for your user list when viewing the ALF score, the numbers should add up to what you see in our spreadsheet.

Obviously there is much more to comment about here, but I wanted to share this information first.

jehunter5811 · January 22, 2019, 9:43pm

I have thoughts on all of this, but that someone was able to vote for their own app and it counted is EXTREMELY problematic. I’m sure the fact that someone who shouldn’t vote was able to can be corrected via code and bug fixes on DE’s end. But the fact that Blockstack did not catch a completely fraudulent vote is not only disappointing personally but frightening from a company/mission perspective.

I am proud of what Graphite has done and I’m happy with the results I’ve seen personally, but I cannot participate in something that can be gamed. I expect a response to this specific issue soon. Very soon.

Gina · January 22, 2019, 10:06pm

Yep, Democracy Earth made an upgrade to the platform that resulted in a bug which was quickly escalated and fixed (as in @benoror post). After the problem was fixed, all non-whitelisted votes were removed.

We have verified the voters line by line several times , and can confirm that votes were only counted from whitelisted stack holders . Zero unverified or self-voting submissions made it to App Mining calculations.

jehunter5811 · January 22, 2019, 10:20pm

Thanks for the quick response @Gina! Are you suggesting that Dan Healy is a token holder? Or are you saying that the conclusions Prabhaav has drawn are incorrect?

I see the 1000 votes from the user critic in the DE vote. As has been the case for everyone else on the platform (I think), that username is tied back to a Blockstack username. A quick search for that username, reveals the profile Prabhaav posted. That profile is part of a keychain that is owned by the Hermes app developer.

I really hope I’m wrong. I hope Prabhaav is wrong.

benoror · January 22, 2019, 10:31pm

I believe what @Gina meant is that even though the votes are reflected in DE, those were not taken into account for the App Mining calculation

muneeb · January 22, 2019, 10:51pm

We can confirm that the only votes that were counted were associated with accounts of verified, whitelisted token holders. We can’t disclose the identities of these token holders.

There may be a world where any developer can convince any token holder to vote a certain way. This will remain true as App Mining is decentralized further and token holders are just Stacks addresses with no identity information linked to them. You can’t practically enforce the alternative — where an entity is in the position of enforcing the “purity” of tokenholders’ votes. There are no easy solutions to this problem. However, the problem becomes smaller and smaller with more participation in App Mining from token holders.

Also, you do not want me or Blockstack PBC to have the power to revoke/cancel a valid token holder’s vote in a decentralized system.

There is an open community meeting around App Mining tomorrow and we should discuss these topics.

P.S: I’ll comment on the other points shortly.

patrick · January 22, 2019, 11:25pm

Prabhaav,

First of all, your feedback is appreciated!

This is an experimental pilot and this is absolutely the time to work out kinks and help iron out issues, especially in the instance that app payouts become more significant and the App Mining mechanism further decentralizes. This is why we are involving the entire community of App Miners to join the recurring monthly call (next one happens to be tomorrow) to continue to iron out the process. Looking forward to seeing you on it. We’ve been acting diligently on incorporating feedback thus far, and will continue to do so. Will cover a few quick points ahead of the community call however:

Gaia integration being compulsory. Everyone agrees it should be there. It’s a question of timing and implementation. Let’s do it thoughtfully.
Hermes appears buggy, and wouldn’t be surprised if most apps have bugs after they’ve passed initial review (e.g. works for some installations, not for others), a spot check of the TryMyUI video shows people successfully using the app. Maybe the app owner needs to update Blockstack.js to resolve the bug issue, and also needs to better maintain the app. There are more solutions and we should usher those in.
We’ve addressed the “integrity of votes” issue. All voters are verified stacks holders. (See above discussion.)
Product Hunt popularity contest: Let’s discuss this on the App Miner community call tomorrow, there are some potential solutions to proactively address this, however it has not been exploited to my knowledge.

We want to work together to figure out a path forward that works for everyone as well. This is an ongoing experiment and are grateful for your feedback — its really helpful.

jehunter5811 · January 22, 2019, 11:27pm

I don’t mean to hijack all of Prabhaav’s points with this but I need someone to unequivocally tell me that “critic” the DE voter is not “critic.id”. No one has said that yet. Because those votes were absolutely counted and it’s a pretty odd coincidence that a voter gets allocated votes, drops all the votes on one app at the same time, and just so happens to have a DE username that matches a Blockstack ID of the developer that benefited from the vote.

patrick · January 22, 2019, 11:45pm

Justin, right on the same page with you.

Here's what we know for a fact at this moment: The user "critic" on Democracy Earth is tied to an investor who has been fully KYC'd as a part of the token sale and is not the creator of the app that benefited from the votes (completely different person, name and all). The above being said, the user who signed in to Democracy Earth either logged in with critic.id (the beneficiary app creator) or changed their username to critic . After logging in they confirmed their email address as that of the KYC'd investor.

These are the facts we know right now and are investigating further as to how this could have happened.

prabhaav · January 23, 2019, 12:06am

It has been taken into consideration: see the spreadsheet. You’ll notice that Hermes DE Theta went from -1 to +0.897

prabhaav · January 23, 2019, 12:07am

The fact that TryMyUI users were given incorrect instructions and there were bugs related to Blockstack affected our PH scoring.

What is the resolution for that? Are you asking us to carry forward a negative score due to no mistake on our part?

patrick · January 23, 2019, 12:46am

Unclear the PH score was due to bugs, but we can find out pretty quickly. As for TryMyUI, their team will be on the call tomorrow so we can review with them. I’ve got access to all 8 of your TryMyUI user testing videos for this run so I’ll double check to see how it affected anything…I imagine there’s way their team can help tweak instructions to make it more fair generally (every app got the same instructions this time). I’ll be ecstatic when app mining runs smoothly, and given this is a pilot run there’s always a fair solution after thoughtful dialectic. Going to continue this discussion on the call tomorrow (it’s being recorded and will be posted to the forum) with the community so they can take part live. Thanks for being patient

jehunter5811 · January 23, 2019, 1:00am

I’m going to continue the conversation in the forum because A) I and others, I imagine, won’t be able to make the conference call, and B) it’s important that anyone be able to see this discussion.

The problem with thoughtful consideration of how to introduce the Gaia storage requirement is that there are already apps on board that don’t use it. So you’ll put yourself in the position of telling those apps to add storage after the fact or grandfather them in.

I struggle with this point a lot because part of the reason a handful of apps participated in the alpha run in October was to provide feedback to help this process going forward. I mentioned this back then. Others did as well. I eventually even wrote a forum post about it. Now we’re deep into apps making significant money without putting in the work to use Blockstack’s core feature. As much as decentralized identity is cool, it’s not Blockstack’s important feature.

I’ve asked a few times about the progress toward Milestone 2’s goal of 1million users. If that goal is fueled by bringing on a bunch of apps that add Blockstack auth, I do not want to be part of that.