Some apps were scored with the incorrect number of Product Hunt upvotes in May 2019. This effects a number of rankings for different apps. We’ll explain what happened, and how we’re fixing the issue. We always strive to be as open as possible, and will continue to own up to any mistakes we’ve made, with plans for how to mitigate them in the future.
Product Hunt provides us with ‘credible upvotes’ data, which uses a private, internal algorithm. Every app has some percentage of non-credible upvotes, and often apps with more upvotes have a higher percentage of non-credible upvotes.
On the 15th of the month, Product Hunt sends us a spreadsheet with the data they’ve collected. These include columns for total votes as well as credible upvotes. They send the spreadsheet in Numbers format.
Here, you can find the file that Product Hunt sent us in May: https://xordrive.io/?p=hankstoever.id_1ebcef99-a7a1-5360-a66c-63f7c162fc14
While formulating results for the June 2019 cohort, we noticed a discrepancy in the number of upvotes for some apps. After going through the data they sent last month, versus the data we used for scoring, I noticed a pattern. In column I of the Numbers spreadsheet, there is some odd indentation. When you copy the raw columns of the spreadsheet that Product Hunt gave us, and paste it into a Google Sheet with the “Paste without formatting” functionality (command-shift-V), the data in the first column overwrites the data in the second column .
This is the root cause of the issue. I always use ‘paste without formatting’ because I don’t want all of the Numbers app’s formatting in our spreadsheets. This is the first time I’ve noticed this kind of behavior.
Due to this error, 14 apps were scored with more upvotes than they should have been.
I have made a new spreadsheet that has the correct results. I have another spreadsheet that analyzes the changes in rank, and how much more money they should have made. Here is the key result for app miners: 18 apps were paid less than they would have been with the correct results .
Next Steps
If an app was paid less than they should have made, we will make up the difference and pay them. If you were paid more than your new rank, no action is required. We will also update the public rankings on app.co and related App Mining spreadsheets.
We are very sorry that this error occurred. In addition to making corrective actions to past payouts and results, we will introduce the following additions to our audit process:
- We will release the raw data that app reviewers give us. This will allow any community members to reformulate results from scratch.
- More intensive internal auditing of results before and during the audit period, including a public checklist that we use during each audit process. Github issue
- In the future, we may extend the audit period. We often receive raw data late in the day on the 15th, and have limited time to put everything together to send audit data to miners. We will consider waiting to send audit data until the 16th, and may also extend the audit period to longer than 48 hours. Github Issue
Please respond to this forum post with any other suggestions or remediation steps you feel is appropriate. App Mining is an iterative program, and we will always be open to owning our mistakes, and strive to improve our processes as the program evolves.