Karma: Exploration of a decentralized, open-source reputation system for the web

Moving this conversation here as an experiment in conversations off of GitHub. @brian Hoffman of OpenBazaar suggested moving the reputation conversation away from https://github.com/OpenBazaar/OpenBazaar/issues/1309

1 Like

Looping in @drwasho @greg @muneeb.


Side thoughts: we don’t necessarily want the Blockstore DHT to be open to everyone without any barriers, as I have doubts about our ability to mitigate against spam attacks. Thus the client, while of course open source, can be used to ensure that only networks that comply to certain standards for assessing reputation can push data to the Blockstore DHT.

Our basic idea about protection against Sybils / Spam in DHT is to require “proof of Bitcoin ownership”. You don’t need to spend the bitcoins but you need to keep them in an address that you own to join the DHT. IMO, as soon as we start getting into “gateways” to restrict access to a DHT in a centralized manner, we also start losing most benefits of the DHT as well and can then just have a storage daemon on top of S3 or something and give API access credentials to respective networks/parties for writing/reading data.

If we assume that our Sybil protection technique works, would you feel more comfortable using the DHT? Or are there other spam concerns you have? Would love to hear more.

P.S: In the current DHT implementation of Blockstore we have a protection in-place that stops spammer from overriding your data. I can provide more details on how it works, but that’s already a really good property to have.

A couple other ad hoc thoughts. Do we plan to have some kind of break-in period for reputation/karma? Should we have some threshold of incoming ratings before users can create ratings against others? This would be similar to having either op privs or comments turned on once you’ve been vetted by the community. In a decentralized network you would not have this simple method of controlling it obviously, but is there some metric we can use to automate the on boarding of a rating entity? This could help to prevent some level of trolls that want to jump on and just create a new identity and start fake rating everyone left and right or jamming spam ratings into the DHT. What is the spam control method in Blockstore @muneeb?

Two ways of doing it:

  1. At the application level you can require a minimum reputation before the user/identity can start participating in rating others. There is a Sybil problem here as well, where a botnet can create ‘n’ identities and they all rate each other and then start to mess with the reputation of others. Here is a good paper about this. The basic idea is Sybils tend to form disconnected graphs (lots of links between Sybils but not a lot of links going from a Sybil cluster to real users).

  2. You can restrict users by time of registration. We have “age” of when a username was registered (it’s measured in blocks of course!). This just delays the attack, in the sense that someone will need to “plan ahead” and have identities with a certain age. This has a nice side-effect that it gives normal users and your systems more time to detect if some identity is a Sybil or not.

As far as Blockstore, we’re basically linking our spam protection with transactions on Bitcoin. You can only write data where the hash of the data is already in the Bitcoin blockchain. So for someone to write junk content, they’d need to send a lot of Bitcoin transactions (and hence spend money). Similarly, you can’t override data that someone else wrote because the DHT only accepts data where hash(data) is in the blockchain i.e., it’s a content-addressable storage system.

P.S: I know Lakshmi (author of the paper I linked) fairly well and am happy to put you guys in touch with him.

I would follow Stack Overflow very closely on this. What they’ve built is probably one of the best reputation systems of all time and they’ve experimented with a lot of things.

How do you expect to ensure that a rating can only be recorded between two users that have actually engaged in some action together (like a trade in the case of OpenBazaar)? One idea is to have each party pre-authorize a rating by co-signing a message. This should be done before the transaction/trade begins. Then, after the trade is completed, each party can rate the other and reference the pre-authorization to show that they have the ability to rate the other / the rating can be successfully recorded by the rating crawlers.

Next, how would we encourage users to rate each other honestly? Users should be compelled to give positive ratings slightly more than they should to give negative ratings. Stack Overflow accomplishes this by taking away 1 karma point each time you downvote a question. Most people will value their personal 1 point higher than they would value subtracting 5 or 10 from another person. The only exception to this is in the case when it is actually important to the user to punish a bad actor, which ends up being when the user is suffiently unhappy (which is an approximation for when the bad actor is actually bad).

Also, does it make sense to put ratings in the DHT? What about the alternative which is having each user store their own ratings and having various parties mirror them? The DHT can then focus on storing identity data and linking to the hashes for the various ratings.

I still don’t buy into the using DHT storage as long term persistence. I think it’s more of a caching layer and so a history for reputation doesn’t seem like a good use of the DHT. Perhaps if we had some form of short-lived reputation storage, but it wouldn’t allow any kind of long-term reliable history.

Dio’s paper on Web of Trust addressed this self-rating issue as well. https://gist.github.com/dionyziz/e3b296861175e0ebea4b

If users store their own ratings @ryan I would think they would only project the positive ratings rather than any of the negative feedback.

@brian when you say you don’t buy using DHT for long-term storage, are you referring to identity data or rating files? If it’s the latter then I definitely agree with you, as I said in my previous comment.

As far as what you said about ratings, with my proposal, users don’t store the ratings about them. They store the ratings they made of other people. In addition, as soon as a mirror picks it up, even the rating user can’t delete it by ending it’s hosting of the file. That said, you could add a “revoke rating” capability.

In OpenBazaar I’m ultimately not expecting that the buyer will run a node all the time, or even run a node at all. But from my discussion with Chris Pacia, all the data is shared between nodes on the network right as a function of the number of nodes online?

Also there’s a privacy concern here in that I don’t want to disclose that I bought anti-fungal cream off you by leaving a rating.

Cream works well... itching stopped within 15 minutes of application.

At least for OpenBazaar, we have to find another way to validate that a legitimate trade on the network has occurred.

Also I’m working on v2 of my proposal that I’ll post here for discussion.


Ok I’ve been thinking about this issue all day and I think we need to address a fundamental question of ‘should’ before we dig into the details of ‘how’.

My argument is that a global reputation rating is essentially meaningless.

Let’s imagine that we have a reputation scale (like Klout) from 0-100. Alice is our user and she has a score of 40. This reputation score is calculated using a magical algorithm factoring in Alice’s reputation across various networks (e.g. OpenBazaar, Mine, Uber for X etc).

Within the OpenBazaar network, how am I supposed to determine whether trade with Alice is low or high risk? What if the bulk of her reputation score comes from networks outside of OpenBazaar? Just because Alice is a great friend, great gal, doesn’t mean she’ll be a great shoe merchant.

Reputation only matters within context… and aggregation is the enemy.

Even in the OpenBazaar network, a reputation of 40 (within a range of 0-100) is also effectively meaningless. Why? There are several roles within the OpenBazaar network: merchant, buyer, notary, arbiter, lender, service provider etc. Where is the bulk of her reputation coming from? Perhaps her rating as a merchant is responsible for her score. Should I then trust her as a notary?

Fundamentally, reputation is the sum of positive/neutral/negative interactions at performing a specific action. Context is everything.

In OpenBazaar, I proposed that ratings need to be defined by their role within the network. Within that role, there are even more context-specific ratings that exist. For example, a merchant also needs to be rated on how accurate the item description was, shipping time, customer service etc.

So going forward, I think this discussion should focus on how network-specific reputation data could be stored… leaving the responsibility of defining how reputation is calculated to that network. We shouldn’t be tempted to calculate one reputation score to rule them all.

Side note: This doesn’t invalidate the work of developing a vouch-based web-of-trust, as suggested by Dionysis, as there’s some value in plotting how many people endorse you.

1 Like

@drwasho I’m with you re “network-specific” reputation. Aggregation can always be performed at run-time and there is no need to throw away the raw data and store only the aggregated values (unless you’re running low on storage space, which we’re not).

I’m thinking of this as an “app specific” rating rather than network-specific, but I think we’re talking about the same thing.

1 Like

@brianhoffman Yes, so DHT is not persistent storage (although we have debated using it like that and pretty much decided against it). DHT is mainly there to avoid blockchain bloat and store small records. These records in the DHT can then point to persistent storage. So, you can think of the DHT as the “discovery layer” for finding out a) RSA / PGP keys of the owner, and b) pointers to where the (signed) data is actually stored.

@drwasho @muneeb yes I agree, there shouldn’t be any global reputation rating. The way I see it is there should be a standard system for ratings and a standard format. Each rating can have a “type”. And then individual applications can crawl the data and hone in on the subset of ratings that are relevant and then come up with their own form of calculating the rating score, if at all.


I agree with the realization that global reputation means nothing. It’s all contextual. I think the global issue is tying different contexts to the same individual actor. Perhaps we are delving into local reputation vs. non-local. I think there are times when an application or group may assign a reputation/karma to a user without really interacting with them and this can be applied in a broad context. This makes it easy to farm out to other sites, but doesn’t mean as much as a local reputation within a specific context (e.g. OpenBazaar). An important part of what we seem to be trying to build is making it standardized and simple to switch between reputation contexts for a specific user. How can I accumulate separate contextual karma scores that all chain up to me as an individual?

Here you create a root identity that branches as many times as necessary for these silos of reputation. They have different meaning within their applications or usages, but stay separate from others. My reputation as a Reddit user should not be conflated with my work reputation for instance. It shouldn’t even really matter where the reputation resides as long as it some how cryptographically ties back to your original identity. Facebook could somehow “link” their internal private identity score to your root identity and if they allow others to query it fine. If not, doesn’t matter.

I don’t believe in global reputation but I do think there may be some concept of composite reputation where like community reputations maybe can be composited to form some kind of group score (i.e. social media reputation, bitcoin reputation, legal reputation). That would be an interesting exploration.

1 Like

The more I’ve been thinking about this problem, as it relates to storing the reputation data, especially in light of the comments made in this thread, I am more convinced that we need to turn to Chris Pacia’s Subspace network.

Overview: https://medium.com/@chrispacia/subspace-73059a1cff71
Messaging Client: http://www.bitcoinauthenticator.org/subspace/
Subspace & OpenBazaar: https://gist.github.com/drwasho/8daf0e95d42ab2266e6b

The architecture of the network closely matches comments made by @ryan and others, a DHT used as a routing table for finding data stored on a distributed federated network of Subspace servers.

At this point, please read over Chris’ overview of the project first before reading my statements below.

Subspace is more than just an anonymous encrypted messaging client and proxy for broadcasting Bitcoin transactions. For this discussion, the application of Subspace as a file storage platform is what I want to focus on.

Theoretically, Subspace nodes/servers (can be a regular computer [extra HDD space ala StorJ] or a dedicated server) can store any encrypted data, typically for a price with market-driven forces guiding that entire process. However, we can consider integrating another model into this system.

This model would be to gratuitously crowd-share data relevant to specific networks.

This would work in Subspace by implementing a “datastream subscription” within the app. So if I download Subspace and subscribe to the OpenBazaar rating/review datastream, I’ll begin to download this data from other nodes on the network. The DHT will list my node as a host for some of this data accordingly.

For OpenBazaar, we could deploy a few Subspace servers to specifically store rating/review data. Other users may deploy nodes or share HDD space to contribute to this effort. Thus far, the incentive for this is purely non-monetary (additional thought may reveal some financial incentives).

So when a user fires up Subspace, they may subscribe to the Onename, Mine, and OpenBazaar datastreams in addition to hosting other data on their drive for a financial incentive.

Earlier this year I discovered Open Badges, a Mozilla project for digital crentials that seems to have gained some traction. I realized that I found it that I may have found the decentralized solution for reputation that I’d been looking for. At it’s core, it’s a pretty simple system: crypto-signed JSON blobs in the OpenBadges format. The cool thing about these badges is they can be used to “endorse” someone for pretty much anything.This means that the context in which you are endorsing someone can be as broad or specific as you want.

So you can endorse someone for “selling coffee,” there’d be a field for the endorser, the endorsee, a link to a badge image file, and then the whole blob would be crypto-signed with the key associated with the endorser’s identity (blockname? passname?). The endorsee would add it to their “reputation folder” which can be linked to from their Blockstore profile.

One question I’ve been thinking about that I don’t think I have completely figured out: how to store negative reputation ratings/reviews. A lot of people probably wouldn’t store negative things about themselves, and we can’t rely on the reviewers to do it. So does the reviewer pay a small fee to some hosting service to do this? If so, how do other people discover this negative rating? The host could also be a directory which is mirrored or federated with other directories for persistence/ censorship-resistance, and a list of these directories could be added by users and/or client app developers, which then gets pinged every time a user wants to see the negative reputation ratings/ reviews another user has received.

Getting warmer? :sunglasses:

Where do we store negative reviews? We are looking currently at having this be a responsibility of notary peers within OpenBazaar. Because they’re already a pseudo trusted third party they could be used to validate existence of negative feedback.

Is is at all possible to associate an OP_RETURN tx on the blockchain with a previous OP_RETURN? So I could attach a hash of my negative review to the GUID of the entity I am reviewing.