Let's start with a hypothetical. Let's say you walk into a casino.
To your left is a completely normal game of blackjack.
To your right is a game of poker... only in this game, the first person to sit down gets to look at everyone else's cards during the game.
This modification breaks the game. If one player can see every other card, s/he is guaranteed to control the game; see who is going to win the hand, bet accordingly. Unless they are (or can influence) the first player, no rational person would choose to play that game.
The fundamental principle is fairness: are all individuals treated reasonably, consistently and equally.
The purpose: promote morality (or whatever).
The real purpose: people refuse to use or rely on systems they don't perceive as fair.
This idea becomes more important as the stakes of the system increase. You might be interested in playing modified-poker when the chips are valueless and the table is full of friends... but the decision changes entirely when the buy-in is 6 figures.
In 2020, Vitalik Buterin distilled this concept into this principle:
When building mechanisms that decide high-stakes outcomes, it’s very important for those mechanisms to be credibly neutral.
A mechanism is a tool that takes in inputs from multiple people, and uses these inputs to make some kind of decision that people care about in a way that is consistent with its participants’ values.
A mechanism is an algorithm plus incentives.
A mechanism is credibly neutral if it does not discriminate for or against any specific people, treating everyone fairly, to the extent that it’s possible in today's diverse and complicated world.
It is not just neutrality that is required here, it's credible neutrality. It is not enough for a mechanism to be fairly designed, it’s also vital that the mechanism convince a large and diverse group of people that it is fair (or at least makes basic efforts to be).
This should be easy to see just by looking at the mechanism's design.
Mechanisms such as blockchains, political systems and social media are designed to facilitate cooperation across large, and diverse, groups of people.
Credible neutrality is about both bringing a new participant to a mechanism and keeping its existing users.
Everyone participating wants to be sure that everyone else will not abandon the mechanism the next day.
Of course, neutrality is never total. For example, capitalism discriminates in favor of concentrated interests and the wealthy, and against the poor and those who rely heavily on public goods.
Furthermore, credible neutrality isn't the only thing we care about; it’s very possible for a mechanism to be both credibly neutral and terrible.But, still, some mechanisms are much more neutral than others.
Thus, credible neutrality must be balanced against another idea, the principle of efficacy.
Sometimes, if a credibly neutral mechanism has not yet been found, an biased mechanism should be adopted in the short term.
But recognizing credible neutrality as something that is valuable, and striving to get closer to that ideal, is important.
Credible neutrality is a systems design principle. The goal is to create systems that are well understood to be fair and provide a level playing field.
When we build credibly neutral systems, we are building for the future.
We are building legitimacy.
Source Material - Twitter Link
Source Material - PDF