Peer prediction

Literature Review
1

Summary & takeaways

Peer prediction is a branch of mechanism design concerned with eliciting truthful information from individual agents in situations where verifying the truth is difficult or impossible, thus making usage of traditional proper scoring rules impossible. For that, they rely on comparison of agents’ reports with peers (called “references”) and produce a modified scoring rule on this basis.

As a reminder, as usual in elicitation / scoring-rule-based mechanisms, we consider agents who receive signals and make reports to the mechanism. The designer is tasked with producing incentives that elicit truthful reports.

Peer prediction mechanisms introduce agents’ signal posterior, the belief that another agent receives a signal given the current agent’s signal, typically denoted p_i(\cdot|\cdot) for agent i.

All peer prediction mechanisms rely on the following constraints:

  • Stochastic relevance: different agent’s signal observations are correlated, ie. p_i(\cdot,s) ≠ p_i(\cdot, s') for s ≠ s'.
  • Belief model constraints: each mechanism has a different but necessary belief model constraint, guaranteeing effectiveness of the scoring rule.

For example:

  • Output Agreement, p_i(s|s) > p_i(s′|s)
  • 1/p Mechanism, p_i(s|s)/y(s) > p_i(s′|s)/y(s′)
  • Shadowing Method, p_i(s|s) − y(s) > p_i(s′|s) − y(s′).

Literature

Mechanisms

Eliciting informative feedback: The peer-prediction method.

A Bayesian truth serum for subjective data

Learning the prior in minimal peer prediction

  • Authors: Witkowski, Jens, and David C. Parkes
  • Year: 2013
  • Description: Non-minimal mechanism that allows learning the prior.

Mathematical models

A geometric perspective on minimal peer prediction

The limits of multi-task peer prediction

1 Like