nltk.metrics.AnnotationTask

class nltk.metrics.AnnotationTask

Bases: object

Represents an annotation task, i.e. people assign labels to items.

Notation tries to match notation in Artstein and Poesio (2007).

In general, coders and items can be represented as any hashable object. Integers, for example, are fine, though strings are more readable. Labels must support the distance functions applied to them, so e.g. a string-edit-distance makes no sense if your labels are integers, whereas interval distance needs numeric values. A notable case of this is the MASI metric, which requires Python sets.

__init__(data=None, distance=<function binary_distance>)

Initialize an annotation task.

The data argument can be None (to create an empty annotation task) or a sequence of 3-tuples, each representing a coder’s labeling of an item: (coder,item,label)

The distance argument is a function taking two arguments (labels) and producing a numerical distance. The distance from a label to itself should be zero: distance(l,l) = 0

load_array(array)

Load an sequence of annotation results, appending to any data already loaded.

The argument is a sequence of 3-tuples, each representing a coder’s labeling of an item:

(coder,item,label)

agr(cA, cB, i, data=None)

Agreement between two coders on a given item

Nk(k)

Nik(i, k)

Nck(c, k)

N(**kwargs)

Implements the “n-notation” used in Artstein and Poesio (2007)

@deprecated: Use Nk, Nik or Nck instead

Ao(cA, cB)

Observed agreement between two coders on all items.

avg_Ao()

Average observed agreement across all coders and items.

Do_Kw_pairwise(cA, cB, max_distance=1.0)

The observed disagreement for the weighted kappa coefficient.

Do_Kw(max_distance=1.0)

Averaged over all labelers

S()

Bennett, Albert and Goldstein 1954

pi()

Scott 1955; here, multi-pi. Equivalent to K from Siegel and Castellan (1988).

Ae_kappa(cA, cB)

kappa_pairwise(cA, cB)

kappa()

Cohen 1960 Averages naively over kappas for each coder pair.

multi_kappa()

Davies and Fleiss 1982 Averages over observed and expected agreements for each coder pair.

Disagreement(label_freqs)

alpha()

Krippendorff 1980

weighted_kappa_pairwise(cA, cB, max_distance=1.0)

Cohen 1968

weighted_kappa(max_distance=1.0)

Cohen 1968