nltk.translate.chrf_score module

ChrF score implementation

nltk.translate.chrf_score.sentence_chrf(reference, hypothesis, min_len=1, max_len=6, beta=3.0, ignore_whitespace=True)[source]
Calculates the sentence level CHRF (Character n-gram F-score) described in

This implementation of CHRF only supports a single reference at the moment.

For details not reported in the paper, consult Maja Popovic’s original implementation: https://github.com/m-popovic/chrF

The code should output results equivalent to running CHRF++ with the following options: -nw 0 -b 3

An example from the original BLEU paper https://www.aclweb.org/anthology/P02-1040.pdf

>>> ref1 = str('It is a guide to action that ensures that the military '
...            'will forever heed Party commands').split()
>>> hyp1 = str('It is a guide to action which ensures that the military '
...            'always obeys the commands of the party').split()
>>> hyp2 = str('It is to insure the troops forever hearing the activity '
...            'guidebook that party direct').split()
>>> sentence_chrf(ref1, hyp1) 
0.6349...
>>> sentence_chrf(ref1, hyp2) 
0.3330...

The infamous “the the the … ” example

>>> ref = 'the cat is on the mat'.split()
>>> hyp = 'the the the the the the the'.split()
>>> sentence_chrf(ref, hyp)  
0.1468...

An example to show that this function allows users to use strings instead of tokens, i.e. list(str) as inputs.

>>> ref1 = str('It is a guide to action that ensures that the military '
...            'will forever heed Party commands')
>>> hyp1 = str('It is a guide to action which ensures that the military '
...            'always obeys the commands of the party')
>>> sentence_chrf(ref1, hyp1) 
0.6349...
>>> type(ref1) == type(hyp1) == str
True
>>> sentence_chrf(ref1.split(), hyp1.split()) 
0.6349...

To skip the unigrams and only use 2- to 3-grams:

>>> sentence_chrf(ref1, hyp1, min_len=2, max_len=3) 
0.6617...
Parameters

  • references (list(str) / str) – reference sentence

  • hypothesis (list(str) / str) – a hypothesis sentence

  • min_len (int) – The minimum order of n-gram this function should extract.

  • max_len (int) – The maximum order of n-gram this function should extract.

  • beta (float) – the parameter to assign more importance to recall over precision

  • ignore_whitespace (bool) – ignore whitespace characters in scoring

Returns

the sentence level CHRF score.

Return type

float

nltk.translate.chrf_score.chrf_precision_recall_fscore_support(reference, hypothesis, n, beta=3.0, epsilon=1e-16)[source]

This function computes the precision, recall and fscore from the ngram overlaps. It returns the support which is the true positive score.

By underspecifying the input type, the function will be agnostic as to how it computes the ngrams and simply take the whichever element in the list; it could be either token or character.

Parameters

  • reference (list) – The reference sentence.

  • hypothesis (list) – The hypothesis sentence.

  • n (int) – Extract up to the n-th order ngrams

  • beta (float) – The parameter to assign more importance to recall over precision.

  • epsilon (float) – The fallback value if the hypothesis or reference is empty.

Returns

Returns the precision, recall and f-score and support (true positive).

Return type

tuple(float)

nltk.translate.chrf_score.corpus_chrf(references, hypotheses, min_len=1, max_len=6, beta=3.0, ignore_whitespace=True)[source]

Calculates the corpus level CHRF (Character n-gram F-score), it is the macro-averaged value of the sentence/segment level CHRF score.

This implementation of CHRF only supports a single reference at the moment.

>>> ref1 = str('It is a guide to action that ensures that the military '
...            'will forever heed Party commands').split()
>>> ref2 = str('It is the guiding principle which guarantees the military '
...            'forces always being under the command of the Party').split()
>>>
>>> hyp1 = str('It is a guide to action which ensures that the military '
...            'always obeys the commands of the party').split()
>>> hyp2 = str('It is to insure the troops forever hearing the activity '
...            'guidebook that party direct')
>>> corpus_chrf([ref1, ref2, ref1, ref2], [hyp1, hyp2, hyp2, hyp1]) 
0.3910...
Parameters

  • references (list(list(str))) – a corpus of list of reference sentences, w.r.t. hypotheses

  • hypotheses (list(list(str))) – a list of hypothesis sentences

  • min_len (int) – The minimum order of n-gram this function should extract.

  • max_len (int) – The maximum order of n-gram this function should extract.

  • beta (float) – the parameter to assign more importance to recall over precision

  • ignore_whitespace (bool) – ignore whitespace characters in scoring

Returns

the sentence level CHRF score.

Return type

float