nltk.translate.IBMModel3

class nltk.translate.IBMModel3

Bases: IBMModel

Translation model that considers how a word can be aligned to multiple words in another language

>>> bitext = []
>>> bitext.append(AlignedSent(['klein', 'ist', 'das', 'haus'], ['the', 'house', 'is', 'small']))
>>> bitext.append(AlignedSent(['das', 'haus', 'war', 'ja', 'groß'], ['the', 'house', 'was', 'big']))
>>> bitext.append(AlignedSent(['das', 'buch', 'ist', 'ja', 'klein'], ['the', 'book', 'is', 'small']))
>>> bitext.append(AlignedSent(['ein', 'haus', 'ist', 'klein'], ['a', 'house', 'is', 'small']))
>>> bitext.append(AlignedSent(['das', 'haus'], ['the', 'house']))
>>> bitext.append(AlignedSent(['das', 'buch'], ['the', 'book']))
>>> bitext.append(AlignedSent(['ein', 'buch'], ['a', 'book']))
>>> bitext.append(AlignedSent(['ich', 'fasse', 'das', 'buch', 'zusammen'], ['i', 'summarize', 'the', 'book']))
>>> bitext.append(AlignedSent(['fasse', 'zusammen'], ['summarize']))

>>> ibm3 = IBMModel3(bitext, 5)

>>> print(round(ibm3.translation_table['buch']['book'], 3))
1.0
>>> print(round(ibm3.translation_table['das']['book'], 3))
0.0
>>> print(round(ibm3.translation_table['ja'][None], 3))
1.0

>>> print(round(ibm3.distortion_table[1][1][2][2], 3))
1.0
>>> print(round(ibm3.distortion_table[1][2][2][2], 3))
0.0
>>> print(round(ibm3.distortion_table[2][2][4][5], 3))
0.75

>>> print(round(ibm3.fertility_table[2]['summarize'], 3))
1.0
>>> print(round(ibm3.fertility_table[1]['book'], 3))
1.0

>>> print(ibm3.p1)
0.054...

>>> test_sentence = bitext[2]
>>> test_sentence.words
['das', 'buch', 'ist', 'ja', 'klein']
>>> test_sentence.mots
['the', 'book', 'is', 'small']
>>> test_sentence.alignment
Alignment([(0, 0), (1, 1), (2, 2), (3, None), (4, 3)])

__init__(sentence_aligned_corpus, iterations, probability_tables=None)

Train on sentence_aligned_corpus and create a lexical translation model, a distortion model, a fertility model, and a model for generating NULL-aligned words.

Translation direction is from AlignedSent.mots to AlignedSent.words.

Parameters

• sentence_aligned_corpus (list(AlignedSent)) – Sentence-aligned parallel corpus

• iterations (int) – Number of iterations to run training algorithm

• probability_tables (dict[str]: object) – Optional. Use this to pass in custom probability values. If not specified, probabilities will be set to a uniform distribution, or some other sensible value. If specified, all the following entries must be present: translation_table, alignment_table, fertility_table, p1, distortion_table. See IBMModel for the type and purpose of these tables.

reset_probabilities()

set_uniform_probabilities(sentence_aligned_corpus)

Initialize probability tables to a uniform distribution

Derived classes should implement this accordingly.

train(parallel_corpus)

maximize_distortion_probabilities(counts)

prob_t_a_given_s(alignment_info)

Probability of target sentence and an alignment given the source sentence

MIN_PROB = 1e-12

best_model2_alignment(sentence_pair, j_pegged=None, i_pegged=0)

Finds the best alignment according to IBM Model 2

Used as a starting point for hill climbing in Models 3 and above, because it is easier to compute than the best alignments in higher models

Parameters

• sentence_pair (AlignedSent) – Source and target language sentence pair to be word-aligned

• j_pegged (int) – If specified, the alignment point of j_pegged will be fixed to i_pegged

• i_pegged (int) – Alignment point to j_pegged

hillclimb(alignment_info, j_pegged=None)

Starting from the alignment in alignment_info, look at neighboring alignments iteratively for the best one

There is no guarantee that the best alignment in the alignment space will be found, because the algorithm might be stuck in a local maximum.

Parameters

j_pegged (int) – If specified, the search will be constrained to alignments where j_pegged remains unchanged

Returns

The best alignment found from hill climbing

Return type

AlignmentInfo

init_vocab(sentence_aligned_corpus)

maximize_fertility_probabilities(counts)

maximize_lexical_translation_probabilities(counts)

maximize_null_generation_probabilities(counts)

neighboring(alignment_info, j_pegged=None)

Determine the neighbors of alignment_info, obtained by moving or swapping one alignment point

Parameters

j_pegged (int) – If specified, neighbors that have a different alignment point from j_pegged will not be considered

Returns

A set neighboring alignments represented by their AlignmentInfo

Return type

set(AlignmentInfo)

prob_of_alignments(alignments)

sample(sentence_pair)

Sample the most probable alignments from the entire alignment space

First, determine the best alignment according to IBM Model 2. With this initial alignment, use hill climbing to determine the best alignment according to a higher IBM Model. Add this alignment and its neighbors to the sample set. Repeat this process with other initial alignments obtained by pegging an alignment point.

Hill climbing may be stuck in a local maxima, hence the pegging and trying out of different alignments.

Parameters

sentence_pair (AlignedSent) – Source and target language sentence pair to generate a sample of alignments from

Returns

A set of best alignments represented by their AlignmentInfo and the best alignment of the set for convenience

Return type

set(AlignmentInfo), AlignmentInfo