Correctly encoding possibly offensive words with their correct
frequency and the possibly_offensive flag set.
Continuing to encode with zero frequency only distracters or
words that should never come up.
https://paste.googleplex.com/5167060875214848
Bug: 11031090
Change-Id: Ia394b1827f292ff8d4791cc2f3e6e50b5aff4cbe
en, en_GB, en_US:
Add "id" -> "I'd" whitelist entry
Reinstate "id" and "ID" in the respective dicts
fr:
Remove many words that are not French
Change "google" to "Google"
pt_BR:
Delete "idéia"
Change-Id: I942266ac7995345580926f60de45d202aa257ae7
>>> dictionaries/pt_BR_wordlist.combined.gz
Header :
date : 1355802839 <=> 1357790917
version : 29 <=> 30
Body :
Added: à 30
Added: é 30
Added: ò 30
Added: ô 30
>>> dictionaries/pt_PT_wordlist.combined.gz
Header :
date : 1355802856 <=> 1357790930
version : 29 <=> 30
Body :
Added: à 30
Added: é 30
Added: ò 30
Added: ô 30
>>> java/res/raw/main_pt_br.dict
Header :
date : 1355802839 <=> 1357790917
version : 29 <=> 30
Body :
Added: à 30
Added: é 30
Added: ò 30
Added: ô 30
Bug: 7966948
Change-Id: I71c0986cf616d67926d0a6a0e53099b04b0427d5
cs, da, de, el, es, fi, fr, hr, it, lt, lv, nb, nl, pl,
pt_BR, pt_PT, sl, sr, sv, tr : rescale frequencies to match
spec. This has no large effect in the practice except the
dictionary will become stronger vs spatial model (especially in
lower count corpora, like lt, lv, sr)
en* : Small changes (rounding going the other way essentially)
ru : the above rescaling, and remove the following words:
Дре, ОСТа, Планше, легкими, легком, легкому, легкости,
легкую, нелегкие, нелегкий, нелегким, нелегкое, нелегкой,
нелегкую, полулегком and add нелёгкие, нелёгкое, нелёгкую;
other accented forms were already in the dictionary.
Change-Id: I40386c2ebd4d2be38874e822bde89db7cb512ae6
Adrian Velicu
Jean Chalard