009e02ce4a
Further fixes to treat 0-frequency words
...
Previously, when both legitimate 0-frequency words (such as
distracters) and offensive words were encoded in the same
way, distracters would never show up when the user blocked
offensive words (the default setting, as well as the setting
for regression tests).
When b/11031090 was fixed and a separate encoding was used
for offensive words, 0-frequency words would no longer be
blocked when they were an "exact match" (where case
mismatches and accent mismatches would be considered an
"exact match"). The exact match boosting functionality meant
that, for example, when the user typed "mt" they would be
suggested the word "Mt", although they most probably meant
to type "my".
For this reason, we introduced this change, which does the
following:
* Defines the "perfect match" as a really exact match, with
no room for case or accent mismatches
* When the target word has probability zero (as "Mt" does,
because it is a distracter), ONLY boost its score if it is a
perfect match.
By doing this, when the user types "mt", the word "Mt" will
NOT be boosted, and they will get "my". However, if the user
makes an explicit effort to type "Mt", we do boost the word
"Mt" so that the user's input is not autocorrected to "My".
Bug: 11031090
Change-Id: I92ee1b4e742645d52e2f7f8c4390920481e8fff0
2014-10-31 15:58:50 +09:00
10416241f7
Block offensive words in multi-word suggestions
...
If the user has chosen to block offensive words and types
"aaaxbb", where "aaa" is an offensive word and "bb" is not,
we should not suggest "aaa bb".
Bug: 11031090
Change-Id: Ie23b8dd5d347bc26b1c046c3f5e8dfbc259bf528
2014-10-31 15:58:50 +09:00
aa20342d7e
Merge "Using "blacklist" flag as "possibly offensive""
2014-10-31 06:49:29 +00:00
7c87859d4c
Using "blacklist" flag as "possibly offensive"
...
Bug: 11031090
Change-Id: I5cc0d006ab003656498eb82b0875eb9c051d331e
2014-10-31 14:33:05 +09:00
0cd1f222fd
Fix: native unit test build.
...
Change-Id: Id2bd4b60d6a4023815a630ebb3059a435b72c193
2014-10-31 12:50:45 +09:00
bcb52d73e2
Enable count based dynamic ngram language model for v403.
...
Bug: 14425059
Change-Id: Icc15e14cfd77d37cd75f75318fd0fa36f9ca7a5b
2014-10-30 23:38:19 +09:00
660b00477c
Add DynamicLanguageModelProbabilityUtils.
...
Bug: 14425059
Change-Id: Ia58ab3f0ead02798046d182a9464dcbd95f086bc
2014-10-30 21:33:57 +09:00
0a9c3f30b6
Add method to encode probability.
...
Bug: 14425059
Change-Id: I3e5d359ba5fa38f1669f0e98dfae792ff53efbf8
2014-10-30 12:42:35 +09:00
c2ba0ce411
Fix: TRT and ime-simulator bulid.
...
Change-Id: I1697a907562d1ed6aff2b001763d1594263ba0d3
2014-10-30 01:01:40 +09:00
afe67611c3
Merge "Add a class to have global counters for LanguageModelDictContent."
2014-10-29 12:18:12 +00:00
6b0561f9d2
Add a class to have global counters for LanguageModelDictContent.
...
Bug: 14425059
Change-Id: I08ec19903432356b6028853fd73b4eefce20218e
2014-10-29 21:05:41 +09:00
dabc12974c
Merge "Improve space substitution error correction."
2014-10-28 09:26:40 +00:00
8a809f3433
Improve space substitution error correction.
...
Bug: 17432052
[Category diff]
+1 262
-1 93
+2 2
-2 18
+3 18
-3 2
+4 111
-4 148
+5 295
-5 217
+6 51
-6 276
+7 139
-7 124
[Weighted category diff]
+1 276
-1 100
+2 4
-2 20
+3 20
-3 4
+4 118
-4 160
+5 309
-5 225
+6 52
-6 298
+7 163
-7 135
show diff for ./en_user_log_phones_2011_08.csv
+1 173
-1 28
+2 2
-2 17
+3 17
-3 2
+4 63
-4 82
+5 120
-5 51
+6 24
-6 220
+7 88
-7 87
Change-Id: I9d673acb0ff632828ae2e0ead56e76e3a20411c6
2014-10-28 17:11:14 +09:00
3844f74aff
Fix: deleted PtNode handling in v403.
...
If a word is once deleted, the word never gets into the
personalized dictionaries due to this bug.
Change-Id: Ife4e3fe1ba0615b4135e6291d2151b0db7d3f940
2014-10-27 15:32:05 +09:00
69402dc992
Merge "Enable Address Sanitizer for native host test 2nd try"
2014-10-23 16:07:39 +00:00
e65973882d
Merge "Fix: Personalized dicts suggest invalid words with v403."
2014-10-23 10:33:09 +00:00
090c3819d7
Fix: Personalized dicts suggest invalid words with v403.
...
Bug: 14425059
Change-Id: I45ae00069dd3b7c461dd9a1f3558b96af0a1c975
2014-10-23 19:26:01 +09:00
5c4bec31d1
Enable Address Sanitizer for native host test 2nd try
...
This CL enables Address Sanitizer for native host test. Note that
production build is not affected with this change. ASan is enabled
only in static lib for test executables.
Change-Id: I2c8e99b8c55e611e86f74579f24a63ac949bb02d
2014-10-23 10:16:55 +00:00
2db1e56ff4
Merge "Stop building host native test in unbundled build"
2014-10-23 09:39:56 +00:00
ba35bb83a8
Stop building host native test in unbundled build
...
It turned out that building native code for host environment
is not supported in NDK build. Hence this CL makes the host
native test available only as a part of platform build to
avoid accidental build breakage in unbundled build.
BUG: 18095678
Change-Id: If608da166d5a478358e6890b8db526b4c2c0ab41
2014-10-23 18:31:06 +09:00
16cc3992d7
Use trigrams for personalization dict.
...
5Bug: 14425059
Change-Id: I73cf6904e569d60996a3b079f16ea6df0cb90f02
2014-10-23 14:32:45 +09:00
9c0b3419da
Merge "Revert "Enable ASan (Address Sanitizer) for native host test""
2014-10-22 10:54:18 +00:00
b9dc32ffd5
Revert "Enable ASan (Address Sanitizer) for native host test"
...
This reverts commit af2673f17d
2014-10-22 10:51:33 +00:00
0672e8554f
Merge "Enable ASan (Address Sanitizer) for native host test"
2014-10-22 10:13:08 +00:00
af2673f17d
Enable ASan (Address Sanitizer) for native host test
...
This CL enables Address Sanitizer for native host test. Note that
production build is not affected with this change. ASan is enabled
only in static lib for test executables.
Change-Id: Idbe1f2e4502dfce9b6fb0253d7ebda8d37fbf84e
2014-10-22 19:08:58 +09:00
b5ef884fbb
Support dumping ngram entries.
...
Bug: 14425059
Change-Id: Ib03a0c3d166ed6f1e60c67127b28006d55143b6b
2014-10-22 18:15:53 +09:00
c9865785f4
Support ngram entry migration.
...
Bug: 14425059
Change-Id: I98cb9fa303af2d93a0a3512e8732231c564e3c5d
2014-10-22 11:31:16 +09:00
0b8bb0c21b
Fix debug build.
...
Change-Id: Id94636714d04a8828718b87741c0ee62a14cb3b4
2014-10-21 20:20:11 +09:00
dfc82fa366
Merge changes I210acb81,Ie9508788
...
* changes:
Make NgramProperty have NgramContext.
Create .cpp file for NgramContext.
2014-10-21 10:28:25 +00:00
88bb28c132
Make NgramProperty have NgramContext.
...
Bug: 14425059
Change-Id: I210acb816b122857dbbe1ee4dd6a35c5335bf2bf
2014-10-21 17:12:32 +09:00
f87bb77a91
Create .cpp file for NgramContext.
...
Bug: 14425059
Change-Id: Ie950878817b9c80cc9c970e1a84880c9b9ab228a
2014-10-21 17:04:56 +09:00
fa1e65cb3a
Merge "Use EntryCounters during GC."
2014-10-21 07:55:04 +00:00
c51b9b5b3f
Merge "Renaming "blacklist" flag to "possibly offensive""
2014-10-21 07:39:18 +00:00
47fc656cd7
Use EntryCounters during GC.
...
Bug: 14425059
Change-Id: I61eb798686dc753fb6c0fe99a0719c1732198f30
2014-10-21 16:36:03 +09:00
e8750d970e
Introduce EntryCounters to count entries in a dictionary.
...
Bug: 14425059
Change-Id: Ic13ba827d96fa4a147485ba92fdb37e23e04e8e8
2014-10-21 15:46:14 +09:00
05172bf1a5
Renaming "blacklist" flag to "possibly offensive"
...
No behaviour changes.
Unified the overloaded FusionDictionary::add method to always take an
isPossiblyOffensive argument.
Bug: 11031090
Change-Id: I5741a023ca1ce842d2cf10d4f6c926b0efabaa78
2014-10-21 11:51:47 +09:00
1085fef8d0
Change entry count limit.
...
Unigram 10K, Bigram 30K, Trigram 30K.
Change-Id: Ibd19c6a2b618499df1c70000bad7b47498187f0a
2014-10-20 15:01:49 +09:00
f4928ad4dd
Merge "Update useless n-gram entry detection logic during GC."
2014-10-15 21:44:45 +00:00
3601c214f8
Update useless n-gram entry detection logic during GC.
...
Bug: 14425059
Change-Id: Ib939deae5b60167751dee07965bb1ef1a43c4625
2014-10-15 20:43:27 +09:00
183e21c36c
Merge "Use better conditional probability for ngram entries."
2014-10-15 09:27:21 +00:00
72d17d9209
Use better conditional probability for ngram entries.
...
Old:
P(W | W_prev) = f(W, W_prev) + C
New:
P(W | W_prev) = f(W, W_prev) / f(W_prev)
Bug: 14425059
Bug: 16547409
Change-Id: I4d13be6de2c6bad6bad7fb22320a23ba4ecd361c
2014-10-15 18:23:00 +09:00
c2429c54ac
Merge "Move entry updating method to language model dict content."
2014-10-15 04:51:04 +00:00
5400701908
Move entry updating method to language model dict content.
...
Bug: 14425059
Change-Id: I710055490d141539458cbf968adf5a7ccffd9552
2014-10-15 12:29:31 +09:00
d8ccb9093b
Quit using weightChildNode for ADDITIONAL_PROXIMITY and SUBSTITUTION.
...
[Category diff]
+1 0
-1 1
+2 0
-2 0
+3 0
-3 0
+4 1
-4 1
+5 8
-5 7
+6 0
-6 1
+7 1
-7 0
[Weighted category diff]
+1 0
-1 1
+2 0
-2 0
+3 0
-3 0
+4 1
-4 1
+5 8
-5 7
+6 0
-6 1
+7 1
-7 0
Bug: 13756409
Change-Id: I6ac3567545676bbefbee3e87dda54bc083c15fb6
2014-10-14 20:20:55 +09:00
d1471ee053
Merge "Remove shouldBlockAutoCorrectionBySafetyNet"
2014-10-14 10:52:32 +00:00
29b4f7aa67
Remove shouldBlockAutoCorrectionBySafetyNet
...
Bug: 13756409
[Category diff]
+1 27
-1 0
+2 0
-2 0
+3 0
-3 1
+4 11
-4 0
+5 51
-5 0
+6 0
-6 38
+7 0
-7 50
[Weighted category diff]
+1 28
-1 0
+2 0
-2 0
+3 0
-3 1
+4 11
-4 0
+5 51
-5 0
+6 0
-6 39
+7 0
-7 50
show diff for ./en_user_log_phones_2011_08.csv
+1 4
+4 5
+5 7
-6 9
-7 7
The increase of false positives comes from the spaceless
typing test cases that are synthetic data.
Change-Id: I4ea77aa56ebfaa5518c71107169e1d2332de6327
2014-10-14 11:20:33 +09:00
d70b8ff291
Fix: BoS bigram from user history dictionary is too strong.
...
They can be always stronger than BoS predictions from the
contextual dictionary.
Bug: 17961731
Change-Id: I70297d82436c10c790bdfad6f3dfefdb4bb2f852
2014-10-13 08:52:08 +00:00
7d5e1cb265
[ML23] Introduce a different accuracy/performance tradeoff
...
Bug: 11230254
Change-Id: Ic09518c818ae7b68942b1c63160dd462e5922cb5
2014-10-10 18:02:52 +09:00
229f354fdc
Merge "Make members of classes that are used with std::vector const"
2014-10-10 05:39:57 +00:00
b559c65e7e
Merge "Rename prev_words_info.h to ngram_context.h"
2014-10-09 12:47:24 +00:00
10fa30e380
Merge "Rename PrevWordsInfo to NgramContext."
2014-10-09 12:47:17 +00:00
620ebde704
Make members of classes that are used with std::vector const
...
Change-Id: Id93fb87f5630230fc3f9cd339e12f3b0e2006ea9
2014-10-09 21:28:40 +09:00
45783013bf
Rename prev_words_info.h to ngram_context.h
...
Bug: 14425059
Change-Id: I0e906631ecad2361a8198b3f9e3394bb22c5bf83
2014-10-09 21:28:19 +09:00
44efbe64b1
Fixing misspelled word
...
Change-Id: I51d77e271143d40256b39e5c60a3065d9fdf63fb
2014-10-09 19:26:54 +09:00
72e2383d11
Rename PrevWordsInfo to NgramContext.
...
Bug: 14425059
Change-Id: I30703fc80e9450d4e2dbfec965e7f9f4468f6a11
2014-10-09 17:34:32 +09:00
ab4437f468
Rename updateCounter to updateEntriesForWordWithNgramContext.
...
Bug: 14425059
Change-Id: Id9b0dd7e32c711ed4292981517c3febd5fe9e897
2014-10-09 17:34:29 +09:00
d7a51c242b
Extend jni method to dump ngram entries.
...
Bug: 14425059
Change-Id: Ib03b58525fae12f254b45630bee31ecbdde227b1
2014-10-09 15:26:10 +09:00
948ef10d03
Merge "Improve bigram probability computation for decaying dicts."
2014-10-06 13:06:29 +00:00
aae1a062eb
Improve bigram probability computation for decaying dicts.
...
Without personalization:
Total words: 1079345, Success Num: 819749, Success Percentage: 75.949%
Bad Failures, with auto-correction (typed word == expected word, output word != expected word): 1754, Bad Failure Percentage: 0.163%
Failures, with auto-correction (F-C): 28463, F-C Percentage: 2.637%
Max Keystrokes: 6074285, Min Keystrokes: 4649326, Keystroke Saving Percentage:23.459%
With current probability computing logic:
Total words: 1079382, Success Num: 838329, Success Percentage: 77.667%
Bad Failures, with auto-correction (typed word == expected word, output word != expected word): 1332, Bad Failure Percentage: 0.123%
Failures, with auto-correction (F-C): 28558, F-C Percentage: 2.646%
Max Keystrokes: 6074503, Min Keystrokes: 4474102, Keystroke Saving Percentage:26.346%
Remove isof files.
With new probability computing logic:
Total words: 1079356, Success Num: 844954, Success Percentage: 78.283%
Bad Failures, with auto-correction (typed word == expected word, output word != expected word): 1306, Bad Failure Percentage: 0.121%
Failures, with auto-correction (F-C): 27214, F-C Percentage: 2.521%
Max Keystrokes: 6074477, Min Keystrokes: 4243021, Keystroke Saving Percentage:30.150%
Remove isof files.
Bug: 16547409
Change-Id: I3d2a49c7aaa2c0f6835c52ef72d22466ee225789
2014-10-06 22:03:11 +09:00
552470c882
Merge "Make sure to suppress BoS prediction until input twice."
2014-10-06 10:46:31 +00:00
c7d199e770
Merge "Fix: BoS prediction is shown after inputting just once."
2014-10-06 10:38:18 +00:00
1c2f00f6b5
Make sure to suppress BoS prediction until input twice.
...
Change-Id: I98d91f264d5d1d3f5bcda1fd9ec885779ba2f746
2014-10-06 19:36:04 +09:00
ca17ed7d9d
Fix: BoS prediction is shown after inputting just once.
...
Change-Id: Ibba209f47cb5b1a4b08281689d607711b8dcfad4
2014-10-06 19:28:39 +09:00
36c4eaadfb
Show prediction results in debug build.
...
Bug: 16547409
Change-Id: If85418583998cd639c794bf5d5cfbbb972c34f72
2014-10-06 18:36:54 +09:00
16e1615301
Fix: ProbabilityDictContent can be wrongly large.
...
It can be twice as large as it should be (80KB larger).
Change-Id: If94f748f8c48a442b3c95ac989099aaed2aa2f86
2014-10-06 11:55:07 +09:00
3d7681d8e5
Merge "Remove -fno-inline from profiling build."
2014-10-03 09:35:14 +00:00
4ef27c0358
[ML13] Fix the locale passing in ProximityInfo
...
The locale is used to determine additional proximity characters. This
is dependent on the dictionary language, but was passed as a function
of the layout, which is wrong and would have given bad suggestions in
multi-lingual mode.
Ideally, additional proximity characters should be inserted in the
dictionary header, but for now it's a rather simple change to get
it from the dictionary's locale instead of the proximity info locale.
Also, that allows us to remove completely the locale parameter from
proximity info, which is a much needed change.
This change has zero effect on unit tests and on regression tests.
Bug: 11230254
Change-Id: If95157155db7dccd1f00b8ba55ccb3600283f9e4
2014-10-03 18:16:34 +09:00
3da610d5a6
Remove -fno-inline from profiling build.
...
Bug: 17797064
Change-Id: I0ab60e2da8c9f509de472afbc3ad32e41e69e73c
2014-10-03 18:16:19 +09:00
29777e3a8a
Implement updateCounter() by using existing entry adding methods.
...
Bug: 14425059
Change-Id: I0b6cb80e1fb8f738e9c7d9e80fbc0c479546b879
2014-10-01 19:59:39 +09:00
ce5fd94b97
Add jni method to update entries for a word efficiently.
...
Bug: 14425059
Change-Id: Ic628939ea68cdee17573409883a8403c73b78905
2014-10-01 17:40:20 +09:00
287e155e44
Move HistoricalInfo to property and use it in *Property.
...
Bug: 14425059
Change-Id: Icccccabad98fb543c6a6be2844cfc0086d80b739
2014-10-01 11:39:33 +09:00
79bb37d499
Rename BigramProperty to NgramProperty.
...
Remaining work is changing bigram to ngram for supporting
ngram entry counting, dumping, and migration.
Bug: 14425059
Change-Id: Ifba288a1166996d62a5e57698f63537ea0a2a8ee
2014-09-29 19:10:39 +09:00
cb4f544198
Quit reading unigram probability in Ver4PatriciaTrieNodeReader.
...
Bug: 14425059
Change-Id: I4fc7b0e236151a2c64e7131772264024c6597633
2014-09-25 11:41:50 +09:00
2842e50c4b
Use std::move for dictionary properties.
...
Change-Id: I15056b36b7493f4bac1dbcbb46a0b44343ede153
2014-09-25 11:36:52 +09:00
ed77b37e6b
Merge "Use CodePointArrayView in WordProperty."
2014-09-24 07:38:37 +00:00
7313b0debe
Merge "Refactoring method to get code points and code point count."
2014-09-24 07:24:48 +00:00
bd0ac97917
Fix: native unit test.
...
Bug: 14425059
Change-Id: I7c5103eac35e01a5732f1f4594d730df06dfca96
2014-09-24 15:26:26 +09:00
80d139a694
Use CodePointArrayView in WordProperty.
...
Change-Id: I45a9755c413003831788d190beb499fee8ce63aa
2014-09-24 14:15:36 +09:00
65a7ccfa00
Refactoring method to get code points and code point count.
...
Bug: 14425059
Change-Id: I4731bd6076d34556e46e6714180fed324fb6aba3
2014-09-24 14:15:36 +09:00
7d911d6f91
Move word flags to language model dict content.
...
Bug: 14425059
Change-Id: I64712e5c83d0bc241e6f0f16117ab47b5d75bd4b
2014-09-24 14:15:34 +09:00
ddfaeff544
Prepare supporting n-gram for user history dictionary.
...
Bug:17097992
Change-Id: Ic8bfde3d4cc0e720bf7681e08e16fb2ad94d5670
2014-09-22 18:18:50 +09:00
5a20827fc7
Fix native crash.
...
Change-Id: Ia8871db0248ddbc821605079e2a3e93859f4c9b7
2014-09-19 21:04:15 +09:00
5c6db929e4
Merge "Native side reads character table"
2014-09-19 09:58:40 +00:00
fb2bde5a68
Native side reads character table
...
Bug:17097992
Change-Id: Ibcfc67833a6754fe6a2d82a3e3023b33886b9ea2
2014-09-19 18:00:16 +09:00
6da9b21191
[ML8] Add a language weight
...
...and rename an improperly named normalization value
Bug: 11230254
Change-Id: I0f5633148a9f66dbfd7d28540b8a8985131c4549
2014-09-19 13:44:42 +09:00
9db2cb4a37
Make MAX_PREV_WORD_COUNT_FOR_N_GRAM 2.
...
Bug: 14425059
Change-Id: Iefacd7e4a6c256658a58856240310eeb91ae32db
2014-09-18 19:18:55 +09:00
8806248386
Pass prevWordCount to native code for getSuggestion.
...
Bug: 14425059
Change-Id: Id2b49a448982c997e5c0a22445b115e675156c3e
2014-09-18 19:18:55 +09:00
b349b57476
Merge "Use ReadOnlyByteArrayView in Ver2ParticiaTrieNodeReader."
2014-09-17 12:53:54 +00:00
a533e18c07
Merge "Use ReadOnlyByteArrayView in Ver2PtNodeArrayReader."
2014-09-17 12:53:48 +00:00
4a55d41410
Merge "Use ReadOnlyByteArrayView in ShortcutListPolicy"
2014-09-17 12:51:06 +00:00
c1040a255c
Merge "Use ReadOnlyByteArrayView in BigramListPolicy."
2014-09-17 12:50:51 +00:00
5849feeee1
Merge "Use ReadOnlyByteArrayView in PatriciaTriePolicy."
2014-09-17 12:50:16 +00:00
198a47a161
Merge "Add firstOrDefault and lastOrDefault to IntArrayView."
2014-09-17 12:43:10 +00:00
6258c57c32
Use ReadOnlyByteArrayView in Ver2ParticiaTrieNodeReader.
...
Change-Id: Ie3325bb9666c7ff6d519a1b72ed2fe9fc4f140a9
2014-09-17 21:42:30 +09:00
a892628fa3
Use ReadOnlyByteArrayView in Ver2PtNodeArrayReader.
...
Change-Id: Ia1065465abdf7c64fadb3ff1efd2a4d2df343264
2014-09-17 21:42:30 +09:00
59ebd51718
Use ReadOnlyByteArrayView in ShortcutListPolicy
...
Change-Id: I03a6f49c9005306bcc5fce8b7e4d37b8d30b9faa
2014-09-17 21:42:29 +09:00
d01eb3c94c
Use ReadOnlyByteArrayView in BigramListPolicy.
...
Change-Id: I0f8e8044c8803a7a542f97be5b8f08fdb5ae3926
2014-09-17 21:42:26 +09:00
180e7b4c07
Use ReadOnlyByteArrayView in PatriciaTriePolicy.
...
Change-Id: I4ef01c0af73f27c256cc1a719343be836913dc71
2014-09-17 21:38:46 +09:00
09c154925f
Add firstOrDefault and lastOrDefault to IntArrayView.
...
Change-Id: I854c02eff3fa0b53c72a5f1cabce001f4854ada0
2014-09-17 21:16:31 +09:00
fc7d0540fe
Use CodePointArrayView in DictionaryUtils.
...
Change-Id: I9ae308e60124ea5acb4ee09847c4fdd58ff168e2
2014-09-17 20:13:36 +09:00
Adrian Velicu
Keisuke Kuroyanagi
Yohei Yukawa
Jean Chalard
Akifumi Yoshimoto