Commit Graph

1009 Commits (d3b93cc95061ada6a162d7989e313f818e6d4e15)

Author SHA1 Message Date
Keisuke Kuroyanagi 88bc312ad3 Move dictionary code to top level dictionary dir.
Bug: 18725954
Change-Id: Ia442ba4b5d84311057d83edf6e7aeb151d6a820b
2014-12-17 16:02:09 +09:00
Keisuke Kuroyanagi 0bb038a19d Remove needless include.
Bug: 18725954
Change-Id: I3c823fda1b7daf41d82b118d9495f5f2356a1a5e
2014-12-15 18:54:42 +09:00
Keisuke Kuroyanagi 5e1b225082 Add missing error check in TrieMap.
Bug: 18725954
Change-Id: I8fcb0d15dda3f823a4575fe707bcdec57ff5e89b
2014-12-15 18:53:22 +09:00
Keisuke Kuroyanagi ad546afbaa Remove dependency on jni.h from WordProperty.
Bug: 18725954
Change-Id: Ic97d3a56b036ff042322c9f794504064046fd7d7
2014-12-15 18:09:45 +09:00
Keisuke Kuroyanagi 52e92f812b Merge "Quit auto-correct explicit accented letters to base letters." 2014-12-09 10:24:56 +00:00
Keisuke Kuroyanagi 515c508135 Quit auto-correct explicit accented letters to base letters.
Bug: 7677193
Change-Id: I66eddbf27a9db8682c0347a1be19922792a3bea7
2014-12-09 19:23:27 +09:00
Keisuke Kuroyanagi 60021bbdc2 Enable Quadgram for personalized dicts.
Before:
Total words: 1134659, Success Num: 944709, Success Percentage: 83.259%
Bad Failures, with auto-correction (typed word == expected word, output word != expected word): 1258, Bad Failure Percentage: 0.111%
Failures, with auto-correction (F-C): 28013, F-C Percentage: 2.469%
Max Keystrokes: 6072844, Min Keystrokes: 3347332, Keystroke Saving Percentage:44.880%

After:
Total words: 1134665, Success Num: 945026, Success Percentage: 83.287%
Bad Failures, with auto-correction (typed word == expected word, output word != expected word): 1271, Bad Failure Percentage: 0.112%
Failures, with auto-correction (F-C): 27756, F-C Percentage: 2.446%
Max Keystrokes: 6072850, Min Keystrokes: 3290996, Keystroke Saving Percentage:45.808%

Change-Id: I16af52a3e9c371b95fd6f0741f45ee6b2443bea6
2014-11-25 19:07:13 +09:00
Keisuke Kuroyanagi 78212a6d3d Use enum to specify ngram type.
Change-Id: Ie28768ceadcd7a2d940c57eb30be7d4c364e509f
2014-11-25 19:07:10 +09:00
Jean Chalard 2a3ed8c988 Fix the base character of D with stroke
Bug: 18436480
Change-Id: Ic92cae7c85c07c8f62a5b6e69d2b71e204aff50d
2014-11-19 17:26:00 +09:00
Keisuke Kuroyanagi 580420d21b Implement IntArrayView::split for dicttoolkit.
Bug: 10059681
Change-Id: Ic29e79d049bb532727cf5cb1e529fec5d35156ed
2014-11-11 15:06:48 +09:00
Keisuke Kuroyanagi b23f03488f Merge "Use reference instead of pointer for WordProperty()." 2014-11-10 18:32:24 +00:00
Keisuke Kuroyanagi 7d5420aa5e Make profiler use getTimeInMicroSec().
Bug: 17797064
Change-Id: Ie992c9454edfc3bf93d5ea367c3a4427b513a205
2014-11-11 01:38:49 +09:00
Keisuke Kuroyanagi bbf0d4141b Use reference instead of pointer for WordProperty().
Change-Id: Idf03e97661d64186c752e35964d641a5528be5b1
2014-11-10 09:15:11 +09:00
Keisuke Kuroyanagi 2cf5550749 Fix: BoS prediction after inputting just once.
Change-Id: Ib69569ab6b6edfcc8c1d2c621b95de4127789ab6
2014-11-01 17:58:22 +09:00
Keisuke Kuroyanagi b3bae2e89b Merge "Update v4 format version from 402 to 403." 2014-10-31 14:19:44 +00:00
Keisuke Kuroyanagi ef931546a0 Merge "Add hacks for better handling count value during migration." 2014-10-31 13:53:57 +00:00
Keisuke Kuroyanagi a88c9682fc Merge "Change v403 historical info format." 2014-10-31 13:38:38 +00:00
Keisuke Kuroyanagi ea468cc9de Update v4 format version from 402 to 403.
Without personalization:
Total words: 1134774, Success Num: 899230, Success Percentage: 79.243%
Bad Failures, with auto-correction (typed word == expected word, output word != expected word): 1871, Bad Failure Percentage: 0.165%
Failures, with auto-correction (F-C): 29084, F-C Percentage: 2.563%
Max Keystrokes: 6072959, Min Keystrokes: 4436090, Keystroke Saving Percentage:26.953%

Before:
Total words: 1134646, Success Num: 925194, Success Percentage: 81.540%
Bad Failures, with auto-correction (typed word == expected word, output word != expected word): 1316, Bad Failure Percentage: 0.116%
Failures, with auto-correction (F-C): 28288, F-C Percentage: 2.493%
Max Keystrokes: 6072831, Min Keystrokes: 3946188, Keystroke Saving Percentage:35.019%

After
Total words: 1134659, Success Num: 944746, Success Percentage: 83.263%
Bad Failures, with auto-correction (typed word == expected word, output word != expected word): 1258, Bad Failure Percentage: 0.111%
Failures, with auto-correction (F-C): 28016, F-C Percentage: 2.469%
Max Keystrokes: 6072844, Min Keystrokes: 3387333, Keystroke Saving Percentage:44.222%

Change-Id: I3af42ec37a11847c0429c28616e726f6a339247f
2014-10-31 17:23:39 +09:00
Keisuke Kuroyanagi c611989929 Add hacks for better handling count value during migration.
Bug: 14425059
Change-Id: Ib050574aa7c4babd4285322a11c3af9be9fbab1e
2014-10-31 17:22:13 +09:00
Keisuke Kuroyanagi 2383575d2d Change v403 historical info format.
count -> 2B, level -> 0B.

Change-Id: I3b241126f56eb33cdf09cb1ebfed04f534e4ec48
2014-10-31 17:22:13 +09:00
Adrian Velicu 009e02ce4a Further fixes to treat 0-frequency words
Previously, when both legitimate 0-frequency words (such as
distracters) and offensive words were encoded in the same
way, distracters would never show up when the user blocked
offensive words (the default setting, as well as the setting
for regression tests).

When b/11031090 was fixed and a separate encoding was used
for offensive words, 0-frequency words would no longer be
blocked when they were an "exact match" (where case
mismatches and accent mismatches would be considered an
"exact match"). The exact match boosting functionality meant
that, for example, when the user typed "mt" they would be
suggested the word "Mt", although they most probably meant
to type "my".

For this reason, we introduced this change, which does the
following:
* Defines the "perfect match" as a really exact match, with
no room for case or accent mismatches
* When the target word has probability zero (as "Mt" does,
because it is a distracter), ONLY boost its score if it is a
perfect match.

By doing this, when the user types "mt", the word "Mt" will
NOT be boosted, and they will get "my". However, if the user
makes an explicit effort to type "Mt", we do boost the word
"Mt" so that the user's input is not autocorrected to "My".

Bug: 11031090
Change-Id: I92ee1b4e742645d52e2f7f8c4390920481e8fff0
2014-10-31 15:58:50 +09:00
Adrian Velicu 10416241f7 Block offensive words in multi-word suggestions
If the user has chosen to block offensive words and types
"aaaxbb", where "aaa" is an offensive word and "bb" is not,
we should not suggest "aaa bb".

Bug: 11031090
Change-Id: Ie23b8dd5d347bc26b1c046c3f5e8dfbc259bf528
2014-10-31 15:58:50 +09:00
Adrian Velicu 7c87859d4c Using "blacklist" flag as "possibly offensive"
Bug: 11031090
Change-Id: I5cc0d006ab003656498eb82b0875eb9c051d331e
2014-10-31 14:33:05 +09:00
Keisuke Kuroyanagi bcb52d73e2 Enable count based dynamic ngram language model for v403.
Bug: 14425059

Change-Id: Icc15e14cfd77d37cd75f75318fd0fa36f9ca7a5b
2014-10-30 23:38:19 +09:00
Keisuke Kuroyanagi 660b00477c Add DynamicLanguageModelProbabilityUtils.
Bug: 14425059
Change-Id: Ia58ab3f0ead02798046d182a9464dcbd95f086bc
2014-10-30 21:33:57 +09:00
Keisuke Kuroyanagi 0a9c3f30b6 Add method to encode probability.
Bug: 14425059
Change-Id: I3e5d359ba5fa38f1669f0e98dfae792ff53efbf8
2014-10-30 12:42:35 +09:00
Keisuke Kuroyanagi c2ba0ce411 Fix: TRT and ime-simulator bulid.
Change-Id: I1697a907562d1ed6aff2b001763d1594263ba0d3
2014-10-30 01:01:40 +09:00
Keisuke Kuroyanagi afe67611c3 Merge "Add a class to have global counters for LanguageModelDictContent." 2014-10-29 12:18:12 +00:00
Keisuke Kuroyanagi 6b0561f9d2 Add a class to have global counters for LanguageModelDictContent.
Bug: 14425059
Change-Id: I08ec19903432356b6028853fd73b4eefce20218e
2014-10-29 21:05:41 +09:00
Keisuke Kuroyanagi dabc12974c Merge "Improve space substitution error correction." 2014-10-28 09:26:40 +00:00
Keisuke Kuroyanagi 8a809f3433 Improve space substitution error correction.
Bug: 17432052

[Category diff]
+1     262
-1      93
+2       2
-2      18
+3      18
-3       2
+4     111
-4     148
+5     295
-5     217
+6      51
-6     276
+7     139
-7     124

[Weighted category diff]
+1     276
-1     100
+2       4
-2      20
+3      20
-3       4
+4     118
-4     160
+5     309
-5     225
+6      52
-6     298
+7     163
-7     135

show diff for ./en_user_log_phones_2011_08.csv
+1     173
-1      28
+2       2
-2      17
+3      17
-3       2
+4      63
-4      82
+5     120
-5      51
+6      24
-6     220
+7      88
-7      87

Change-Id: I9d673acb0ff632828ae2e0ead56e76e3a20411c6
2014-10-28 17:11:14 +09:00
Keisuke Kuroyanagi 3844f74aff Fix: deleted PtNode handling in v403.
If a word is once deleted, the word never gets into the
personalized dictionaries due to this bug.

Change-Id: Ife4e3fe1ba0615b4135e6291d2151b0db7d3f940
2014-10-27 15:32:05 +09:00
Keisuke Kuroyanagi 090c3819d7 Fix: Personalized dicts suggest invalid words with v403.
Bug: 14425059
Change-Id: I45ae00069dd3b7c461dd9a1f3558b96af0a1c975
2014-10-23 19:26:01 +09:00
Keisuke Kuroyanagi 16cc3992d7 Use trigrams for personalization dict.
5Bug: 14425059
Change-Id: I73cf6904e569d60996a3b079f16ea6df0cb90f02
2014-10-23 14:32:45 +09:00
Keisuke Kuroyanagi b5ef884fbb Support dumping ngram entries.
Bug: 14425059
Change-Id: Ib03a0c3d166ed6f1e60c67127b28006d55143b6b
2014-10-22 18:15:53 +09:00
Keisuke Kuroyanagi c9865785f4 Support ngram entry migration.
Bug: 14425059
Change-Id: I98cb9fa303af2d93a0a3512e8732231c564e3c5d
2014-10-22 11:31:16 +09:00
Keisuke Kuroyanagi 0b8bb0c21b Fix debug build.
Change-Id: Id94636714d04a8828718b87741c0ee62a14cb3b4
2014-10-21 20:20:11 +09:00
Keisuke Kuroyanagi dfc82fa366 Merge changes I210acb81,Ie9508788
* changes:
  Make NgramProperty have NgramContext.
  Create .cpp file for NgramContext.
2014-10-21 10:28:25 +00:00
Keisuke Kuroyanagi 88bb28c132 Make NgramProperty have NgramContext.
Bug: 14425059
Change-Id: I210acb816b122857dbbe1ee4dd6a35c5335bf2bf
2014-10-21 17:12:32 +09:00
Keisuke Kuroyanagi f87bb77a91 Create .cpp file for NgramContext.
Bug: 14425059

Change-Id: Ie950878817b9c80cc9c970e1a84880c9b9ab228a
2014-10-21 17:04:56 +09:00
Keisuke Kuroyanagi fa1e65cb3a Merge "Use EntryCounters during GC." 2014-10-21 07:55:04 +00:00
Adrian Velicu c51b9b5b3f Merge "Renaming "blacklist" flag to "possibly offensive"" 2014-10-21 07:39:18 +00:00
Keisuke Kuroyanagi 47fc656cd7 Use EntryCounters during GC.
Bug: 14425059
Change-Id: I61eb798686dc753fb6c0fe99a0719c1732198f30
2014-10-21 16:36:03 +09:00
Keisuke Kuroyanagi e8750d970e Introduce EntryCounters to count entries in a dictionary.
Bug: 14425059

Change-Id: Ic13ba827d96fa4a147485ba92fdb37e23e04e8e8
2014-10-21 15:46:14 +09:00
Adrian Velicu 05172bf1a5 Renaming "blacklist" flag to "possibly offensive"
No behaviour changes.
Unified the overloaded FusionDictionary::add method to always take an
isPossiblyOffensive argument.

Bug: 11031090
Change-Id: I5741a023ca1ce842d2cf10d4f6c926b0efabaa78
2014-10-21 11:51:47 +09:00
Keisuke Kuroyanagi 1085fef8d0 Change entry count limit.
Unigram 10K, Bigram 30K, Trigram 30K.

Change-Id: Ibd19c6a2b618499df1c70000bad7b47498187f0a
2014-10-20 15:01:49 +09:00
Keisuke Kuroyanagi f4928ad4dd Merge "Update useless n-gram entry detection logic during GC." 2014-10-15 21:44:45 +00:00
Keisuke Kuroyanagi 3601c214f8 Update useless n-gram entry detection logic during GC.
Bug: 14425059
Change-Id: Ib939deae5b60167751dee07965bb1ef1a43c4625
2014-10-15 20:43:27 +09:00
Keisuke Kuroyanagi 183e21c36c Merge "Use better conditional probability for ngram entries." 2014-10-15 09:27:21 +00:00
Keisuke Kuroyanagi 72d17d9209 Use better conditional probability for ngram entries.
Old:
P(W | W_prev) = f(W, W_prev) + C
New:
P(W | W_prev) = f(W, W_prev) / f(W_prev)

Bug: 14425059
Bug: 16547409

Change-Id: I4d13be6de2c6bad6bad7fb22320a23ba4ecd361c
2014-10-15 18:23:00 +09:00