nyancrimew
/
LatinIME
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
42,008 Commits
1 Branch
0 Tags
363 MiB
Commit Graph

1426 Commits (main)

Author SHA1 Message Date
Keisuke Kuroyanagi e8750d970e Introduce EntryCounters to count entries in a dictionary. 
Bug: 14425059

Change-Id: Ic13ba827d96fa4a147485ba92fdb37e23e04e8e8
 2014-10-21 15:46:14 +09:00
Adrian Velicu 05172bf1a5 Renaming "blacklist" flag to "possibly offensive" 
No behaviour changes.
Unified the overloaded FusionDictionary::add method to always take an
isPossiblyOffensive argument.

Bug: 11031090
Change-Id: I5741a023ca1ce842d2cf10d4f6c926b0efabaa78
 2014-10-21 11:51:47 +09:00
Keisuke Kuroyanagi 1085fef8d0 Change entry count limit. 
Unigram 10K, Bigram 30K, Trigram 30K.

Change-Id: Ibd19c6a2b618499df1c70000bad7b47498187f0a
 2014-10-20 15:01:49 +09:00
Keisuke Kuroyanagi f4928ad4dd Merge "Update useless n-gram entry detection logic during GC." 2014-10-15 21:44:45 +00:00
Keisuke Kuroyanagi 3601c214f8 Update useless n-gram entry detection logic during GC. 
Bug: 14425059
Change-Id: Ib939deae5b60167751dee07965bb1ef1a43c4625
 2014-10-15 20:43:27 +09:00
Keisuke Kuroyanagi 183e21c36c Merge "Use better conditional probability for ngram entries." 2014-10-15 09:27:21 +00:00
Keisuke Kuroyanagi 72d17d9209 Use better conditional probability for ngram entries. 
Old:
P(W | W_prev) = f(W, W_prev) + C
New:
P(W | W_prev) = f(W, W_prev) / f(W_prev)

Bug: 14425059
Bug: 16547409

Change-Id: I4d13be6de2c6bad6bad7fb22320a23ba4ecd361c
 2014-10-15 18:23:00 +09:00
Keisuke Kuroyanagi c2429c54ac Merge "Move entry updating method to language model dict content." 2014-10-15 04:51:04 +00:00
Keisuke Kuroyanagi 5400701908 Move entry updating method to language model dict content. 
Bug: 14425059
Change-Id: I710055490d141539458cbf968adf5a7ccffd9552
 2014-10-15 12:29:31 +09:00
Keisuke Kuroyanagi d8ccb9093b Quit using weightChildNode for ADDITIONAL_PROXIMITY and SUBSTITUTION. 
[Category diff]
+1       0
-1       1
+2       0
-2       0
+3       0
-3       0
+4       1
-4       1
+5       8
-5       7
+6       0
-6       1
+7       1
-7       0

[Weighted category diff]
+1       0
-1       1
+2       0
-2       0
+3       0
-3       0
+4       1
-4       1
+5       8
-5       7
+6       0
-6       1
+7       1
-7       0

Bug: 13756409
Change-Id: I6ac3567545676bbefbee3e87dda54bc083c15fb6
 2014-10-14 20:20:55 +09:00
Keisuke Kuroyanagi d1471ee053 Merge "Remove shouldBlockAutoCorrectionBySafetyNet" 2014-10-14 10:52:32 +00:00
Keisuke Kuroyanagi 29b4f7aa67 Remove shouldBlockAutoCorrectionBySafetyNet 
Bug: 13756409

[Category diff]
+1      27
-1       0
+2       0
-2       0
+3       0
-3       1
+4      11
-4       0
+5      51
-5       0
+6       0
-6      38
+7       0
-7      50

[Weighted category diff]
+1      28
-1       0
+2       0
-2       0
+3       0
-3       1
+4      11
-4       0
+5      51
-5       0
+6       0
-6      39
+7       0
-7      50

show diff for ./en_user_log_phones_2011_08.csv
+1       4
+4       5
+5       7
-6       9
-7       7

The increase of false positives comes from the spaceless
typing test cases that are synthetic data.

Change-Id: I4ea77aa56ebfaa5518c71107169e1d2332de6327
 2014-10-14 11:20:33 +09:00
Keisuke Kuroyanagi d70b8ff291 Fix: BoS bigram from user history dictionary is too strong. 
They can be always stronger than BoS predictions from the
contextual dictionary.

Bug: 17961731
Change-Id: I70297d82436c10c790bdfad6f3dfefdb4bb2f852
 2014-10-13 08:52:08 +00:00
Jean Chalard 7d5e1cb265 [ML23] Introduce a different accuracy/performance tradeoff 
Bug: 11230254
Change-Id: Ic09518c818ae7b68942b1c63160dd462e5922cb5
 2014-10-10 18:02:52 +09:00
Keisuke Kuroyanagi 229f354fdc Merge "Make members of classes that are used with std::vector const" 2014-10-10 05:39:57 +00:00
Keisuke Kuroyanagi b559c65e7e Merge "Rename prev_words_info.h to ngram_context.h" 2014-10-09 12:47:24 +00:00
Keisuke Kuroyanagi 10fa30e380 Merge "Rename PrevWordsInfo to NgramContext." 2014-10-09 12:47:17 +00:00
Keisuke Kuroyanagi 620ebde704 Make members of classes that are used with std::vector const 
Change-Id: Id93fb87f5630230fc3f9cd339e12f3b0e2006ea9
 2014-10-09 21:28:40 +09:00
Keisuke Kuroyanagi 45783013bf Rename prev_words_info.h to ngram_context.h 
Bug: 14425059
Change-Id: I0e906631ecad2361a8198b3f9e3394bb22c5bf83
 2014-10-09 21:28:19 +09:00
Adrian Velicu 44efbe64b1 Fixing misspelled word 
Change-Id: I51d77e271143d40256b39e5c60a3065d9fdf63fb
 2014-10-09 19:26:54 +09:00
Keisuke Kuroyanagi 72e2383d11 Rename PrevWordsInfo to NgramContext. 
Bug: 14425059
Change-Id: I30703fc80e9450d4e2dbfec965e7f9f4468f6a11
 2014-10-09 17:34:32 +09:00
Keisuke Kuroyanagi ab4437f468 Rename updateCounter to updateEntriesForWordWithNgramContext. 
Bug: 14425059
Change-Id: Id9b0dd7e32c711ed4292981517c3febd5fe9e897
 2014-10-09 17:34:29 +09:00
Keisuke Kuroyanagi d7a51c242b Extend jni method to dump ngram entries. 
Bug: 14425059
Change-Id: Ib03b58525fae12f254b45630bee31ecbdde227b1
 2014-10-09 15:26:10 +09:00
Keisuke Kuroyanagi 948ef10d03 Merge "Improve bigram probability computation for decaying dicts." 2014-10-06 13:06:29 +00:00
Keisuke Kuroyanagi aae1a062eb Improve bigram probability computation for decaying dicts. 
Without personalization:
Total words: 1079345, Success Num: 819749, Success Percentage: 75.949%
Bad Failures, with auto-correction (typed word == expected word, output word != expected word): 1754, Bad Failure Percentage: 0.163%
Failures, with auto-correction (F-C): 28463, F-C Percentage: 2.637%
Max Keystrokes: 6074285, Min Keystrokes: 4649326, Keystroke Saving Percentage:23.459%

With current probability computing logic:
Total words: 1079382, Success Num: 838329, Success Percentage: 77.667%
Bad Failures, with auto-correction (typed word == expected word, output word != expected word): 1332, Bad Failure Percentage: 0.123%
Failures, with auto-correction (F-C): 28558, F-C Percentage: 2.646%
Max Keystrokes: 6074503, Min Keystrokes: 4474102, Keystroke Saving Percentage:26.346%
Remove isof files.

With new probability computing logic:
Total words: 1079356, Success Num: 844954, Success Percentage: 78.283%
Bad Failures, with auto-correction (typed word == expected word, output word != expected word): 1306, Bad Failure Percentage: 0.121%
Failures, with auto-correction (F-C): 27214, F-C Percentage: 2.521%
Max Keystrokes: 6074477, Min Keystrokes: 4243021, Keystroke Saving Percentage:30.150%
Remove isof files.

Bug: 16547409
Change-Id: I3d2a49c7aaa2c0f6835c52ef72d22466ee225789
 2014-10-06 22:03:11 +09:00
Keisuke Kuroyanagi 552470c882 Merge "Make sure to suppress BoS prediction until input twice." 2014-10-06 10:46:31 +00:00
Keisuke Kuroyanagi c7d199e770 Merge "Fix: BoS prediction is shown after inputting just once." 2014-10-06 10:38:18 +00:00
Keisuke Kuroyanagi 1c2f00f6b5 Make sure to suppress BoS prediction until input twice. 
Change-Id: I98d91f264d5d1d3f5bcda1fd9ec885779ba2f746
 2014-10-06 19:36:04 +09:00
Keisuke Kuroyanagi ca17ed7d9d Fix: BoS prediction is shown after inputting just once. 
Change-Id: Ibba209f47cb5b1a4b08281689d607711b8dcfad4
 2014-10-06 19:28:39 +09:00
Keisuke Kuroyanagi 36c4eaadfb Show prediction results in debug build. 
Bug: 16547409
Change-Id: If85418583998cd639c794bf5d5cfbbb972c34f72
 2014-10-06 18:36:54 +09:00
Keisuke Kuroyanagi 16e1615301 Fix: ProbabilityDictContent can be wrongly large. 
It can be twice as large as it should be (80KB larger).

Change-Id: If94f748f8c48a442b3c95ac989099aaed2aa2f86
 2014-10-06 11:55:07 +09:00
Keisuke Kuroyanagi 3d7681d8e5 Merge "Remove -fno-inline from profiling build." 2014-10-03 09:35:14 +00:00
Jean Chalard 4ef27c0358 [ML13] Fix the locale passing in ProximityInfo 
The locale is used to determine additional proximity characters. This
is dependent on the dictionary language, but was passed as a function
of the layout, which is wrong and would have given bad suggestions in
multi-lingual mode.

Ideally, additional proximity characters should be inserted in the
dictionary header, but for now it's a rather simple change to get
it from the dictionary's locale instead of the proximity info locale.

Also, that allows us to remove completely the locale parameter from
proximity info, which is a much needed change.

This change has zero effect on unit tests and on regression tests.

Bug: 11230254
Change-Id: If95157155db7dccd1f00b8ba55ccb3600283f9e4
 2014-10-03 18:16:34 +09:00
Keisuke Kuroyanagi 3da610d5a6 Remove -fno-inline from profiling build. 
Bug: 17797064
Change-Id: I0ab60e2da8c9f509de472afbc3ad32e41e69e73c
 2014-10-03 18:16:19 +09:00
Keisuke Kuroyanagi 29777e3a8a Implement updateCounter() by using existing entry adding methods. 
Bug: 14425059
Change-Id: I0b6cb80e1fb8f738e9c7d9e80fbc0c479546b879
 2014-10-01 19:59:39 +09:00
Keisuke Kuroyanagi ce5fd94b97 Add jni method to update entries for a word efficiently. 
Bug: 14425059
Change-Id: Ic628939ea68cdee17573409883a8403c73b78905
 2014-10-01 17:40:20 +09:00
Keisuke Kuroyanagi 287e155e44 Move HistoricalInfo to property and use it in *Property. 
Bug: 14425059
Change-Id: Icccccabad98fb543c6a6be2844cfc0086d80b739
 2014-10-01 11:39:33 +09:00
Keisuke Kuroyanagi 79bb37d499 Rename BigramProperty to NgramProperty. 
Remaining work is changing bigram to ngram for supporting
ngram entry counting, dumping, and migration.

Bug: 14425059
Change-Id: Ifba288a1166996d62a5e57698f63537ea0a2a8ee
 2014-09-29 19:10:39 +09:00
Keisuke Kuroyanagi cb4f544198 Quit reading unigram probability in Ver4PatriciaTrieNodeReader. 
Bug: 14425059
Change-Id: I4fc7b0e236151a2c64e7131772264024c6597633
 2014-09-25 11:41:50 +09:00
Keisuke Kuroyanagi 2842e50c4b Use std::move for dictionary properties. 
Change-Id: I15056b36b7493f4bac1dbcbb46a0b44343ede153
 2014-09-25 11:36:52 +09:00
Keisuke Kuroyanagi ed77b37e6b Merge "Use CodePointArrayView in WordProperty." 2014-09-24 07:38:37 +00:00
Keisuke Kuroyanagi 7313b0debe Merge "Refactoring method to get code points and code point count." 2014-09-24 07:24:48 +00:00
Keisuke Kuroyanagi bd0ac97917 Fix: native unit test. 
Bug: 14425059
Change-Id: I7c5103eac35e01a5732f1f4594d730df06dfca96
 2014-09-24 15:26:26 +09:00
Keisuke Kuroyanagi 80d139a694 Use CodePointArrayView in WordProperty. 
Change-Id: I45a9755c413003831788d190beb499fee8ce63aa
 2014-09-24 14:15:36 +09:00
Keisuke Kuroyanagi 65a7ccfa00 Refactoring method to get code points and code point count. 
Bug: 14425059
Change-Id: I4731bd6076d34556e46e6714180fed324fb6aba3
 2014-09-24 14:15:36 +09:00
Keisuke Kuroyanagi 7d911d6f91 Move word flags to language model dict content. 
Bug: 14425059
Change-Id: I64712e5c83d0bc241e6f0f16117ab47b5d75bd4b
 2014-09-24 14:15:34 +09:00
Keisuke Kuroyanagi ddfaeff544 Prepare supporting n-gram for user history dictionary. 
Bug:17097992
Change-Id: Ic8bfde3d4cc0e720bf7681e08e16fb2ad94d5670
 2014-09-22 18:18:50 +09:00
Keisuke Kuroyanagi 5a20827fc7 Fix native crash. 
Change-Id: Ia8871db0248ddbc821605079e2a3e93859f4c9b7
 2014-09-19 21:04:15 +09:00
Akifumi Yoshimoto 5c6db929e4 Merge "Native side reads character table" 2014-09-19 09:58:40 +00:00
Akifumi Yoshimoto fb2bde5a68 Native side reads character table 
Bug:17097992
Change-Id: Ibcfc67833a6754fe6a2d82a3e3023b33886b9ea2
 2014-09-19 18:00:16 +09:00
Jean Chalard 6da9b21191 [ML8] Add a language weight 
...and rename an improperly named normalization value

Bug: 11230254
Change-Id: I0f5633148a9f66dbfd7d28540b8a8985131c4549
 2014-09-19 13:44:42 +09:00
Keisuke Kuroyanagi 9db2cb4a37 Make MAX_PREV_WORD_COUNT_FOR_N_GRAM 2. 
Bug: 14425059
Change-Id: Iefacd7e4a6c256658a58856240310eeb91ae32db
 2014-09-18 19:18:55 +09:00
Keisuke Kuroyanagi 8806248386 Pass prevWordCount to native code for getSuggestion. 
Bug: 14425059
Change-Id: Id2b49a448982c997e5c0a22445b115e675156c3e
 2014-09-18 19:18:55 +09:00
Keisuke Kuroyanagi b349b57476 Merge "Use ReadOnlyByteArrayView in Ver2ParticiaTrieNodeReader." 2014-09-17 12:53:54 +00:00
Keisuke Kuroyanagi a533e18c07 Merge "Use ReadOnlyByteArrayView in Ver2PtNodeArrayReader." 2014-09-17 12:53:48 +00:00
Keisuke Kuroyanagi 4a55d41410 Merge "Use ReadOnlyByteArrayView in ShortcutListPolicy" 2014-09-17 12:51:06 +00:00
Keisuke Kuroyanagi c1040a255c Merge "Use ReadOnlyByteArrayView in BigramListPolicy." 2014-09-17 12:50:51 +00:00
Keisuke Kuroyanagi 5849feeee1 Merge "Use ReadOnlyByteArrayView in PatriciaTriePolicy." 2014-09-17 12:50:16 +00:00
Keisuke Kuroyanagi 198a47a161 Merge "Add firstOrDefault and lastOrDefault to IntArrayView." 2014-09-17 12:43:10 +00:00
Keisuke Kuroyanagi 6258c57c32 Use ReadOnlyByteArrayView in Ver2ParticiaTrieNodeReader. 
Change-Id: Ie3325bb9666c7ff6d519a1b72ed2fe9fc4f140a9
 2014-09-17 21:42:30 +09:00
Keisuke Kuroyanagi a892628fa3 Use ReadOnlyByteArrayView in Ver2PtNodeArrayReader. 
Change-Id: Ia1065465abdf7c64fadb3ff1efd2a4d2df343264
 2014-09-17 21:42:30 +09:00
Keisuke Kuroyanagi 59ebd51718 Use ReadOnlyByteArrayView in ShortcutListPolicy 
Change-Id: I03a6f49c9005306bcc5fce8b7e4d37b8d30b9faa
 2014-09-17 21:42:29 +09:00
Keisuke Kuroyanagi d01eb3c94c Use ReadOnlyByteArrayView in BigramListPolicy. 
Change-Id: I0f8e8044c8803a7a542f97be5b8f08fdb5ae3926
 2014-09-17 21:42:26 +09:00
Keisuke Kuroyanagi 180e7b4c07 Use ReadOnlyByteArrayView in PatriciaTriePolicy. 
Change-Id: I4ef01c0af73f27c256cc1a719343be836913dc71
 2014-09-17 21:38:46 +09:00
Keisuke Kuroyanagi 09c154925f Add firstOrDefault and lastOrDefault to IntArrayView. 
Change-Id: I854c02eff3fa0b53c72a5f1cabce001f4854ada0
 2014-09-17 21:16:31 +09:00
Keisuke Kuroyanagi fc7d0540fe Use CodePointArrayView in DictionaryUtils. 
Change-Id: I9ae308e60124ea5acb4ee09847c4fdd58ff168e2
 2014-09-17 20:13:36 +09:00
Keisuke Kuroyanagi 3e75c59133 Use CodePointArrayView in Dictionary. 
Change-Id: I63fa0a8348f6de6ec7a424a8033e936b4af72beb
 2014-09-17 20:13:36 +09:00
Keisuke Kuroyanagi 89a074fade Use IntArrayView in DynamicPtUpdatingHelper. 
Change-Id: Ifa185eba7dd9abee1b35e49fe559be1042ca63d7
 2014-09-17 20:13:36 +09:00
Keisuke Kuroyanagi 4926b90ec5 Support n-gram for look-up. 
Bug: 14425059
Change-Id: I19523c29fb802cd65158c7540d1608e7f55c4ca7
 2014-09-17 16:20:00 +09:00
Keisuke Kuroyanagi 5520e84e16 Support removing n-gram entry in structure policy. 
Bug: 14425059
Change-Id: Ifd2ac97a15dcbb936e2f068866a9d1ee68759ba2
 2014-09-16 15:46:28 +09:00
Keisuke Kuroyanagi 0c0b8207cd Support adding n-gram entry in structure policy. 
Bug: 14425059
Change-Id: Ia207d3c4735b1d6d43f18d18a70d28af613cb458
 2014-09-16 15:37:09 +09:00
Keisuke Kuroyanagi 7eb6e28b9e Update input checking logic for adding/removing n-gram. 
Bug: 14425059
Change-Id: Ifb4e3c3f06aaf9a16f632ff208ba76cf741cea83
 2014-09-16 14:14:22 +09:00
Keisuke Kuroyanagi d2230525bc Have mPrevWordCount in DicNodeProperties. 
Bug: 14425059
Change-Id: I5ce22bace4ec08d0da4e5c167288a742c4426c33
 2014-09-16 12:46:16 +09:00
Keisuke Kuroyanagi 36ba139ca6 Support decaying dict in getWordProbability(). 
Bug: 14425059
Change-Id: I24db3f9131c2999fc388035dc365c7faaef3bdb1
 2014-09-14 17:29:50 +09:00
Keisuke Kuroyanagi c43b6664fa Use passed previous word count in PrevWordsInfo. 
Bug: 14425059

Change-Id: I04007bdacf0176a05be7a27ef1c20c5b851d8bed
 2014-09-14 17:29:38 +09:00
Keisuke Kuroyanagi 537f6eea8a Use WordIdArrayView for prevWordIds. 
Bug: 14425059
Change-Id: Ia84fb997d89564e60111b46ca83bbfa3b187f316
 2014-09-11 19:36:22 +09:00
Keisuke Kuroyanagi a3b0eb1685 Implement IntArrayView.copyToBuffer(). 
Bug: 14425059
Change-Id: Ic1b7de72701eae0c0b1bb28d12514099446a66a9
 2014-09-11 15:21:52 +09:00
Keisuke Kuroyanagi d53aea5af9 Remove unigram probability from dicNode. 
Bug: 14425059
Change-Id: Ie848e8568bb4dbb1d8358e823a881d9157a1aad3
 2014-09-10 21:21:25 +09:00
Keisuke Kuroyanagi c32356c229 Quit using dicNode.getUnigramProbability(). 
Bug: 14425059
Change-Id: I192070cc11e5d46c8413ebc19982d6a8c93577fc
 2014-09-10 21:21:25 +09:00
Keisuke Kuroyanagi 521e2382da Use CodePointArrayView to create children DicNodes. 
Change-Id: Ie940b6595f3f3f804fbb8dd03c710ea062b75af3
 2014-09-10 21:21:23 +09:00
Keisuke Kuroyanagi 87a5c76906 Use WordAttributes for checking flags. 
Bug: 14425059
Change-Id: Idee84478a482a0e7b5cc53e5dbd4e2484584ba79
 2014-09-10 19:51:57 +09:00
Keisuke Kuroyanagi 2111e3abc9 Introduce WordAttributes to get word probability and flags. 
Bug: 14425059

Change-Id: Iee11d038e0893d7ddd6c52447907f8c55fecb6a5
 2014-09-10 19:51:48 +09:00
Keisuke Kuroyanagi 11a48f92a5 Use getProbabilityOfWordInContext for prediction. 
Bug: 14425059
Change-Id: I9d5c905a0adda3503c593bfbf0bb9af8d1686f5d
 2014-09-10 19:51:14 +09:00
Keisuke Kuroyanagi 395fe8e98d Implement LanguageModelDictContent.getWordProbability(). 
Bug: 14425059
Change-Id: I290a05cee6f341caa25fb222892505529cef1eb7
 2014-09-10 19:51:12 +09:00
Keisuke Kuroyanagi 9f8da0f833 Use MultiBigramMap in structure policy. 
Bug: 14425059
Change-Id: I4d78da4839ef177e0223e6e5bcf0ebd7315c3099
 2014-09-09 17:53:44 +09:00
Keisuke Kuroyanagi 138647876b Merge "Rename probability to unigramProbability." 2014-09-09 06:16:08 +00:00
Keisuke Kuroyanagi 72ee1c7ee5 Merge "Remove mHasChildrenPtNodes from DicNodeProperties." 2014-09-09 06:03:36 +00:00
Keisuke Kuroyanagi 3e6c5edcf9 Merge "Add FormatUtilsTest." 2014-09-09 05:24:28 +00:00
Keisuke Kuroyanagi 5d95e8d747 Add FormatUtilsTest. 
Change-Id: I39ee1d18e0b70d9f4504d29a94cabc9354e4527e
 2014-09-09 14:10:53 +09:00
Keisuke Kuroyanagi 9c42ad47d4 Rename probability to unigramProbability. 
Bug: 14425059
Change-Id: I6a204c3b8fb257d037ad95a1a455ae6fb89068fd
 2014-09-09 14:09:01 +09:00
Keisuke Kuroyanagi d028294890 Remove mHasChildrenPtNodes from DicNodeProperties. 
Bug: 14425059
Change-Id: I3a9511e7f7c3a722f9942f525530f04def5965da
 2014-09-09 14:08:41 +09:00
Keisuke Kuroyanagi 3f779718cc Merge "Remove DicNode.getPtNodePos()." 2014-09-05 10:15:45 +00:00
Keisuke Kuroyanagi 9ff6fee838 Remove DicNode.getPtNodePos(). 
Bug: 14425059
Change-Id: If6e291d23e68342792febb85f8a576ce785b3845
 2014-09-05 17:27:01 +09:00
Keisuke Kuroyanagi 5cfa023777 Merge "Add HeaderReadWriteUtilsTest" 2014-09-04 07:30:59 +00:00
Keisuke Kuroyanagi fbddeedff6 Add HeaderReadWriteUtilsTest 
Change-Id: I08aeaa3e5852008874e38ed4799ab8989759e861
 2014-09-04 15:32:30 +09:00
Keisuke Kuroyanagi 94e4cd25a8 Use word id to get code ponits of the word. 
Bug: 14425059
Change-Id: I81accffcdf5abe447c33ffc3a8e8315f9a4cde7f
 2014-09-03 18:55:31 +09:00
Keisuke Kuroyanagi ac983b13a9 Use word id to get shortcut iterator. 
Bug: 14425059
Change-Id: I1b35a139bd29f70b328cbc82648783b99f633d72
 2014-09-03 18:33:10 +09:00
Keisuke Kuroyanagi 847a026cd8 Make dictionary structure policy return shortcut iterator. 
Bug: 14425059

Change-Id: I0da22c41f818673430c285103af340397aaba9fb
 2014-09-03 18:20:14 +09:00
Keisuke Kuroyanagi 259ee830f5 Fix: Missing include. 
Change-Id: I23cb5fffa474d916f3e80b951bfb30902498e043
 2014-09-03 17:03:04 +09:00
Keisuke Kuroyanagi cc6081c51b Merge "Use word id for methods related to n-grams." 2014-09-03 07:42:02 +00:00
Keisuke Kuroyanagi 89a003b12b Use word id for methods related to n-grams. 
Bug: 14425059

Change-Id: I81e5d3793527776d3c9faa5594005ddbd4a71354
 2014-09-03 16:32:43 +09:00
Keisuke Kuroyanagi 29dcf97e74 Add CharUtilsTest. 
Change-Id: I69b57d88a5376393b5d09d8b8e056c82cb8b8a3c
 2014-09-03 13:57:32 +09:00
Keisuke Kuroyanagi 69732169cc Add DamerauLevenshteinEditDistancePolicyTest 
Change-Id: Ia36bc6ddd0ec553fee151153061428c7c66d32d2
 2014-09-02 19:47:43 +09:00
Keisuke Kuroyanagi 985b2c2e91 Add GeometryUtilsTest. 
Change-Id: I6ea9ef137c2930ca8e8745fec829b9bcaf1d10b9
 2014-09-02 17:10:42 +09:00
Keisuke Kuroyanagi 93e3b5a16f Add TerminalPositionLookupTableTest. 
Change-Id: I4a3ab4c94a7759d7f24c7edc9c167fe6bbdd3eb7
 2014-08-29 14:16:15 +09:00
Keisuke Kuroyanagi 40b6d6cbd1 Add DicNodePoolTest. 
Change-Id: I9bec708def3af6bea1c82b9576b80ce1b7901ddc
 2014-08-29 13:02:22 +09:00
Keisuke Kuroyanagi e394a0c6ff Implement IntArrayView::skip(). 
Change-Id: Ib7f13963ac1d9c64eff2c08391154c6a25024126
 2014-08-28 12:12:56 +09:00
Keisuke Kuroyanagi 733a023b64 Remove redundant code. 
Change-Id: I173eac15dc128e84893881ec4ee0ad1ad2fce44b
 2014-08-27 20:28:48 +09:00
Keisuke Kuroyanagi 6ae4d79d81 Use IntArrayView for dictionary structure policy. 
Change-Id: I0dc94908259d70d5085ff22abf422d90affb1452
 2014-08-27 20:28:43 +09:00
Keisuke Kuroyanagi b85bf4ebb8 Merge "Use word id to construct DicNode instead of isTerminal flag." 2014-08-27 11:21:41 +00:00
Keisuke Kuroyanagi fe395232d6 Remove bigram dict content. 
Bug: 14425059
Change-Id: I75918c6761a50832da511088eb83becd56b23662
 2014-08-27 20:05:59 +09:00
Keisuke Kuroyanagi 758d093644 Get entry count after truncation using LanguageModelDictContent. 
Bug: 14425059
Change-Id: I41b237c1c22c21740946d52e3be9d6f963c9cd54
 2014-08-27 20:04:39 +09:00
Keisuke Kuroyanagi c7f1de826c Merge "Use LanguageModelDictContent in getWordProperty()." 2014-08-27 10:58:28 +00:00
Keisuke Kuroyanagi 7c08ede12b Merge "Use LanguageModelDictContent in iterateNgramEntries()." 2014-08-27 10:44:21 +00:00
Keisuke Kuroyanagi 82f7d3a9de Merge "Add a method to iterate entries in LanguageModelDictContent." 2014-08-27 10:35:38 +00:00
Keisuke Kuroyanagi 78aea9f133 Add TimeKeeperTest. 
Change-Id: Id35823414f8552fd6820a5a559d6c9d54c7d6cb6
 2014-08-27 14:39:27 +09:00
Keisuke Kuroyanagi 7d47500357 Use word id to construct DicNode instead of isTerminal flag. 
Bug: 14425059
Change-Id: I8484d34756bd76668ece34211e7366a4758d7bf5
 2014-08-26 14:33:19 +09:00
Keisuke Kuroyanagi 8b4409f4b9 Use LanguageModelDictContent in getWordProperty(). 
Bug: 14425059
Change-Id: Ic230f764ff5570f24ce6ce930023798718f326df
 2014-08-26 12:05:02 +09:00
Keisuke Kuroyanagi 1c14effab5 Use LanguageModelDictContent in iterateNgramEntries(). 
Bug: 14425059
Change-Id: I8a5028657ea5a4eeee8ef40db52a80f09b878972
 2014-08-26 12:03:14 +09:00
Keisuke Kuroyanagi 07b3b41c25 Add a method to iterate entries in LanguageModelDictContent. 
Bug: 14425059
Change-Id: I4e9c3a97891c020f762fa709f806d333c067f496
 2014-08-26 12:01:08 +09:00
Keisuke Kuroyanagi 30e6b21af0 Add SparseTableTest. 
Change-Id: I5684a24f9584cd326e014274c2298773aa3a91ca
 2014-08-25 22:27:01 +09:00
Keisuke Kuroyanagi c04ae77583 Add ByteArrayUtilsTest. 
Change-Id: I4695a2573b4d2d8331fc9ce1e9ff83397791a586
 2014-08-25 16:52:12 +09:00
Keisuke Kuroyanagi 295e60234c Get bigram probability using language model dict content. 
Bug: 14425059
Change-Id: Ia0493ed94e9bd47e9396d8cda94511b47820db0d
 2014-08-22 20:13:06 +09:00
Keisuke Kuroyanagi 063f86d40f Truncate entries in language model dict content. 
Bug: 14425059

Change-Id: I023c1d5109a2c43fcea3bb11a0fd7198c82891ba
 2014-08-22 20:13:04 +09:00
Keisuke Kuroyanagi 9aa6699107 Update probabilities in language model dict content for GC. 
Bug: 14425059
Change-Id: I354408afd8e5c1955ff0acea3d0243d628fe3843
 2014-08-22 20:07:54 +09:00
Keisuke Kuroyanagi 9c9f2d06bc Fix TrieMap: Cannot create next level map of small value entry. 
Bug: 14425059
Change-Id: I864f81ea7dd1d847ef2b016366ce046b1c43d5e5
 2014-08-20 20:48:25 +09:00
Keisuke Kuroyanagi ace03d7919 Merge "Add BoS flag in probability entry." 2014-08-16 04:15:21 +00:00
Keisuke Kuroyanagi 623067a183 Add BoS flag in probability entry. 
Bug: 14425059

Change-Id: I50439630034ada0280c44cbbb308aa0b95b72048
 2014-08-19 11:49:05 +09:00
Keisuke Kuroyanagi bfcd5efd50 Merge "Use byte array view in ver4 dict contents." 2014-08-16 04:15:21 +00:00
Keisuke Kuroyanagi 1f6e52ef02 Use byte array view in ver4 dict contents. 
Change-Id: Icf79a51a200f7ccd775264d1a83dd61e7dcfbab2
 2014-08-18 22:46:10 +09:00
Keisuke Kuroyanagi 2f34f0d1a8 Merge "Remove entry from language model dict content." 2014-08-16 04:15:21 +00:00
Keisuke Kuroyanagi d3097c67ca Remove entry from language model dict content. 
Bug: 14425059
Change-Id: Iea51c0ae908d499da19839de06222a1c4d19088e
 2014-08-18 12:34:50 +09:00
Keisuke Kuroyanagi b4531d861e Add method to remove entry from language model dict content. 
Bug: 14425059
Change-Id: Id21af0110e770caa3e95cb5d7ba8b3d1af8e0b12
 2014-08-18 12:34:48 +09:00
Keisuke Kuroyanagi 0db136e51f Merge "Add method to remove entry from language model dict content." 2014-08-16 04:12:14 +00:00
Keisuke Kuroyanagi 7be252d479 Use %zd to print size_t value. 
Change-Id: Id46f50353b489d84b9a97dd4ddc5519bd99492cd
 2014-08-15 16:50:23 +09:00
Keisuke Kuroyanagi ed1ab3b363 Improve accent error handling in exact match logic. 
Bug: 16963362
Change-Id: I50a48806bc3147388d165faab6b572415a851f29
 2014-08-15 16:50:23 +09:00
Keisuke Kuroyanagi 5409957440 Merge "Remove unigram for v402 with historical info." 2014-08-15 05:02:04 +00:00
Keisuke Kuroyanagi 8890b01550 Remove unigram for v402 with historical info. 
Bug: 15531638
Change-Id: If1b73ac693e45a80df987ea16b2dece5597697e8
 2014-08-15 15:47:53 +09:00
Keisuke Kuroyanagi d9d94f40c7 Merge "Make FormatUtils use ByteArrayView." 2014-08-15 05:03:06 +00:00
Keisuke Kuroyanagi 4fbb2148ee Make FormatUtils use ByteArrayView. 
Change-Id: I472b238a0d59f0092ee1f5f3b12ad63823025fae
 2014-08-14 18:09:30 +09:00
Keisuke Kuroyanagi 5fe1bed2e4 Add TrieMap::remove(). 
Bug: 14425059
Change-Id: Ifb9a873eab79efdd6bee8e0da73c0cf37727350e
 2014-08-13 21:10:48 +09:00
Keisuke Kuroyanagi 9a23f0fba2 Add bigrams to language model content. 
Bug: 14425059

Change-Id: Id81e3775ea0104750a23e3dca62c00681ed8dc2e
 2014-08-12 20:32:42 +09:00
Keisuke Kuroyanagi 9069d30043 Use IntArrayView to add/remove n-gram entry. 
Bug: 14425059
Change-Id: I734f12dfde5161abdfb019a6391677dba8984ff4
 2014-08-06 12:52:01 +09:00
Keisuke Kuroyanagi da8cda0578 Clean up IntArrayViewTest 
Change-Id: I30138f0722571100fe13efc96f8332efde846c2e
 2014-08-06 10:09:02 +09:00
Keisuke Kuroyanagi 03dc44f543 Add/Get n-gram probability entry in languageModelDictContent 
Bug: 14425059
Change-Id: I7926c3812f89b9a71fe1873a5bc32f793f91b640
 2014-08-06 00:42:56 +00:00
Keisuke Kuroyanagi 851e0458fe Remove ProbabilityDictContent and use LanguageModelDictContent 
Bug: 14425059
Change-Id: I1bb9e78ecb24139b87c99be6722e37eec0a2285d
 2014-08-05 14:13:07 +09:00
Keisuke Kuroyanagi 0889484266 Add methods for unigrams to LanguageModelDictContent. 
Bug: 14425059
Change-Id: I0a6b480a3d4735787ffac68c47b4ffefc3f1b8a5
 2014-08-05 12:38:55 +09:00
Keisuke Kuroyanagi c4696b2eb6 Save language model in the body buffer. 
Bug: 14425059
Change-Id: Iaec277f7bed03d6c6780c6ce90fbe5fe799e175e
 2014-08-01 20:19:16 +09:00
Keisuke Kuroyanagi c0c674cdc0 Make MmappedBuffer use byte array view. 
Bug: 16691311
Change-Id: I2122c01ee27c33e11dec52643925c069927bea2b
 2014-08-01 19:26:01 +09:00
Keisuke Kuroyanagi 0257e40345 Use ReadWriteByteArrayView in BufferWithExtendableBuffer. 
Bug: 16691311
Change-Id: Ibd68354a53cebf284a886d76bd8ce9848a51346f
 2014-08-01 08:44:32 +00:00