fb2bde5a68
Native side reads character table
...
Bug:17097992
Change-Id: Ibcfc67833a6754fe6a2d82a3e3023b33886b9ea2
2014-09-19 18:00:16 +09:00
b349b57476
Merge "Use ReadOnlyByteArrayView in Ver2ParticiaTrieNodeReader."
2014-09-17 12:53:54 +00:00
a533e18c07
Merge "Use ReadOnlyByteArrayView in Ver2PtNodeArrayReader."
2014-09-17 12:53:48 +00:00
4a55d41410
Merge "Use ReadOnlyByteArrayView in ShortcutListPolicy"
2014-09-17 12:51:06 +00:00
c1040a255c
Merge "Use ReadOnlyByteArrayView in BigramListPolicy."
2014-09-17 12:50:51 +00:00
5849feeee1
Merge "Use ReadOnlyByteArrayView in PatriciaTriePolicy."
2014-09-17 12:50:16 +00:00
198a47a161
Merge "Add firstOrDefault and lastOrDefault to IntArrayView."
2014-09-17 12:43:10 +00:00
6258c57c32
Use ReadOnlyByteArrayView in Ver2ParticiaTrieNodeReader.
...
Change-Id: Ie3325bb9666c7ff6d519a1b72ed2fe9fc4f140a9
2014-09-17 21:42:30 +09:00
a892628fa3
Use ReadOnlyByteArrayView in Ver2PtNodeArrayReader.
...
Change-Id: Ia1065465abdf7c64fadb3ff1efd2a4d2df343264
2014-09-17 21:42:30 +09:00
59ebd51718
Use ReadOnlyByteArrayView in ShortcutListPolicy
...
Change-Id: I03a6f49c9005306bcc5fce8b7e4d37b8d30b9faa
2014-09-17 21:42:29 +09:00
d01eb3c94c
Use ReadOnlyByteArrayView in BigramListPolicy.
...
Change-Id: I0f8e8044c8803a7a542f97be5b8f08fdb5ae3926
2014-09-17 21:42:26 +09:00
180e7b4c07
Use ReadOnlyByteArrayView in PatriciaTriePolicy.
...
Change-Id: I4ef01c0af73f27c256cc1a719343be836913dc71
2014-09-17 21:38:46 +09:00
09c154925f
Add firstOrDefault and lastOrDefault to IntArrayView.
...
Change-Id: I854c02eff3fa0b53c72a5f1cabce001f4854ada0
2014-09-17 21:16:31 +09:00
89a074fade
Use IntArrayView in DynamicPtUpdatingHelper.
...
Change-Id: Ifa185eba7dd9abee1b35e49fe559be1042ca63d7
2014-09-17 20:13:36 +09:00
4926b90ec5
Support n-gram for look-up.
...
Bug: 14425059
Change-Id: I19523c29fb802cd65158c7540d1608e7f55c4ca7
2014-09-17 16:20:00 +09:00
5520e84e16
Support removing n-gram entry in structure policy.
...
Bug: 14425059
Change-Id: Ifd2ac97a15dcbb936e2f068866a9d1ee68759ba2
2014-09-16 15:46:28 +09:00
0c0b8207cd
Support adding n-gram entry in structure policy.
...
Bug: 14425059
Change-Id: Ia207d3c4735b1d6d43f18d18a70d28af613cb458
2014-09-16 15:37:09 +09:00
7eb6e28b9e
Update input checking logic for adding/removing n-gram.
...
Bug: 14425059
Change-Id: Ifb4e3c3f06aaf9a16f632ff208ba76cf741cea83
2014-09-16 14:14:22 +09:00
36ba139ca6
Support decaying dict in getWordProbability().
...
Bug: 14425059
Change-Id: I24db3f9131c2999fc388035dc365c7faaef3bdb1
2014-09-14 17:29:50 +09:00
c43b6664fa
Use passed previous word count in PrevWordsInfo.
...
Bug: 14425059
Change-Id: I04007bdacf0176a05be7a27ef1c20c5b851d8bed
2014-09-14 17:29:38 +09:00
537f6eea8a
Use WordIdArrayView for prevWordIds.
...
Bug: 14425059
Change-Id: Ia84fb997d89564e60111b46ca83bbfa3b187f316
2014-09-11 19:36:22 +09:00
d53aea5af9
Remove unigram probability from dicNode.
...
Bug: 14425059
Change-Id: Ie848e8568bb4dbb1d8358e823a881d9157a1aad3
2014-09-10 21:21:25 +09:00
521e2382da
Use CodePointArrayView to create children DicNodes.
...
Change-Id: Ie940b6595f3f3f804fbb8dd03c710ea062b75af3
2014-09-10 21:21:23 +09:00
87a5c76906
Use WordAttributes for checking flags.
...
Bug: 14425059
Change-Id: Idee84478a482a0e7b5cc53e5dbd4e2484584ba79
2014-09-10 19:51:57 +09:00
2111e3abc9
Introduce WordAttributes to get word probability and flags.
...
Bug: 14425059
Change-Id: Iee11d038e0893d7ddd6c52447907f8c55fecb6a5
2014-09-10 19:51:48 +09:00
395fe8e98d
Implement LanguageModelDictContent.getWordProbability().
...
Bug: 14425059
Change-Id: I290a05cee6f341caa25fb222892505529cef1eb7
2014-09-10 19:51:12 +09:00
9f8da0f833
Use MultiBigramMap in structure policy.
...
Bug: 14425059
Change-Id: I4d78da4839ef177e0223e6e5bcf0ebd7315c3099
2014-09-09 17:53:44 +09:00
72ee1c7ee5
Merge "Remove mHasChildrenPtNodes from DicNodeProperties."
2014-09-09 06:03:36 +00:00
d028294890
Remove mHasChildrenPtNodes from DicNodeProperties.
...
Bug: 14425059
Change-Id: I3a9511e7f7c3a722f9942f525530f04def5965da
2014-09-09 14:08:41 +09:00
3f779718cc
Merge "Remove DicNode.getPtNodePos()."
2014-09-05 10:15:45 +00:00
9ff6fee838
Remove DicNode.getPtNodePos().
...
Bug: 14425059
Change-Id: If6e291d23e68342792febb85f8a576ce785b3845
2014-09-05 17:27:01 +09:00
5cfa023777
Merge "Add HeaderReadWriteUtilsTest"
2014-09-04 07:30:59 +00:00
fbddeedff6
Add HeaderReadWriteUtilsTest
...
Change-Id: I08aeaa3e5852008874e38ed4799ab8989759e861
2014-09-04 15:32:30 +09:00
94e4cd25a8
Use word id to get code ponits of the word.
...
Bug: 14425059
Change-Id: I81accffcdf5abe447c33ffc3a8e8315f9a4cde7f
2014-09-03 18:55:31 +09:00
ac983b13a9
Use word id to get shortcut iterator.
...
Bug: 14425059
Change-Id: I1b35a139bd29f70b328cbc82648783b99f633d72
2014-09-03 18:33:10 +09:00
847a026cd8
Make dictionary structure policy return shortcut iterator.
...
Bug: 14425059
Change-Id: I0da22c41f818673430c285103af340397aaba9fb
2014-09-03 18:20:14 +09:00
259ee830f5
Fix: Missing include.
...
Change-Id: I23cb5fffa474d916f3e80b951bfb30902498e043
2014-09-03 17:03:04 +09:00
89a003b12b
Use word id for methods related to n-grams.
...
Bug: 14425059
Change-Id: I81e5d3793527776d3c9faa5594005ddbd4a71354
2014-09-03 16:32:43 +09:00
93e3b5a16f
Add TerminalPositionLookupTableTest.
...
Change-Id: I4a3ab4c94a7759d7f24c7edc9c167fe6bbdd3eb7
2014-08-29 14:16:15 +09:00
6ae4d79d81
Use IntArrayView for dictionary structure policy.
...
Change-Id: I0dc94908259d70d5085ff22abf422d90affb1452
2014-08-27 20:28:43 +09:00
b85bf4ebb8
Merge "Use word id to construct DicNode instead of isTerminal flag."
2014-08-27 11:21:41 +00:00
fe395232d6
Remove bigram dict content.
...
Bug: 14425059
Change-Id: I75918c6761a50832da511088eb83becd56b23662
2014-08-27 20:05:59 +09:00
758d093644
Get entry count after truncation using LanguageModelDictContent.
...
Bug: 14425059
Change-Id: I41b237c1c22c21740946d52e3be9d6f963c9cd54
2014-08-27 20:04:39 +09:00
c7f1de826c
Merge "Use LanguageModelDictContent in getWordProperty()."
2014-08-27 10:58:28 +00:00
7c08ede12b
Merge "Use LanguageModelDictContent in iterateNgramEntries()."
2014-08-27 10:44:21 +00:00
82f7d3a9de
Merge "Add a method to iterate entries in LanguageModelDictContent."
2014-08-27 10:35:38 +00:00
7d47500357
Use word id to construct DicNode instead of isTerminal flag.
...
Bug: 14425059
Change-Id: I8484d34756bd76668ece34211e7366a4758d7bf5
2014-08-26 14:33:19 +09:00
8b4409f4b9
Use LanguageModelDictContent in getWordProperty().
...
Bug: 14425059
Change-Id: Ic230f764ff5570f24ce6ce930023798718f326df
2014-08-26 12:05:02 +09:00
1c14effab5
Use LanguageModelDictContent in iterateNgramEntries().
...
Bug: 14425059
Change-Id: I8a5028657ea5a4eeee8ef40db52a80f09b878972
2014-08-26 12:03:14 +09:00
07b3b41c25
Add a method to iterate entries in LanguageModelDictContent.
...
Bug: 14425059
Change-Id: I4e9c3a97891c020f762fa709f806d333c067f496
2014-08-26 12:01:08 +09:00
30e6b21af0
Add SparseTableTest.
...
Change-Id: I5684a24f9584cd326e014274c2298773aa3a91ca
2014-08-25 22:27:01 +09:00
c04ae77583
Add ByteArrayUtilsTest.
...
Change-Id: I4695a2573b4d2d8331fc9ce1e9ff83397791a586
2014-08-25 16:52:12 +09:00
295e60234c
Get bigram probability using language model dict content.
...
Bug: 14425059
Change-Id: Ia0493ed94e9bd47e9396d8cda94511b47820db0d
2014-08-22 20:13:06 +09:00
063f86d40f
Truncate entries in language model dict content.
...
Bug: 14425059
Change-Id: I023c1d5109a2c43fcea3bb11a0fd7198c82891ba
2014-08-22 20:13:04 +09:00
9aa6699107
Update probabilities in language model dict content for GC.
...
Bug: 14425059
Change-Id: I354408afd8e5c1955ff0acea3d0243d628fe3843
2014-08-22 20:07:54 +09:00
9c9f2d06bc
Fix TrieMap: Cannot create next level map of small value entry.
...
Bug: 14425059
Change-Id: I864f81ea7dd1d847ef2b016366ce046b1c43d5e5
2014-08-20 20:48:25 +09:00
ace03d7919
Merge "Add BoS flag in probability entry."
2014-08-16 04:15:21 +00:00
623067a183
Add BoS flag in probability entry.
...
Bug: 14425059
Change-Id: I50439630034ada0280c44cbbb308aa0b95b72048
2014-08-19 11:49:05 +09:00
bfcd5efd50
Merge "Use byte array view in ver4 dict contents."
2014-08-16 04:15:21 +00:00
1f6e52ef02
Use byte array view in ver4 dict contents.
...
Change-Id: Icf79a51a200f7ccd775264d1a83dd61e7dcfbab2
2014-08-18 22:46:10 +09:00
2f34f0d1a8
Merge "Remove entry from language model dict content."
2014-08-16 04:15:21 +00:00
d3097c67ca
Remove entry from language model dict content.
...
Bug: 14425059
Change-Id: Iea51c0ae908d499da19839de06222a1c4d19088e
2014-08-18 12:34:50 +09:00
b4531d861e
Add method to remove entry from language model dict content.
...
Bug: 14425059
Change-Id: Id21af0110e770caa3e95cb5d7ba8b3d1af8e0b12
2014-08-18 12:34:48 +09:00
0db136e51f
Merge "Add method to remove entry from language model dict content."
2014-08-16 04:12:14 +00:00
7be252d479
Use %zd to print size_t value.
...
Change-Id: Id46f50353b489d84b9a97dd4ddc5519bd99492cd
2014-08-15 16:50:23 +09:00
5409957440
Merge "Remove unigram for v402 with historical info."
2014-08-15 05:02:04 +00:00
8890b01550
Remove unigram for v402 with historical info.
...
Bug: 15531638
Change-Id: If1b73ac693e45a80df987ea16b2dece5597697e8
2014-08-15 15:47:53 +09:00
d9d94f40c7
Merge "Make FormatUtils use ByteArrayView."
2014-08-15 05:03:06 +00:00
4fbb2148ee
Make FormatUtils use ByteArrayView.
...
Change-Id: I472b238a0d59f0092ee1f5f3b12ad63823025fae
2014-08-14 18:09:30 +09:00
5fe1bed2e4
Add TrieMap::remove().
...
Bug: 14425059
Change-Id: Ifb9a873eab79efdd6bee8e0da73c0cf37727350e
2014-08-13 21:10:48 +09:00
9a23f0fba2
Add bigrams to language model content.
...
Bug: 14425059
Change-Id: Id81e3775ea0104750a23e3dca62c00681ed8dc2e
2014-08-12 20:32:42 +09:00
9069d30043
Use IntArrayView to add/remove n-gram entry.
...
Bug: 14425059
Change-Id: I734f12dfde5161abdfb019a6391677dba8984ff4
2014-08-06 12:52:01 +09:00
03dc44f543
Add/Get n-gram probability entry in languageModelDictContent
...
Bug: 14425059
Change-Id: I7926c3812f89b9a71fe1873a5bc32f793f91b640
2014-08-06 00:42:56 +00:00
851e0458fe
Remove ProbabilityDictContent and use LanguageModelDictContent
...
Bug: 14425059
Change-Id: I1bb9e78ecb24139b87c99be6722e37eec0a2285d
2014-08-05 14:13:07 +09:00
0889484266
Add methods for unigrams to LanguageModelDictContent.
...
Bug: 14425059
Change-Id: I0a6b480a3d4735787ffac68c47b4ffefc3f1b8a5
2014-08-05 12:38:55 +09:00
c4696b2eb6
Save language model in the body buffer.
...
Bug: 14425059
Change-Id: Iaec277f7bed03d6c6780c6ce90fbe5fe799e175e
2014-08-01 20:19:16 +09:00
c0c674cdc0
Make MmappedBuffer use byte array view.
...
Bug: 16691311
Change-Id: I2122c01ee27c33e11dec52643925c069927bea2b
2014-08-01 19:26:01 +09:00
0257e40345
Use ReadWriteByteArrayView in BufferWithExtendableBuffer.
...
Bug: 16691311
Change-Id: Ibd68354a53cebf284a886d76bd8ce9848a51346f
2014-08-01 08:44:32 +00:00
dc3856d758
Add LanguageModelDictContent.
...
This class will replace BigramDictContent and
ProbabilityDictContent.
Bug: 14425059
Change-Id: I3d15c833957e27b2f5999386db042188272bbb4b
2014-08-01 12:45:00 +09:00
90b7c1729f
Remove DictContent.
...
Bug: 14425059
Change-Id: I74fa4b6ba4605447c1c87427371e4be5eb8e7ae6
2014-08-01 12:06:21 +09:00
0e6a1d1020
Remove getBigramsIteratorOfPtNode from structure policy.
...
Bug: 14425059
Change-Id: I427fbfd026c1e33b5e9cf78949fe862d8748b7ec
2014-08-01 12:06:21 +09:00
de5c3a2562
Construct TrieMap from existing buffer.
...
Bug: 14425059
Change-Id: Ia1c58d88540f55fb38fe0bdfcd7c0a34ebc7415e
2014-07-31 09:06:22 +00:00
3437032a91
Merge "Add TrieMap.save()." into lmp-dev
2014-07-24 16:04:26 +00:00
60ae3e0be5
Add TrieMap.save().
...
Bug: 14425059
Change-Id: Ie0b4fe8b9a157da0401bb277c703ab084d5bf9cc
2014-07-31 06:01:52 +00:00
5c1decfbb9
Add entry iteration method to TrieMap.
...
Bug: 14425059
Change-Id: I79420b755f29f651d8eed61e7e48b6eb001d8dd2
2014-07-30 19:04:07 +09:00
d624091afb
Merge "Fix native debug build." into lmp-dev
2014-07-25 11:19:20 +00:00
d3d7c31ca9
Fix native debug build.
...
Change-Id: I95ff6b14d7f444911ec76698381b8d2d8da94776
2014-07-31 14:36:12 +09:00
5a7b634aaf
Merge "Add entry iteration method to TrieMap." into lmp-dev
2014-07-24 16:04:26 +00:00
a704ad254e
Merge "Remove PrevWordsInfo.getBigramsIteratorForPrediction()." into lmp-dev
2014-07-25 11:19:20 +00:00
bd1f59bda5
Remove PrevWordsInfo.getBigramsIteratorForPrediction().
...
This changes the behavior of predictions for some rare cases.
For example, let's consider the case of the prev word is
"ABC", "ABC" and "abc" is in the dict, "ABC" doesn't have
any bigrams, and "abc" has bigrams.
Without this change, "abc"'s bigrams are used for
prediction. But, with this change, "abc"'s bigrams are not
used for prediction. This strategy is the same as the the
current way of bigram probability computation during
traversal (for suggestion).
Bug: 14425059
Change-Id: I857ff39aa70aea265efa651451a63b759bb47d48
2014-07-30 17:26:26 +09:00
2d57b3339a
Add a method to iterate ngram entries.
...
Bug: 14425059
Change-Id: I9962c546504288f6c22b1a7368f775decd229c62
2014-07-30 10:52:56 +09:00
f40adc97cb
Merge "Add a method to iterate ngram entries." into lmp-dev
2014-07-24 16:04:26 +00:00
0b1fa0c1c7
Cleanup: C++11 doesn't require a space between angle brackets
...
Change-Id: Id0777cf3deac94395bd63aee40b49375dff3e35a
2014-07-30 11:32:05 +09:00
6c3b0d3caa
Merge "Add TrieMap." into lmp-dev
2014-07-25 11:19:20 +00:00
de3121dead
Add TrieMap.
...
Bug: 14425059
Change-Id: Ief36f808ed4f3e35bfab07837ea314d2564973ab
2014-07-29 17:00:29 +09:00
2e32e83b6e
Merge "Get n-gram probability in structure policy." into lmp-dev
2014-07-24 16:04:26 +00:00
1229879e7c
Get n-gram probability in structure policy.
...
Bug: 14425059
Change-Id: Id955a2e07dbdfb90ae55720f0c92597e0594c0d9
2014-07-29 12:01:18 +09:00
96d47fe745
Change getUnigramProbabilityOfPtNode to getProbabilityOfPtNode()
...
Bug: 14425059
Change-Id: I92403d750d2979408b41554dcd79b0ed31b41161
2014-07-29 11:35:57 +09:00
179f38cc8c
Merge "Change getUnigramProbabilityOfPtNode to getProbabilityOfPtNode()" into lmp-dev
2014-07-25 11:19:20 +00:00
3225b6fe66
Add boundary check for ver2 bigram reading.
...
Bug: 16330528
Change-Id: I6aca6c7a735e2a652eb325572d44dff660789cff
2014-07-28 21:48:00 +09:00
Akifumi Yoshimoto
Keisuke Kuroyanagi
Ken Wakasa