6ae4d79d81
Use IntArrayView for dictionary structure policy.
...
Change-Id: I0dc94908259d70d5085ff22abf422d90affb1452
2014-08-27 20:28:43 +09:00
b85bf4ebb8
Merge "Use word id to construct DicNode instead of isTerminal flag."
2014-08-27 11:21:41 +00:00
fe395232d6
Remove bigram dict content.
...
Bug: 14425059
Change-Id: I75918c6761a50832da511088eb83becd56b23662
2014-08-27 20:05:59 +09:00
758d093644
Get entry count after truncation using LanguageModelDictContent.
...
Bug: 14425059
Change-Id: I41b237c1c22c21740946d52e3be9d6f963c9cd54
2014-08-27 20:04:39 +09:00
c7f1de826c
Merge "Use LanguageModelDictContent in getWordProperty()."
2014-08-27 10:58:28 +00:00
7c08ede12b
Merge "Use LanguageModelDictContent in iterateNgramEntries()."
2014-08-27 10:44:21 +00:00
82f7d3a9de
Merge "Add a method to iterate entries in LanguageModelDictContent."
2014-08-27 10:35:38 +00:00
7d47500357
Use word id to construct DicNode instead of isTerminal flag.
...
Bug: 14425059
Change-Id: I8484d34756bd76668ece34211e7366a4758d7bf5
2014-08-26 14:33:19 +09:00
8b4409f4b9
Use LanguageModelDictContent in getWordProperty().
...
Bug: 14425059
Change-Id: Ic230f764ff5570f24ce6ce930023798718f326df
2014-08-26 12:05:02 +09:00
1c14effab5
Use LanguageModelDictContent in iterateNgramEntries().
...
Bug: 14425059
Change-Id: I8a5028657ea5a4eeee8ef40db52a80f09b878972
2014-08-26 12:03:14 +09:00
07b3b41c25
Add a method to iterate entries in LanguageModelDictContent.
...
Bug: 14425059
Change-Id: I4e9c3a97891c020f762fa709f806d333c067f496
2014-08-26 12:01:08 +09:00
30e6b21af0
Add SparseTableTest.
...
Change-Id: I5684a24f9584cd326e014274c2298773aa3a91ca
2014-08-25 22:27:01 +09:00
c04ae77583
Add ByteArrayUtilsTest.
...
Change-Id: I4695a2573b4d2d8331fc9ce1e9ff83397791a586
2014-08-25 16:52:12 +09:00
295e60234c
Get bigram probability using language model dict content.
...
Bug: 14425059
Change-Id: Ia0493ed94e9bd47e9396d8cda94511b47820db0d
2014-08-22 20:13:06 +09:00
063f86d40f
Truncate entries in language model dict content.
...
Bug: 14425059
Change-Id: I023c1d5109a2c43fcea3bb11a0fd7198c82891ba
2014-08-22 20:13:04 +09:00
9aa6699107
Update probabilities in language model dict content for GC.
...
Bug: 14425059
Change-Id: I354408afd8e5c1955ff0acea3d0243d628fe3843
2014-08-22 20:07:54 +09:00
9c9f2d06bc
Fix TrieMap: Cannot create next level map of small value entry.
...
Bug: 14425059
Change-Id: I864f81ea7dd1d847ef2b016366ce046b1c43d5e5
2014-08-20 20:48:25 +09:00
ace03d7919
Merge "Add BoS flag in probability entry."
2014-08-16 04:15:21 +00:00
623067a183
Add BoS flag in probability entry.
...
Bug: 14425059
Change-Id: I50439630034ada0280c44cbbb308aa0b95b72048
2014-08-19 11:49:05 +09:00
bfcd5efd50
Merge "Use byte array view in ver4 dict contents."
2014-08-16 04:15:21 +00:00
1f6e52ef02
Use byte array view in ver4 dict contents.
...
Change-Id: Icf79a51a200f7ccd775264d1a83dd61e7dcfbab2
2014-08-18 22:46:10 +09:00
2f34f0d1a8
Merge "Remove entry from language model dict content."
2014-08-16 04:15:21 +00:00
d3097c67ca
Remove entry from language model dict content.
...
Bug: 14425059
Change-Id: Iea51c0ae908d499da19839de06222a1c4d19088e
2014-08-18 12:34:50 +09:00
b4531d861e
Add method to remove entry from language model dict content.
...
Bug: 14425059
Change-Id: Id21af0110e770caa3e95cb5d7ba8b3d1af8e0b12
2014-08-18 12:34:48 +09:00
0db136e51f
Merge "Add method to remove entry from language model dict content."
2014-08-16 04:12:14 +00:00
7be252d479
Use %zd to print size_t value.
...
Change-Id: Id46f50353b489d84b9a97dd4ddc5519bd99492cd
2014-08-15 16:50:23 +09:00
ed1ab3b363
Improve accent error handling in exact match logic.
...
Bug: 16963362
Change-Id: I50a48806bc3147388d165faab6b572415a851f29
2014-08-15 16:50:23 +09:00
5409957440
Merge "Remove unigram for v402 with historical info."
2014-08-15 05:02:04 +00:00
8890b01550
Remove unigram for v402 with historical info.
...
Bug: 15531638
Change-Id: If1b73ac693e45a80df987ea16b2dece5597697e8
2014-08-15 15:47:53 +09:00
d9d94f40c7
Merge "Make FormatUtils use ByteArrayView."
2014-08-15 05:03:06 +00:00
4fbb2148ee
Make FormatUtils use ByteArrayView.
...
Change-Id: I472b238a0d59f0092ee1f5f3b12ad63823025fae
2014-08-14 18:09:30 +09:00
5fe1bed2e4
Add TrieMap::remove().
...
Bug: 14425059
Change-Id: Ifb9a873eab79efdd6bee8e0da73c0cf37727350e
2014-08-13 21:10:48 +09:00
9a23f0fba2
Add bigrams to language model content.
...
Bug: 14425059
Change-Id: Id81e3775ea0104750a23e3dca62c00681ed8dc2e
2014-08-12 20:32:42 +09:00
9069d30043
Use IntArrayView to add/remove n-gram entry.
...
Bug: 14425059
Change-Id: I734f12dfde5161abdfb019a6391677dba8984ff4
2014-08-06 12:52:01 +09:00
03dc44f543
Add/Get n-gram probability entry in languageModelDictContent
...
Bug: 14425059
Change-Id: I7926c3812f89b9a71fe1873a5bc32f793f91b640
2014-08-06 00:42:56 +00:00
851e0458fe
Remove ProbabilityDictContent and use LanguageModelDictContent
...
Bug: 14425059
Change-Id: I1bb9e78ecb24139b87c99be6722e37eec0a2285d
2014-08-05 14:13:07 +09:00
0889484266
Add methods for unigrams to LanguageModelDictContent.
...
Bug: 14425059
Change-Id: I0a6b480a3d4735787ffac68c47b4ffefc3f1b8a5
2014-08-05 12:38:55 +09:00
c4696b2eb6
Save language model in the body buffer.
...
Bug: 14425059
Change-Id: Iaec277f7bed03d6c6780c6ce90fbe5fe799e175e
2014-08-01 20:19:16 +09:00
c0c674cdc0
Make MmappedBuffer use byte array view.
...
Bug: 16691311
Change-Id: I2122c01ee27c33e11dec52643925c069927bea2b
2014-08-01 19:26:01 +09:00
0257e40345
Use ReadWriteByteArrayView in BufferWithExtendableBuffer.
...
Bug: 16691311
Change-Id: Ibd68354a53cebf284a886d76bd8ce9848a51346f
2014-08-01 08:44:32 +00:00
dc3856d758
Add LanguageModelDictContent.
...
This class will replace BigramDictContent and
ProbabilityDictContent.
Bug: 14425059
Change-Id: I3d15c833957e27b2f5999386db042188272bbb4b
2014-08-01 12:45:00 +09:00
90b7c1729f
Remove DictContent.
...
Bug: 14425059
Change-Id: I74fa4b6ba4605447c1c87427371e4be5eb8e7ae6
2014-08-01 12:06:21 +09:00
0e6a1d1020
Remove getBigramsIteratorOfPtNode from structure policy.
...
Bug: 14425059
Change-Id: I427fbfd026c1e33b5e9cf78949fe862d8748b7ec
2014-08-01 12:06:21 +09:00
de5c3a2562
Construct TrieMap from existing buffer.
...
Bug: 14425059
Change-Id: Ia1c58d88540f55fb38fe0bdfcd7c0a34ebc7415e
2014-07-31 09:06:22 +00:00
3437032a91
Merge "Add TrieMap.save()." into lmp-dev
2014-07-24 16:04:26 +00:00
60ae3e0be5
Add TrieMap.save().
...
Bug: 14425059
Change-Id: Ie0b4fe8b9a157da0401bb277c703ab084d5bf9cc
2014-07-31 06:01:52 +00:00
5c1decfbb9
Add entry iteration method to TrieMap.
...
Bug: 14425059
Change-Id: I79420b755f29f651d8eed61e7e48b6eb001d8dd2
2014-07-30 19:04:07 +09:00
57574b98c2
Merge "Remove doesAutoCorrectValidWord()." into lmp-dev
2014-07-25 11:19:20 +00:00
5b69472d56
Remove doesAutoCorrectValidWord().
...
Possibly offensive words are wrongly demoted when input by
gesture.
Change-Id: Ibb2eb73d4191dea6cb767643341fe96465c1f70c
2014-07-31 14:48:16 +09:00
d3d7c31ca9
Fix native debug build.
...
Change-Id: I95ff6b14d7f444911ec76698381b8d2d8da94776
2014-07-31 14:36:12 +09:00
d624091afb
Merge "Fix native debug build." into lmp-dev
2014-07-25 11:19:20 +00:00
f40adc97cb
Merge "Add a method to iterate ngram entries." into lmp-dev
2014-07-24 16:04:26 +00:00
2d57b3339a
Add a method to iterate ngram entries.
...
Bug: 14425059
Change-Id: I9962c546504288f6c22b1a7368f775decd229c62
2014-07-30 10:52:56 +09:00
5a7b634aaf
Merge "Add entry iteration method to TrieMap." into lmp-dev
2014-07-24 16:04:26 +00:00
a704ad254e
Merge "Remove PrevWordsInfo.getBigramsIteratorForPrediction()." into lmp-dev
2014-07-25 11:19:20 +00:00
bd1f59bda5
Remove PrevWordsInfo.getBigramsIteratorForPrediction().
...
This changes the behavior of predictions for some rare cases.
For example, let's consider the case of the prev word is
"ABC", "ABC" and "abc" is in the dict, "ABC" doesn't have
any bigrams, and "abc" has bigrams.
Without this change, "abc"'s bigrams are used for
prediction. But, with this change, "abc"'s bigrams are not
used for prediction. This strategy is the same as the the
current way of bigram probability computation during
traversal (for suggestion).
Bug: 14425059
Change-Id: I857ff39aa70aea265efa651451a63b759bb47d48
2014-07-30 17:26:26 +09:00
0b1fa0c1c7
Cleanup: C++11 doesn't require a space between angle brackets
...
Change-Id: Id0777cf3deac94395bd63aee40b49375dff3e35a
2014-07-30 11:32:05 +09:00
6c3b0d3caa
Merge "Add TrieMap." into lmp-dev
2014-07-25 11:19:20 +00:00
de3121dead
Add TrieMap.
...
Bug: 14425059
Change-Id: Ief36f808ed4f3e35bfab07837ea314d2564973ab
2014-07-29 17:00:29 +09:00
2e32e83b6e
Merge "Get n-gram probability in structure policy." into lmp-dev
2014-07-24 16:04:26 +00:00
1229879e7c
Get n-gram probability in structure policy.
...
Bug: 14425059
Change-Id: Id955a2e07dbdfb90ae55720f0c92597e0594c0d9
2014-07-29 12:01:18 +09:00
96d47fe745
Change getUnigramProbabilityOfPtNode to getProbabilityOfPtNode()
...
Bug: 14425059
Change-Id: I92403d750d2979408b41554dcd79b0ed31b41161
2014-07-29 11:35:57 +09:00
179f38cc8c
Merge "Change getUnigramProbabilityOfPtNode to getProbabilityOfPtNode()" into lmp-dev
2014-07-25 11:19:20 +00:00
3225b6fe66
Add boundary check for ver2 bigram reading.
...
Bug: 16330528
Change-Id: I6aca6c7a735e2a652eb325572d44dff660789cff
2014-07-28 21:48:00 +09:00
d4c9d50ef0
Add BufferWithExtendablebufferTest
...
Bug: 13754552
Change-Id: Ib9c330dc85bdb5d4d3d5bc15cc3ce5f28faa0b12
2014-07-25 07:02:10 +09:00
ff6ca09905
Add BufferWithExtendableBuffer::extend()
...
Bug: 14425059
Change-Id: I13dd8412ba4d16a1325ac2210b3852d580e71ad3
2014-07-25 07:02:05 +09:00
6810e8df1e
Refactoring checkAndPrepareWriting.
...
Bug: 14425059
Change-Id: I98940f3841a6c61b0482440b4da21f415dc843f6
2014-07-24 10:24:42 +09:00
b22f95ec8a
Remove isUpdatable from constructors of dict contents.
...
Change-Id: I2d54f477d9b341e944e265786a734f23d152bb81
2014-07-11 15:23:55 +09:00
2ac934296c
Concatenate dict buffers other than header to a single file.
...
Bug: 13664080
Change-Id: I34c9d8046b339c9b855be378a5fad907382d1359
2014-07-11 15:15:47 +09:00
198be3a6c5
Fix: wrong bigram entry pos can be used during GC.
...
Bug: 14425059
Change-Id: I4e7c977c5ef184953682175b118de5473cdb64a6
2014-07-09 11:18:35 +09:00
b00973952f
Get bigrams iterator via dict structure policy.
...
Bug: 14425059
Change-Id: Ie4c6afbb6a3b707b5aa8e9e4f941c9da6e6bcdbf
2014-07-08 16:23:31 +09:00
804f7450fc
Use linked list for bigram list.
...
BinaryDictionaryTests for VERSION4_DEV:
Before
Time: 36.461
After
Time: 33.031
Bug: 14425059
Change-Id: I9ca2714f450f61f713df6ebd34c953dece991cdb
2014-07-07 21:09:25 +09:00
6afd14dc94
Remove unused method.
...
Bug: 13664080
Change-Id: I47fd90f01ea9a18b11037c966150fa5fd926101b
2014-06-27 15:23:38 +09:00
8681bef03c
Remove BigramDictionary form native code.
...
Bug: 14425059
Change-Id: I48d1a4ae6a4317afaaf67748060618f8f9485d40
2014-06-26 15:15:13 +09:00
f7322b166b
Make getNextWordAndNextToken output code point count.
...
Bug: 14425059
Change-Id: Id1c2927c8a1be0b1680206d444dbdf0c9453dceb
2014-06-23 19:08:30 +09:00
24d49ee577
Remove hasBigrams/hasShortcut flags from ver4 PtNode.
...
Always check bigram/shortcut content instead of having
flags in each PtNode.
Bug: 14425059
Change-Id: I9eb816681bf22fc12e3ab3c92a178a7990b90e93
2014-06-23 16:49:49 +09:00
0fbca1ac23
Remove word from personalized dicts when it's canceled.
...
Bug: 15555552
Bug: 15552347
Change-Id: Ia4d8c79d079e3ce233d46d4bbad4fff1be5c5c39
2014-06-20 14:46:13 +09:00
89899e14df
Merge "Update unit test for beginning-of-sentence."
2014-06-10 22:31:57 +00:00
e041bfab5c
Update unit test for beginning-of-sentence.
...
Bug: 14119293
Change-Id: I594a1ca1096f0c12c678dd30eb75de905e55c5bb
2014-06-12 16:07:39 +09:00
22931cd941
Enable Beginning-of-Sentence prediction for contextual dict.
...
Bug: 14161647
Bug: 14119293
Change-Id: I0c00f13966db88e4de85e245e7bced43c9d474b2
2014-06-12 12:26:18 +09:00
86fe8081b3
Merge "Add removeUnigramEntry() to native dictionary policy."
2014-06-10 22:29:12 +00:00
f12985245c
Add removeUnigramEntry() to native dictionary policy.
...
Bug: 15552347
Change-Id: Ic713f79ef26b5963060304df9c0a3a402c3cc6d4
2014-06-12 12:14:11 +09:00
07e1412631
Purge dict reading code for 401 and create for 402.
...
Change-Id: Ib0150b3306bb2db8aff443d1158fecc8464cbbdb
2014-06-10 19:22:40 +09:00
f9ce867d80
Add boundary check for v4 bigram reading.
...
Bug: 14496386
Change-Id: Iedd3445c3222a777a2476beed7d9eb53773f406c
2014-05-27 19:29:35 +09:00
dfca51726e
Return whether the dynamic dict operation was success.
...
Bug: 12184250
Change-Id: Iee7e00c1e84c95551a077f4dd023c0a9b9ac9466
2014-05-27 17:30:00 +09:00
a37f374ad1
Version up dynamic dict format from 401 to 402.
...
Change-Id: Ibea36af905ade773ae3db3a5456f7b5a0ad7d220
2014-05-23 20:20:56 +09:00
1adca93381
Extend jni methods and enable Beginning-of-Sentence.
...
Bug: 14119293
Change-Id: I78fc877367dd0d6240eeacb750b6d2d0b93cba83
2014-05-23 19:58:58 +09:00
96990ca773
Support Beginning-of-Sentence in native code
...
Bug: 14119293
Change-Id: I0f382e33a19bf481823b23405d454de61ec835ff
2014-05-23 17:23:08 +09:00
6600340af5
Fix: PtNodeParams.representsNonWordInfo()
...
Change-Id: I7ab4fb280e8ac89f81d0b32b0663a86c9b69aa5d
2014-05-23 13:49:18 +09:00
2a015dcb25
Add Beginning-of-Sentence info in UnigramProperty.
...
Bug: 14119293
Bug: 14425059
Change-Id: I8a894352568377d32468e5563f4e89af00d22944
2014-05-23 00:07:14 +09:00
9f8c9a0161
Use PrevWordsInfo to add/remove n(bi)-gram in native code.
...
Bug: 14119293
Bug: 14425059
Change-Id: I4b9a46bfd670b35195418eaee51456d44fb91b6d
2014-05-21 18:30:34 +09:00
e810a266fd
Merge "Quit use bigram probability diff for ver4 dict."
2014-05-15 07:07:25 +00:00
64341927d2
Quit use bigram probability diff for ver4 dict.
...
Change-Id: I2cfcfbcf351877d1dff466a24974dbb05908f14e
2014-05-15 16:02:58 +09:00
a709bc7f93
Merge "Reduce the cost for handling intentional omission."
2014-05-15 03:31:39 +00:00
39e5a15af0
Merge "Always keep PtNodes that represent non-word info during GC."
2014-05-15 02:38:57 +00:00
91d7538225
Merge "Add VERSION_4_DEV(402) in native code."
2014-05-14 13:40:10 +00:00
7116ea98f4
Add VERSION_4_DEV(402) in native code.
...
Bug: 13406708
Change-Id: I96cfacf524d670a6a5637a96a63bcd47aaf09ca0
2014-05-14 22:37:05 +09:00
cf700695ea
Always keep PtNodes that represent non-word info during GC.
...
Bug: 14119293
Change-Id: Ie50b341a512be6a6efe3e05b60e1d09560b01d1f
2014-05-14 22:29:21 +09:00
2dc8e36f8e
Use nullptr instead of MmappedBufferPtr(nullptr).
...
Change-Id: I9bc35f723f1c105ef6155a49630b8cd492900bc8
2014-05-14 22:05:00 +09:00
728141d55a
Merge "Generate dict code for version 401."
2014-05-14 10:51:12 +00:00
Keisuke Kuroyanagi
Ken Wakasa