69ccac6e51
Remove unused code.
...
Bug: 12810574
Change-Id: If0ef02a984469a3b6e0c00b1c3c8d98d0d2b5466
2014-02-10 15:05:11 +09:00
8ffc631826
Make PtNode have ProbabilityInfo instead of raw value.
...
Bug: 11281877
Bug: 12810574
Change-Id: Id1cda0afc74c4e30633c735729143491b2274a7b
2014-02-10 15:05:08 +09:00
ab6a93773b
Use native logic to read Ver4 dict.
...
Bug: 11281877
Bug: 12810574
Change-Id: Ief371d3ef61818e4e031de4659aee3c9584c7379
2014-02-06 21:55:37 +09:00
fd018c1588
Merge "Use CombinedFormatUtils to convert dict elements to strings."
2014-02-06 07:29:30 +00:00
b24de426fc
Use CombinedFormatUtils to convert dict elements to strings.
...
Bug: 11281877
Bug: 12810574
Change-Id: Ib631f75eab73abc9877a7698171c45e8f2fc7600
2014-02-06 16:09:25 +09:00
337dce8074
Merge "Consolidate WordProperty and Word."
2014-02-06 07:02:07 +00:00
5f5feeba13
Consolidate WordProperty and Word.
...
Bug: 11281877
Bug: 12810574
Change-Id: I9dc99188f80f25a8780c1860dab46e4aa80a23e5
2014-02-06 15:13:33 +09:00
239f7daf4a
Merge "Make WeightedString have ProbabilityInfo."
2014-02-05 12:53:14 +00:00
df1d3e733e
Make WeightedString have ProbabilityInfo.
...
Bug: 11281877
Bug: 12810574
Change-Id: I265e3d8654c75766cd0e0d09d67ef62b4566298a
2014-02-05 21:44:55 +09:00
e42380a471
Merge "Separate ProbabilityInfo form WordProperty."
2014-02-05 12:37:49 +00:00
75a3df30f6
Separate ProbabilityInfo form WordProperty.
...
Bug: 11281877
Bug: 12810574
Change-Id: I0c77d976c915bdf2a2cddabd0cbedc2fb6691c7b
2014-02-05 21:31:35 +09:00
43cf9076b2
[HD04] Make the locale mandatory.
...
Bug: 11281748
Change-Id: I69281b0053bec404c3e3c713ade3f65a140f51b1
2014-02-05 17:35:25 +09:00
b986f78ba8
Separate header class from FormatSpec.
...
Bug: 12810574
Change-Id: Iacf1cd05a268bf690ab864b5e32a18a4b0ccc693
2014-02-04 21:36:04 +09:00
79b2e4d86c
[HD03] Straighten out attribute key names in Java.
...
Bug: 11281748
Change-Id: I1d813bdacd45bcfd9c4cc73ac1d67c5c89854e86
2014-01-31 14:46:07 +09:00
971bbcf562
Fix historical info of bigrams reading.
...
Bug: 12809791
Change-Id: I879279a0d0e1007a7c6cb87a627bf046340dda98
2014-01-30 16:19:22 +09:00
26bd46095a
Reading dictionary containing timestamps in Java Side.
...
Just skipping historical information fields.
Bug: 11281877
Change-Id: I43d2adaa576b7da11ed3ca54990265dbb6f53b08
2014-01-29 20:19:24 +09:00
fde4c5f128
Fix unit tests by adding @UsedForTesting
...
Change-Id: I2d2d226e21a46239077394ab09db88a4bd563460
2014-01-19 13:37:23 -08:00
8214a8c2cf
Fix possible NPE.
...
Bug: 11933670
Change-Id: Ib0ce853e691bdadffb73874ea9f188cfbec8bc93
2014-01-06 17:56:49 +09:00
4196360912
Run GC when needed, instead of too late.
...
Bug: 12164855
Change-Id: Ie4d8be9f2652b01470bcc06911e2191e09fb3149
2013-12-17 18:08:48 +09:00
97a553ae69
Fix wrong dereference reading code.
...
Bug: 12164855
Change-Id: I26faf27edfa303510197bccaa3521578f649741d
2013-12-17 16:42:22 +09:00
b54c747d7c
Merge "Move DictUpdater to the tests directory."
2013-12-17 05:33:16 +00:00
4fdcefe504
Move DictUpdater to the tests directory.
...
Bug: 11245133
Change-Id: I0907a091ac3ae960eaf3b27da78dbb48a24b2ea1
2013-12-17 14:31:25 +09:00
b5215d456a
Increment version number of ver4 format to 401.
...
Bug: 11073222
Change-Id: I3878aef02040b1ef73165645fc70ede586289c51
2013-12-16 23:02:54 +09:00
56e7e38d37
Fix: incompatible PtNode array size reading.
...
Bug: 11073222
Change-Id: I681be725b5c6dcc10aa536b00327698a7df75277
2013-12-16 22:42:01 +09:00
5590e9d040
Merge "Make the bad header a checked exception and fix dicttool test"
2013-12-16 07:18:44 +00:00
3df617323e
Make the bad header a checked exception and fix dicttool test
...
Change-Id: If780a2a1cb722aee83d910904c0a6315add9552d
2013-12-16 15:40:45 +09:00
b868375763
Fix failing tests
...
- Version 3 is not supported
- Now passing the right string to open v4 dicts. Fix the tests for this.
Change-Id: I7829330c3568a715b96396ba4e4e69c6e17775ab
2013-12-16 14:32:19 +09:00
1dc7eca114
Remove references to the v3 dictionary.
...
Change-Id: I811c8c923ad67a6d9bfdd11bdef8991eb7135c27
2013-12-13 18:53:41 +09:00
a245d15da5
Have dicttool use the native library to generate v4 dicts.
...
Yay !
Change-Id: Iea8ced9e81031b9ab7eff05ad9ef7215be248de9
2013-12-13 18:18:20 +09:00
7b55cd3e2b
Remove flags from Java side.
...
This simplifies the code quite a bit.
- GERMAN_UMLAUTS are now handled through a key-value attribute.
The dictionary generator does not need to know about it any more.
- FRENCH_LIGATURES are deprecated as we handle them with shortcuts now.
- CONTAINS_BIGRAMS is deprecated. Bigram processing is always applied
regardless of this flag.
Bug: 11281748
Change-Id: If567e52e245a9342adc7f3104a0f7d8d782df8c1
2013-12-13 18:15:05 +09:00
2fa3693c26
Reset to 9bd6dac470
...
The bulk merge from -bayo to klp-dev should not have been merged to master.
Change-Id: I527a03a76f5247e4939a672f27c314dc11cbb854
2013-12-13 17:13:32 +09:00
db4f373004
Fix: PtNode array size writirng when array size > 127. DO NOT MERGE.
...
This is a manual cherrypick of Ib729ceedbc8ef837e50490439817b36039ae2b4e.
Bug: 11772864
Change-Id: I5ecbe729dbdd24e194e48b4d68b17af8549c4726
2013-11-21 08:45:51 +00:00
7abdab1c6f
Merge "Add a time stamp for unigrams."
2013-10-11 10:17:01 +00:00
be22a01df3
Merge "Add resolveBigramPositions."
2013-10-11 07:45:20 +00:00
c32962b8f1
Add a time stamp for unigrams.
...
Bug: 10920255
Change-Id: I26d2cce3c322a4ff39a614f8615f43fb7bd3baed
2013-10-11 15:08:09 +09:00
9514ed5c2a
Add the new format of bigram entries.
...
In new format, each bigram entry has flags (1 byte), a terminal id (3 byte),
a time-stamp (4 byte), a counter (1 byte) and a level (1 byte).
Bug: 10920255
Bug: 10920165
Change-Id: I0f7fc125a6178e6d25a07e8462afc41a7f57e3e1
2013-10-11 14:50:41 +09:00
67c758f160
Add resolveBigramPositions.
...
Bug: 11073222
Change-Id: I680e4304c6ab701e9aa132e1e140d789097a53ae
2013-10-09 17:39:22 +09:00
ca9c3c0613
Fix a possible IOOB
...
We want to use StringUtils here, but it's full of references to
stuff not accessible host-side like JsonReader and TextUtils
and SettingsValues :/
Bug: 11061476
Change-Id: I3c0194979833ede283b4f9190335dba5376fe6fc
2013-10-07 23:16:57 +09:00
e4e0add9fb
Add Ver4DictUpdater.
...
Change-Id: I986ab26faf535fc4bc98443053f534eced9d048f
2013-10-04 17:33:29 +09:00
75d60e821c
Refactor BinaryDictIOUtilsTests.
...
Change-Id: I2208378b33038771b460abb33f9a690872e998e2
2013-10-04 14:19:13 +09:00
14d31d4640
Add AbstractDictDecoder.
...
Change-Id: Ie69d84e090f69dc3ea1f5de73ad8c954ecd2c6a7
2013-10-04 14:16:59 +09:00
73b9d3b879
Separate shortcut targets from the trie file.
...
Bug: 10920165
Change-Id: I340759eadbde7fb64cb3b9a3c619ee3a768cedf8
2013-10-04 11:16:10 +09:00
3dd77a6d66
Add SparseTableContentWriter to Ver4DictEncoder.
...
Bug: 10920165
Change-Id: I6372492e97297baad4c5aeeb3fb36dcccd7a944b
2013-10-03 20:31:01 +09:00
d6e307a4b7
Add DictUpdater.
...
Change-Id: Ic586e46e5a9f59de53d53e59886d635345940974
2013-10-03 20:16:34 +09:00
3aa8977cb2
Remove some unused variables.
...
Change-Id: Iaf1556fec194d17cb4318f2bdcc837f8d79449ef
2013-10-02 18:26:03 +09:00
4284e9aae7
Make SparseTable have multiple content tables.
...
Bug: 10920165
Change-Id: Ie9008452ee292fb0b1fec66e2ffed228c4af6c3e
2013-10-02 15:36:13 +09:00
fa946d4a0f
Fix a test and crash with a better error message when reading
...
When there are too many bigrams, we stop reading the file,
so the file pointer is in an inconsistent place. This means we
have no idea what's going to happen next. It's better to crash
right away.
Change-Id: Id3b7b78cbe4fda3493b3c9c46758763e1ab5f6a3
2013-10-02 11:48:47 +09:00
bc4926235d
Add skipPtNode to DictDecoders.
...
Change-Id: I042ff041b68572182c87dc87db6a6aa2bbbefc6c
2013-10-01 17:21:21 +09:00
25f4782853
Merge "Separate bigram entries from the trie file."
2013-09-30 08:00:01 +00:00
fd46e87da2
Separate bigram entries from the trie file.
...
Bug: 10920165
Change-Id: I42eb7f228994a56c2f31a1c83904c8c360b6d088
2013-09-30 16:42:42 +09:00
dc4f357f6b
Change scopes of some methods in DynamicBinaryDictIOUtils.
...
Change-Id: I1371d41760b38834d1d5ff40e682ac0a6fc9f2e9
2013-09-30 11:32:07 +09:00
5ed30a7660
Set header attributes for ExpandableBinaryDictionary.
...
Bug: 6669677
Change-Id: I2462777eaa07600f1c774226adfc4f351882f523
2013-09-28 12:50:09 +09:00
e3e10317b0
Merge "Create empty dictionary file in native code."
2013-09-26 08:03:38 +00:00
d188af7022
Add SparseTable.
...
Bug: 10920165
Change-Id: I749dd0269e788799e30b10beb2671813d40ce15f
2013-09-26 16:16:30 +09:00
5ef6209656
Create empty dictionary file in native code.
...
Bug: 6669677
Change-Id: I46d23deee1fd091678f4593561cb8687eb815212
2013-09-26 12:59:02 +09:00
22c5c450fe
Make Ver4DictEncoder write an address table of terminal nodes.
...
Bug: 10920165
Change-Id: I86017456ea9fa5d6e12b57172c34f5ed4f88ef94
2013-09-25 17:03:54 +09:00
14087ba52c
Add Ver4DictDecoder.
...
Bug: 9618601
Change-Id: I43c5840505c6a847aaf4893a400392ccd45903c0
2013-09-19 16:11:23 +09:00
d0c87576ff
Fix getChildrenPosition.
...
Change-Id: Ic39592dbf0d4e6d2e4879cdb1a95b65d46a57e92
2013-09-18 15:50:10 +09:00
a141d8ef7d
Add Ver4DictEncoder.
...
Bug: 9618601
Change-Id: I161d2845906f07c1251deb8005fdffe49c5b7940
2013-09-13 17:33:51 +09:00
0e40cd0c40
Add getDictDecoder.
...
Bug: 9618601
Change-Id: I173100ac704c03f7d5d0d53477e83cab5d1110d4
2013-09-12 20:14:09 +09:00
be470f06e4
Hide DictBuffer from BinaryDictDecoderUtils.
...
Bug: 9618601
Change-Id: I87515bc1555bdfe09426a056ceda74d8e63eba82
2013-09-12 16:49:54 +09:00
7547a7042c
Add writePtNode to DictEncoder.
...
Bug: 9618601
Change-Id: Ie4529444780edee8d0f0d5f23c5619c1a3344286
2013-09-11 18:27:06 +09:00
95bc256f41
Add a flag to readDictioanryBinary in DictDecoder.
...
Change-Id: I356adb72047ebc43c924fbff1ff45e7460508a31
2013-09-11 18:20:56 +09:00
8a06b26cb4
Check the version of format options before writing.
...
Change-Id: I3951e4395a2da225e240e856c23339b679e6155e
2013-09-10 18:59:31 +09:00
70e81af726
Hide physical structure of the dictionary from BinaryDictEncoderUtils.
...
Bug: 9618601
Change-Id: I01b8da42058d7c05bf7f4e6d43718032c2ad12c5
2013-09-09 14:55:33 +09:00
563bcb5117
Rename write(|Signed)VariableAddress and to write(|Signed)ChildrenPosition.
...
Change-Id: Ib018bde346744b470a4fb13b0c57e3a633b8bb20
2013-09-06 23:53:58 +09:00
6156892b7e
Add computeChildrenPosition and simplify writePlacedNode.
...
Change-Id: Idcf0e056bf61e29844d4d0e91c482cd78c41704c
2013-09-06 23:52:10 +09:00
55f5f7a005
[Refactor] Add writeDictionaryHeader.
...
Change-Id: I69026c47ce1d23f5c39c99ace76fa6b96cd1ce1b
2013-09-05 16:19:26 +09:00
663f5375d1
Add comments about how to calculate parent and children addresses.
...
Change-Id: Idf37d9a08a7b464a14c8755bedb755005e49ba33
2013-09-05 15:10:28 +09:00
d5a3c593db
Remove a broken dictionary file when fail to read a dictionary file.
...
Bug: 10434720
Change-Id: Ibdf05a39113538546b8fcf9d59af7dddf7ca27fc
2013-08-26 18:08:21 +09:00
b50a7a4bc7
Simplify computeActualNodeArraySize.
...
Change-Id: I128d02866c2cc70c42263496927327b8a2130139
2013-08-26 08:30:23 +00:00
752a33640c
[Refactor] Add DictDecoder.readUnigramsAndBigramsBinary.
...
Change-Id: I259db91d837c67cbcb3b6dc504b21dca23a6a5be
2013-08-26 17:24:38 +09:00
bb5b84a826
[Refactor] Add DictDecoder.getTerminalPosition.
...
Change-Id: I9d04f64a58f5481cbb64cf1c09b5c485dd4176b4
2013-08-26 16:14:59 +09:00
576f625ee1
Rename CharGroup to PtNode.
...
Bug: 10233675
Change-Id: I7b0eb07d195cd386cd0d9e97cd59bf48fcf24107
2013-08-26 15:58:30 +09:00
8f4d5d5f23
Fix getDictionaryFileHeader.
...
Change-Id: I5186f41b0f38eb08f4154f7b3de7ffc2119fc44b
2013-08-23 22:39:36 +09:00
e9a10ff0f0
Add DictDecoder.readDictionaryBinary.
...
Bug: 10434720
Change-Id: I14690a6e0f922ed1bab3a4b6c9a457ae84d4c1a4
2013-08-23 20:29:25 +09:00
aa4168ee09
Fix writePlacedNode.
...
Change-Id: I1d6b086f1d9f0dbd8d74f964e29ae62c533af978
2013-08-22 23:02:08 +09:00
e301085a70
Move findWordByBinaryDictReader to BinaryDictIOUtilsTests.
...
Change-Id: I443238fd816dea9650dcbbeb3ea757f9674fa52f
2013-08-22 14:49:49 +09:00
c922c8a504
Add DictEncoder.
...
Change-Id: I41049b9118b58838e5dedf8e5618d939ca70c5ef
2013-08-22 11:53:41 +09:00
558e34c7bd
Make readPtNode be called with the address from the beginning of the file.
...
Change-Id: I8939fdfb4f79e55bcd7393633784effb30df3f8f
2013-08-21 20:02:18 +09:00
a306e08753
Rename BinaryDictEncoder to BinaryDictEncoderUtils.
...
Change-Id: I4dabf17da7003b1d8204a83dbd10e5be6e8fd805
2013-08-21 18:54:34 +09:00
107a5f6fb8
Add PtNodeReader.
...
Change-Id: Ic918822fc1b3a8a7c39ffbcf7defde2c5bf888db
2013-08-21 18:43:18 +09:00
065aad9501
Add DictDecoder.
...
Change-Id: Ia1c32f21fe07081ce04d093660e18146b93275a4
2013-08-20 17:43:13 +09:00
112257e40f
Rename BinaryDictDecoder to Ver3DictDecoder.
...
Change-Id: Ibf9b95b658df6e2c2218bdb62e2380f326a03832
2013-08-20 17:11:51 +09:00
66004ce2de
Remove populateOptions.
...
Change-Id: I1a1830aaa8ea586b68fc34ff3a27ae52b810e8af
2013-08-20 16:06:52 +09:00
77bce05e6f
[Refactor] Rename BinaryDictReader and BinaryDictDecoder.
...
BinaryDictReader -> BinaryDictDecoder.
BinaryDictDecoder -> BianryDictDecoderUtils.
Change-Id: Iadf2153b379b760538ecda488dda4f17225e5f37
2013-08-19 19:36:31 +09:00
d794b42f98
Add HeaderReaderInterface.
...
Change-Id: I298f86b70d18cd08b240509b6f757c72e1a59ffe
2013-08-19 11:15:03 +09:00
3a73b37b30
Make BinaryDictIOUtils and DynamicBinaryIOUtils use BinaryDictReader.
...
Change-Id: I191dfe0e05ff3c2c5af99e8beebbb73b097748a3
2013-08-16 21:06:23 +09:00
e72c4e5fc7
Remove a static buffer for thread safety.
...
Change-Id: I335c35eb182ff63abb8a5b04c053a98d44b7c6ce
2013-08-16 20:22:46 +09:00
47bac6ebf2
Merge "Remove unnecessary caching."
2013-08-16 08:28:19 +00:00
6e26cc3f5d
Remove unnecessary caching.
...
Change-Id: Ic4ccab9d344b30b72fca1503827eec1c628fa4ac
2013-08-16 17:10:45 +09:00
af30cbf0ee
Rename Node to PtNodeArray
...
Bug: 10247660
Change-Id: I1a0ac19f58f96adb5efac5fd35c6404831618c99
2013-08-16 16:24:54 +09:00
94460eba11
[Refactor] Divide BinaryDictInputOutput into BinaryDictEncoder and BinaryDictDecoder.
...
Change-Id: I7c3269d77e3e3b567e459dcaa1bc029903941744
2013-08-15 20:23:07 +09:00
e7870a2c0d
Add an initial JNI interface to dicttool.
...
Bug: 10100269
Change-Id: I883992c2033e7d9e7c754c0bf653767728b221b6
2013-08-15 17:58:55 +09:00
117f18e844
Revert "[Refactor] Divide BinaryDictInputOutput into BinaryDictInputUtils and BinaryDictOutputUtils."
...
This reverts commit 4c63d0614e
2013-08-15 08:54:29 +00:00
4c63d0614e
[Refactor] Divide BinaryDictInputOutput into BinaryDictInputUtils and BinaryDictOutputUtils.
...
Change-Id: I0d476abe763c11ba9005152f928e8dccf15ac9de
2013-08-15 15:46:58 +09:00
1db93c9c04
[Refactor] Move some helper methods to BinaryDictIOUtils.
...
Change-Id: Ib817a975dc1f82241f732b236c44b042fda25b3c
2013-08-15 10:49:40 +09:00
3edb62c69b
Move some methods in BinaryDictIOUtils to DynamicBinaryDictIOUtils.
...
Change-Id: I9ba55582c533fef0eb3e60c46bf23c8b16ee1ff4
2013-08-14 19:33:36 +09:00
f795f2b789
Merge "Add FusionDictionaryBufferFromWritableByteBufferFactory."
2013-08-14 10:26:21 +00:00
665592774c
Move some constants in BinaryDictInputOutput to FormatSpec.
...
Change-Id: I6b12faf35b65238b9a64c82d4d1a6050f980e72e
2013-08-14 19:19:27 +09:00
Keisuke Kuroyanagi
Jean Chalard
Ken Wakasa
Yuichiro Hanada