c2fd53ee0e
Remove ver4 dict updater.
...
Change-Id: I468994c98d091be621b9fb3fbe6405c67fc6a465
2013-12-17 18:17:51 +09:00
5590e9d040
Merge "Make the bad header a checked exception and fix dicttool test"
2013-12-16 07:18:44 +00:00
3df617323e
Make the bad header a checked exception and fix dicttool test
...
Change-Id: If780a2a1cb722aee83d910904c0a6315add9552d
2013-12-16 15:40:45 +09:00
b868375763
Fix failing tests
...
- Version 3 is not supported
- Now passing the right string to open v4 dicts. Fix the tests for this.
Change-Id: I7829330c3568a715b96396ba4e4e69c6e17775ab
2013-12-16 14:32:19 +09:00
a245d15da5
Have dicttool use the native library to generate v4 dicts.
...
Yay !
Change-Id: Iea8ced9e81031b9ab7eff05ad9ef7215be248de9
2013-12-13 18:18:20 +09:00
7b55cd3e2b
Remove flags from Java side.
...
This simplifies the code quite a bit.
- GERMAN_UMLAUTS are now handled through a key-value attribute.
The dictionary generator does not need to know about it any more.
- FRENCH_LIGATURES are deprecated as we handle them with shortcuts now.
- CONTAINS_BIGRAMS is deprecated. Bigram processing is always applied
regardless of this flag.
Bug: 11281748
Change-Id: If567e52e245a9342adc7f3104a0f7d8d782df8c1
2013-12-13 18:15:05 +09:00
2fa3693c26
Reset to 9bd6dac470
...
The bulk merge from -bayo to klp-dev should not have been merged to master.
Change-Id: I527a03a76f5247e4939a672f27c314dc11cbb854
2013-12-13 17:13:32 +09:00
73665510ca
Show more messages when reading a compressed combined format file.
...
Change-Id: I51a1b9454fcfe656e0fcf762dcfd9ecbadde86c3
2013-10-08 17:05:39 +09:00
48e01ec111
Make dicttool read the compressed combined format.
...
Change-Id: Ib39fa110402895a655f4e705caae53397ace9259
2013-09-30 14:59:19 +09:00
51a590b2fe
Fix getDictionary.
...
Change-Id: I6bc3ec8dd4397a9aaf9dca2f16ce8a1929a47e9e
2013-09-26 15:26:31 +09:00
fa68e2cdf5
Add a new option for version 4 to dicttool.
...
Change-Id: I18fd48c1f6921758d30330fbc77f4a917c33f1c8
2013-09-19 11:59:42 +09:00
0e40cd0c40
Add getDictDecoder.
...
Bug: 9618601
Change-Id: I173100ac704c03f7d5d0d53477e83cab5d1110d4
2013-09-12 20:14:09 +09:00
95bc256f41
Add a flag to readDictioanryBinary in DictDecoder.
...
Change-Id: I356adb72047ebc43c924fbff1ff45e7460508a31
2013-09-11 18:20:56 +09:00
576f625ee1
Rename CharGroup to PtNode.
...
Bug: 10233675
Change-Id: I7b0eb07d195cd386cd0d9e97cd59bf48fcf24107
2013-08-26 15:58:30 +09:00
e9a10ff0f0
Add DictDecoder.readDictionaryBinary.
...
Bug: 10434720
Change-Id: I14690a6e0f922ed1bab3a4b6c9a457ae84d4c1a4
2013-08-23 20:29:25 +09:00
c922c8a504
Add DictEncoder.
...
Change-Id: I41049b9118b58838e5dedf8e5618d939ca70c5ef
2013-08-22 11:53:41 +09:00
a306e08753
Rename BinaryDictEncoder to BinaryDictEncoderUtils.
...
Change-Id: I4dabf17da7003b1d8204a83dbd10e5be6e8fd805
2013-08-21 18:54:34 +09:00
112257e40f
Rename BinaryDictDecoder to Ver3DictDecoder.
...
Change-Id: Ibf9b95b658df6e2c2218bdb62e2380f326a03832
2013-08-20 17:11:51 +09:00
77bce05e6f
[Refactor] Rename BinaryDictReader and BinaryDictDecoder.
...
BinaryDictReader -> BinaryDictDecoder.
BinaryDictDecoder -> BianryDictDecoderUtils.
Change-Id: Iadf2153b379b760538ecda488dda4f17225e5f37
2013-08-19 19:36:31 +09:00
af30cbf0ee
Rename Node to PtNodeArray
...
Bug: 10247660
Change-Id: I1a0ac19f58f96adb5efac5fd35c6404831618c99
2013-08-16 16:24:54 +09:00
a94eb97413
Remove unused imports.
...
Change-Id: Ib46da6efeaa8f83a9cda24c947fc2e667a1468e8
2013-08-16 11:19:53 +09:00
35f41a9f18
Fix build.
...
Change-Id: I80d177f3c9bff1fb90029edeafa4a62d0b99e09e
2013-08-15 20:47:42 +09:00
94460eba11
[Refactor] Divide BinaryDictInputOutput into BinaryDictEncoder and BinaryDictDecoder.
...
Change-Id: I7c3269d77e3e3b567e459dcaa1bc029903941744
2013-08-15 20:23:07 +09:00
e7870a2c0d
Add an initial JNI interface to dicttool.
...
Bug: 10100269
Change-Id: I883992c2033e7d9e7c754c0bf653767728b221b6
2013-08-15 17:58:55 +09:00
117f18e844
Revert "[Refactor] Divide BinaryDictInputOutput into BinaryDictInputUtils and BinaryDictOutputUtils."
...
This reverts commit 4c63d0614e
2013-08-15 08:54:29 +00:00
4c63d0614e
[Refactor] Divide BinaryDictInputOutput into BinaryDictInputUtils and BinaryDictOutputUtils.
...
Change-Id: I0d476abe763c11ba9005152f928e8dccf15ac9de
2013-08-15 15:46:58 +09:00
3feacba1eb
Add BinaryDictReader.
...
Bug: 9618601
Change-Id: Ief07fa0c3c4f7f5999a3fafcef4e47b6b6fd8143
2013-08-13 19:55:05 +09:00
b03447e1af
Move a couple classes to the utils package
...
Change-Id: Ia14a2011d79bad7cd02697b9254705f6e2099442
2013-07-19 10:46:46 +09:00
4d5939fc0a
Remove dependencies to android source in dicttool
...
bug: 9889653
Change-Id: I2630c56548f9146ade85b17f891932dae2082a85
2013-07-17 13:16:02 +09:00
cc54032b39
Fix build.
...
Change-Id: I51c0b7ba5641c1e7c1d63a4e5a6307b08d6f4886
2013-07-16 19:09:14 -07:00
b6cc4333a8
Generalize constuctors & add tests
...
If there is any two-args constructor with the right arg types
in a test class, use it. Also, add a test suite to dicttool test.
Also, have the return value of dicttool reflect success or
failure.
Change-Id: I51ab2a4abb93a0a483e8d6ef3c39d1ff1bce1dbd
2013-07-04 17:13:58 +09:00
4b7acd1df6
Add args to dicttool test.
...
Change-Id: I0667e0a5a6f6db3964cfcca5c8f083b9ceb41a2e
2013-07-01 15:28:32 +09:00
e28eba5074
Move util classes to the latin/utils directory
...
Change-Id: I1c5b27c8edf231680edb8d96f63b9d04cfc6a6fa
2013-06-24 17:04:40 +09:00
23d4eb55ba
Add tests to dicttool test.
...
Bug: 8526576
Change-Id: Idd6f9cd076d5915361c68f5c29afbba67dd54eba
2013-06-20 17:29:37 +09:00
d055cca7d2
Add an initial test command to dicttool.
...
Also fix some warnings
This is a preliminary change to help track b/8526576
Bug: 8526576
Change-Id: If2ed74bb92f9b5fb7bbaf9c4851d9d0b5514b405
2013-06-19 20:08:54 +09:00
a411595b16
Fix two nasty bugs with surrogate pairs.
...
The important bug is in findWordInTree. The problem, which is
not obvious, is that we were calling codePointAt() with the
code point index in the string, instead of the char index.
The other bug this change fixes was harmless in the practice,
because it's in the iteration which is only used for debug and
pretty printing purposes. It's very similar in that it would
substract a length in code point to a length in chars and
truncate a StringBuilder at that length, so it would fail in a
quite similar manner. This changes the meaning of the "length"
attribute in Position, but it's clearer this way anyway.
Bug: 8450145
Change-Id: If396f883a9e6449de39351553ba83f5be5bd30f0
2013-04-01 17:06:19 +09:00
02996dc56a
Clean up dependency for dicttool
...
bug: 7635215
Change-Id: Ie50297ab592261390885c593341b6c46161b2e63
2012-11-29 19:09:01 +09:00
5b1037fa94
Fix a bug in the diff tool
...
Fix reporting of removed attributes multiple times instead
of just one in some cases. Also, report the frequency of
the removed bigram in the same cases.
Change-Id: I1455fa7553fcebc3005c222bf5aa9afcf1678c31
2012-11-08 11:57:45 +09:00
271e731eb4
Add a missing report, and fix a wrong report
...
Deleted words would be reported, but added words wouldn't
Also, shortcuts would be reported as bigrams
Change-Id: I017653f2afa65cf16924017a0be2ee2cbd6ca10d
2012-11-01 14:25:53 +09:00
8c02f3005a
Add a "no changes" message in the diff command
...
Also fix a bug where no locale would be interpreted as a
different locale instead of the same.
Change-Id: I296feef45e39a9e89509cb315e789676755141ab
2012-11-01 13:16:58 +09:00
acafc073b9
Fix an NPE in recent code
...
Change-Id: I5c76edff5c3c250167d425a737b09b10e25f9f84
2012-11-01 12:48:37 +09:00
1b67a2bd8b
Merge "Use spaces instead of tabs in the combined format."
2012-10-31 02:33:55 -07:00
8f18c261cd
Use spaces instead of tabs in the combined format.
...
Change-Id: Ie8a4cdc44f87839e29bfbffc03d4853b7b5cc5f6
2012-10-31 18:32:09 +09:00
51a0ef8c59
Add a plumbing option to dicttool info.
...
Also align the `porcelain' option to the diff command
that was used mistakenly.
Bug: 7388665
Change-Id: Ic0e1b98c62ce37b2e909384a0370af4458563703
2012-10-31 16:35:22 +09:00
a8058d169d
Cleanup after dicttool
...
Arrange for temporary files to be deleted.
Also, add a security on stack overflows.
Change-Id: I9da2ebefb06409a71b235243ea835ce10d6a9b81
2012-10-30 19:57:02 +09:00
ee0b5488b5
Implement the package command
...
Bug: 7388852
Change-Id: Ib6a50e02894961b4c7d3f57bc0fc70c3b3bfe4e9
2012-10-30 18:42:58 +09:00
ba98b63983
Add the unpackage command
...
Bug: 7388852
Change-Id: I7e5376f5d646e669884ef78e4e0dee62c99d4d6b
2012-10-30 18:20:26 +09:00
e44358d2eb
Add an empty crypt command to aosp dicttool
...
This also makes the AdditionalCommandList useless,
so let's remove it
Bug: 7388852
Change-Id: I308f54ca6f778afe01c1233801a9cc22ebb3d1e9
2012-10-30 17:44:39 +09:00
e5b68fb51d
Accept comments in the combined file format
...
Change-Id: I17b745281b8b7b1922e1c2d3717986e9bb11aa0b
2012-10-29 18:08:50 +09:00
f41389a74b
Remove warnings
...
Thanks Eclipse
Change-Id: I88e3979ed22be5d8be5a5accdde417c6b1a8bf2d
2012-10-29 14:24:16 +09:00
74d66a5513
Add word arguments to the info command (A10)
...
With this, it's now possible to get all the information about a word from
any dictionary.
Bug: 7388857
Change-Id: Ifc9d539f52f87044a1974d95bf35d87465e84c47
2012-10-29 12:41:57 +09:00
a23e333079
Implement the word-level diff (A9)
...
Bug: 7388857
Change-Id: I4c4560d4f4b579936a44cdf409a4c27300b65610
2012-10-29 12:31:22 +09:00
0c5b93800e
Add the dicttool diff command. (A8)
...
This is an initial implementation, it does not
do everything just yet.
Bug: 7388857
Change-Id: I11543973a4657e122ca3475225059b06b9a3f2b9
2012-10-29 11:15:15 +09:00
6ecc50a867
Add a reporting option to the getDictionary method (A7)
...
This is refactoring to help future changes
Change-Id: I8c4372383a2fff5805b96cc47df5012e4b231004
2012-10-29 10:39:40 +09:00
47cac57e45
Finish up the "info" command in dicttool. (A6)
...
Bug: 7388857
Change-Id: I704f12a6be76ce1644ec5e8dd3b667f112e9c04a
2012-10-25 19:15:24 +09:00
f1d35ac5dc
Implement a part of the "info" command (A5)
...
Bug: 7388857
Change-Id: Id1bba7a33ab6195d5711bebb1df5c18c9c08925c
2012-10-25 18:19:10 +09:00
3862b9e819
Add symmetry and pre-emptively improve performance (A4)
...
The test for binary dictionary will soon entail decrypting and
uncompressing files if necessary to test for their headers, and will
become much slower than it is. It's better to be able to detect the
XML format too, and leave the slower test for last.
Bug: 7388852
Change-Id: I6b9a7944de80217e1571cab65dcd1cff347b3046
2012-10-25 17:20:24 +09:00
0044df6cf2
Add automatic decryption capabilities. (A3)
...
Bug: 7388852
Change-Id: Ie952b1ad6cfad2f83793bd968e159d3cfbf20a47
2012-10-25 16:47:32 +09:00
b3c98901c5
Add auto detection and decoding of dictionary files. (A2)
...
Bug: 7388852
Change-Id: I25e755fc15f5b383acc046f668e9681efa4f0c2f
2012-10-25 16:40:15 +09:00
77fe603a3d
Some initial refactoring in dicttool. (A1)
...
Bug: 7388852
Change-Id: I5ff70d12f3a8096ae6fb8cd4883a32ffe1683c9b
2012-10-25 16:40:11 +09:00
9bb4eebf48
Add reader and writer for the combined dict format.
...
This introduces a new textual format for the dictionary that combines
words, bigrams and shortcuts to avoid complexity. It is also extensible
to n-grams to fool-prof for the future, and easier to read than XML.
Bug: 7388540
Change-Id: I942bbad51bd0c905a5a54c278667563fd6dd66ec
2012-10-23 15:06:25 +09:00
cbb3252731
Merge "Use a buffered writer for XML output"
2012-10-22 18:54:38 -07:00
000a020e32
Use a buffered writer for XML output
...
Change-Id: I85aa133985282993542659db2ee61960b02042d7
2012-10-23 10:42:22 +09:00
15f6d4ae34
Add @UsedForTesting and @ExternallyReferenced annotations
...
Bug: 7268357
Change-Id: I0b7e0c19f04af9ae30874d0a4c26ad81bc80be8c
2012-10-22 11:18:43 -07:00
243c1fecc6
Sync package name and source path
...
Change-Id: I8d3ded01c508c3afaa0d8afa9ce1445117626ddc
2012-10-19 18:14:19 -07:00
99b84b42f9
Use a freq of 1 instead of 0 for non-word shortcuts.
...
Also fix a spelling mistake in a comment
Bug: 7301525
Change-Id: I4437403dce620fed03871485ee04f13c51ce34fc
2012-10-09 19:16:07 +09:00
cc958dd96e
Refactor BinaryDictInputOutput.
...
Change-Id: Idb4b635fcac70cc988e0dd3ce3bf121fba12099c
2012-09-14 11:08:01 +09:00
1a347723c5
Move FormatOptions and FileHeader to FormatSpec.
...
Change-Id: I232e35598635113bf2c81825669c744aadc79efe
2012-09-13 16:35:41 +09:00
debe3b4555
Have makedict support the new file format.
...
Change-Id: Idcbba71b397f87a5d59e16bb4061d62261c73282
2012-09-10 17:47:06 +09:00
60af4f03a8
Fix dicttool_aosp build
...
Change-Id: Iabcf420b4120e3d275fdbcb23939a0ab6052ddf0
2012-09-10 12:56:10 +09:00
83dfe0fd8c
Add FormatOptions.
...
Change-Id: Ibad05a5f9143de1156b2c897593ec89b0a0b07e7
2012-09-05 18:05:43 +09:00
72b1c93941
Reinstate the shortcut-only attribute
...
Also add the blacklist attribute
Bug: 7005742
Bug: 2704000
Change-Id: Icbe60bdf25bfb098d9e3f20870be30d6aef07c9d
2012-08-31 22:11:52 +09:00
d4fe7fda30
Use ByteBuffer when reading FusionDictionary from file.
...
Change-Id: Ia71561648e17f846d277c22309ac37c21c67a537
2012-08-24 13:31:08 +09:00
4af100759b
(DO NOT MERGE) Remove useless module tags from makefiles
...
Change-Id: If77482da66e80b762cf205369f7d53d069cd8072
2012-08-17 20:12:49 +09:00
565fdeee7d
Reinstate putting shortcut-only entries in the dictionary
...
Not sure at which point this functionality got lost, but
anyway, it's back
Bug: 6906525
Change-Id: Ie87baa0dacdcee90e029917ca866099593d5d048
2012-08-10 15:51:53 +09:00
31b90a368f
Support a syntax to add whitelist entries in shortcuts
...
Bug: 6906525
Change-Id: Ie33586c07973bf76e38390766175ba424d72d655
2012-08-10 11:26:40 +09:00
ccd8ed53b3
Support stdin and stdout for compress/uncompress
...
This will improve flexibility in dictionary generation by
allowing pipelined commands.
Change-Id: Ia07bdc82c8bd740b5dfee6b5fd1a067f4b8b7ba5
2012-08-09 14:25:13 +09:00
4c573f3c64
Rename dicttool to dicttool_aosp and split commands
...
Change-Id: I83d948be2db5cd0203e876f019697e1d5e038818
2012-08-08 17:15:58 +09:00
04cccea295
Move makedict to live under dicttool (A5)
...
This removes the makedict directory and makes a new target
to replace it. The old command is still working the same
way it used to.
Change-Id: Ie73b12a7ef101dd2b282e0cb4206ac263af1093e
2012-08-04 02:14:13 +09:00
3598053d41
Move dicttool tests and make a script to run them (A4)
...
Change-Id: If582e81741c5edaf345dcb55517dcad0baa64c9e
2012-08-04 01:55:33 +09:00
54e84a00fc
Make a makedict command for dicttool (A3)
...
This behaves exactly as the old makedict command. Further
changes will redirect the calls to makedict to this, so as
to consolidate similar code.
Groundwork for
Bug: 6429606
Change-Id: Ibeadbf48bec70f988a15ca36ebf5d1ce3b5b54ea
2012-08-04 01:11:46 +09:00
82cd20ee98
Simplification (A2)
...
A single command at a time and no arity gives opportunity to simplify
the argument parsing code.
Change-Id: Iff7bab78ede7448a662092f221c590a9eedae2a5
2012-08-03 23:02:39 +09:00
77c8c73837
Remove arity from dicttool commands (A1)
...
This unnecessarily complicates and limits commands.
It's simpler to just have one command by invocation.
Groundwork for
Bug: 6429606
Change-Id: I24cf267a9cbc937a5ba53942b29e28e7095d2fd2
2012-08-03 23:02:06 +09:00
ddcb4847df
Add a tool to do intendance tasks on dictionaries.
...
Bug: 6429606
Change-Id: I0c7258d992a4bfa9707002f8fbc425ccde7c6172
2012-05-28 18:46:09 +09:00
Keisuke Kuroyanagi
Jean Chalard
Ken Wakasa
Yuichiro Hanada
Ying Wang
Tadashi G. Takaoka