nyancrimew/LatinIME
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
33772 commits 1 branch 0 tags 363 MiB
Commit graph

41 commits

Author SHA1 Message Date
Jean Chalard
5564317f83 Genericize getting a raw dictionary 
This will allow for not copying the whole dictionary when only
the header is needed.

Change-Id: Ie4a649b507ccd4a430201824ed87b8b8bbf55e9f
 2014-11-06 13:12:39 +09:00
Jean Chalard
ae55db95a7 Large simplification in obtaining a raw dictionary 
That is where the last refactorings were leading. This code is
simpler, but it's far more flexible. Importantly, it only makes
a single copy instead of making a full disk copy for every
intermediate step.
Next we're going to make the "copy" part modular for processes
that don't need to copy the whole file.

Change-Id: Ief32ac665d804b9b20c44f443a9c87452ceb367a
 2014-11-05 12:27:35 +09:00
Jean Chalard
afdde63374 Some more simplification of DecoderSpec works 
Change-Id: I23fa4e4ed96228406e70aa94d84fd7b8d3f69347
 2014-10-23 16:57:14 +09:00
Jean Chalard
52e92b8a3f Simplify handling of steps in DecoderChainSpec 
This is a preliminary refactoring change to improve performance
in dicttool diagnostic tools.

Change-Id: I9a59328af62e336809246be5bebbbf2e154366b3
 2014-10-23 16:57:11 +09:00
Tadashi G. Takaoka
5f00fe09e9 Fix some compiler warnings 
This CL fixes the following compiler warnings.

- Indirect access to static member
- Access to a non-accessible member of an enclosing type
- Parameter assignment
- Method can be static
- Local variable declaration hides another field or variable
- Value of local variable is not used
- Unused import
- Unused private member
- Unnecessary 'else' statement
- Unnecessary declaration of throw exception
- Redundant type arguments
- Missing '@Override' annotation
- Unused '@SuppressWarning' annotations

Bug: 18003991
Change-Id: Icfebe753e53a2cc621848f769d6a3d7ce501ebc7
 2014-10-21 19:28:37 +09:00
Adrian Velicu
05172bf1a5 Renaming "blacklist" flag to "possibly offensive" 
No behaviour changes.
Unified the overloaded FusionDictionary::add method to always take an
isPossiblyOffensive argument.

Bug: 11031090
Change-Id: I5741a023ca1ce842d2cf10d4f6c926b0efabaa78
 2014-10-21 11:51:47 +09:00
Tadashi G. Takaoka
fec4769e0b Refactor dicttool with try-with-resource 
This CL must be checked in together with Idd7c744d0f.

Change-Id: Ia0ff09a054c1852b39cdce22a4377108afb254e2
 2014-06-22 23:20:37 -07:00
Tadashi G. Takaoka
a91561aa58 Use Java 7 diamond operator 
Change-Id: If16ef50ae73147594615d0f49d6a22621eaf1aef
 2014-05-24 01:05:42 +09:00
Keisuke Kuroyanagi
93cda5bb39 Move code only used for dicttool and tests under tests. 
Bug: 13035567
Change-Id: I13c6df013ef2b67c9bf67455d9c32d283bf9ea2e
 2014-03-27 15:30:32 +09:00
Keisuke Kuroyanagi
f14cf3e64c Fix: dicttool build. 
Change-Id: I5c3bcbe9f3054bdd1a760398fe11344e0e05ac6a
 2014-03-07 13:01:48 +00:00
Jean Chalard
890b44e537 Correctly read the header of APK-embedded dicts 
Bug: 13164518
Change-Id: I8768ad887af8b89ad9f29637f606c3c68629c7ca
 2014-02-24 22:54:01 +09:00
Keisuke Kuroyanagi
8e3a1d0f89 Remove unused argument from readDictionaryBinary. 
Bug: 12810574
Change-Id: Ice415ebd8d11162facca3fe8927ef8a616b11424
 2014-02-14 19:02:15 +09:00
Keisuke Kuroyanagi
8ffc631826 Make PtNode have ProbabilityInfo instead of raw value. 
Bug: 11281877
Bug: 12810574
Change-Id: Id1cda0afc74c4e30633c735729143491b2274a7b
 2014-02-10 15:05:08 +09:00
Keisuke Kuroyanagi
5f5feeba13 Consolidate WordProperty and Word. 
Bug: 11281877
Bug: 12810574
Change-Id: I9dc99188f80f25a8780c1860dab46e4aa80a23e5
 2014-02-06 15:13:33 +09:00
Keisuke Kuroyanagi
b986f78ba8 Separate header class from FormatSpec. 
Bug: 12810574
Change-Id: Iacf1cd05a268bf690ab864b5e32a18a4b0ccc693
 2014-02-04 21:36:04 +09:00
Jean Chalard
79b2e4d86c [HD03] Straighten out attribute key names in Java. 
Bug: 11281748
Change-Id: I1d813bdacd45bcfd9c4cc73ac1d67c5c89854e86
 2014-01-31 14:46:07 +09:00
Jean Chalard
3df617323e Make the bad header a checked exception and fix dicttool test 
Change-Id: If780a2a1cb722aee83d910904c0a6315add9552d
 2013-12-16 15:40:45 +09:00
Jean Chalard
a245d15da5 Have dicttool use the native library to generate v4 dicts. 
Yay !

Change-Id: Iea8ced9e81031b9ab7eff05ad9ef7215be248de9
 2013-12-13 18:18:20 +09:00
Jean Chalard
7b55cd3e2b Remove flags from Java side. 
This simplifies the code quite a bit.
- GERMAN_UMLAUTS are now handled through a key-value attribute.
  The dictionary generator does not need to know about it any more.
- FRENCH_LIGATURES are deprecated as we handle them with shortcuts now.
- CONTAINS_BIGRAMS is deprecated. Bigram processing is always applied
  regardless of this flag.

Bug: 11281748
Change-Id: If567e52e245a9342adc7f3104a0f7d8d782df8c1
 2013-12-13 18:15:05 +09:00
Yuichiro Hanada
48e01ec111 Make dicttool read the compressed combined format. 
Change-Id: Ib39fa110402895a655f4e705caae53397ace9259
 2013-09-30 14:59:19 +09:00
Yuichiro Hanada
0e40cd0c40 Add getDictDecoder. 
Bug: 9618601
Change-Id: I173100ac704c03f7d5d0d53477e83cab5d1110d4
 2013-09-12 20:14:09 +09:00
Yuichiro Hanada
95bc256f41 Add a flag to readDictioanryBinary in DictDecoder. 
Change-Id: I356adb72047ebc43c924fbff1ff45e7460508a31
 2013-09-11 18:20:56 +09:00
Yuichiro Hanada
576f625ee1 Rename CharGroup to PtNode. 
Bug: 10233675
Change-Id: I7b0eb07d195cd386cd0d9e97cd59bf48fcf24107
 2013-08-26 15:58:30 +09:00
Yuichiro Hanada
e9a10ff0f0 Add DictDecoder.readDictionaryBinary. 
Bug: 10434720
Change-Id: I14690a6e0f922ed1bab3a4b6c9a457ae84d4c1a4
 2013-08-23 20:29:25 +09:00
Yuichiro Hanada
c922c8a504 Add DictEncoder. 
Change-Id: I41049b9118b58838e5dedf8e5618d939ca70c5ef
 2013-08-22 11:53:41 +09:00
Yuichiro Hanada
a306e08753 Rename BinaryDictEncoder to BinaryDictEncoderUtils. 
Change-Id: I4dabf17da7003b1d8204a83dbd10e5be6e8fd805
 2013-08-21 18:54:34 +09:00
Yuichiro Hanada
112257e40f Rename BinaryDictDecoder to Ver3DictDecoder. 
Change-Id: Ibf9b95b658df6e2c2218bdb62e2380f326a03832
 2013-08-20 17:11:51 +09:00
Yuichiro Hanada
77bce05e6f [Refactor] Rename BinaryDictReader and BinaryDictDecoder. 
BinaryDictReader -> BinaryDictDecoder.
BinaryDictDecoder -> BianryDictDecoderUtils.

Change-Id: Iadf2153b379b760538ecda488dda4f17225e5f37
 2013-08-19 19:36:31 +09:00
Jean Chalard
af30cbf0ee Rename Node to PtNodeArray 
Bug: 10247660
Change-Id: I1a0ac19f58f96adb5efac5fd35c6404831618c99
 2013-08-16 16:24:54 +09:00
Yuichiro Hanada
a94eb97413 Remove unused imports. 
Change-Id: Ib46da6efeaa8f83a9cda24c947fc2e667a1468e8
 2013-08-16 11:19:53 +09:00
Yuichiro Hanada
94460eba11 [Refactor] Divide BinaryDictInputOutput into BinaryDictEncoder and BinaryDictDecoder. 
Change-Id: I7c3269d77e3e3b567e459dcaa1bc029903941744
 2013-08-15 20:23:07 +09:00
Ken Wakasa
117f18e844 Revert "[Refactor] Divide BinaryDictInputOutput into BinaryDictInputUtils and BinaryDictOutputUtils." 
This reverts commit 4c63d0614e.

Change-Id: I1fa277d720bab4d895259df7d6d82eebfa5eb6c5
 2013-08-15 08:54:29 +00:00
Yuichiro Hanada
4c63d0614e [Refactor] Divide BinaryDictInputOutput into BinaryDictInputUtils and BinaryDictOutputUtils. 
Change-Id: I0d476abe763c11ba9005152f928e8dccf15ac9de
 2013-08-15 15:46:58 +09:00
Yuichiro Hanada
3feacba1eb Add BinaryDictReader. 
Bug: 9618601

Change-Id: Ief07fa0c3c4f7f5999a3fafcef4e47b6b6fd8143
 2013-08-13 19:55:05 +09:00
Jean Chalard
23d4eb55ba Add tests to dicttool test. 
Bug: 8526576
Change-Id: Idd6f9cd076d5915361c68f5c29afbba67dd54eba
 2013-06-20 17:29:37 +09:00
Jean Chalard
a411595b16 Fix two nasty bugs with surrogate pairs. 
The important bug is in findWordInTree. The problem, which is
not obvious, is that we were calling codePointAt() with the
code point index in the string, instead of the char index.
The other bug this change fixes was harmless in the practice,
because it's in the iteration which is only used for debug and
pretty printing purposes. It's very similar in that it would
substract a length in code point to a length in chars and
truncate a StringBuilder at that length, so it would fail in a
quite similar manner. This changes the meaning of the "length"
attribute in Position, but it's clearer this way anyway.

Bug: 8450145
Change-Id: If396f883a9e6449de39351553ba83f5be5bd30f0
 2013-04-01 17:06:19 +09:00
Jean Chalard
a8058d169d Cleanup after dicttool 
Arrange for temporary files to be deleted.
Also, add a security on stack overflows.

Change-Id: I9da2ebefb06409a71b235243ea835ce10d6a9b81
 2012-10-30 19:57:02 +09:00
Jean Chalard
b3c98901c5 Add auto detection and decoding of dictionary files. (A2) 
Bug: 7388852
Change-Id: I25e755fc15f5b383acc046f668e9681efa4f0c2f
 2012-10-25 16:40:15 +09:00
Tadashi G. Takaoka
15f6d4ae34 Add @UsedForTesting and @ExternallyReferenced annotations 
Bug: 7268357
Change-Id: I0b7e0c19f04af9ae30874d0a4c26ad81bc80be8c
 2012-10-22 11:18:43 -07:00
Jean Chalard
72b1c93941 Reinstate the shortcut-only attribute 
Also add the blacklist attribute

Bug: 7005742
Bug: 2704000
Change-Id: Icbe60bdf25bfb098d9e3f20870be30d6aef07c9d
 2012-08-31 22:11:52 +09:00
Jean Chalard
3598053d41 Move dicttool tests and make a script to run them (A4) 
Change-Id: If582e81741c5edaf345dcb55517dcad0baa64c9e
 2012-08-04 01:55:33 +09:00