Commit graph

121 commits

Author SHA1 Message Date
Yuichiro Hanada
1db93c9c04 [Refactor] Move some helper methods to BinaryDictIOUtils.
Change-Id: Ib817a975dc1f82241f732b236c44b042fda25b3c
2013-08-15 10:49:40 +09:00
Yuichiro Hanada
3edb62c69b Move some methods in BinaryDictIOUtils to DynamicBinaryDictIOUtils.
Change-Id: I9ba55582c533fef0eb3e60c46bf23c8b16ee1ff4
2013-08-14 19:33:36 +09:00
Ken Wakasa
f795f2b789 Merge "Add FusionDictionaryBufferFromWritableByteBufferFactory." 2013-08-14 10:26:21 +00:00
Yuichiro Hanada
665592774c Move some constants in BinaryDictInputOutput to FormatSpec.
Change-Id: I6b12faf35b65238b9a64c82d4d1a6050f980e72e
2013-08-14 19:19:27 +09:00
Yuichiro Hanada
bbc8a930f7 Add FusionDictionaryBufferFromWritableByteBufferFactory.
Change-Id: I23de0a178e7f11f2cf301fd433cde60c6152055b
2013-08-14 17:07:44 +09:00
Yuichiro Hanada
3feacba1eb Add BinaryDictReader.
Bug: 9618601

Change-Id: Ief07fa0c3c4f7f5999a3fafcef4e47b6b6fd8143
2013-08-13 19:55:05 +09:00
Yuichiro Hanada
b7bb9c9722 Make readHeader check the header size before using it.
Change-Id: I5dc3e2b674f7343ef57317fde6bdb7349a7fe04c
2013-08-13 17:06:25 +09:00
Yuichiro Hanada
7ec9db2c34 Remove the code and comments about version 1 format.
Change-Id: I827052f234eeaa4dbcfd37da69a99866896a158b
2013-08-09 16:05:07 +09:00
Yuichiro Hanada
7d1ae52ded Fix unit tests.
Change-Id: Ic0013089625e112aaccc888d462330640ef7cc6f
2013-08-08 19:12:35 +09:00
Jean Chalard
93445b4821 Fix some warnings
Change-Id: I7290cd1fb675a1b85b9b6ac2d464c932b5bca1dd
2013-07-31 16:17:01 +09:00
Satoshi Kataoka
ffcbbaf127 Refactor on UserHistoryDictionary
Bug: 9429906
Change-Id: I576a91643bdaf5017cc826ac2e07a74a9a275d60
2013-07-26 13:00:19 +09:00
Jean Chalard
25de86a6a2 [FD4] Separate cached address before/after update for groups
This should fix bug#8526576 for good.

Bug: 8526576
Change-Id: I473aad26b69d64efa09d2ec9d8e69f29f5cf4819
2013-07-24 18:40:14 +09:00
Jean Chalard
429db8d61e [FD3] Split stackNodes into two methods.
In the future we need to have a method that computes only
from the size, as we used to have, to initialize the cached
addresses, and a much simpler and faster method to copy
the cached sizes.

Bug: 8526576
Change-Id: I6a5a790303ab8f3bf957c7ca266eb12da7c1ad9e
2013-07-24 17:26:16 +09:00
Jean Chalard
91cbe3566d [FD2] Separate cached address before/after update for nodes.
Bug: 8526576
Change-Id: Ib9f8594a9e12dc75eba296faff2612c4bd7483d3
2013-07-23 17:52:54 +09:00
Jean Chalard
257750d988 [FD1] Move parents' address computation outside
There is no need to do it repeatedly in this loop: it's
clearer and faster to do it at the end only.

Bug: 8526576
Change-Id: I707571179c89479830891ec6d4fd06a9fffed7c1
2013-07-17 20:47:53 +09:00
Jean Chalard
c2e9c511cb Fix Binary dict tests
There are two problems here. The first one is the tests would send
an invalid unicode character. Although we could want dicttool to
handle this more gracefully, it's fine for now.

The second problem is much more serious. If a node has more than
128 children, then the java code will crash trying to read the
dictionary back because of a bug that this change fixes. In
theory, it's possible that happens when we try to load the user
history dictionary back from the disk - native code is not affected
so there is no other point that may cause a problem.
In the practice, that means you'd need to have 129 words with a
common prefix (including empty string) but all different after
this. It's almost impossible with Google Keyboard since there are
only so many keys on the keyboard that you can make a word out
of, and then again you'd have to do it repeatedly until it
actually enters the user history dictionary, wait for it to get
saved on the disk.
The bad news is, if you manage to get this far, the keyboard will
crash every time and won't be able to get up until you clear
data for the package.
The good news is, the dictionary itself is not corrupted and only
the reading code is wrong. So updating to a newer version would
actually even recover from this situation.

All in all, considering how almost-impossible this is to trigger,
I don't think even a single user actually did hit this bug.

Bug: 8583091
Change-Id: Iabb2a7f47cbd9ed3193d2a3487318d280753e071
2013-04-15 12:48:16 +09:00
Jean Chalard
ca0fdbbe2e Fix two bugs in dicttool
Both bugs only affect debug mode. One has the wrong object tested
with equals, the other has the iteration failing in some cases.

Change-Id: Ie9100d257a3f9e3be340cf3e38116f63417bdc1a
2013-04-10 22:10:31 +09:00
Jean Chalard
a411595b16 Fix two nasty bugs with surrogate pairs.
The important bug is in findWordInTree. The problem, which is
not obvious, is that we were calling codePointAt() with the
code point index in the string, instead of the char index.
The other bug this change fixes was harmless in the practice,
because it's in the iteration which is only used for debug and
pretty printing purposes. It's very similar in that it would
substract a length in code point to a length in chars and
truncate a StringBuilder at that length, so it would fail in a
quite similar manner. This changes the meaning of the "length"
attribute in Position, but it's clearer this way anyway.

Bug: 8450145
Change-Id: If396f883a9e6449de39351553ba83f5be5bd30f0
2013-04-01 17:06:19 +09:00
Jean Chalard
c6799ffeab Send the dictionaries descriptions to the dict pack
Bug: 8255795
Change-Id: I12a5922f50c2d2e3aa639457abcc1483e6a48721
2013-02-23 01:46:39 -08:00
Jean Chalard
2521edec09 Fix a bug with the passed dictionary id
We used to make the dictionary that we passed to the
dictionary pack as an initial value based on the locale.
This is wrong - it should be read from the dictionary.
This change fixes that.

Bug: 7005813
Change-Id: Ib08ed31dd9c216f6f7b9c6c3174ca514bf96e06f
2013-02-22 20:49:48 -08:00
Jean Chalard
af4a7e8c4b Create methods in LatinIME to make the current dict lists
Bug: 7005813
Change-Id: I82232af8e3071333b6fd01e4453b6b3c0a3ddb1f
2013-01-31 09:26:52 +09:00
Tadashi G. Takaoka
8aa9963a89 Fix Apache license comment
Change-Id: Ic56167f952a7f4449da366e1e81610e72c966086
2013-01-21 22:23:37 +09:00
Jean Chalard
fbc5e9b334 [AD3] Implement the interface to choose a local dictionary
Bug: 7702011
Change-Id: Id3b9c58dbbf5097e4d6ce986d20924eae19f9690
2013-01-21 15:40:46 +09:00
Jean Chalard
1d15fe7e51 [AD2] Add a helper method to read an arbitrary dict header
Bug: 7702011
Change-Id: Ib88f6dc222892831ae6932635b65fd2595b16b43
2013-01-18 20:34:28 +09:00
Ken Wakasa
b6ca354431 Small code cleanups
Multi-project commit with I249d5fbe

Change-Id: Ia28c4e970992aa1299a30e604eaa5d096655c3a5
2013-01-07 12:13:42 +09:00
Ken Wakasa
45239029ce Remove trailing spaces
Change-Id: I260b85ef9e91d17f97d6e405d2d92a65b443df44
2012-12-19 15:36:55 +09:00
Jean Chalard
f1b464da31 Remove a useless member
Change-Id: Id13e0aeec6ec3655d6bb0edc7f8f7821e7dc5a36
2012-12-11 19:15:24 +09:00
Jean Chalard
2da8866518 Remove a couple Eclipse and Android Lint warnings
Change-Id: I0c29c5d2abcbf80759b996d34b534deb083cd7d3
2012-12-06 21:30:51 +09:00
Jean Chalard
51a0ef8c59 Add a plumbing option to dicttool info.
Also align the `porcelain' option to the diff command
that was used mistakenly.

Bug: 7388665
Change-Id: Ic0e1b98c62ce37b2e909384a0370af4458563703
2012-10-31 16:35:22 +09:00
Jean Chalard
f41389a74b Remove warnings
Thanks Eclipse

Change-Id: I88e3979ed22be5d8be5a5accdde417c6b1a8bf2d
2012-10-29 14:24:16 +09:00
Jean Chalard
a23e333079 Implement the word-level diff (A9)
Bug: 7388857
Change-Id: I4c4560d4f4b579936a44cdf409a4c27300b65610
2012-10-29 12:31:22 +09:00
Jean Chalard
47cac57e45 Finish up the "info" command in dicttool. (A6)
Bug: 7388857
Change-Id: I704f12a6be76ce1644ec5e8dd3b667f112e9c04a
2012-10-25 19:15:24 +09:00
Jean Chalard
b3c98901c5 Add auto detection and decoding of dictionary files. (A2)
Bug: 7388852
Change-Id: I25e755fc15f5b383acc046f668e9681efa4f0c2f
2012-10-25 16:40:15 +09:00
Jean Chalard
ddb0bcc051 Fix a bug where a bigram would be ignored
Bug: 7403386
Change-Id: I89f495d07f7059a9f1ccd97d487c2f2657a8ebd2
2012-10-24 13:24:59 +09:00
Jean Chalard
c59c741987 Return the correct bigram frequency
The "correct" bigram frequency is now returned by the reading
code. However, as the binary format represents the frequency
in a lossy manner, the frequency is not guaranteed to be the
exact same as the one in the source text format - only a close
enough value. It is however the exact same value seen by the
native code.

Bug: 7395653
Change-Id: I49199ef18901c671189912b3550623e9643baedd
2012-10-23 17:17:37 +09:00
Tadashi G. Takaoka
15f6d4ae34 Add @UsedForTesting and @ExternallyReferenced annotations
Bug: 7268357
Change-Id: I0b7e0c19f04af9ae30874d0a4c26ad81bc80be8c
2012-10-22 11:18:43 -07:00
Yuichiro Hanada
d2579c4832 fix writeCharGroup.
Change-Id: Ib841afaba0a20c3b300eb7d3e9133243f9f3ae58
2012-10-05 14:54:17 +09:00
Yuichiro Hanada
3c6d9fe148 Add insertWord.
bug: 6669677

Change-Id: Ide55a4931071de9cd42c1cddae63ddd531d2feba
2012-10-04 17:19:47 +09:00
Yuichiro Hanada
c3a98ca306 Add writeNode.
Change-Id: I088bb6ea43ce0841d725e48b677d429e1155569d
2012-10-04 14:28:42 +09:00
Yuichiro Hanada
38712ff27d Add updateParentAddresses.
Change-Id: Iac210131b7c003ef363e1138bf22f777a37c6a89
2012-10-03 19:37:17 +09:00
Yuichiro Hanada
a853356b82 Add isDeletedGroup.
Change-Id: I83f09c068868e5e6e1b46f494a6ef957f0b466d8
2012-10-03 02:19:41 -07:00
Yuichiro Hanada
7223cc2ef1 Add MAX_BIGRAMS_IN_A_GROUP.
Change-Id: I128d5deb8e523045d7ad77d7a8fd3db944f71238
2012-10-03 18:10:06 +09:00
Yuichiro Hanada
4ad4ff618f Add makeCharGroupFlags.
Change-Id: Id2c580f21b77f66a97c5fbdf4542fdafe6c43614
2012-10-03 14:33:59 +09:00
Yuichiro Hanada
7f438aa12f Make writeCharGroup return a size of a new group.
bug: 6669677

Change-Id: I56f6a07b04b08443f2c052927404318c2018fc9d
2012-10-01 22:02:04 +09:00
Yuichiro Hanada
fb7e08ea8f Add writeCharGroup.
bug: 6669677

Change-Id: I36792ba9c511a5148c963096cc93ca8c2e0ee04e
2012-10-01 21:50:38 +09:00
Yuichiro Hanada
f3aed3ea26 Add updateChildrenAddress.
Change-Id: Ic06a755d85612476e719e580469dc1cd9447286c
2012-09-28 18:45:56 +09:00
Tadashi G. Takaoka
a28a05e971 Cleanup: Make some classes as final
Change-Id: I6009b3c1950ba32b7f1e205a3db2307fe0cd688e
2012-09-27 19:03:30 +09:00
Yuichiro Hanada
84d858ed5e Use BinaryDictInputOutput to save UserHistoryDictionary.
bug: 6669677

Change-Id: I08193c26f76dbd48168f8ac02c1b737525bfc7b2
2012-09-27 12:02:17 +09:00
Yuichiro Hanada
2aea34fb31 Add updateParentAddress.
bug: 6669677

Change-Id: I353f8ae53720cdf7a809271a28cb703709609f53
2012-09-26 17:18:01 +09:00
Yuichiro Hanada
2ee70804e9 Add moved char groups.
bug: 6669677

Change-Id: I372f841044fe8e076a50a80ac10b715e5f8fd4eb
2012-09-26 17:01:48 +09:00