nyancrimew/LatinIME
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
27622 commits 1 branch 0 tags 363 MiB
Commit graph

915 commits

Author SHA1 Message Date
Keisuke Kuroyanagi
4095177c4f Implement GC for terminal Id lookup table. 
Bug: 11073222
Change-Id: I8889ba808755ac39ae0abcfe2f6f0c4ee3a01a03
 2013-11-22 20:27:44 +09:00
Keisuke Kuroyanagi
f65dbee3b6 Implement ver4 dict shortcut writing. 
Bug: 11073222
Change-Id: I74247c104bdc2e26c77dd75bc28d2aa209296f63
 2013-11-22 19:37:35 +09:00
Keisuke Kuroyanagi
bcde771489 Implement ver4 dictionary GC. 
Bug: 11391130
Change-Id: I8a8c5b24b5922df49f27341a2f94d85513bb7166
 2013-11-22 19:37:33 +09:00
Keisuke Kuroyanagi
1b0e574830 Refactoring: Make GC listeners version independent. 
Bug: 11073222
Change-Id: I831517045edf934e09e9441deb9b57dbc78ff2ee
 2013-11-21 21:07:57 +09:00
Keisuke Kuroyanagi
9882fbb2ee Merge "Implement Ver4PatriciaTriePolicy::needsToRunGC()." 2013-11-21 08:40:26 +00:00
Keisuke Kuroyanagi
4af48b1a7a Implement Ver4PatriciaTriePolicy::needsToRunGC(). 
Bug: 11073222
Change-Id: Ia3fd1be94addb2ba76579e27763f84c5e5156643
 2013-11-21 16:12:09 +09:00
Keisuke Kuroyanagi
15c8b84af8 Merge "Implement Ver4PatriciaTriePolicy::flush()." 2013-11-20 05:35:16 +00:00
Keisuke Kuroyanagi
dc14d5fbba Implement Ver4PatriciaTriePolicy::flush(). 
Bug: 11073222
Change-Id: I7da5f7f8b7918ce6cc35d36e1ce993840cad797c
 2013-11-20 14:12:04 +09:00
Keisuke Kuroyanagi
36c3075c7d Reduce MAX_DICT_EXTENDED_REGION_SIZE to 32K. 
Bug: 11740462
Change-Id: I21cb956c692a1bcc5515f47b904f0c9b3dfdc3fd
 2013-11-18 23:45:45 +09:00
Keisuke Kuroyanagi
80bd4a7585 Implement writing empty ver4 dictionary to file. 
Bug: 11073222
Change-Id: Ic1a9295953d091c8e8982264ffff15669c30544c
 2013-11-18 14:57:53 +09:00
Keisuke Kuroyanagi
2687484585 Create empty ver4 dictionary buffer on memory. 
Bug: 11073222
Change-Id: Ia1f4d7782e019fdc6a4162f942f433d8b150b49d
 2013-11-13 16:40:04 +09:00
Keisuke Kuroyanagi
ff4b1d1fd0 Implement ver4 dictionary bigram removing methods. 
Bug: 11073222
Change-Id: I0c42e283d1ff983dc2c51e91b167bc39cdfd3da8
 2013-11-13 16:40:01 +09:00
Keisuke Kuroyanagi
fe05173198 Merge "Implement ver4 dictionary bigram writing methods." 2013-11-13 05:23:57 +00:00
Keisuke Kuroyanagi
99808b7b0d Implement ver4 dictionary bigram writing methods. 
Bug: 11073222
Change-Id: Ibb0e697fe14dac1edb1f3074fb59e2cdc91039d4
 2013-11-12 21:10:01 +09:00
Keisuke Kuroyanagi
180419b20c Merge "Implement ver4 dictionary unigram writing methods." 2013-11-07 09:42:40 +00:00
Keisuke Kuroyanagi
3fe9458b6d Implement ver4 dictionary unigram writing methods. 
Bug: 11073222
Change-Id: Ibdb6846fee98919bb5f845170c19d7d571fcb88d
 2013-11-07 17:58:27 +09:00
Keisuke Kuroyanagi
fc1824ea55 Merge "Use sysconf(_SC_PAGESIZE) instead of getpagesize()." 2013-11-07 06:19:48 +00:00
Keisuke Kuroyanagi
75534b98ca Use sysconf(_SC_PAGESIZE) instead of getpagesize(). 
Bug: 11563061
Change-Id: I1350f0b1cb8e2bf17cdce34b7429763bbb6b2856
 2013-11-07 14:53:20 +09:00
Keisuke Kuroyanagi
956ca35157 Refactoring: Separate updating methods from writing methods. 
Bug: 11073222

Change-Id: I5e9df7276a8c167f31423bebf30bd26bde74fc92
 2013-11-06 19:22:37 +09:00
Keisuke Kuroyanagi
0be85cfdf7 Refactoring: PtNodeWriter to separate version specific writing methods. 
Bug: 11073222

Change-Id: Id2ade16316f33e17ead5faf45404a2054c303d74
 2013-11-06 19:22:34 +09:00
Keisuke Kuroyanagi
b460c85589 Implement ver4 shortcut reading method. 
Bug: 11073222
Change-Id: I3738c7ce07f500920bde8d3f985cf6e8ecb40b6a
 2013-11-06 18:09:35 +09:00
Keisuke Kuroyanagi
9b3e59d644 Implement ver4 bigram reading method. 
Bug: 11073222
Change-Id: I7b3408938f304da361201892e0a1342fdf92e62e
 2013-11-05 16:00:26 +09:00
Keisuke Kuroyanagi
6e15f8a17f Implement TerminalPositionLookupTable. 
Bug: 11073222
Change-Id: I2ffe8ffc290fb69afc0eca35b5fed7157f5108fd
 2013-11-01 17:54:23 -07:00
Keisuke Kuroyanagi
1c62341de6 Implement SparseTable in native code. 
Bug: 11073222
Change-Id: Ia2cbb2fecfae0c9f621600483d96a55aef75c1b8
 2013-11-01 11:03:41 -07:00
Keisuke Kuroyanagi
b3691b5642 Implement ver4 dict PtNode reading methods. 
Bug: 11073222
Change-Id: I03af717ff02a621d9be8eb554d8f1438a500339b
 2013-10-31 11:01:41 -07:00
Keisuke Kuroyanagi
ecbd8af169 Implement ProbabilityDictContent. 
Bug: 11073222
Change-Id: Ia57c940fe3507a53b9d32aa6b9ebc5581c08d11f
 2013-10-28 21:41:41 -07:00
Keisuke Kuroyanagi
b8a1e8fa33 Add uint reading method to BufferWithExtendableBuffer. 
Bug: 11073222
Change-Id: Iff503cba883dd6f4ce9079783ad6a2edf464ffba
 2013-10-28 15:32:57 -07:00
Keisuke Kuroyanagi
41b77116d3 Merge "Refactoring: Move some methods to the reading helper." 2013-10-28 22:12:06 +00:00
Keisuke Kuroyanagi
2271c23dc6 Open dictionary files always using O_RDONLY. 
We never directly write to opened and mmapped files.

Bug: 11391620

Change-Id: Ic52fbe6018b8b3c1b6ca687461f29185809217c2
 2013-10-25 17:35:37 -07:00
Keisuke Kuroyanagi
3f0a8e3fb0 Refactoring: Move some methods to the reading helper. 
To handle multiple dictionary formats in helpers.

Bug: 11073222
Change-Id: I376bbdea934e13588feca02b9865105742f1a058
 2013-10-25 12:32:36 -07:00
Keisuke Kuroyanagi
c481d0556f Refactoring: Introduce PtNode and PtNodeReader. 
To handle multiple dictionary formats in helpers.

Bug: 11073222

Change-Id: Iaef7be08534f9010e837ffcf8c8292b174b64d2b
 2013-10-25 11:18:55 -07:00
Keisuke Kuroyanagi
75d7f0fbf5 Introduce Ver4DictBuffers to handle multiple buffers. 
Bug: 11073222
Change-Id: Icacca8133c0030b19ca826fbf9525f41a1ac6cc8
 2013-10-24 23:45:01 -07:00
Keisuke Kuroyanagi
1e34cc1698 Introduce DictContent for ver4 dict. 
Bug: 11073222
Change-Id: I88ab3948a98388931d81c97825c9d2c76e15a44b
 2013-10-24 17:42:56 -07:00
Keisuke Kuroyanagi
e5cfb77694 Introduce ExclusiveOwnershipPointer. 
Bug: 11073222
Change-Id: I841527f0247f2ade07796406729b3a4a8958aa57
 2013-10-24 17:28:30 -07:00
Keisuke Kuroyanagi
4fd413b7e4 Merge "Tracking minor errors to categorize exact match." 2013-10-21 12:32:03 +00:00
Keisuke Kuroyanagi
eb5049b28d Tracking minor errors to categorize exact match. 
Bug: 8844931

Change-Id: I012529f5fc0d094ef64a41c075fe548b6b439265
 2013-10-21 21:01:59 +09:00
Keisuke Kuroyanagi
0913aaa56f Merge "Add utility methods related to buffers for ver4 dict." 2013-10-21 11:03:08 +00:00
Keisuke Kuroyanagi
72877b15ea Add utility methods related to buffers for ver4 dict. 
Bug: 11073222
Change-Id: Ieca15ffa09fd36c6602f6d2afab5779800c5a422
 2013-10-21 19:56:50 +09:00
Keisuke Kuroyanagi
53686e0f5f Fix: Suggested words from user history are invalid. 
- Suggestions form user history can contain invalid words.
- isValidWord always returns false.

Bug: 11139426
Change-Id: I6075b275603332ddb00f4a9284afcaa82d824270
 2013-10-18 19:53:57 +09:00
Keisuke Kuroyanagi
2628268ef4 Add ver4 dictionary structure policy file. 
Bug: 11073222
Change-Id: Ia9e1a4842d42a581812a8c3dd4a85e7ee86dd529
 2013-10-16 18:46:32 +09:00
Keisuke Kuroyanagi
427d69e218 Moving files for dictionary structure policies. 
Bug: 11073222
Change-Id: Ic40f6258e2b0c2790d526aff31312c112760839d
 2013-10-16 18:46:32 +09:00
Keisuke Kuroyanagi
c1fce1a720 Make "node"s clear by renaming to PtNode or DicNode. 
Bug: 10233675
Change-Id: I248b927c724253f593e3806e82b4e1fed2c025fb
 2013-10-16 18:46:28 +09:00
Keisuke Kuroyanagi
e1bd7d5d39 Initial step to support version 4 format in native code. 
Bug: 11073222
Change-Id: I633229381b28cef23edef2b37e5b0bf6e15af468
 2013-10-09 12:54:33 +09:00
Keisuke Kuroyanagi
6d91e4ce74 Fix offdevice test build. 
Change-Id: I42143bdc4f6f5aa4f55530de84819ab722a22f72
 2013-10-08 16:05:42 +09:00
Keisuke Kuroyanagi
6d3d53955e Merge "Fix a bug: wrong parent position is written when moving." 2013-10-08 04:48:53 +00:00
Keisuke Kuroyanagi
802e6383ea Fix a bug: wrong parent position is written when moving. 
Change-Id: I93d33ba9105eb85327108cca5706d8db3399de6a
 2013-10-08 12:44:00 +09:00
Jean Chalard
86f7711407 Merge "Don't auto-correct to non-whitelist shortcuts" 2013-10-07 10:04:18 +00:00
Jean Chalard
f309f2288b Don't auto-correct to non-whitelist shortcuts 
Bug: 11076722
Change-Id: I0de10ab24d33ec31f45664fe92d80d5a7bec9c20
 2013-10-07 19:01:57 +09:00
Keisuke Kuroyanagi
cfb018ba6d Fix bug and Add large test for decaying dictionary. 
- GC gets failure when the dictionary become empty.
- Useless unigrams are sometimes not removed.

Bug: 10197478
Change-Id: I8d1479c01efba61a81f03bc077da6bcb4797a940
 2013-10-07 17:31:01 +09:00
Keisuke Kuroyanagi
15b1a0fd14 Parameter optimization. 
[Category diff]
+1     194
-1      21
+2      15
-2       1
+3       1
-3      15
+4     152
-4      90
+5     108
-5     197
+6      94
-6     136
+7      28
-7     132

[Weighted category diff]
+1     230
-1      23
+2      15
-2       1
+3       1
-3      15
+4     180
-4     106
+5     122
-5     224
+6     107
-6     158
+7      29
-7     157

show diff for ./en_user_log_phones_2011_08.csv
+1     171
-1       9
+2       7
-2       1
+3       1
-3       7
+4     123
-4      76
+5      84
-5     157
+6      66
-6     119
+7      20
-7     103

Before
---- count ----
Count(1):75.64%, 177913
Count(2):1.06%, 2510
Count(3):0.56%, 1319
Count(4):6.63%, 15612
Count(5):9.87%, 23223
Count(6):2.79%, 6580
Count(7):3.41%, 8038
---- weight ----
Weight(1):96.88%, 2009907
Weight(2):0.24%, 5157
Weight(3):0.09%, 2025
Weight(4):0.77%, 16095
Weight(5):1.11%, 23077
Weight(6):0.39%, 8172
Weight(7):0.48%, 10146
---- score ----
Conservative score = 66.406
Normal score = 81.512
Agressive score = 89.064
Insane score = 92.841

After
---- count ----
Count(1):75.71%, 178087
Count(2):1.07%, 2524
Count(3):0.55%, 1305
Count(4):6.66%, 15677
Count(5):9.83%, 23133
Count(6):2.78%, 6539
Count(7):3.37%, 7930
---- weight ----
Weight(1):96.89%, 2010118
Weight(2):0.24%, 5171
Weight(3):0.09%, 2011
Weight(4):0.78%, 16184
Weight(5):1.1%, 22972
Weight(6):0.39%, 8109
Weight(7):0.48%, 10014
---- score ----
Conservative score = 66.431
Normal score = 81.531
Agressive score = 89.08
Insane score = 92.855

No diffs for 10+ freq user log.

Change-Id: Ica4a7604507a2c91aaa7daf5bacf4fce4039f547
 2013-10-07 12:39:51 +09:00