nyancrimew/LatinIME
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
27606 commits 1 branch 0 tags 363 MiB
Commit graph

911 commits

Author SHA1 Message Date
Keisuke Kuroyanagi
9882fbb2ee Merge "Implement Ver4PatriciaTriePolicy::needsToRunGC()." 2013-11-21 08:40:26 +00:00
Keisuke Kuroyanagi
4af48b1a7a Implement Ver4PatriciaTriePolicy::needsToRunGC(). 
Bug: 11073222
Change-Id: Ia3fd1be94addb2ba76579e27763f84c5e5156643
 2013-11-21 16:12:09 +09:00
Keisuke Kuroyanagi
15c8b84af8 Merge "Implement Ver4PatriciaTriePolicy::flush()." 2013-11-20 05:35:16 +00:00
Keisuke Kuroyanagi
dc14d5fbba Implement Ver4PatriciaTriePolicy::flush(). 
Bug: 11073222
Change-Id: I7da5f7f8b7918ce6cc35d36e1ce993840cad797c
 2013-11-20 14:12:04 +09:00
Keisuke Kuroyanagi
36c3075c7d Reduce MAX_DICT_EXTENDED_REGION_SIZE to 32K. 
Bug: 11740462
Change-Id: I21cb956c692a1bcc5515f47b904f0c9b3dfdc3fd
 2013-11-18 23:45:45 +09:00
Keisuke Kuroyanagi
80bd4a7585 Implement writing empty ver4 dictionary to file. 
Bug: 11073222
Change-Id: Ic1a9295953d091c8e8982264ffff15669c30544c
 2013-11-18 14:57:53 +09:00
Keisuke Kuroyanagi
2687484585 Create empty ver4 dictionary buffer on memory. 
Bug: 11073222
Change-Id: Ia1f4d7782e019fdc6a4162f942f433d8b150b49d
 2013-11-13 16:40:04 +09:00
Keisuke Kuroyanagi
ff4b1d1fd0 Implement ver4 dictionary bigram removing methods. 
Bug: 11073222
Change-Id: I0c42e283d1ff983dc2c51e91b167bc39cdfd3da8
 2013-11-13 16:40:01 +09:00
Keisuke Kuroyanagi
fe05173198 Merge "Implement ver4 dictionary bigram writing methods." 2013-11-13 05:23:57 +00:00
Keisuke Kuroyanagi
99808b7b0d Implement ver4 dictionary bigram writing methods. 
Bug: 11073222
Change-Id: Ibb0e697fe14dac1edb1f3074fb59e2cdc91039d4
 2013-11-12 21:10:01 +09:00
Keisuke Kuroyanagi
180419b20c Merge "Implement ver4 dictionary unigram writing methods." 2013-11-07 09:42:40 +00:00
Keisuke Kuroyanagi
3fe9458b6d Implement ver4 dictionary unigram writing methods. 
Bug: 11073222
Change-Id: Ibdb6846fee98919bb5f845170c19d7d571fcb88d
 2013-11-07 17:58:27 +09:00
Keisuke Kuroyanagi
fc1824ea55 Merge "Use sysconf(_SC_PAGESIZE) instead of getpagesize()." 2013-11-07 06:19:48 +00:00
Keisuke Kuroyanagi
75534b98ca Use sysconf(_SC_PAGESIZE) instead of getpagesize(). 
Bug: 11563061
Change-Id: I1350f0b1cb8e2bf17cdce34b7429763bbb6b2856
 2013-11-07 14:53:20 +09:00
Keisuke Kuroyanagi
956ca35157 Refactoring: Separate updating methods from writing methods. 
Bug: 11073222

Change-Id: I5e9df7276a8c167f31423bebf30bd26bde74fc92
 2013-11-06 19:22:37 +09:00
Keisuke Kuroyanagi
0be85cfdf7 Refactoring: PtNodeWriter to separate version specific writing methods. 
Bug: 11073222

Change-Id: Id2ade16316f33e17ead5faf45404a2054c303d74
 2013-11-06 19:22:34 +09:00
Keisuke Kuroyanagi
b460c85589 Implement ver4 shortcut reading method. 
Bug: 11073222
Change-Id: I3738c7ce07f500920bde8d3f985cf6e8ecb40b6a
 2013-11-06 18:09:35 +09:00
Keisuke Kuroyanagi
9b3e59d644 Implement ver4 bigram reading method. 
Bug: 11073222
Change-Id: I7b3408938f304da361201892e0a1342fdf92e62e
 2013-11-05 16:00:26 +09:00
Keisuke Kuroyanagi
6e15f8a17f Implement TerminalPositionLookupTable. 
Bug: 11073222
Change-Id: I2ffe8ffc290fb69afc0eca35b5fed7157f5108fd
 2013-11-01 17:54:23 -07:00
Keisuke Kuroyanagi
1c62341de6 Implement SparseTable in native code. 
Bug: 11073222
Change-Id: Ia2cbb2fecfae0c9f621600483d96a55aef75c1b8
 2013-11-01 11:03:41 -07:00
Keisuke Kuroyanagi
b3691b5642 Implement ver4 dict PtNode reading methods. 
Bug: 11073222
Change-Id: I03af717ff02a621d9be8eb554d8f1438a500339b
 2013-10-31 11:01:41 -07:00
Keisuke Kuroyanagi
ecbd8af169 Implement ProbabilityDictContent. 
Bug: 11073222
Change-Id: Ia57c940fe3507a53b9d32aa6b9ebc5581c08d11f
 2013-10-28 21:41:41 -07:00
Keisuke Kuroyanagi
b8a1e8fa33 Add uint reading method to BufferWithExtendableBuffer. 
Bug: 11073222
Change-Id: Iff503cba883dd6f4ce9079783ad6a2edf464ffba
 2013-10-28 15:32:57 -07:00
Keisuke Kuroyanagi
41b77116d3 Merge "Refactoring: Move some methods to the reading helper." 2013-10-28 22:12:06 +00:00
Keisuke Kuroyanagi
2271c23dc6 Open dictionary files always using O_RDONLY. 
We never directly write to opened and mmapped files.

Bug: 11391620

Change-Id: Ic52fbe6018b8b3c1b6ca687461f29185809217c2
 2013-10-25 17:35:37 -07:00
Keisuke Kuroyanagi
3f0a8e3fb0 Refactoring: Move some methods to the reading helper. 
To handle multiple dictionary formats in helpers.

Bug: 11073222
Change-Id: I376bbdea934e13588feca02b9865105742f1a058
 2013-10-25 12:32:36 -07:00
Keisuke Kuroyanagi
c481d0556f Refactoring: Introduce PtNode and PtNodeReader. 
To handle multiple dictionary formats in helpers.

Bug: 11073222

Change-Id: Iaef7be08534f9010e837ffcf8c8292b174b64d2b
 2013-10-25 11:18:55 -07:00
Keisuke Kuroyanagi
75d7f0fbf5 Introduce Ver4DictBuffers to handle multiple buffers. 
Bug: 11073222
Change-Id: Icacca8133c0030b19ca826fbf9525f41a1ac6cc8
 2013-10-24 23:45:01 -07:00
Keisuke Kuroyanagi
1e34cc1698 Introduce DictContent for ver4 dict. 
Bug: 11073222
Change-Id: I88ab3948a98388931d81c97825c9d2c76e15a44b
 2013-10-24 17:42:56 -07:00
Keisuke Kuroyanagi
e5cfb77694 Introduce ExclusiveOwnershipPointer. 
Bug: 11073222
Change-Id: I841527f0247f2ade07796406729b3a4a8958aa57
 2013-10-24 17:28:30 -07:00
Keisuke Kuroyanagi
4fd413b7e4 Merge "Tracking minor errors to categorize exact match." 2013-10-21 12:32:03 +00:00
Keisuke Kuroyanagi
eb5049b28d Tracking minor errors to categorize exact match. 
Bug: 8844931

Change-Id: I012529f5fc0d094ef64a41c075fe548b6b439265
 2013-10-21 21:01:59 +09:00
Keisuke Kuroyanagi
0913aaa56f Merge "Add utility methods related to buffers for ver4 dict." 2013-10-21 11:03:08 +00:00
Keisuke Kuroyanagi
72877b15ea Add utility methods related to buffers for ver4 dict. 
Bug: 11073222
Change-Id: Ieca15ffa09fd36c6602f6d2afab5779800c5a422
 2013-10-21 19:56:50 +09:00
Keisuke Kuroyanagi
53686e0f5f Fix: Suggested words from user history are invalid. 
- Suggestions form user history can contain invalid words.
- isValidWord always returns false.

Bug: 11139426
Change-Id: I6075b275603332ddb00f4a9284afcaa82d824270
 2013-10-18 19:53:57 +09:00
Keisuke Kuroyanagi
2628268ef4 Add ver4 dictionary structure policy file. 
Bug: 11073222
Change-Id: Ia9e1a4842d42a581812a8c3dd4a85e7ee86dd529
 2013-10-16 18:46:32 +09:00
Keisuke Kuroyanagi
427d69e218 Moving files for dictionary structure policies. 
Bug: 11073222
Change-Id: Ic40f6258e2b0c2790d526aff31312c112760839d
 2013-10-16 18:46:32 +09:00
Keisuke Kuroyanagi
c1fce1a720 Make "node"s clear by renaming to PtNode or DicNode. 
Bug: 10233675
Change-Id: I248b927c724253f593e3806e82b4e1fed2c025fb
 2013-10-16 18:46:28 +09:00
Keisuke Kuroyanagi
e1bd7d5d39 Initial step to support version 4 format in native code. 
Bug: 11073222
Change-Id: I633229381b28cef23edef2b37e5b0bf6e15af468
 2013-10-09 12:54:33 +09:00
Keisuke Kuroyanagi
6d91e4ce74 Fix offdevice test build. 
Change-Id: I42143bdc4f6f5aa4f55530de84819ab722a22f72
 2013-10-08 16:05:42 +09:00
Keisuke Kuroyanagi
6d3d53955e Merge "Fix a bug: wrong parent position is written when moving." 2013-10-08 04:48:53 +00:00
Keisuke Kuroyanagi
802e6383ea Fix a bug: wrong parent position is written when moving. 
Change-Id: I93d33ba9105eb85327108cca5706d8db3399de6a
 2013-10-08 12:44:00 +09:00
Jean Chalard
86f7711407 Merge "Don't auto-correct to non-whitelist shortcuts" 2013-10-07 10:04:18 +00:00
Jean Chalard
f309f2288b Don't auto-correct to non-whitelist shortcuts 
Bug: 11076722
Change-Id: I0de10ab24d33ec31f45664fe92d80d5a7bec9c20
 2013-10-07 19:01:57 +09:00
Keisuke Kuroyanagi
cfb018ba6d Fix bug and Add large test for decaying dictionary. 
- GC gets failure when the dictionary become empty.
- Useless unigrams are sometimes not removed.

Bug: 10197478
Change-Id: I8d1479c01efba61a81f03bc077da6bcb4797a940
 2013-10-07 17:31:01 +09:00
Keisuke Kuroyanagi
15b1a0fd14 Parameter optimization. 
[Category diff]
+1     194
-1      21
+2      15
-2       1
+3       1
-3      15
+4     152
-4      90
+5     108
-5     197
+6      94
-6     136
+7      28
-7     132

[Weighted category diff]
+1     230
-1      23
+2      15
-2       1
+3       1
-3      15
+4     180
-4     106
+5     122
-5     224
+6     107
-6     158
+7      29
-7     157

show diff for ./en_user_log_phones_2011_08.csv
+1     171
-1       9
+2       7
-2       1
+3       1
-3       7
+4     123
-4      76
+5      84
-5     157
+6      66
-6     119
+7      20
-7     103

Before
---- count ----
Count(1):75.64%, 177913
Count(2):1.06%, 2510
Count(3):0.56%, 1319
Count(4):6.63%, 15612
Count(5):9.87%, 23223
Count(6):2.79%, 6580
Count(7):3.41%, 8038
---- weight ----
Weight(1):96.88%, 2009907
Weight(2):0.24%, 5157
Weight(3):0.09%, 2025
Weight(4):0.77%, 16095
Weight(5):1.11%, 23077
Weight(6):0.39%, 8172
Weight(7):0.48%, 10146
---- score ----
Conservative score = 66.406
Normal score = 81.512
Agressive score = 89.064
Insane score = 92.841

After
---- count ----
Count(1):75.71%, 178087
Count(2):1.07%, 2524
Count(3):0.55%, 1305
Count(4):6.66%, 15677
Count(5):9.83%, 23133
Count(6):2.78%, 6539
Count(7):3.37%, 7930
---- weight ----
Weight(1):96.89%, 2010118
Weight(2):0.24%, 5171
Weight(3):0.09%, 2011
Weight(4):0.78%, 16184
Weight(5):1.1%, 22972
Weight(6):0.39%, 8109
Weight(7):0.48%, 10014
---- score ----
Conservative score = 66.431
Normal score = 81.531
Agressive score = 89.08
Insane score = 92.855

No diffs for 10+ freq user log.

Change-Id: Ica4a7604507a2c91aaa7daf5bacf4fce4039f547
 2013-10-07 12:39:51 +09:00
Keisuke Kuroyanagi
3e5f4b53b0 Merge "Check the elapsed time for decaying." 2013-10-04 07:16:17 +00:00
Keisuke Kuroyanagi
67c855ea6f Check the elapsed time for decaying. 
Bug: 6669677
Change-Id: I0401a4c0908c702ce65abfa5e017dd16cb1296f9
 2013-10-04 16:11:11 +09:00
Xiaojun Bi
0b62621446 Merge "Optimize the parameters for the typing algorithm" 2013-10-04 03:25:43 +00:00
Xiaojun Bi
2ed09c214a Optimize the parameters for the typing algorithm 
The new set of parameters are optimized for both
keystroke savings and regression test scores.

This change is corresponding to the patch 6 of
Change-Id: Ib8439a85de26cbbba58837344717f0acbfcf0ab0
in the regression test repository.

Here is the link to the doc analyzing the
“nnow → know” case:
https://docs.google.com/a/google.com/document/d/1XIHfAHLlP1f3F8R-u7yxPGWcP58tokygzN_lQKXFFgM/edit

Here is the link to the spreadsheet showing
the diff of the words with frequencies >= 10
in en_user_log_phones_2011_08.csv.
https://docs.google.com/a/google.com/spreadsheet/ccc?key=0Am_RQlV4zC6SdFFVTlhJT0RLUENCWWZOZ3JQTUpMQ2c&usp=drive_web#gid=0

Updated TRT Results for this change:
[Category diff]
+1     783
-1     717
+2      20
-2      34
+3      34
-3      20
+4     575
-4     777
+5     765
-5     755
+6     785
-6     666
+7     479
-7     472

[Weighted category diff]
+1     999
-1     965
+2      21
-2      35
+3      35
-3      21
+4     804
-4     980
+5     970
-5     971
+6     954
-6     831
+7     645
-7     625

Bug: 10613502

Change-Id: I5cd447673a196ad5af641a2118602bec879bdefb
 2013-10-03 19:30:26 -07:00