nyancrimew/LatinIME
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
28847 commits 1 branch 0 tags 363 MiB
Commit graph

890 commits

Author SHA1 Message Date
Keisuke Kuroyanagi
ecbd8af169 Implement ProbabilityDictContent. 
Bug: 11073222
Change-Id: Ia57c940fe3507a53b9d32aa6b9ebc5581c08d11f
 2013-10-28 21:41:41 -07:00
Keisuke Kuroyanagi
b8a1e8fa33 Add uint reading method to BufferWithExtendableBuffer. 
Bug: 11073222
Change-Id: Iff503cba883dd6f4ce9079783ad6a2edf464ffba
 2013-10-28 15:32:57 -07:00
Keisuke Kuroyanagi
41b77116d3 Merge "Refactoring: Move some methods to the reading helper." 2013-10-28 22:12:06 +00:00
Keisuke Kuroyanagi
2271c23dc6 Open dictionary files always using O_RDONLY. 
We never directly write to opened and mmapped files.

Bug: 11391620

Change-Id: Ic52fbe6018b8b3c1b6ca687461f29185809217c2
 2013-10-25 17:35:37 -07:00
Keisuke Kuroyanagi
3f0a8e3fb0 Refactoring: Move some methods to the reading helper. 
To handle multiple dictionary formats in helpers.

Bug: 11073222
Change-Id: I376bbdea934e13588feca02b9865105742f1a058
 2013-10-25 12:32:36 -07:00
Keisuke Kuroyanagi
c481d0556f Refactoring: Introduce PtNode and PtNodeReader. 
To handle multiple dictionary formats in helpers.

Bug: 11073222

Change-Id: Iaef7be08534f9010e837ffcf8c8292b174b64d2b
 2013-10-25 11:18:55 -07:00
Keisuke Kuroyanagi
75d7f0fbf5 Introduce Ver4DictBuffers to handle multiple buffers. 
Bug: 11073222
Change-Id: Icacca8133c0030b19ca826fbf9525f41a1ac6cc8
 2013-10-24 23:45:01 -07:00
Keisuke Kuroyanagi
1e34cc1698 Introduce DictContent for ver4 dict. 
Bug: 11073222
Change-Id: I88ab3948a98388931d81c97825c9d2c76e15a44b
 2013-10-24 17:42:56 -07:00
Keisuke Kuroyanagi
e5cfb77694 Introduce ExclusiveOwnershipPointer. 
Bug: 11073222
Change-Id: I841527f0247f2ade07796406729b3a4a8958aa57
 2013-10-24 17:28:30 -07:00
Keisuke Kuroyanagi
4fd413b7e4 Merge "Tracking minor errors to categorize exact match." 2013-10-21 12:32:03 +00:00
Keisuke Kuroyanagi
eb5049b28d Tracking minor errors to categorize exact match. 
Bug: 8844931

Change-Id: I012529f5fc0d094ef64a41c075fe548b6b439265
 2013-10-21 21:01:59 +09:00
Keisuke Kuroyanagi
0913aaa56f Merge "Add utility methods related to buffers for ver4 dict." 2013-10-21 11:03:08 +00:00
Keisuke Kuroyanagi
72877b15ea Add utility methods related to buffers for ver4 dict. 
Bug: 11073222
Change-Id: Ieca15ffa09fd36c6602f6d2afab5779800c5a422
 2013-10-21 19:56:50 +09:00
Keisuke Kuroyanagi
53686e0f5f Fix: Suggested words from user history are invalid. 
- Suggestions form user history can contain invalid words.
- isValidWord always returns false.

Bug: 11139426
Change-Id: I6075b275603332ddb00f4a9284afcaa82d824270
 2013-10-18 19:53:57 +09:00
Keisuke Kuroyanagi
2628268ef4 Add ver4 dictionary structure policy file. 
Bug: 11073222
Change-Id: Ia9e1a4842d42a581812a8c3dd4a85e7ee86dd529
 2013-10-16 18:46:32 +09:00
Keisuke Kuroyanagi
427d69e218 Moving files for dictionary structure policies. 
Bug: 11073222
Change-Id: Ic40f6258e2b0c2790d526aff31312c112760839d
 2013-10-16 18:46:32 +09:00
Keisuke Kuroyanagi
c1fce1a720 Make "node"s clear by renaming to PtNode or DicNode. 
Bug: 10233675
Change-Id: I248b927c724253f593e3806e82b4e1fed2c025fb
 2013-10-16 18:46:28 +09:00
Keisuke Kuroyanagi
e1bd7d5d39 Initial step to support version 4 format in native code. 
Bug: 11073222
Change-Id: I633229381b28cef23edef2b37e5b0bf6e15af468
 2013-10-09 12:54:33 +09:00
Keisuke Kuroyanagi
6d91e4ce74 Fix offdevice test build. 
Change-Id: I42143bdc4f6f5aa4f55530de84819ab722a22f72
 2013-10-08 16:05:42 +09:00
Keisuke Kuroyanagi
6d3d53955e Merge "Fix a bug: wrong parent position is written when moving." 2013-10-08 04:48:53 +00:00
Keisuke Kuroyanagi
802e6383ea Fix a bug: wrong parent position is written when moving. 
Change-Id: I93d33ba9105eb85327108cca5706d8db3399de6a
 2013-10-08 12:44:00 +09:00
Jean Chalard
86f7711407 Merge "Don't auto-correct to non-whitelist shortcuts" 2013-10-07 10:04:18 +00:00
Jean Chalard
f309f2288b Don't auto-correct to non-whitelist shortcuts 
Bug: 11076722
Change-Id: I0de10ab24d33ec31f45664fe92d80d5a7bec9c20
 2013-10-07 19:01:57 +09:00
Keisuke Kuroyanagi
cfb018ba6d Fix bug and Add large test for decaying dictionary. 
- GC gets failure when the dictionary become empty.
- Useless unigrams are sometimes not removed.

Bug: 10197478
Change-Id: I8d1479c01efba61a81f03bc077da6bcb4797a940
 2013-10-07 17:31:01 +09:00
Keisuke Kuroyanagi
15b1a0fd14 Parameter optimization. 
[Category diff]
+1     194
-1      21
+2      15
-2       1
+3       1
-3      15
+4     152
-4      90
+5     108
-5     197
+6      94
-6     136
+7      28
-7     132

[Weighted category diff]
+1     230
-1      23
+2      15
-2       1
+3       1
-3      15
+4     180
-4     106
+5     122
-5     224
+6     107
-6     158
+7      29
-7     157

show diff for ./en_user_log_phones_2011_08.csv
+1     171
-1       9
+2       7
-2       1
+3       1
-3       7
+4     123
-4      76
+5      84
-5     157
+6      66
-6     119
+7      20
-7     103

Before
---- count ----
Count(1):75.64%, 177913
Count(2):1.06%, 2510
Count(3):0.56%, 1319
Count(4):6.63%, 15612
Count(5):9.87%, 23223
Count(6):2.79%, 6580
Count(7):3.41%, 8038
---- weight ----
Weight(1):96.88%, 2009907
Weight(2):0.24%, 5157
Weight(3):0.09%, 2025
Weight(4):0.77%, 16095
Weight(5):1.11%, 23077
Weight(6):0.39%, 8172
Weight(7):0.48%, 10146
---- score ----
Conservative score = 66.406
Normal score = 81.512
Agressive score = 89.064
Insane score = 92.841

After
---- count ----
Count(1):75.71%, 178087
Count(2):1.07%, 2524
Count(3):0.55%, 1305
Count(4):6.66%, 15677
Count(5):9.83%, 23133
Count(6):2.78%, 6539
Count(7):3.37%, 7930
---- weight ----
Weight(1):96.89%, 2010118
Weight(2):0.24%, 5171
Weight(3):0.09%, 2011
Weight(4):0.78%, 16184
Weight(5):1.1%, 22972
Weight(6):0.39%, 8109
Weight(7):0.48%, 10014
---- score ----
Conservative score = 66.431
Normal score = 81.531
Agressive score = 89.08
Insane score = 92.855

No diffs for 10+ freq user log.

Change-Id: Ica4a7604507a2c91aaa7daf5bacf4fce4039f547
 2013-10-07 12:39:51 +09:00
Keisuke Kuroyanagi
3e5f4b53b0 Merge "Check the elapsed time for decaying." 2013-10-04 07:16:17 +00:00
Keisuke Kuroyanagi
67c855ea6f Check the elapsed time for decaying. 
Bug: 6669677
Change-Id: I0401a4c0908c702ce65abfa5e017dd16cb1296f9
 2013-10-04 16:11:11 +09:00
Xiaojun Bi
0b62621446 Merge "Optimize the parameters for the typing algorithm" 2013-10-04 03:25:43 +00:00
Xiaojun Bi
2ed09c214a Optimize the parameters for the typing algorithm 
The new set of parameters are optimized for both
keystroke savings and regression test scores.

This change is corresponding to the patch 6 of
Change-Id: Ib8439a85de26cbbba58837344717f0acbfcf0ab0
in the regression test repository.

Here is the link to the doc analyzing the
“nnow → know” case:
https://docs.google.com/a/google.com/document/d/1XIHfAHLlP1f3F8R-u7yxPGWcP58tokygzN_lQKXFFgM/edit

Here is the link to the spreadsheet showing
the diff of the words with frequencies >= 10
in en_user_log_phones_2011_08.csv.
https://docs.google.com/a/google.com/spreadsheet/ccc?key=0Am_RQlV4zC6SdFFVTlhJT0RLUENCWWZOZ3JQTUpMQ2c&usp=drive_web#gid=0

Updated TRT Results for this change:
[Category diff]
+1     783
-1     717
+2      20
-2      34
+3      34
-3      20
+4     575
-4     777
+5     765
-5     755
+6     785
-6     666
+7     479
-7     472

[Weighted category diff]
+1     999
-1     965
+2      21
-2      35
+3      35
-3      21
+4     804
-4     980
+5     970
-5     971
+6     954
-6     831
+7     645
-7     625

Bug: 10613502

Change-Id: I5cd447673a196ad5af641a2118602bec879bdefb
 2013-10-03 19:30:26 -07:00
Keisuke Kuroyanagi
864ee881bd Merge "Fix UserHistoryDictionaryTests." 2013-10-03 12:02:34 +00:00
Keisuke Kuroyanagi
11f7cae094 Fix UserHistoryDictionaryTests. 
Bug: 6669677
Bug: 10667710

Change-Id: I6cdc6a6c9cacc7f276fda3a26ec31e3eb928471c
 2013-10-03 20:55:34 +09:00
Keisuke Kuroyanagi
9cb1e78a44 Fix: Native crash when an empty attribute is written. 
Bug: 10964805
Change-Id: Ia3cd891d20aee2226092f0f646d8f280901786fc
 2013-10-03 17:30:54 +09:00
Keisuke Kuroyanagi
3d92954120 Fix build. 
Change-Id: Ib33b27214e67378646e44f5b45bda298f960524f
 2013-10-02 23:02:42 +09:00
Keisuke Kuroyanagi
7c4dcf1e91 Use probability table for decaying dictionaries. 
Bug: 6669677
Change-Id: Ib5994a2e343dfcdaf23755e523f52846709b1c6c
 2013-10-02 18:01:47 +09:00
Keisuke Kuroyanagi
c76bbceedc Stochastic decay. 
Bug: 6669677
Change-Id: Ib2d9228b951c77dab7a8675ce9db60677e87e771
 2013-10-02 17:51:12 +09:00
Keisuke Kuroyanagi
13d5dc914a Change name of DecayingUtils. 
Bug: 6669677
Change-Id: I97839b2184b95ae1659c55a9b31d22e2d601cba1
 2013-10-02 16:23:44 +09:00
Jean Chalard
74577bedb3 Merge "Implement the heuristic for auto-commit." 2013-10-01 12:10:57 +00:00
Jean Chalard
ff9476f10a Merge "Fix typos" 2013-10-01 12:10:37 +00:00
Jean Chalard
3e954347e3 Fix typos 
I think I may pass out from the sheer ugliness

Change-Id: I1e709f03ef47ec713552d41fd9e37e01b810ae88
 2013-10-01 21:08:14 +09:00
Jean Chalard
459cd6f8ef Implement the heuristic for auto-commit. 
Bug: 9059617
Change-Id: I066abf018df5aaeabf415425dd822ebe233e6008
 2013-10-01 21:00:40 +09:00
Keisuke Kuroyanagi
6995310996 Separate "GC" and "GC with decay". 
Bug: 6669677
Change-Id: I9d6aba76cef2616f0549e612db9701e1d6a19467
 2013-10-01 18:09:31 +09:00
Jean Chalard
bb57090f1d Create the wiring for auto-commit confidence computation. 
Bug: 9059617
Change-Id: I7e4e05ffa8c68cfebbe362a977fd176555cad12a
 2013-10-01 15:04:15 +09:00
Jean Chalard
bf1beefd08 Merge "Surface the distance after first word for autocommit." 2013-09-30 14:06:02 +00:00
Jean Chalard
da06e385f5 Surface the distance after first word for autocommit. 
Bug: 9059617
Change-Id: Ie9b4cc8148ae8e0ff437b3337ab6c1bde95500f5
 2013-09-30 23:02:04 +09:00
Keisuke Kuroyanagi
fd02b2d6ee Implement simple dictionary decay. 
Groundwork and implement simple decay.
Increment probability when typed and decrement probability
at GC.

Bug: 6669677

Change-Id: Ib12caead0cbeef4ce7808fe8ac0b00ee331523fe
 2013-09-30 20:53:35 +09:00
Keisuke Kuroyanagi
c18510049a Prepare dictionary decay. 
Bug: 6669677
Change-Id: I8fbae190dd44a6bdbee7e9b6d3a16208322727f7
 2013-09-30 13:57:54 +09:00
Keisuke Kuroyanagi
1cf4789ba6 Merge "Set header attributes for ExpandableBinaryDictionary." 2013-09-29 05:14:22 +00:00
Keisuke Kuroyanagi
5ed30a7660 Set header attributes for ExpandableBinaryDictionary. 
Bug: 6669677

Change-Id: I2462777eaa07600f1c774226adfc4f351882f523
 2013-09-28 12:50:09 +09:00
Keisuke Kuroyanagi
68bc4ba5c2 Merge "Add boundary checking for PtNode Array reading." 2013-09-27 14:25:24 +00:00
Keisuke Kuroyanagi
8b580819a5 Merge "Add unigram/bigram counting." 2013-09-27 14:14:49 +00:00