2687484585
Create empty ver4 dictionary buffer on memory.
...
Bug: 11073222
Change-Id: Ia1f4d7782e019fdc6a4162f942f433d8b150b49d
2013-11-13 16:40:04 +09:00
ff4b1d1fd0
Implement ver4 dictionary bigram removing methods.
...
Bug: 11073222
Change-Id: I0c42e283d1ff983dc2c51e91b167bc39cdfd3da8
2013-11-13 16:40:01 +09:00
fe05173198
Merge "Implement ver4 dictionary bigram writing methods."
2013-11-13 05:23:57 +00:00
99808b7b0d
Implement ver4 dictionary bigram writing methods.
...
Bug: 11073222
Change-Id: Ibb0e697fe14dac1edb1f3074fb59e2cdc91039d4
2013-11-12 21:10:01 +09:00
180419b20c
Merge "Implement ver4 dictionary unigram writing methods."
2013-11-07 09:42:40 +00:00
3fe9458b6d
Implement ver4 dictionary unigram writing methods.
...
Bug: 11073222
Change-Id: Ibdb6846fee98919bb5f845170c19d7d571fcb88d
2013-11-07 17:58:27 +09:00
fc1824ea55
Merge "Use sysconf(_SC_PAGESIZE) instead of getpagesize()."
2013-11-07 06:19:48 +00:00
75534b98ca
Use sysconf(_SC_PAGESIZE) instead of getpagesize().
...
Bug: 11563061
Change-Id: I1350f0b1cb8e2bf17cdce34b7429763bbb6b2856
2013-11-07 14:53:20 +09:00
956ca35157
Refactoring: Separate updating methods from writing methods.
...
Bug: 11073222
Change-Id: I5e9df7276a8c167f31423bebf30bd26bde74fc92
2013-11-06 19:22:37 +09:00
0be85cfdf7
Refactoring: PtNodeWriter to separate version specific writing methods.
...
Bug: 11073222
Change-Id: Id2ade16316f33e17ead5faf45404a2054c303d74
2013-11-06 19:22:34 +09:00
b460c85589
Implement ver4 shortcut reading method.
...
Bug: 11073222
Change-Id: I3738c7ce07f500920bde8d3f985cf6e8ecb40b6a
2013-11-06 18:09:35 +09:00
9b3e59d644
Implement ver4 bigram reading method.
...
Bug: 11073222
Change-Id: I7b3408938f304da361201892e0a1342fdf92e62e
2013-11-05 16:00:26 +09:00
6e15f8a17f
Implement TerminalPositionLookupTable.
...
Bug: 11073222
Change-Id: I2ffe8ffc290fb69afc0eca35b5fed7157f5108fd
2013-11-01 17:54:23 -07:00
1c62341de6
Implement SparseTable in native code.
...
Bug: 11073222
Change-Id: Ia2cbb2fecfae0c9f621600483d96a55aef75c1b8
2013-11-01 11:03:41 -07:00
b3691b5642
Implement ver4 dict PtNode reading methods.
...
Bug: 11073222
Change-Id: I03af717ff02a621d9be8eb554d8f1438a500339b
2013-10-31 11:01:41 -07:00
ecbd8af169
Implement ProbabilityDictContent.
...
Bug: 11073222
Change-Id: Ia57c940fe3507a53b9d32aa6b9ebc5581c08d11f
2013-10-28 21:41:41 -07:00
b8a1e8fa33
Add uint reading method to BufferWithExtendableBuffer.
...
Bug: 11073222
Change-Id: Iff503cba883dd6f4ce9079783ad6a2edf464ffba
2013-10-28 15:32:57 -07:00
41b77116d3
Merge "Refactoring: Move some methods to the reading helper."
2013-10-28 22:12:06 +00:00
2271c23dc6
Open dictionary files always using O_RDONLY.
...
We never directly write to opened and mmapped files.
Bug: 11391620
Change-Id: Ic52fbe6018b8b3c1b6ca687461f29185809217c2
2013-10-25 17:35:37 -07:00
3f0a8e3fb0
Refactoring: Move some methods to the reading helper.
...
To handle multiple dictionary formats in helpers.
Bug: 11073222
Change-Id: I376bbdea934e13588feca02b9865105742f1a058
2013-10-25 12:32:36 -07:00
c481d0556f
Refactoring: Introduce PtNode and PtNodeReader.
...
To handle multiple dictionary formats in helpers.
Bug: 11073222
Change-Id: Iaef7be08534f9010e837ffcf8c8292b174b64d2b
2013-10-25 11:18:55 -07:00
75d7f0fbf5
Introduce Ver4DictBuffers to handle multiple buffers.
...
Bug: 11073222
Change-Id: Icacca8133c0030b19ca826fbf9525f41a1ac6cc8
2013-10-24 23:45:01 -07:00
1e34cc1698
Introduce DictContent for ver4 dict.
...
Bug: 11073222
Change-Id: I88ab3948a98388931d81c97825c9d2c76e15a44b
2013-10-24 17:42:56 -07:00
e5cfb77694
Introduce ExclusiveOwnershipPointer.
...
Bug: 11073222
Change-Id: I841527f0247f2ade07796406729b3a4a8958aa57
2013-10-24 17:28:30 -07:00
4fd413b7e4
Merge "Tracking minor errors to categorize exact match."
2013-10-21 12:32:03 +00:00
eb5049b28d
Tracking minor errors to categorize exact match.
...
Bug: 8844931
Change-Id: I012529f5fc0d094ef64a41c075fe548b6b439265
2013-10-21 21:01:59 +09:00
0913aaa56f
Merge "Add utility methods related to buffers for ver4 dict."
2013-10-21 11:03:08 +00:00
72877b15ea
Add utility methods related to buffers for ver4 dict.
...
Bug: 11073222
Change-Id: Ieca15ffa09fd36c6602f6d2afab5779800c5a422
2013-10-21 19:56:50 +09:00
53686e0f5f
Fix: Suggested words from user history are invalid.
...
- Suggestions form user history can contain invalid words.
- isValidWord always returns false.
Bug: 11139426
Change-Id: I6075b275603332ddb00f4a9284afcaa82d824270
2013-10-18 19:53:57 +09:00
2628268ef4
Add ver4 dictionary structure policy file.
...
Bug: 11073222
Change-Id: Ia9e1a4842d42a581812a8c3dd4a85e7ee86dd529
2013-10-16 18:46:32 +09:00
427d69e218
Moving files for dictionary structure policies.
...
Bug: 11073222
Change-Id: Ic40f6258e2b0c2790d526aff31312c112760839d
2013-10-16 18:46:32 +09:00
c1fce1a720
Make "node"s clear by renaming to PtNode or DicNode.
...
Bug: 10233675
Change-Id: I248b927c724253f593e3806e82b4e1fed2c025fb
2013-10-16 18:46:28 +09:00
e1bd7d5d39
Initial step to support version 4 format in native code.
...
Bug: 11073222
Change-Id: I633229381b28cef23edef2b37e5b0bf6e15af468
2013-10-09 12:54:33 +09:00
6d91e4ce74
Fix offdevice test build.
...
Change-Id: I42143bdc4f6f5aa4f55530de84819ab722a22f72
2013-10-08 16:05:42 +09:00
6d3d53955e
Merge "Fix a bug: wrong parent position is written when moving."
2013-10-08 04:48:53 +00:00
802e6383ea
Fix a bug: wrong parent position is written when moving.
...
Change-Id: I93d33ba9105eb85327108cca5706d8db3399de6a
2013-10-08 12:44:00 +09:00
86f7711407
Merge "Don't auto-correct to non-whitelist shortcuts"
2013-10-07 10:04:18 +00:00
f309f2288b
Don't auto-correct to non-whitelist shortcuts
...
Bug: 11076722
Change-Id: I0de10ab24d33ec31f45664fe92d80d5a7bec9c20
2013-10-07 19:01:57 +09:00
cfb018ba6d
Fix bug and Add large test for decaying dictionary.
...
- GC gets failure when the dictionary become empty.
- Useless unigrams are sometimes not removed.
Bug: 10197478
Change-Id: I8d1479c01efba61a81f03bc077da6bcb4797a940
2013-10-07 17:31:01 +09:00
15b1a0fd14
Parameter optimization.
...
[Category diff]
+1 194
-1 21
+2 15
-2 1
+3 1
-3 15
+4 152
-4 90
+5 108
-5 197
+6 94
-6 136
+7 28
-7 132
[Weighted category diff]
+1 230
-1 23
+2 15
-2 1
+3 1
-3 15
+4 180
-4 106
+5 122
-5 224
+6 107
-6 158
+7 29
-7 157
show diff for ./en_user_log_phones_2011_08.csv
+1 171
-1 9
+2 7
-2 1
+3 1
-3 7
+4 123
-4 76
+5 84
-5 157
+6 66
-6 119
+7 20
-7 103
Before
---- count ----
Count(1):75.64%, 177913
Count(2):1.06%, 2510
Count(3):0.56%, 1319
Count(4):6.63%, 15612
Count(5):9.87%, 23223
Count(6):2.79%, 6580
Count(7):3.41%, 8038
---- weight ----
Weight(1):96.88%, 2009907
Weight(2):0.24%, 5157
Weight(3):0.09%, 2025
Weight(4):0.77%, 16095
Weight(5):1.11%, 23077
Weight(6):0.39%, 8172
Weight(7):0.48%, 10146
---- score ----
Conservative score = 66.406
Normal score = 81.512
Agressive score = 89.064
Insane score = 92.841
After
---- count ----
Count(1):75.71%, 178087
Count(2):1.07%, 2524
Count(3):0.55%, 1305
Count(4):6.66%, 15677
Count(5):9.83%, 23133
Count(6):2.78%, 6539
Count(7):3.37%, 7930
---- weight ----
Weight(1):96.89%, 2010118
Weight(2):0.24%, 5171
Weight(3):0.09%, 2011
Weight(4):0.78%, 16184
Weight(5):1.1%, 22972
Weight(6):0.39%, 8109
Weight(7):0.48%, 10014
---- score ----
Conservative score = 66.431
Normal score = 81.531
Agressive score = 89.08
Insane score = 92.855
No diffs for 10+ freq user log.
Change-Id: Ica4a7604507a2c91aaa7daf5bacf4fce4039f547
2013-10-07 12:39:51 +09:00
3e5f4b53b0
Merge "Check the elapsed time for decaying."
2013-10-04 07:16:17 +00:00
67c855ea6f
Check the elapsed time for decaying.
...
Bug: 6669677
Change-Id: I0401a4c0908c702ce65abfa5e017dd16cb1296f9
2013-10-04 16:11:11 +09:00
0b62621446
Merge "Optimize the parameters for the typing algorithm"
2013-10-04 03:25:43 +00:00
2ed09c214a
Optimize the parameters for the typing algorithm
...
The new set of parameters are optimized for both
keystroke savings and regression test scores.
This change is corresponding to the patch 6 of
Change-Id: Ib8439a85de26cbbba58837344717f0acbfcf0ab0
in the regression test repository.
Here is the link to the doc analyzing the
“nnow → know” case:
https://docs.google.com/a/google.com/document/d/1XIHfAHLlP1f3F8R-u7yxPGWcP58tokygzN_lQKXFFgM/edit
Here is the link to the spreadsheet showing
the diff of the words with frequencies >= 10
in en_user_log_phones_2011_08.csv.
https://docs.google.com/a/google.com/spreadsheet/ccc?key=0Am_RQlV4zC6SdFFVTlhJT0RLUENCWWZOZ3JQTUpMQ2c&usp=drive_web#gid=0
Updated TRT Results for this change:
[Category diff]
+1 783
-1 717
+2 20
-2 34
+3 34
-3 20
+4 575
-4 777
+5 765
-5 755
+6 785
-6 666
+7 479
-7 472
[Weighted category diff]
+1 999
-1 965
+2 21
-2 35
+3 35
-3 21
+4 804
-4 980
+5 970
-5 971
+6 954
-6 831
+7 645
-7 625
Bug: 10613502
Change-Id: I5cd447673a196ad5af641a2118602bec879bdefb
2013-10-03 19:30:26 -07:00
864ee881bd
Merge "Fix UserHistoryDictionaryTests."
2013-10-03 12:02:34 +00:00
11f7cae094
Fix UserHistoryDictionaryTests.
...
Bug: 6669677
Bug: 10667710
Change-Id: I6cdc6a6c9cacc7f276fda3a26ec31e3eb928471c
2013-10-03 20:55:34 +09:00
9cb1e78a44
Fix: Native crash when an empty attribute is written.
...
Bug: 10964805
Change-Id: Ia3cd891d20aee2226092f0f646d8f280901786fc
2013-10-03 17:30:54 +09:00
3d92954120
Fix build.
...
Change-Id: Ib33b27214e67378646e44f5b45bda298f960524f
2013-10-02 23:02:42 +09:00
7c4dcf1e91
Use probability table for decaying dictionaries.
...
Bug: 6669677
Change-Id: Ib5994a2e343dfcdaf23755e523f52846709b1c6c
2013-10-02 18:01:47 +09:00
c76bbceedc
Stochastic decay.
...
Bug: 6669677
Change-Id: Ib2d9228b951c77dab7a8675ce9db60677e87e771
2013-10-02 17:51:12 +09:00
13d5dc914a
Change name of DecayingUtils.
...
Bug: 6669677
Change-Id: I97839b2184b95ae1659c55a9b31d22e2d601cba1
2013-10-02 16:23:44 +09:00
74577bedb3
Merge "Implement the heuristic for auto-commit."
2013-10-01 12:10:57 +00:00
ff9476f10a
Merge "Fix typos"
2013-10-01 12:10:37 +00:00
3e954347e3
Fix typos
...
I think I may pass out from the sheer ugliness
Change-Id: I1e709f03ef47ec713552d41fd9e37e01b810ae88
2013-10-01 21:08:14 +09:00
459cd6f8ef
Implement the heuristic for auto-commit.
...
Bug: 9059617
Change-Id: I066abf018df5aaeabf415425dd822ebe233e6008
2013-10-01 21:00:40 +09:00
6995310996
Separate "GC" and "GC with decay".
...
Bug: 6669677
Change-Id: I9d6aba76cef2616f0549e612db9701e1d6a19467
2013-10-01 18:09:31 +09:00
bb57090f1d
Create the wiring for auto-commit confidence computation.
...
Bug: 9059617
Change-Id: I7e4e05ffa8c68cfebbe362a977fd176555cad12a
2013-10-01 15:04:15 +09:00
bf1beefd08
Merge "Surface the distance after first word for autocommit."
2013-09-30 14:06:02 +00:00
da06e385f5
Surface the distance after first word for autocommit.
...
Bug: 9059617
Change-Id: Ie9b4cc8148ae8e0ff437b3337ab6c1bde95500f5
2013-09-30 23:02:04 +09:00
fd02b2d6ee
Implement simple dictionary decay.
...
Groundwork and implement simple decay.
Increment probability when typed and decrement probability
at GC.
Bug: 6669677
Change-Id: Ib12caead0cbeef4ce7808fe8ac0b00ee331523fe
2013-09-30 20:53:35 +09:00
c18510049a
Prepare dictionary decay.
...
Bug: 6669677
Change-Id: I8fbae190dd44a6bdbee7e9b6d3a16208322727f7
2013-09-30 13:57:54 +09:00
1cf4789ba6
Merge "Set header attributes for ExpandableBinaryDictionary."
2013-09-29 05:14:22 +00:00
5ed30a7660
Set header attributes for ExpandableBinaryDictionary.
...
Bug: 6669677
Change-Id: I2462777eaa07600f1c774226adfc4f351882f523
2013-09-28 12:50:09 +09:00
68bc4ba5c2
Merge "Add boundary checking for PtNode Array reading."
2013-09-27 14:25:24 +00:00
8b580819a5
Merge "Add unigram/bigram counting."
2013-09-27 14:14:49 +00:00
31097a57cc
Add unigram/bigram counting.
...
Bug: 6669677
Change-Id: I05ea2201d822dddf062b08c8467daa336760e16c
2013-09-27 23:12:12 +09:00
67cb282b64
Add boundary checking for PtNode Array reading.
...
Bug: 6669677
Change-Id: Iea5e19d98d2fc26f137046dd5f8e339239672351
2013-09-27 22:28:54 +09:00
85463c7f40
Merge "Use base lower character to get point to key length."
2013-09-26 08:28:41 +00:00
e639d7346c
Use base lower character to get point to key length.
...
Bug: 9490364
Change-Id: I3755f00ab22f926f087fc917b95a2db7d722fa00
2013-09-26 16:36:05 +09:00
5ef6209656
Create empty dictionary file in native code.
...
Bug: 6669677
Change-Id: I46d23deee1fd091678f4593561cb8687eb815212
2013-09-26 12:59:02 +09:00
10b56df486
Merge "Prepare dictionary file creating in native code."
2013-09-26 02:38:19 +00:00
c6015327fb
Merge "Add empty PtNode checking for PatriciaTriePolicy."
2013-09-26 02:32:03 +00:00
9155eec0d9
Add empty PtNode checking for PatriciaTriePolicy.
...
Bug: 9503570
Change-Id: I784d29c307c1040699fa0aadfc6a47d97f03ffe4
2013-09-26 11:19:04 +09:00
36f4592128
Merge "Refactoring header attribute reading."
2013-09-26 01:57:22 +00:00
1592eb8a3a
Prepare dictionary file creating in native code.
...
Bug: 6669677
Change-Id: I7c476a6e99ec7ac883f05d84ef306265255154da
2013-09-26 10:48:43 +09:00
989596844e
Refactoring header attribute reading.
...
Bug: 6669677
Change-Id: Ifc11da614d5c331ac61019a324e3a0ff187329cd
2013-09-26 10:45:03 +09:00
009dcac33f
Boundary check for children creating of PatriciaTriePolicy.
...
Bug: 10537529
Change-Id: I0cd8b6de230795498395ca08789ccc1c74780f2c
2013-09-25 19:54:31 +09:00
3d2802ef2e
Allow large memory space for GC.
...
Bug: 6669677
Change-Id: I021c92a1ca5cd28f745cbfe729c99e346c58074d
2013-09-24 19:21:17 +09:00
f3cd38f2c7
GC step 5. Add large tests and fix bugs.
...
Bug: 6669677
Change-Id: I41e44a64730f9c66645995095a7ecff51ffd5351
2013-09-24 16:32:25 +09:00
2cfe7f9e3b
GC step 4. Update all positions in new dict and add a test.
...
Bug: 6669677
Change-Id: I035d848a210653dcd3c1a1f21781968adced9225
2013-09-24 15:29:56 +09:00
0355bf2d4c
GC step 3. Place and write valid PtNodes.
...
Bug: 6669677
Change-Id: Id2b7359bafb34acf12dc3bf4c595a03073fda805
2013-09-20 21:11:49 +09:00
77ef75cbe6
GC step 2. Finding garbage bigram entries.
...
Bug: 6669677
Change-Id: Ie3f406776ab56bf589ac10e15e2534ff8cb1f275
2013-09-20 20:04:57 +09:00
2a64726a16
Step 1 to implement GC. Finding garbage PtNodes.
...
Bug: 6669677
Change-Id: I3551fe2f16a09d2bf7761f4e1d73ebd4a03380e7
2013-09-20 15:37:14 +09:00
b7e8a9abf5
Groundwork for implementing GC.
...
Bug: 6669677
Change-Id: Ia2b2f338795ab62cc9eef73731285b2a5a0ac5e4
2013-09-19 14:48:59 +09:00
e28927dfb6
Change name of header_reading_utlis.*
...
Bug: 6669677
Change-Id: I91389050148237580ef640d3a58b935ea752954e
2013-09-18 18:27:01 +09:00
484fa7b59c
Add dictionary header writing methods.
...
Bug: 6669677
Change-Id: I392ac4776b55779903cbaa17e683005d80017a41
2013-09-18 18:08:33 +09:00
72e53a1e38
Merge "Implement dictionary flush."
2013-09-18 05:20:05 +00:00
1560543754
Implement dictionary flush.
...
Bug: 6669677
Change-Id: I1d70ac6319fb6d8700074de382568c2108887960
2013-09-18 13:19:14 +09:00
9465819cf6
Merge "Add BinaryDictionary.getBigramProbabilityNative()."
2013-09-18 04:10:38 +00:00
4d02a2d44d
Add BinaryDictionary.getBigramProbabilityNative().
...
Bug: 10197478
Change-Id: Id16e3ee22a17679c46cc7a0199a9e28065177aa3
2013-09-18 11:18:28 +09:00
4e17b42d0f
Output input index of second word's first char.
...
This is for partial commit.
Bug: 9059617
Change-Id: I77a0245b63c29b7234fd9b06d362e05aebf30051
2013-09-17 20:31:34 +09:00
ac7e005ef0
Merge "Prepare dictionary saving in native code."
2013-09-17 09:18:32 +00:00
d0246277fd
Prepare dictionary saving in native code.
...
Bug: 6669677
Change-Id: I0787f66409e2b183c6328fa05805945520e1e93a
2013-09-17 18:07:16 +09:00
d9f450ef00
Implement removeBigramWords() for DynamicPatriciaTrie.
...
Bug: 6669677
Change-Id: I99cb517511b4c968b8bf937baab5d7f42b0f534e
2013-09-17 18:05:32 +09:00
78b55a31cb
Fix handling multi-bytes characters and add a test.
...
Bug: 6669677
Change-Id: Id2154db47adea2929559a4187a726f9dfa83363e
2013-09-17 15:11:24 +09:00
6d1a29eaab
Fix build.
...
Change-Id: I1c3c2996ca11bfe8864df774e9280893cf17c0d4
2013-09-17 13:03:12 +09:00
cd6a0430c7
Implement addBigramWords() for DynamicPatriciaTrie.
...
Bug: 6669677
Change-Id: Ia6aa76c212165690191c65fee9dacdc44be5acd5
2013-09-17 12:52:21 +09:00
cb816e5e16
Merge NOT_A_VALID_WORD_POS into NOT_A_DICT_POS.
...
Change-Id: I2ac3aa2a19d14e8c4159a59334e7816bd1409688
2013-09-16 18:37:23 +09:00
9ea9c61c99
Move methods using ByteArrayUtils from .h to .cpp.
...
Bug: 10402083
Change-Id: Ic5f1aa11eb403f5bdc83de8e38a6457200519cae
2013-09-16 18:37:08 +09:00
f874728496
Add boundary check for PtNode reading position.
...
Bug: 10743878
Change-Id: I6727b12b0ef98bb9cfb3ca95aeab2c79749f47a2
2013-09-16 18:34:34 +09:00
Keisuke Kuroyanagi
Jean Chalard
Xiaojun Bi