- Notifications
You must be signed in to change notification settings - Fork5
Commit7730f48
committed
Teach UtfToLocal/LocalToUtf to support algorithmic encoding conversions.
Until now, these functions have only supported encoding conversions usinglookup tables, which is fine as long as there's not too many code pointsto convert. However, GB18030 expects all 1.1 million Unicode code pointsto be convertible, which would require a ridiculously-sized lookup table.Fortunately, a large fraction of those conversions can be expressed througharithmetic, ie the conversions are one-to-one in certain defined ranges.To support that, provide a callback function that is used after consultingthe lookup tables. (This patch doesn't actually change anything about theGB18030 conversion behavior, just provide infrastructure for fixing it.)Since this requires changing the APIs of UtfToLocal/LocalToUtf anyway,take the opportunity to rearrange their argument lists into what seemsto me a saner order. And beautify the call sites by using lengthof()instead of error-prone sizeof() arithmetic.In passing, also mark all the lookup tables used by these calls "const".This moves an impressive amount of stuff into the text segment, at leaston my machine, and is safer anyhow.1 parent83e176e commit7730f48
File tree
108 files changed
+538
-408
lines changed- src
- backend/utils/mb
- Unicode
- conversion_procs
- euc_tw_and_big5
- utf8_and_big5
- utf8_and_cyrillic
- utf8_and_euc2004
- utf8_and_euc_cn
- utf8_and_euc_jp
- utf8_and_euc_kr
- utf8_and_euc_tw
- utf8_and_gb18030
- utf8_and_gbk
- utf8_and_iso8859
- utf8_and_johab
- utf8_and_sjis
- utf8_and_sjis2004
- utf8_and_uhc
- utf8_and_win
- include/mb
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
108 files changed
+538
-408
lines changedLines changed: 2 additions & 2 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
97 | 97 |
| |
98 | 98 |
| |
99 | 99 |
| |
100 |
| - | |
| 100 | + | |
101 | 101 |
| |
102 | 102 |
| |
103 | 103 |
| |
| |||
185 | 185 |
| |
186 | 186 |
| |
187 | 187 |
| |
188 |
| - | |
| 188 | + | |
189 | 189 |
| |
190 | 190 |
| |
191 | 191 |
| |
|
Lines changed: 2 additions & 2 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
55 | 55 |
| |
56 | 56 |
| |
57 | 57 |
| |
58 |
| - | |
| 58 | + | |
59 | 59 |
| |
60 | 60 |
| |
61 | 61 |
| |
| |||
109 | 109 |
| |
110 | 110 |
| |
111 | 111 |
| |
112 |
| - | |
| 112 | + | |
113 | 113 |
| |
114 | 114 |
| |
115 | 115 |
| |
|
Lines changed: 4 additions & 4 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
72 | 72 |
| |
73 | 73 |
| |
74 | 74 |
| |
75 |
| - | |
| 75 | + | |
76 | 76 |
| |
77 | 77 |
| |
78 | 78 |
| |
| |||
133 | 133 |
| |
134 | 134 |
| |
135 | 135 |
| |
136 |
| - | |
| 136 | + | |
137 | 137 |
| |
138 | 138 |
| |
139 | 139 |
| |
| |||
256 | 256 |
| |
257 | 257 |
| |
258 | 258 |
| |
259 |
| - | |
| 259 | + | |
260 | 260 |
| |
261 | 261 |
| |
262 | 262 |
| |
| |||
283 | 283 |
| |
284 | 284 |
| |
285 | 285 |
| |
286 |
| - | |
| 286 | + | |
287 | 287 |
| |
288 | 288 |
| |
289 | 289 |
| |
|
Lines changed: 2 additions & 2 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
136 | 136 |
| |
137 | 137 |
| |
138 | 138 |
| |
139 |
| - | |
| 139 | + | |
140 | 140 |
| |
141 | 141 |
| |
142 | 142 |
| |
| |||
263 | 263 |
| |
264 | 264 |
| |
265 | 265 |
| |
266 |
| - | |
| 266 | + | |
267 | 267 |
| |
268 | 268 |
| |
269 | 269 |
| |
|
Lines changed: 2 additions & 2 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
55 | 55 |
| |
56 | 56 |
| |
57 | 57 |
| |
58 |
| - | |
| 58 | + | |
59 | 59 |
| |
60 | 60 |
| |
61 | 61 |
| |
| |||
109 | 109 |
| |
110 | 110 |
| |
111 | 111 |
| |
112 |
| - | |
| 112 | + | |
113 | 113 |
| |
114 | 114 |
| |
115 | 115 |
| |
|
Lines changed: 2 additions & 2 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
71 | 71 |
| |
72 | 72 |
| |
73 | 73 |
| |
74 |
| - | |
| 74 | + | |
75 | 75 |
| |
76 | 76 |
| |
77 | 77 |
| |
| |||
138 | 138 |
| |
139 | 139 |
| |
140 | 140 |
| |
141 |
| - | |
| 141 | + | |
142 | 142 |
| |
143 | 143 |
| |
144 | 144 |
| |
|
Lines changed: 2 additions & 2 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
52 | 52 |
| |
53 | 53 |
| |
54 | 54 |
| |
55 |
| - | |
| 55 | + | |
56 | 56 |
| |
57 | 57 |
| |
58 | 58 |
| |
| |||
106 | 106 |
| |
107 | 107 |
| |
108 | 108 |
| |
109 |
| - | |
| 109 | + | |
110 | 110 |
| |
111 | 111 |
| |
112 | 112 |
| |
|
Lines changed: 4 additions & 4 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
72 | 72 |
| |
73 | 73 |
| |
74 | 74 |
| |
75 |
| - | |
| 75 | + | |
76 | 76 |
| |
77 | 77 |
| |
78 | 78 |
| |
| |||
99 | 99 |
| |
100 | 100 |
| |
101 | 101 |
| |
102 |
| - | |
| 102 | + | |
103 | 103 |
| |
104 | 104 |
| |
105 | 105 |
| |
| |||
185 | 185 |
| |
186 | 186 |
| |
187 | 187 |
| |
188 |
| - | |
| 188 | + | |
189 | 189 |
| |
190 | 190 |
| |
191 | 191 |
| |
| |||
212 | 212 |
| |
213 | 213 |
| |
214 | 214 |
| |
215 |
| - | |
| 215 | + | |
216 | 216 |
| |
217 | 217 |
| |
218 | 218 |
| |
|
Lines changed: 2 additions & 2 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
72 | 72 |
| |
73 | 73 |
| |
74 | 74 |
| |
75 |
| - | |
| 75 | + | |
76 | 76 |
| |
77 | 77 |
| |
78 | 78 |
| |
| |||
122 | 122 |
| |
123 | 123 |
| |
124 | 124 |
| |
125 |
| - | |
| 125 | + | |
126 | 126 |
| |
127 | 127 |
| |
128 | 128 |
| |
|
Lines changed: 2 additions & 2 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
88 | 88 |
| |
89 | 89 |
| |
90 | 90 |
| |
91 |
| - | |
| 91 | + | |
92 | 92 |
| |
93 | 93 |
| |
94 | 94 |
| |
| |||
140 | 140 |
| |
141 | 141 |
| |
142 | 142 |
| |
143 |
| - | |
| 143 | + | |
144 | 144 |
| |
145 | 145 |
| |
146 | 146 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 |
| - | |
| 1 | + | |
2 | 2 |
| |
3 | 3 |
| |
4 | 4 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 | 3 |
| |
4 |
| - | |
| 4 | + | |
5 | 5 |
| |
6 | 6 |
| |
7 | 7 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 | 3 |
| |
4 |
| - | |
| 4 | + | |
5 | 5 |
| |
6 | 6 |
| |
7 | 7 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 |
| - | |
| 1 | + | |
2 | 2 |
| |
3 | 3 |
| |
4 | 4 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 | 1 |
| |
2 | 2 |
| |
3 |
| - | |
| 3 | + | |
4 | 4 |
| |
5 | 5 |
| |
6 | 6 |
| |
|
Lines changed: 1 addition & 1 deletion
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 |
| - | |
| 1 | + | |
2 | 2 |
| |
3 | 3 |
| |
4 | 4 |
| |
|
0 commit comments
Comments
(0)