Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commita5be6e9

Browse files
committed
Fix comments that claimed that mblen() only looks at first byte.
GB18030's mblen() function looks at the first and the second byte of themultibyte character, to determine its length. copy.c had made theassumption that mblen() only looks at the first byte, but it turns out towork out fine, because of the way the GB18030 encoding works. COPY willsee a 4-byte encoded character as two 2-byte encoded characters, which isenough for COPY's purposes. It cannot mix those up with delimiter orescaping characters, because only single-byte ASCII characters aresupported as delimiters or escape characters.Discussion:https://www.postgresql.org/message-id/7704d099-9643-2a55-fb0e-becd64400dcb%40iki.fi
1 parent7c079d7 commita5be6e9

File tree

2 files changed

+31
-8
lines changed

2 files changed

+31
-8
lines changed

‎src/backend/commands/copy.c

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4121,9 +4121,14 @@ CopyReadLineText(CopyState cstate)
41214121
{
41224122
intmblen;
41234123

4124+
/*
4125+
* It is enough to look at the first byte in all our encodings, to
4126+
* get the length. (GB18030 is a bit special, but still works for
4127+
* our purposes; see comment in pg_gb18030_mblen())
4128+
*/
41244129
mblen_str[0]=c;
4125-
/* All our encodings only read the first byte to get the length */
41264130
mblen=pg_encoding_mblen(cstate->file_encoding,mblen_str);
4131+
41274132
IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen-1);
41284133
IF_NEED_REFILL_AND_EOF_BREAK(mblen-1);
41294134
raw_buf_ptr+=mblen-1;

‎src/backend/utils/mb/wchar.c

Lines changed: 25 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -15,16 +15,23 @@
1515

1616

1717
/*
18-
* conversion to pg_wchar is done by "table driven."
19-
* to add an encoding support, define mb2wchar_with_len(), mblen(), dsplen()
20-
* for the particular encoding. Note that if the encoding is only
21-
* supported in the client, you don't need to define
22-
* mb2wchar_with_len() function (SJIS is the case).
18+
* Operations on multi-byte encodings are driven by a table of helper
19+
* functions.
20+
*
21+
* To add an encoding support, define mblen(), dsplen() and verifier() for
22+
* the encoding. For server-encodings, also define mb2wchar() and wchar2mb()
23+
* conversion functions.
2324
*
2425
* These functions generally assume that their input is validly formed.
2526
* The "verifier" functions, further down in the file, have to be more
26-
* paranoid. We expect that mblen() does not need to examine more than
27-
* the first byte of the character to discover the correct length.
27+
* paranoid.
28+
*
29+
* We expect that mblen() does not need to examine more than the first byte
30+
* of the character to discover the correct length. GB18030 is an exception
31+
* to that rule, though, as it also looks at second byte. But even that
32+
* behaves in a predictable way, if you only pass the first byte: it will
33+
* treat 4-byte encoded characters as two 2-byte encoded characters, which is
34+
* good enough for all current uses.
2835
*
2936
* Note: for the display output of psql to work properly, the return values
3037
* of the dsplen functions must conform to the Unicode standard. In particular
@@ -1073,6 +1080,17 @@ pg_uhc_dsplen(const unsigned char *s)
10731080
* GB18030
10741081
*Added by Bill Huang <bhuang@redhat.com>,<bill_huanghb@ybb.ne.jp>
10751082
*/
1083+
1084+
/*
1085+
* Unlike all other mblen() functions, this also looks at the second byte of
1086+
* the input. However, if you only pass the first byte of a multi-byte
1087+
* string, and \0 as the second byte, this still works in a predictable way:
1088+
* a 4-byte character will be reported as two 2-byte characters. That's
1089+
* enough for all current uses, as a client-only encoding. It works that
1090+
* way, because in any valid 4-byte GB18030-encoded character, the third and
1091+
* fourth byte look like a 2-byte encoded character, when looked at
1092+
* separately.
1093+
*/
10761094
staticint
10771095
pg_gb18030_mblen(constunsignedchar*s)
10781096
{

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp