Repository files navigation

A simple one header solution to supporting utf8 strings in C and C++.

Functions provided from the C header string.h but with a utf8* prefix instead of the str* prefix:

API function docs

Functions provided from the C header strings.h but with a utf8* prefix instead of the str* prefix:

Functions provided that are unique to utf8.h:

Just#include "utf8.h" in your code!

The current supported platforms are Linux, macOS and Windows.

The current supported compilers are gcc, clang, MSVC's cl.exe, and clang-cl.exe.

The utf8.h API matches the string.h API as much as possible by design. There are a few major differences though.

utf8.h uses char8_t* in C++ 20 instead of char*

Anywhere in the string.h or strings.h documentation where it refers to 'bytes' I have changed that to utf8 codepoints. For instance, utf8len will return the number of utf8 codepoints in a utf8 string - which does not necessarily equate to the number of bytes.

intutf8casecmp(constvoid*src1,constvoid*src2);

Return less than 0, 0, greater than 0 ifsrc1 < src2,src1 == src2,src1 > src2 respectively, case insensitive.

void*utf8cat(void*dst,constvoid*src);

Append the utf8 stringsrc onto the utf8 stringdst.

void*utf8chr(constvoid*src,utf8_int32_tchr);

Find the first match of the utf8 codepointchr in the utf8 stringsrc.

intutf8cmp(constvoid*src1,constvoid*src2);

Return less than 0, 0, greater than 0 ifsrc1 < src2,
src1 == src2,src1 > src2 respectively.

void*utf8cpy(void*dst,constvoid*src);

Copy the utf8 stringsrc onto the memory allocated indst.

size_tutf8cspn(constvoid*src,constvoid*reject);

Number of utf8 codepoints in the utf8 stringsrc that consists entirely
of utf8 codepoints not from the utf8 stringreject.

void*utf8dup(constvoid*src);

Duplicate the utf8 stringsrc by getting its size,mallocing a new buffer
copying over the data, and returning that. Or 0 ifmalloc failed.

size_tutf8len(constvoid*str);

Number of utf8 codepoints in the utf8 stringstr,
excluding the null terminating byte.

size_tutf8nlen(constvoid*str,size_tn);

Similar toutf8len, except that only at mostn bytes ofsrc are looked.

intutf8ncasecmp(constvoid*src1,constvoid*src2,size_tn);

Return less than 0, 0, greater than 0 ifsrc1 < src2,src1 == src2,
src1 > src2 respectively, case insensitive. Checking at mostn
bytes of each utf8 string.

void*utf8ncat(void*dst,constvoid*src,size_tn);

Append the utf8 stringsrc onto the utf8 stringdst,
writing at mostn+1 bytes. Can produce an invalid utf8
string ifn falls partway through a utf8 codepoint.

intutf8ncmp(constvoid*src1,constvoid*src2,size_tn);

Return less than 0, 0, greater than 0 ifsrc1 < src2,
src1 == src2,src1 > src2 respectively. Checking at mostn
bytes of each utf8 string.

void*utf8ncpy(void*dst,constvoid*src,size_tn);

Copy the utf8 stringsrc onto the memory allocated indst.
Copies at mostn bytes. Ifn falls partway through a utf8codepoint, or ifdst doesn't have enough room for a nullterminator, the final string will be cut short to preserveutf8 validity.

void*utf8pbrk(constvoid*str,constvoid*accept);

Locates the first occurrence in the utf8 stringstr of any byte in the
utf8 stringaccept, or 0 if no match was found.

void*utf8rchr(constvoid*src,utf8_int32_tchr);

Find the last match of the utf8 codepointchr in the utf8 stringsrc.

size_tutf8size(constvoid*str);

Number of bytes in the utf8 stringstr,
including the null terminating byte.

size_tutf8size_lazy(constvoid*str);

Similar toutf8size, except that the null terminating byte isexcluded.

size_tutf8nsize_lazy(constvoid*str,size_tn);

Similar toutf8size, except that only at mostn bytes ofsrc are looked andthe null terminating byte isexcluded.

size_tutf8spn(constvoid*src,constvoid*accept);

Number of utf8 codepoints in the utf8 stringsrc that consists entirely
of utf8 codepoints from the utf8 stringaccept.

void*utf8str(constvoid*haystack,constvoid*needle);

The position of the utf8 stringneedle in the utf8 stringhaystack.

void*utf8casestr(constvoid*haystack,constvoid*needle);

The position of the utf8 stringneedle in the utf8 stringhaystack,case insensitive.

void*utf8valid(constvoid*str);

Return 0 on success, or the position of the invalid utf8 codepoint on failure.

void*utf8nvalid(constvoid*str,size_tn);

Similar toutf8valid, except that only at mostn bytes ofsrc are looked.

intutf8makevalid(void*str,utf8_int32_treplacement);

Return 0 on success. Makes thestr valid by replacing invalid sequences withthe 1-bytereplacement codepoint.

void*utf8codepoint(constvoid*str,utf8_int32_t*out_codepoint);

Sets out_codepoint to the current utf8 codepoint instr, and returns theaddress of the next utf8 codepoint after the current one instr.

void*utf8rcodepoint(constvoid*str,utf8_int32_t*out_codepoint);

Sets out_codepoint to the current utf8 codepoint instr, and returns theaddress of the previous utf8 codepoint before the current one instr.

size_tutf8codepointsize(utf8_int32_tchr);

Returns the size of the given codepoint in bytes.

void*utf8catcodepoint(void*utf8_restrictstr,utf8_int32_tchr,size_tn);

Write a codepoint to the given string, and return the address to the nextplace after the written codepoint. Pass how many bytes left in the buffer ton. If there is not enough space for the codepoint, this function returnsnull.

intutf8islower(utf8_int32_tchr);

Returns 1 if the given character is lowercase, or 0 if it is not.

intutf8isupper(utf8_int32_tchr);

Returns 1 if the given character is uppercase, or 0 if it is not.

voidutf8lwr(void*utf8_restrictstr);

Transform the given string into all lowercase codepoints.

voidutf8upr(void*utf8_restrictstr);

Transform the given string into all uppercase codepoints.

utf8_int32_tutf8lwrcodepoint(utf8_int32_tcp);

Make a codepoint lower case if possible.

utf8_int32_tutf8uprcodepoint(utf8_int32_tcp);

Make a codepoint upper case if possible.

Various functions provided will do case insensitive compares, or transform utf8strings from one case to another. Given the vastness of unicode, and the authorslack of understanding beyond latin codepoints on whether case means anything,the following categories are the only ones that will be checked in caseinsensitive code:

• ASCII
• Latin-1 Supplement
• Latin Extended-A
• Latin Extended-B
• Greek and Coptic
• Cyrillic

• Implement utf8coll (akin to strcoll).
• Implement utf8fry (akin to strfry).
• Investigate adding dst buffer sizes for utf8cpy and utf8cat to catch overwrites (as suggested by@FlohOfWoe inhttps://twitter.com/FlohOfWoe/status/618669237771608064)

This is free and unencumbered software released into the public domain.

Anyone is free to copy, modify, publish, use, compile, sell, ordistribute this software, either in source code form or as a compiledbinary, for any purpose, commercial or non-commercial, and by anymeans.

In jurisdictions that recognize copyright laws, the author or authorsof this software dedicate any and all copyright interest in thesoftware to the public domain. We make this dedication for the benefitof the public at large and to the detriment of our heirs andsuccessors. We intend this dedication to be an overt act ofrelinquishment in perpetuity of all present and future rights to thissoftware under copyright law.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OFMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OROTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OROTHER DEALINGS IN THE SOFTWARE.

For more information, please refer tohttp://unlicense.org/