Table of Contents

Introduction

Design

Usage

Examples

Conversion Functions

// This module doesn't require Unicode data, so it can be used as header-only#include"uni_algo/conv.h"// Lenient conversion (cannot fail), "\xD800" is unpaired high surrogate in UTF-16std::string str8 = una::utf16to8(u"Te\xD800st");// std::u8string is supported tooassert(str8 =="Te\xEF\xBF\xBDst");// "\xEF\xBF\xBD" is the replacement character U+FFFD in UTF-8// Strict conversionuna::error error;std::string str8 = una::strict::utf16to8(u"Te\xD800st", error);assert(str8.empty() && error && error.pos() ==2);// Template functions:// Lenient conversion (cannot fail), "\xC2" is truncated sequence in UTF-8std::u16string str16 = una::utf8to16<char,char16_t>("Te\xC2st");assert(str16 ==u"Te\xFFFDst");// "\xFFFD" is the replacement character U+FFFD in UTF-16// Strict conversionuna::error error;std::u16string str16 = una::strict::utf8to16<char,char16_t>("Te\xC2st", error);assert(str16.empty() && error && error.pos() ==2);// Note that most of the functions in the library have template versions because the low level// of the library is type-agnostic; it only requires that the UTF-16 type is enough to store 16-bit// values and the UTF-32 type is enough to store 32-bit values; aside from that, the low-level doesn't// care about the types, so with the template functions you can do some strange stuff like this:std::basic_string<longlong> str16 = una::utf8to16<char,longlong>("😺");std::string str8 = una::utf16to8<longlong,char>(str16);assert(str8 =="😺");// This works perfectly fine; this is allowed and tested. Of course, it is not needed in most// situations, but it can be used to handle std::wstring and wchar_t properly because the only// portable way of using std::wstring is to use it only to store UTF-16 internally, and, for example,// on Linux, convert it to a real UTF-32 std::wstring when it's needed. It can be done like this:// Works on Linux, static_assert on Windowsstd::wstring str32 = una::utf16to32<wchar_t,wchar_t>(L"Test");// It can be helpful if you work with a library that actually does this// for example: https://github.com/taglib/taglib// Aside from such corner cases, you should use non-template functions because they are just shorter.// It's not like only wchar_t is broken, char has the same problem: you never know what it stores.// So most of the functions in the library end with utf8/utf16/utf32. These suffixes show what data// the function works with, the types are irrelevant.// Note that you need to use template functions if you want to use a custom allocator,// because non-template functions always use default std::allocator.

Case Functions

#include"uni_algo/case.h"std::cout << una::cases::to_uppercase_utf8("Straße") <<'\n';std::cout << una::cases::to_lowercase_utf8("ДВА") <<'\n';std::cout << una::cases::to_casefold_utf8("Ligature ffi") <<'\n';std::cout << una::cases::to_titlecase_utf8("teMPuS eDAX reRuM") <<'\n';// Output:// STRASSE// два// ligature ffi// Tempus Edax Rerum// With Greek locale, removes diacritics etc.std::cout << una::cases::to_uppercase_utf8("Ἀριστοτέλης", una::locale("el")) <<'\n';// With Turkish locale, maps i to İ etc.std::cout << una::cases::to_uppercase_utf8("istanbul", una::locale("tr")) <<'\n';// With Dutch locale, maps ij to IJ at the start of a word.std::cout << una::cases::to_titlecase_utf8("ijsland", una::locale("nl")) <<'\n';// Output:// ΑΡΙΣΤΟΤΕΛΗΣ// İSTANBUL// IJslandassert(una::caseless::compare_utf8("ffi","FFI") ==0);una::found found = una::caseless::find_utf8("Ligature fi test","FI");assert(found && found.pos() ==9 && found.end_pos() ==12);// The module provides a very simple collation function too.// Use the Ukrainian alphabet, for examplestd::u32string str32 =U"абвгґдеєжзиіїйклмнопрстуфхцчшщьюяАБВГҐДЕЄЖЗИІЇЙКЛМНОПРСТУФХЦЧШЩЬЮЯ";std::vector<std::string> vec8;for (char32_t c : str32)// Convert the code points to a vector of UTF-8 code units    vec8.emplace_back(una::utf32to8(std::u32string(1, c)));std::shuffle(vec8.begin(), vec8.end(), std::mt19937(42));// Shuffle them just in case// For example, sort them with the binary comparison firststd::sort(vec8.begin(), vec8.end());// Output: ЄІЇАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЬЮЯабвгдежзийклмнопрстуфхцчшщьюяєіїҐґ// Everything is out of place// Sort them with una::casesens::utf8_collatestd::sort(vec8.begin(), vec8.end(), [](auto a,auto b) {returnuna::casesens::collate_utf8(a, b) <0;});std::for_each(vec8.begin(), vec8.end(), [](auto s) { std::cout << s; });std::cout <<'\n';// Output: аАбБвВгГґҐдДеЕєЄжЖзЗиИіІїЇйЙкКлЛмМнНоОпПрРсСтТуУфФхХцЦчЧшШщЩьЬюЮяЯ// This is the correct order for the Ukrainian alphabet// Group them tooauto it = std::unique(vec8.begin(), vec8.end(), [](auto a,auto b) {returnuna::caseless::collate_utf8(a, b) ==0;});vec8.erase(it, vec8.end());// Output: абвгґдеєжзиіїйклмнопрстуфхцчшщьюя

Normalization Functions

#include"uni_algo/norm.h"// Normalization Form C (NFC)// "W" with circumflex == "Ŵ"assert(una::norm::to_nfc_utf8("W\u0302") =="Ŵ");assert(una::norm::is_nfc_utf8("Ŵ") ==true);assert(una::norm::is_nfc_utf8("W\u0302") ==false);// Normalization Form D (NFD)assert(una::norm::to_nfd_utf8("Ŵ") =="W\u0302");assert(una::norm::is_nfd_utf8("Ŵ") ==false);assert(una::norm::is_nfd_utf8("W\u0302") ==true);

Code Point Properties

#include"uni_algo/prop.h"assert(una::codepoint::is_numeric(U'7'));assert(una::codepoint::is_alphabetic(U'W'));assert(una::codepoint::prop{U'W'}.Alphabetic());// Equivalent to the previous one// Other modules can provide more properties#include"uni_algo/case.h"assert(una::codepoint::is_lowercase(U'w'));assert(una::codepoint::prop_case{U'w'}.Lowercase());// Equivalent to the previous one// Code point properties are useful when you want to implement your own Unicode algorithm.// For example, a function that checks that a UTF-8 string contains only alphabetic characters// and works properly for all scripts and all Unicode normalization forms looks something like this:boolis_alphabetic_string_utf8(std::string_view view){// Note that this algorithm uses UTF-8 range view read below about thisfor (char32_t c : view | una::views::utf8)    {if (c ==0x200D || c ==0x200C)// Ignore ZERO WIDTH JOINER/NON-JOINERcontinue;auto prop = una::codepoint::prop{c};if (prop.General_Category_M())// Ignore combining markscontinue;if (!prop.Alphabetic())returnfalse;    }returntrue;}// Note that degenerate cases when a string starts with combining marks or contains// only combining marks are not handled, so the real function will be a bit more complex.

Locale

#include"uni_algo/locale.h"#include"uni_algo/case.h"// Uppercase a string using system localeuna::cases::to_uppercase_utf8("Test", una::locale::system());una::cases::to_uppercase_utf8("Test", una::locale{"en-US"});// Using locale stringuna::cases::to_uppercase_utf8("Test", una::locale::language{"en"});// Using language subtag directly// Parse locale string (can be ill-formed, the parser will normalize it)una::locale locale{"EN_latn_US"};assert(locale.is_empty() ==false);assert(locale == una::locale::language{"en"});assert(locale == una::locale::script{"Latn"});assert(locale == una::locale::region{"US"});assert(locale.to_string() =="en-Latn-US");// Use locale subtags in switch caseswitch (una::locale::system().get_language()){case una::locale::language{"en"}:        std::cout <<"English Language" <<'\n';break;case una::locale::language{"ja"}:        std::cout <<"Japanese Language" <<'\n';break;case una::locale::language{"ar"}:        std::cout <<"Arabic Language" <<'\n';break;}// Note that system locale functions are not available in header-only and single include versions.

Basic Ranges

// These modules don't require Unicode data, so they can be used as header-only#include"uni_algo/ranges.h"#include"uni_algo/ranges_conv.h"// Ranges in this library are compatible with C++20 ranges.// In C++17, the library implements some basic ranges that are similar to C++20 ranges:una::views::reverseuna::views::transformuna::views::filteruna::views::dropuna::views::take// In C++20, you can just use std::ranges variants of these ranges,// but always use una::views::reverse from this library// because std::views::reverse is not good enought for Unicode.// You use these ranges the same way you use C++20 rangesassert((std::string_view{"ⅠⅡⅢⅣⅤⅥⅦⅧⅨⅩ"}        | una::views::utf8        | una::views::reverse | una::views::drop(3) | una::views::take(5)        | una::views::reverse | una::ranges::to_utf8<std::string>())        =="ⅢⅣⅤⅥⅦ");

Conversion Ranges

// This module doesn't require Unicode data, so it can be used as header-only#include"uni_algo/ranges_conv.h"// A simple conversion function using ranges looks like this:std::u16stringutf8to16(std::string_view view){return view | una::views::utf8 | una::ranges::to_utf16<std::u16string>();}// The following examples demonstrate how to work directly with a conversion range view// 3 UTF-8 code pointsstd::string str8 ="😺😼🙀";// These emojis use 1 code point// Make UTF-8 view from std::stringauto view = una::ranges::utf8_view{str8};// Count code pointsstd::cout <<"Number of code points:" << std::distance(view.begin(), view.end()) <<'\n';// Print code pointsfor (auto it = view.begin(); it != view.end(); ++it){    std::cout <<"Code point:" <<std::to_string(*it)              <<" at:" << it.begin() - str8.begin() <<'\n';}// Skip 2 code pointsif (auto it = std::next(view.begin(),2); it != view.end())    std::cout <<"3rd code point at:" << it.begin() - str8.begin() <<'\n';// Find code point 128572if (auto it = std::find(view.begin(), view.end(),128572); it != view.end())    std::cout <<"Code point 128572 at:" << it.begin() - str8.begin() <<'\n';std::cout <<"Reverse order:" <<'\n';for (auto it = view.end(); it != view.begin();){    --it;    std::cout <<"Code point:" <<std::to_string(*it)              <<" at:" << it.begin() - str8.begin() <<'\n';}// Output:// Number of code points: 3// Code point: 128570 at: 0// Code point: 128572 at: 4// Code point: 128576 at: 8// 3rd code point at: 8// Code point 128572 at: 4// Reverse order:// Code point: 128576 at: 8// Code point: 128572 at: 4// Code point: 128570 at: 0

Grapheme/Word Ranges

#include"uni_algo/ranges_grapheme.h"#include"uni_algo/ranges_word.h"// Grapheme/Word aka text segmentation ranges are similar to conversion ranges,// but they return subranges in the form of std::string_view.// They use default grapheme/word boundary rules from The Unicode Standard UAX #29std::string str8 ="Άλμπερτ Αϊνστάιν";// Replace it with una::views::grapheme::utf8 for grapheme segmentationauto view = una::views::word::utf8(str8);// Count words/graphemes (note that a space/punctuation is a word too according to the rules)std::cout <<"Number of words:" << std::distance(view.begin(), view.end()) <<'\n';// Print positions of the words/graphemesfor (auto it = view.begin(); it != view.end(); ++it)    std::cout <<"Word at:" << it.begin() - str8.begin() <<'\n';// Word/grapheme tokenizerstd::cout <<"Words:" <<'\n';// Note that we use the word_only view here to skip all punctuation and spacesfor (std::string_view word : una::views::word_only::utf8(str8))    std::cout << word <<'\n';// Output:// Number of words: 3// Word at: 0// Word at: 14// Word at: 15// Words:// Άλμπερτ// Αϊνστάιν// For example, you need to append or insert a code point or grapheme to a string// To append a code point to a UTF-8 string, all you need is the Conversion modulestd::string str8 ="🏴󠁧󠁢󠁷󠁬󠁳󠁿👨‍👩‍👧🧙‍♀️";// These emojis use multiple code pointsstr8 += una::utf32to8(std::u32string{U'😺'});// This emoji use 1 code point// But if you need to insert a code point, you need text segmentation ranges tooauto view = una::views::grapheme::utf8(str8);// Insert a code point after the 2nd graphemeauto pos = std::next(view.begin(),2).begin() - str8.begin();str8.insert(pos, una::utf32to8(std::u32string{U'😼'}));// This emoji uses 1 code point// Output: 🏴󠁧󠁢󠁷󠁬󠁳󠁿👨‍👩‍👧😼🧙‍♀️😺// The same way a grapheme can be appended or inserted, and of course, you don't even need// the Conversion module if your grapheme or code point is already in UTF-8 encoding.

Normalization Ranges

// For example, we need to remove accents that mark the stressed vowels in a text.// The algorithm is simple: NFD -> Remove grave and acute accents (U+0300 and U+0301) -> NFC// Note that the library has the una::norm::to_unaccent_utf8 function, but it won't produce// the same result because it removes all accents, not specific ones,// but it is possible to implement this algorithm using ranges like this.#include"uni_algo/ranges.h"#include"uni_algo/ranges_conv.h"#include"uni_algo/ranges_norm.h"std::stringremove_grave_and_acute_accents(std::string_view str_view){// Use ranges: UTF-8 -> NFD -> Filter -> NFC -> To UTF-8// Note that in C++20 una::views::filter can be replaced with std::views::filterreturn str_view         | una::views::utf8         | una::views::norm::nfd         |una::views::filter([](char32_t c) {return c !=0x0300 && c !=0x0301; })         | una::views::norm::nfc         | una::ranges::to_utf8<std::string>();}std::cout << remove_grave_and_acute_accents("bótte bòtte") <<'\n';// Output: botte botte

Ranges vs Functions

Unicode Algorithms

Performance

C Language

SQLite Extension

License