Encodings↑

The Basics↑

Acharacter encoding, often shortened toencoding, is a mapping between:

Some character sets contain only 1-byte characters;US-ASCII, for example, has 256 1-byte characters. This string, encoded in US-ASCII, has six characters that are stored as six bytes:

s ='Hello!'.encode(Encoding::US_ASCII)# => "Hello!"s.encoding# => #<Encoding:US-ASCII>s.bytes# => [72, 101, 108, 108, 111, 33]

Other encodings may involve multi-byte characters.UTF-8, for example, encodes more than one million characters, encoding each in one to four bytes. The lowest-valued of these characters correspond to ASCII characters, and so are 1-byte characters:

s ='Hello!'# => "Hello!"s.bytes# => [72, 101, 108, 108, 111, 33]

Other characters, such as the Euro symbol, are multi-byte:

s ="\u20ac"# => "€"s.bytes# => [226, 130, 172]

The Encoding Class↑

Encoding Objects↑

Ruby encodings are defined by constants in class Encoding. There can be only one instance of Encoding for each of these constants. MethodEncoding.list returns an array of Encoding objects (one for each constant):

Encoding.list.size# => 103Encoding.list.first.class# => EncodingEncoding.list.take(3)# => [#<Encoding:ASCII-8BIT>, #<Encoding:UTF-8>, #<Encoding:US-ASCII>]

Names and Aliases↑

MethodEncoding#name returns the name of an Encoding:

Encoding::ASCII_8BIT.name# => "ASCII-8BIT"Encoding::WINDOWS_31J.name# => "Windows-31J"

An Encoding object has zero or more aliases; methodEncoding#names returns an array containing the name and all aliases:

Encoding::ASCII_8BIT.names# => ["ASCII-8BIT", "BINARY"]Encoding::WINDOWS_31J.names#=> ["Windows-31J", "CP932", "csWindows31J", "SJIS", "PCK"]

MethodEncoding.aliases returns a hash of all alias/name pairs:

Encoding.aliases.size# => 71Encoding.aliases.take(3)# => [["BINARY", "ASCII-8BIT"], ["CP437", "IBM437"], ["CP720", "IBM720"]]

MethodEncoding.name_list returns an array of all the encoding names and aliases:

Encoding.name_list.size# => 175Encoding.name_list.take(3)# => ["ASCII-8BIT", "UTF-8", "US-ASCII"]

Methodname_list returns more entries than methodlist because it includes both the names and their aliases.

MethodEncoding.find returns the Encoding for a given name or alias, if it exists:

Encoding.find("US-ASCII")# => #<Encoding:US-ASCII>Encoding.find("US-ASCII").class# => Encoding

Default Encodings↑

MethodEncoding.find, above, also returns a default Encoding for each of these special names:

MethodEncoding.default_external returns the default external Encoding:

Encoding.default_external# => #<Encoding:UTF-8>

MethodEncoding.default_external= sets that value:

Encoding.default_external =Encoding::US_ASCII# => #<Encoding:US-ASCII>Encoding.default_external# => #<Encoding:US-ASCII>

MethodEncoding.default_internal returns the default internal Encoding:

Encoding.default_internal# => nil

MethodEncoding.default_internal= sets the default internal Encoding:

Encoding.default_internal =Encoding::US_ASCII# => #<Encoding:US-ASCII>Encoding.default_internal# => #<Encoding:US-ASCII>

Compatible Encodings↑

MethodEncoding.compatible? returns whether two given objects are encoding-compatible (that is, whether they can be concatenated); returns the Encoding of the concatenated string, ornil if incompatible:

rus ="\u{442 435 441 442}"eng ='text'Encoding.compatible?(rus,eng)# => #<Encoding:UTF-8>s0 ="\xa1\xa1".force_encoding(Encoding::ISO_8859_1)# => "\xA1\xA1"s1 ="\xa1\xa1".force_encoding(Encoding::EUCJP)# => "\x{A1A1}"Encoding.compatible?(s0,s1)# => nil

String Encoding↑

A RubyString object has an encoding that is an instance of class Encoding. The encoding may be retrieved by methodString#encoding.

The default encoding for a string literal is the script encoding; seeScript Encoding.

's'.encoding# => #<Encoding:UTF-8>

The default encoding for a string created with methodString.new is:

In either case, any encoding may be specified:

s =String.new(encoding:Encoding::UTF_8)# => ""s.encoding# => #<Encoding:UTF-8>s =String.new('foo',encoding:Encoding::BINARY)# => "foo"s.encoding# => #<Encoding:BINARY (ASCII-8BIT)>

The encoding for a string may be changed:

s ="R\xC3\xA9sum\xC3\xA9"# => "Résumé"s.encoding# => #<Encoding:UTF-8>s.force_encoding(Encoding::ISO_8859_1)# => "R\xC3\xA9sum\xC3\xA9"s.encoding# => #<Encoding:ISO-8859-1>

Changing the assigned encoding does not alter the content of the string; it changes only the way the content is to be interpreted:

s# => "R\xC3\xA9sum\xC3\xA9"s.force_encoding(Encoding::UTF_8)# => "Résumé"

The actual content of a string may also be altered; seeTranscoding a String.

Here are a couple of useful query methods:

s ="abc".force_encoding(Encoding::UTF_8)# => "abc"s.ascii_only?# => trues ="abc\u{6666}".force_encoding(Encoding::UTF_8)# => "abc晦"s.ascii_only?# => falses ="\xc2\xa1".force_encoding(Encoding::UTF_8)# => "¡"s.valid_encoding?# => trues ="\xc2".force_encoding(Encoding::UTF_8)# => "\xC2"s.valid_encoding?# => false

Symbol and Regexp Encodings↑

The string stored in aSymbol orRegexp object also has an encoding; the encoding may be retrieved by methodSymbol#encoding orRegexp#encoding.

The default encoding for these, however, is:

Filesystem Encoding↑

The filesystem encoding is the default Encoding for a string from the filesystem:

Encoding.find("filesystem")# => #<Encoding:UTF-8>

Locale Encoding↑

The locale encoding is the default encoding for a string from the environment, other than from the filesystem:

Encoding.find('locale')# => #<Encoding:IBM437>

Stream Encodings↑

Certain stream objects can have two encodings; these objects include instances of:

The two encodings are:

External Encoding↑

The external encoding, which is an Encoding object, specifies how bytes read from the stream are to be interpreted as characters.

The default external encoding is:

The default external encoding is returned by methodEncoding.default_external, and may be set by:

You can also set the default external encoding using methodEncoding.default_external=, but doing so may cause problems; strings created before and after the change may have a different encodings.

For an IO or File object, the external encoding may be set by:

For an IO, File, ARGF, or StringIO object, the external encoding may be set by:

Internal Encoding↑

The internal encoding, which is an Encoding object ornil, specifies how characters read from the stream are to be converted to characters in the internal encoding; those characters become a string whose encoding is set to the internal encoding.

The default internal encoding isnil (no conversion). It is returned by methodEncoding.default_internal, and may be set by:

You can also set the default internal encoding using methodEncoding.default_internal=, but doing so may cause problems; strings created before and after the change may have a different encodings.

For an IO or File object, the internal encoding may be set by:

For an IO, File, ARGF, or StringIO object, the internal encoding may be set by:

Script Encoding↑

A Ruby script has a script encoding, which may be retrieved by:

__ENCODING__# => #<Encoding:UTF-8>

The default script encoding is UTF-8; a Ruby source file may set its script encoding with a magic comment on the first line of the file (or second line, if there is a shebang on the first). The comment must contain the wordcoding orencoding, followed by a colon, space and theEncoding name or alias:

# encoding: ISO-8859-1__ENCODING__#=> #<Encoding:ISO-8859-1>

Transcoding↑

Transcoding is the process of changing a sequence of characters from one encoding to another.

As far as possible, the characters remain the same, but the bytes that represent them may change.

The handling for characters that cannot be represented in the destination encoding may be specified by @Encoding+Options.

Transcoding a String↑

Each of these methods transcodes a string:

Transcoding a Stream↑

Each of these methods may transcode a stream; whether it does so depends on the external and internal encodings:

This example writes a string to a file, encoding it as ISO-8859-1, then reads the file into a new string, encoding it as UTF-8:

s ="R\u00E9sum\u00E9"path ='t.tmp'ext_enc =Encoding::ISO_8859_1int_enc =Encoding::UTF_8File.write(path,s,external_encoding:ext_enc)raw_text =File.binread(path)transcoded_text =File.read(path,external_encoding:ext_enc,internal_encoding:int_enc)praw_textptranscoded_text

Output:

"R\xE9sum\xE9""Résumé"

Encoding Options↑

A number of methods in the Ruby core accept keyword arguments as encoding options.

Some of the options specify or utilize areplacementstring, to be used in certain transcoding operations. A replacement string may be in any encoding that can be converted to the encoding of the destination string.

These keyword-value pairs specify encoding options: