Decimal Numbers#

classBasicDecimal128:publicarrow::GenericBasicDecimal<BasicDecimal128,128>#

Represents a signed 128-bit integer in two’s complement.

This class is also compiled into LLVM IR - so, it should not have cpp references like streams and boost.

Subclassed byarrow::Decimal128

Public Functions

inlineconstexprBasicDecimal128(int64_thigh,uint64_tlow)noexcept#

Create aBasicDecimal128 from the two’s complement representation.

BasicDecimal128&Negate()#

Negate the current value (in-place)

BasicDecimal128&Abs()#

Absolute value (in-place)

BasicDecimal128&operator+=(constBasicDecimal128&right)#

Add a number to this one. The result is truncated to 128 bits.

BasicDecimal128&operator-=(constBasicDecimal128&right)#

Subtract a number from this one. The result is truncated to 128 bits.

BasicDecimal128&operator*=(constBasicDecimal128&right)#

Multiply this number by another number. The result is truncated to 128 bits.

DecimalStatusDivide(constBasicDecimal128&divisor,BasicDecimal128*result,BasicDecimal128*remainder)const#

Divide this number by right and return the result.

This operation is not destructive. The answer rounds to zero. Signs work like: 21 / 5 -> 4, 1 -21 / 5 -> -4, -1 21 / -5 -> -4, 1 -21 / -5 -> 4, -1

Parameters:

• divisor –[in] the number to divide by

• result –[out] the quotient

• remainder –[out] the remainder after the division

BasicDecimal128&operator/=(constBasicDecimal128&right)#

In-place division.

BasicDecimal128&operator|=(constBasicDecimal128&right)#

Bitwise “or” between twoBasicDecimal128.

BasicDecimal128&operator&=(constBasicDecimal128&right)#

Bitwise “and” between twoBasicDecimal128.

BasicDecimal128&operator<<=(uint32_tbits)#

Shift left by the given number of bits.

BasicDecimal128&operator>>=(uint32_tbits)#

Shift right by the given number of bits.

Negative values will sign-extend.

inlineconstexprint64_thigh_bits()const#

Get the high bits of the two’s complement representation of the number.

inlineconstexpruint64_tlow_bits()const#

Get the low bits of the two’s complement representation of the number.

voidGetWholeAndFraction(int32_tscale,BasicDecimal128*whole,BasicDecimal128*fraction)const#

separate the integer and fractional parts for the given scale.

DecimalStatusRescale(int32_toriginal_scale,int32_tnew_scale,BasicDecimal128*out)const#

ConvertBasicDecimal128 from one scale to another.

BasicDecimal128IncreaseScaleBy(int32_tincrease_by)const#

Scale up.

BasicDecimal128ReduceScaleBy(int32_treduce_by,boolround=true)const#

Scale down.

• If ‘round’ is true, the right-most digits are dropped and the result value is rounded up (+1 for +ve, -1 for -ve) based on the value of the dropped digits (>= 10^reduce_by / 2).

• If ‘round’ is false, the right-most digits are simply dropped.

boolFitsInPrecision(int32_tprecision)const#

Whether this number fits in the given precision.

Return true if the number of significant digits is less or equal toprecision.

int32_tCountLeadingBinaryZeros()const#

count the number of leading binary zeroes.

inlineconstexprGenericBasicDecimal()noexcept#

Empty constructor creates a decimal with a value of 0.

inlineexplicitconstexprGenericBasicDecimal(constWordArray&array)noexcept#

Create a decimal from the two’s complement representation.

Input array is assumed to be in native endianness.

inlineGenericBasicDecimal(LittleEndianArrayTag,constWordArray&array)noexcept#

Create a decimal from the two’s complement representation.

Input array is assumed to be in little endianness, with native endian elements.

template<typenameT,typename=typenamestd::enable_if<std::is_integral<T>::value&&(sizeof(T)<=sizeof(uint64_t)),T>::type>
inlineconstexprGenericBasicDecimal(Tvalue)noexcept#

Create a decimal from any integer not wider than 64 bits.

inlineexplicitGenericBasicDecimal(constuint8_t*bytes)#

Create a decimal from an array of bytes.

Bytes are assumed to be in native-endian byte order.

Public Static Functions

staticBasicDecimal128Abs(constBasicDecimal128&left)#

Absolute value.

staticconstBasicDecimal128&GetScaleMultiplier(int32_tscale)#

Scale multiplier for given scale value.

staticconstBasicDecimal128&GetHalfScaleMultiplier(int32_tscale)#

Half-scale multiplier for given scale value.

staticconstBasicDecimal128&GetMaxValue()#

Get the maximum valid unscaled decimal value.

staticBasicDecimal128GetMaxValue(int32_tprecision)#

Get the maximum valid unscaled decimal value for the given precision.

staticinlineconstexprBasicDecimal128GetMaxSentinel()#

Get the maximum decimal value (is not a valid value).

staticinlineconstexprBasicDecimal128GetMinSentinel()#

Get the minimum decimal value (is not a valid value).

classDecimal128:publicarrow::BasicDecimal128#

Represents a signed 128-bit integer in two’s complement.

Calculations wrap around and overflow is ignored. The max decimal precision that can be safely represented is 38 significant digits.

For a discussion of the algorithms, look at Knuth’s volume 2, Semi-numerical Algorithms section 4.3.1.

Adapted from the Apache ORC C++ implementation

The implementation is split into two parts :

1. BasicDecimal128

   • can be safely compiled to IR without references to libstdc++.

2. Decimal128

   • has additional functionality on top ofBasicDecimal128 to deal with strings and streams.

Public Functions

inlineconstexprDecimal128(constBasicDecimal128&value)noexcept#

constructor creates aDecimal128 from aBasicDecimal128.

explicitDecimal128(conststd::string&value)#

Parse the number from a base 10 string representation.

inlineconstexprDecimal128()noexcept#

Empty constructor creates aDecimal128 with a value of 0.

inlineResult<std::pair<Decimal128,Decimal128>>Divide(constDecimal128&divisor)const#

Divide this number by right and return the result.

This operation is not destructive. The answer rounds to zero. Signs work like: 21 / 5 -> 4, 1 -21 / 5 -> -4, -1 21 / -5 -> -4, 1 -21 / -5 -> 4, -1

Parameters:

divisor –[in] the number to divide by

Returns:

the pair of the quotient and the remainder

std::stringToString(int32_tscale)const#

Convert theDecimal128 value to a base 10 decimal string with the given scale.

std::stringToIntegerString()const#

Convert the value to an integer string.

explicitoperatorint64_t()const#

Cast this value to an int64_t.

inlineResult<Decimal128>Rescale(int32_toriginal_scale,int32_tnew_scale)const#

ConvertDecimal128 from one scale to another.

template<typenameT,typename=internal::EnableIfIsOneOf<T,int32_t,int64_t>>
inlineResult<T>ToInteger()const#

Convert to a signed integer.

template<typenameT,typename=internal::EnableIfIsOneOf<T,int32_t,int64_t>>
inlineStatusToInteger(T*out)const#

Convert to a signed integer.

floatToFloat(int32_tscale)const#

Convert to a floating-point number (scaled)

doubleToDouble(int32_tscale)const#

Convert to a floating-point number (scaled)

template<typenameT,typename=std::enable_if_t<std::is_floating_point_v<T>>>
inlineTToReal(int32_tscale)const#

Convert to a floating-point number (scaled)

Public Static Functions

staticStatusFromString(std::string_views,Decimal128*out,int32_t*precision,int32_t*scale=NULLPTR)#

Convert a decimal string to aDecimal128 value, optionally including precision and scale if they’re passed in and not null.

staticResult<Decimal128>FromBigEndian(constuint8_t*data,int32_tlength)#

Convert from a big-endian byte representation.

The length must be between 1 and 16.

Returns:

error status if the length is an invalid value

classBasicDecimal256:publicarrow::GenericBasicDecimal<BasicDecimal256,256>#

Subclassed byarrow::Decimal256

Public Functions

BasicDecimal256&Negate()#

Negate the current value (in-place)

BasicDecimal256&Abs()#

Absolute value (in-place)

BasicDecimal256&operator+=(constBasicDecimal256&right)#

Add a number to this one. The result is truncated to 256 bits.

BasicDecimal256&operator-=(constBasicDecimal256&right)#

Subtract a number from this one. The result is truncated to 256 bits.

inlineuint64_tlow_bits()const#

Get the lowest bits of the two’s complement representation of the number.

voidGetWholeAndFraction(int32_tscale,BasicDecimal256*whole,BasicDecimal256*fraction)const#

separate the integer and fractional parts for the given scale.

DecimalStatusRescale(int32_toriginal_scale,int32_tnew_scale,BasicDecimal256*out)const#

ConvertBasicDecimal256 from one scale to another.

BasicDecimal256IncreaseScaleBy(int32_tincrease_by)const#

Scale up.

BasicDecimal256ReduceScaleBy(int32_treduce_by,boolround=true)const#

Scale down.

• If ‘round’ is true, the right-most digits are dropped and the result value is rounded up (+1 for positive, -1 for negative) based on the value of the dropped digits (>= 10^reduce_by / 2).

• If ‘round’ is false, the right-most digits are simply dropped.

boolFitsInPrecision(int32_tprecision)const#

Whether this number fits in the given precision.

Return true if the number of significant digits is less or equal toprecision.

BasicDecimal256&operator*=(constBasicDecimal256&right)#

Multiply this number by another number. The result is truncated to 256 bits.

DecimalStatusDivide(constBasicDecimal256&divisor,BasicDecimal256*result,BasicDecimal256*remainder)const#

Divide this number by right and return the result.

This operation is not destructive. The answer rounds to zero. Signs work like: 21 / 5 -> 4, 1 -21 / 5 -> -4, -1 21 / -5 -> -4, 1 -21 / -5 -> 4, -1

Parameters:

• divisor –[in] the number to divide by

• result –[out] the quotient

• remainder –[out] the remainder after the division

BasicDecimal256&operator<<=(uint32_tbits)#

Shift left by the given number of bits.

BasicDecimal256&operator>>=(uint32_tbits)#

Shift right by the given number of bits.

Negative values will sign-extend.

BasicDecimal256&operator/=(constBasicDecimal256&right)#

In-place division.

inlineconstexprGenericBasicDecimal()noexcept#

Empty constructor creates a decimal with a value of 0.

inlineexplicitconstexprGenericBasicDecimal(constWordArray&array)noexcept#

Create a decimal from the two’s complement representation.

Input array is assumed to be in native endianness.

inlineGenericBasicDecimal(LittleEndianArrayTag,constWordArray&array)noexcept#

Create a decimal from the two’s complement representation.

Input array is assumed to be in little endianness, with native endian elements.

template<typenameT,typename=typenamestd::enable_if<std::is_integral<T>::value&&(sizeof(T)<=sizeof(uint64_t)),T>::type>
inlineconstexprGenericBasicDecimal(Tvalue)noexcept#

Create a decimal from any integer not wider than 64 bits.

inlineexplicitGenericBasicDecimal(constuint8_t*bytes)#

Create a decimal from an array of bytes.

Bytes are assumed to be in native-endian byte order.

Public Static Functions

staticBasicDecimal256Abs(constBasicDecimal256&left)#

Absolute value.

staticconstBasicDecimal256&GetScaleMultiplier(int32_tscale)#

Scale multiplier for given scale value.

staticconstBasicDecimal256&GetHalfScaleMultiplier(int32_tscale)#

Half-scale multiplier for given scale value.

staticBasicDecimal256GetMaxValue(int32_tprecision)#

Get the maximum valid unscaled decimal value for the given precision.

staticinlineconstexprBasicDecimal256GetMaxSentinel()#

Get the maximum decimal value (is not a valid value).

staticinlineconstexprBasicDecimal256GetMinSentinel()#

Get the minimum decimal value (is not a valid value).

classDecimal256:publicarrow::BasicDecimal256#

Represents a signed 256-bit integer in two’s complement.

The max decimal precision that can be safely represented is 76 significant digits.

The implementation is split into two parts :

1. BasicDecimal256

   • can be safely compiled to IR without references to libstdc++.

2. Decimal256

   • (TODO) has additional functionality on top ofBasicDecimal256 to deal with strings and streams.

Public Functions

inlineconstexprDecimal256(constBasicDecimal256&value)noexcept#

constructor creates aDecimal256 from aBasicDecimal256.

explicitDecimal256(conststd::string&value)#

Parse the number from a base 10 string representation.

inlineconstexprDecimal256()noexcept#

Empty constructor creates aDecimal256 with a value of 0.

std::stringToString(int32_tscale)const#

Convert theDecimal256 value to a base 10 decimal string with the given scale.

std::stringToIntegerString()const#

Convert the value to an integer string.

inlineResult<Decimal256>Rescale(int32_toriginal_scale,int32_tnew_scale)const#

ConvertDecimal256 from one scale to another.

inlineResult<std::pair<Decimal256,Decimal256>>Divide(constDecimal256&divisor)const#

Divide this number by right and return the result.

This operation is not destructive. The answer rounds to zero. Signs work like: 21 / 5 -> 4, 1 -21 / 5 -> -4, -1 21 / -5 -> -4, 1 -21 / -5 -> 4, -1

Parameters:

divisor –[in] the number to divide by

Returns:

the pair of the quotient and the remainder

floatToFloat(int32_tscale)const#

Convert to a floating-point number (scaled).

May return infinity in case of overflow.

doubleToDouble(int32_tscale)const#

Convert to a floating-point number (scaled)

template<typenameT,typename=std::enable_if_t<std::is_floating_point_v<T>>>
inlineTToReal(int32_tscale)const#

Convert to a floating-point number (scaled)

Public Static Functions

staticStatusFromString(std::string_views,Decimal256*out,int32_t*precision,int32_t*scale=NULLPTR)#

Convert a decimal string to aDecimal256 value, optionally including precision and scale if they’re passed in and not null.

staticResult<Decimal256>FromBigEndian(constuint8_t*data,int32_tlength)#

Convert from a big-endian byte representation.

The length must be between 1 and 32.

Returns:

error status if the length is an invalid value

Iterators#

template<typenameT>
classIterator:publicarrow::util::EqualityComparable<Iterator<T>>#

A genericIterator that can return errors.

Public Functions

template<typenameWrapped>
inlineexplicitIterator(Wrappedhas_next)#

Iterator may be constructed from any type which has a member function with signature Result<T>Next(); End of iterator is signalled by returning IteratorTraits<T>::End();.

The argument is moved or copied to the heap and kept in a unique_ptr<void>. Only its destructor and its Next method (which are stored in function pointers) are referenced after construction.

This approach is used to dodge MSVC linkage hell (ARROW-6244, ARROW-6558) when using an abstract template base class: instead of being inlined as usual for a template function the base’s virtual destructor will be exported, leading to multiple definition errors when linking to any other TU where the base is instantiated.

inlineResult<T>Next()#

Return the next element of the sequence, IterationTraits<T>::End() when the iteration is completed.

template<typenameVisitor>
inlineStatusVisit(Visitor&&visitor)#

Pass each element of the sequence to a visitor.

Will return any error status returned by the visitor, terminating iteration.

inlineboolEquals(constIterator&other)const#

Iterators will only compare equal if they are both null.

Equality comparability is required to make anIterator of Iterators (to check for the end condition).

inlineResult<std::vector<T>>ToVector()#

Move every element of this iterator into a vector.

classRangeIterator#

template<typenameT>
classVectorIterator#

Simple iterator which yields the elements of a std::vector.

Comparison#

classEqualOptions#

A container of options for equality comparisons.

Public Functions

inlineboolnans_equal()const#

Whether or not NaNs are considered equal.

inlineEqualOptionsnans_equal(boolv)const#

Return a newEqualOptions object with the “nans_equal” property changed.

inlineboolsigned_zeros_equal()const#

Whether or not zeros with differing signs are considered equal.

inlineEqualOptionssigned_zeros_equal(boolv)const#

Return a newEqualOptions object with the “signed_zeros_equal” property changed.

inlinebooluse_atol()const#

Whether the “atol” property is used in the comparison.

This option only affects the Equals methods and has no effect on ApproxEquals methods.

inlineEqualOptionsuse_atol(boolv)const#

Return a newEqualOptions object with the “use_atol” property changed.

inlinedoubleatol()const#

The absolute tolerance for approximate comparisons of floating-point values.

Note that this option is ignored if “use_atol” is set to false.

inlineEqualOptionsatol(doublev)const#

Return a newEqualOptions object with the “atol” property changed.

inlinebooluse_schema()const#

Whether thearrow::Schema property is used in the comparison.

This option only affects the Equals methods and has no effect on ApproxEquals methods.

inlineEqualOptionsuse_schema(boolv)const#

Return a newEqualOptions object with the “use_schema_” property changed.

Setting this option is false making the value ofEqualOptions::use_metadata is ignored.

inlinebooluse_metadata()const#

Whether the “metadata” inarrow::Schema is used in the comparison.

This option only affects the Equals methods and has no effect on the ApproxEquals methods.

Note: This option is only considered whenarrow::EqualOptions::use_schema is set to true.

inlineEqualOptionsuse_metadata(boolv)const#

Return a newEqualOptions object with the “use_metadata” property changed.

inlinestd::ostream*diff_sink()const#

The ostream to which a diff will be formatted if arrays disagree.

If this is null (the default) no diff will be formatted.

inlineEqualOptionsdiff_sink(std::ostream*diff_sink)const#

Return a newEqualOptions object with the “diff_sink” property changed.

This option will be ignored if diff formatting of the types of compared arrays is not supported.

Compression#

enumarrow::Compression::type#

Compression algorithm.

Values:

enumeratorUNCOMPRESSED#

enumeratorSNAPPY#

enumeratorGZIP#

enumeratorBROTLI#

enumeratorZSTD#

enumeratorLZ4#

enumeratorLZ4_FRAME#

enumeratorLZO#

enumeratorBZ2#

enumeratorLZ4_HADOOP#

classCodec#

Compression codec.

Public Functions

virtualintminimum_compression_level()const=0#

Return the smallest supported compression level.

virtualintmaximum_compression_level()const=0#

Return the largest supported compression level.

virtualintdefault_compression_level()const=0#

Return the default compression level.

virtualResult<int64_t>Decompress(int64_tinput_len,constuint8_t*input,int64_toutput_buffer_len,uint8_t*output_buffer)=0#

One-shot decompression function.

output_buffer_len must be correct and therefore be obtained in advance. The actual decompressed length is returned.

Note

One-shot decompression is not always compatible with streaming compression. Depending on the codec (e.g. LZ4), different formats may be used.

virtualResult<int64_t>Compress(int64_tinput_len,constuint8_t*input,int64_toutput_buffer_len,uint8_t*output_buffer)=0#

One-shot compression function.

output_buffer_len must first have been computed using MaxCompressedLen(). The actual compressed length is returned.

Note

One-shot compression is not always compatible with streaming decompression. Depending on the codec (e.g. LZ4), different formats may be used.

virtualResult<std::shared_ptr<Compressor>>MakeCompressor()=0#

Create a streaming compressor instance.

virtualResult<std::shared_ptr<Decompressor>>MakeDecompressor()=0#

Create a streaming compressor instance.

virtualCompression::typecompression_type()const=0#

ThisCodec’s compression type.

inlineconststd::string&name()const#

The name of thisCodec’s compression type.

inlinevirtualintcompression_level()const#

ThisCodec’s compression level, if applicable.

Public Static Functions

staticintUseDefaultCompressionLevel()#

Return special value to indicate that a codec implementation should use its default compression level.

staticconststd::string&GetCodecAsString(Compression::typet)#

Return a string name for compression type.

staticResult<Compression::type>GetCompressionType(conststd::string&name)#

Return compression type for name (all lower case)

staticResult<std::unique_ptr<Codec>>Create(Compression::typecodec,constCodecOptions&codec_options=CodecOptions{})#

Create a codec for the given compression algorithm with CodecOptions.

staticResult<std::unique_ptr<Codec>>Create(Compression::typecodec,intcompression_level)#

Create a codec for the given compression algorithm.

staticboolIsAvailable(Compression::typecodec)#

Return true if support for indicated codec has been enabled.

staticboolSupportsCompressionLevel(Compression::typecodec)#

Return true if indicated codec supports setting a compression level.

staticResult<int>MinimumCompressionLevel(Compression::typecodec)#

Return the smallest supported compression level for the codec Note: This function creates a temporaryCodec instance.

staticResult<int>MaximumCompressionLevel(Compression::typecodec)#

Return the largest supported compression level for the codec Note: This function creates a temporaryCodec instance.

staticResult<int>DefaultCompressionLevel(Compression::typecodec)#

Return the default compression level Note: This function creates a temporaryCodec instance.

classCompressor#

Streaming compressor interface.

Public Functions

virtualResult<CompressResult>Compress(int64_tinput_len,constuint8_t*input,int64_toutput_len,uint8_t*output)=0#

Compress some input.

If bytes_read is 0 on return, then a larger output buffer should be supplied.

virtualResult<FlushResult>Flush(int64_toutput_len,uint8_t*output)=0#

Flush part of the compressed output.

If should_retry is true on return,Flush() should be called again with a larger buffer.

virtualResult<EndResult>End(int64_toutput_len,uint8_t*output)=0#

End compressing, doing whatever is necessary to end the stream.

If should_retry is true on return,End() should be called again with a larger buffer. Otherwise, theCompressor should not be used anymore.

End() impliesFlush().

structCompressResult#

structEndResult#

structFlushResult#

classDecompressor#

Streaming decompressor interface.

Public Functions

virtualResult<DecompressResult>Decompress(int64_tinput_len,constuint8_t*input,int64_toutput_len,uint8_t*output)=0#

Decompress some input.

If need_more_output is true on return, a larger output buffer needs to be supplied.

virtualboolIsFinished()=0#

Return whether the compressed stream is finished.

This is a heuristic. If true is returned, then it is guaranteed that the stream is finished. If false is returned, however, it may simply be that the underlying library isn’t able to provide the information.

virtualStatusReset()=0#

Reinitialize decompressor, making it ready for a new compressed stream.

structDecompressResult#

Visitors#

template<typenameVISITOR,typename...ARGS>
inlineStatusarrow::VisitTypeInline(constDataType&type,VISITOR*visitor,ARGS&&...args)#

Callsvisitor with the corresponding concrete type class.

A visitor is a type that implements specialized logic for each Arrow type. Example usage:

classExampleVisitor{arrow::StatusVisit(constarrow::Int32Type&type){...}arrow::StatusVisit(constarrow::Int64Type&type){...}...}ExampleVisitorvisitor;VisitTypeInline(some_type,&visitor);

Template Parameters:

• VISITOR – Visitor type that implements Visit() for all Arrow types.

• ARGS – Additional arguments, if any, will be passed to the Visit function after thetype argument

Returns:

Status

template<typenameVISITOR,typename...ARGS>
inlineStatusarrow::VisitTypeIdInline(Type::typeid,VISITOR*visitor,ARGS&&...args)#

Callsvisitor with a nullptr of the corresponding concrete type class.

Template Parameters:

• VISITOR – Visitor type that implements Visit() for all Arrow types.

• ARGS – Additional arguments, if any, will be passed to the Visit function after thetype argument

Returns:

Status

template<typenameVISITOR,typename...ARGS>
inlineStatusarrow::VisitScalarInline(constScalar&scalar,VISITOR*visitor,ARGS&&...args)#

Apply the visitors Visit() method specialized to the scalar type.

A visitor is a type that implements specialized logic for each Arrow type. Example usage:

classExampleVisitor{arrow::StatusVisit(arrow::Int32Scalarscalar){...}arrow::StatusVisit(arrow::Int64Scalarscalar){...}...}ExampleVisitorvisitor;VisitScalarInline(some_scalar,&visitor);

Template Parameters:

• VISITOR – Visitor type that implements Visit() for all scalar types.

• ARGS – Additional arguments, if any, will be passed to the Visit function after thescalar argument

Returns:

Status

template<typenameVISITOR,typename...ARGS>
inlineStatusarrow::VisitArrayInline(constArray&array,VISITOR*visitor,ARGS&&...args)#

Apply the visitors Visit() method specialized to the array type.

A visitor is a type that implements specialized logic for each Arrow type. Example usage:

classExampleVisitor{arrow::StatusVisit(arrow::NumericArray<Int32Type>arr){...}arrow::StatusVisit(arrow::NumericArray<Int64Type>arr){...}...}ExampleVisitorvisitor;VisitArrayInline(some_array,&visitor);

Template Parameters:

• VISITOR – Visitor type that implements Visit() for all array types.

• ARGS – Additional arguments, if any, will be passed to the Visit function after thearr argument

Returns:

Status

Type Traits#

These types provide relationships between Arrow types at compiletime.TypeTraits maps Arrow DataTypes to other types, andCTypeTraits maps C types to Arrow types.