Reading CSV files#

Data in a CSV file can either be read in as a single Arrow Table usingTableReader or streamed as RecordBatches usingStreamingReader. SeeTradeoffs for adiscussion of the tradeoffs between the two methods.

Both these readers require anarrow::io::InputStream instancerepresenting the input file. Their behavior can be customized using acombination ofReadOptions,ParseOptions, andConvertOptions.

#include"arrow/csv/api.h"{// ...arrow::io::IOContextio_context=arrow::io::default_io_context();std::shared_ptr<arrow::io::InputStream>input=...;autoread_options=arrow::csv::ReadOptions::Defaults();autoparse_options=arrow::csv::ParseOptions::Defaults();autoconvert_options=arrow::csv::ConvertOptions::Defaults();// Instantiate TableReader from input stream and optionsautomaybe_reader=arrow::csv::TableReader::Make(io_context,input,read_options,parse_options,convert_options);if(!maybe_reader.ok()){// Handle TableReader instantiation error...}std::shared_ptr<arrow::csv::TableReader>reader=*maybe_reader;// Read table from CSV fileautomaybe_table=reader->Read();if(!maybe_table.ok()){// Handle CSV read error// (for example a CSV syntax error or failed type conversion)}std::shared_ptr<arrow::Table>table=*maybe_table;}

#include"arrow/csv/api.h"{// ...arrow::io::IOContextio_context=arrow::io::default_io_context();std::shared_ptr<arrow::io::InputStream>input=...;autoread_options=arrow::csv::ReadOptions::Defaults();autoparse_options=arrow::csv::ParseOptions::Defaults();autoconvert_options=arrow::csv::ConvertOptions::Defaults();// Instantiate StreamingReader from input stream and optionsautomaybe_reader=arrow::csv::StreamingReader::Make(io_context,input,read_options,parse_options,convert_options);if(!maybe_reader.ok()){// Handle StreamingReader instantiation error...}std::shared_ptr<arrow::csv::StreamingReader>reader=*maybe_reader;// Set aside a RecordBatch pointer for re-use while streamingstd::shared_ptr<RecordBatch>batch;while(true){// Attempt to read the first RecordBatcharrow::Statusstatus=reader->ReadNext(&batch);if(!status.ok()){// Handle read error}if(batch==NULL){// Handle end of filebreak;}// Do something with the batch}}

Writing CSV files#

A CSV file is written to aOutputStream.

#include<arrow/csv/api.h>{// Oneshot write// ...std::shared_ptr<arrow::io::OutputStream>output=...;autowrite_options=arrow::csv::WriteOptions::Defaults();if(WriteCSV(table,write_options,output.get()).ok()){// Handle writer error...}}{// Write incrementally// ...std::shared_ptr<arrow::io::OutputStream>output=...;autowrite_options=arrow::csv::WriteOptions::Defaults();automaybe_writer=arrow::csv::MakeCSVWriter(output,schema,write_options);if(!maybe_writer.ok()){// Handle writer instantiation error...}std::shared_ptr<arrow::ipc::RecordBatchWriter>writer=*maybe_writer;// Write batches...if(!writer->WriteRecordBatch(*batch).ok()){// Handle write error...}if(!writer->Close().ok()){// Handle close error...}if(!output->Close().ok()){// Handle file close error...}}

Column names#

There are three possible ways to infer column names from the CSV file:

• By default, the column names are read from the first row in the CSV file

• IfReadOptions::column_names is set, it forces the columnnames in the table to these values (the first row in the CSV file isread as data)

• IfReadOptions::autogenerate_column_names is true, column nameswill be autogenerated with the pattern “f0”, “f1”… (the first row in theCSV file is read as data)

Column selection#

By default, Arrow reads all columns in the CSV file. You can narrow theselection of columns with theConvertOptions::include_columnsoption. If some columns inConvertOptions::include_columnsare missing from the CSV file, an error will be emitted unlessConvertOptions::include_missing_columns is true, in which casethe missing columns are assumed to contain all-null values.

Data types#

By default, the CSV reader infers the most appropriate data type for eachcolumn. Type inference considers the following data types, in order:

• Null

• Int64

• Boolean

• Date32

• Time32 (with seconds unit)

• Timestamp (with seconds unit)

• Timestamp (with nanoseconds unit)

• Float64

• Dictionary<String> (ifConvertOptions::auto_dict_encode is true)

• Dictionary<Binary> (ifConvertOptions::auto_dict_encode is true)

• String

• Binary

It is possible to override type inference for select columns by settingtheConvertOptions::column_types option. Explicit data typescan be chosen from the following list:

• Null

• All Integer types

• Float32 and Float64

• Decimal128

• Boolean

• Date32 and Date64

• Time32 and Time64

• Timestamp

• Binary and Large Binary

• String and Large String (with optional UTF8 input validation)

• Fixed-Size Binary

• Duration (numeric strings matching the schema unit, e.g., “60000” for duration[ms])

• Dictionary with index type Int32 and value type one of the following:Binary, String, LargeBinary, LargeString, Int32, UInt32, Int64, UInt64,Float32, Float64, Decimal128

Other data types do not support conversion from CSV values and will error out.

2021-01-01T00:00:002021-01-01T00:00:00Z

2021-01-01T00:00:002021-01-01T00:00:00Z

2021-01-01T00:00:002021-01-01T00:00:00Z

2021-01-01T00:00:002021-01-01T00:00:00Z

Write Options#

The format of written CSV files can be customized viaWriteOptions.Currently few options are available; more will be added in future releases.

Performance#

By default,TableReader will parallelize reads in order toexploit all CPU cores on your machine. You can change this setting inReadOptions::use_threads. A reasonable expectation is at least100 MB/s per core on a performant desktop or laptop computer (measured insource CSV bytes, not target Arrow data bytes).