Movatterモバイル変換

parquetize (WIP)

This release includes :

• {parquetize} now has a newget_parquet_info function for retrieving metadata fromparquet files. This function is particularly useful for row group size(added by@nbc).

parquetize 0.5.7

This release includes :

• bugfix by@leungi:remove single quotes in SQL statement thatgenerates incorrect SQL syntaxfor connection of type Microsoft SQL Server #45
• {parquetize} now has a minimal version (2.4.0) for{haven} dependency package to ensure that conversions areperformed correctly from SAS files compressed in BINARY mode #46
• csv_to_parquet now has aread_delim_argsargument, allowing passing of arguments toread_delim(added by@nikostr).
• table_to_parquet can now convert files with uppercaseextensions (.SAS7BDAT, .SAV, .DTA)

parquetize 0.5.6.1

This release includes :

fst_to_parquet function

• a newfst_to_parquetfunction that converts a fst file to parquet format.

Other

• Rely more on@inheritParams to simply documentation offunctions arguments #38. This leads to some renaming of arguments (e.gpath_to_csv ->path_to_file…)
• Argumentscompression andcompression_level are now passed to write_parquet_at_onceand write_parquet_by_chunk functions and now available in mainconversion functions ofparquetize #36
• Group@importFrom in a file to facilitate theirmaintenance #37
• work on download_extract tests #43

parquetize 0.5.6

This release includes :

Possibility to use a RDBMSas source

You can convert to parquet any query you want on any DBI compatibleRDBMS :

dbi_connection <- DBI::dbConnect(RSQLite::SQLite(),  system.file("extdata","iris.sqlite",package = "parquetize"))  # Reading iris table from local sqlite database# and conversion to one parquet file :dbi_to_parquet(  conn = dbi_connection,  sql_query = "SELECT * FROM iris",  path_to_parquet = tempdir(),  parquetname = "iris")

You can find more information ondbi_to_parquetdocumentation.

check_parquet function

• a newcheck_parquetfunction that check if a dataset/file is valid and return columns andarrow type

Deprecations

Two arguments are deprecated to avoid confusion with arrow conceptand keep consistency

• chunk_size is replaced bymax_rows (chunksize is an arrow concept).
• chunk_memory_size is replaced bymax_memory for consistency

Other

• refactoring : extract the logic to write parquet files as chunk orat once in write_parquet_by_chunk and write_parquet_at_once
• a big test’s refactoring : all _to_parquet output files are formallyvalidate (readable as parquet, number of lines, partitions, number offiles).
• use cli_abort instead of cli_alert_danger with stop(““)everywhere
• some minors changes
• bugfix: table_to_parquet did not select columns as expected
• bugfix: skip_if_offline tests with download

parquetize 0.5.5

This release includes :

A very importantnew contributor toparquetize !

Due to these numerous contributions,@nbc is now officially part of the projectauthors !

Three arguments deprecation

After a big refactoring, three arguments are deprecated :

• by_chunk :table_to_parquet willautomatically chunked if you use one ofchunk_memory_sizeorchunk_size.
• csv_as_a_zip:csv_to_table will detect iffile is a zip by the extension.
• url_to_csv : usepath_to_csv instead,csv_to_table will detect if the file is remote with thefile path.

They will raise a deprecation warning for the moment.

Chunking by memory size

The possibility to chunk parquet by memory size withtable_to_parquet():table_to_parquet() takes achunk_memory_size argument to convert an input file intoparquet file of roughlychunk_memory_size Mb size when dataare loaded in memory.

Argumentby_chunk is deprecated (see above).

Example of use of the argumentchunk_memory_size:

table_to_parquet(  path_to_table = system.file("examples","iris.sas7bdat", package = "haven"),  path_to_parquet = tempdir(),  chunk_memory_size = 5000, # this will create files of around 5Gb when loaded in memory)

table_to_parquet(  path_to_table = system.file("examples","iris.sas7bdat", package = "haven"),  path_to_parquet = tempdir(),  compression = "zstd",  compression_level = 10,  chunk_memory_size = 5000)

file_path <- download_extract(  "https://www.nomisweb.co.uk/output/census/2021/census2021-ts007.zip",  filename_in_zip = "census2021-ts007-ctry.csv")csv_to_parquet(  file_path,  path_to_parquet = tempdir())