pyarrow.dataset.FileSystemFactoryOptions#

classpyarrow.dataset.FileSystemFactoryOptions(partition_base_dir=None,partitioning=None,exclude_invalid_files=None,listselector_ignore_prefixes=None)#

Bases:_Weakrefable

Influences the discovery of filesystem paths.

Parameters:

partition_base_dirstr, optional

For the purposes of applying the partitioning, paths will bestripped of the partition_base_dir. Files not matching thepartition_base_dir prefix will be skipped for partitioning discovery.The ignored files will still be part of the Dataset, but will nothave partition information.

partitioningPartitioning/PartitioningFactory, optional

Apply the Partitioning to every discovered Fragment. See Partitioning orPartitioningFactory documentation.

exclude_invalid_filesbool, optional (defaultTrue)

If True, invalid files will be excluded (file format specific check).This will incur IO for each files in a serial and single threadedfashion. Disabling this feature will skip the IO, but unsupportedfiles may be present in the Dataset (resulting in an error at scantime).

selector_ignore_prefixeslist, optional

When discovering from a Selector (and not from an explicit file list),ignore files and directories matching any of these prefixes.By default this is [‘.’, ‘_’].

__init__(*args,**kwargs)#

Methods

__init__(*args, **kwargs)

Attributes

exclude_invalid_files	Whether to exclude invalid files.
partition_base_dir	Base directory to strip paths before applying the partitioning.
partitioning	Partitioning to apply to discovered files.
partitioning_factory	PartitioningFactory to apply to discovered files and discover a Partitioning.
selector_ignore_prefixes	List of prefixes.

exclude_invalid_files#

Whether to exclude invalid files.

partition_base_dir#

Base directory to strip paths before applying the partitioning.

partitioning#

Partitioning to apply to discovered files.

NOTE: setting this property will overwrite partitioning_factory.

partitioning_factory#

PartitioningFactory to apply to discovered files anddiscover a Partitioning.

NOTE: setting this property will overwrite partitioning.

selector_ignore_prefixes#

List of prefixes. Files matching one of those prefixes will beignored by the discovery process.