Inheritance Relationships#

Class Documentation#

template<typenameChunkReader,typenameChunkSampler=samplers::RandomSampler,typenameExampleSampler=samplers::RandomSampler>
classChunkDataset:publictorch::data::datasets::StatefulDataset<ChunkDataset<ChunkReader,samplers::RandomSampler,samplers::RandomSampler>,ChunkReader::BatchType,size_t>#

A stateful dataset that support hierarchical sampling and prefetching of entre chunks.

Unlike regular dataset, chunk dataset require two samplers to operate and keeps an internal state.ChunkSampler selects, which chunk to load next, while theExampleSampler determines the order of Examples that are returned in eachget_batch call. The hierarchical sampling approach used here is inspired by this paperhttp://martin.zinkevich.org/publications/nips2010.pdf

Public Types

usingBatchType=std::optional<typenameChunkReader::BatchType>#

usingUnwrappedBatchType=typenameChunkReader::BatchType#

usingBatchRequestType=size_t#

usingChunkSamplerType=ChunkSampler#

usingExampleSamplerType=ExampleSampler#

Public Functions

inlineChunkDataset(ChunkReaderchunk_reader,ChunkSamplerchunk_sampler,ExampleSamplerexample_sampler,ChunkDatasetOptionsoptions,std::function<void(UnwrappedBatchType&)>preprocessing_policy=std::function<void(UnwrappedBatchType&)>())#

inline~ChunkDataset()override#

inlineBatchTypeget_batch(size_tbatch_size)override#

Default get_batch method ofBatchDataset.

This method returnsExample batches created from the preloaded chunks. The implementation is dataset agnostic and does not need overriding in different chunk datasets.

inlineBatchTypeget_batch()#

Helper method around get_batch asbatch_size is not strictly necessary.

inlinevirtualvoidreset()override#

This will clear any internal state and starts the internal prefetching mechanism for the chunk dataset.

inlinevirtualstd::optional<size_t>size()constoverride#

size is not used for chunk dataset.

inlineChunkSamplerType&chunk_sampler()#

inlinevirtualvoidsave(serialize::OutputArchive&archive)constoverride#

Saves the statefulDataset’s state to OutputArchive.

inlinevirtualvoidload(serialize::InputArchive&archive)override#

Deserializes the statefulDataset’s state from thearchive.