archives
packagemodule
This package is not in the latest version of its module.
Details
Validgo.mod file
The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
- Redistributable license
Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
- Tagged version
Modules with tagged versions give importers more predictable builds.
Stable versionWhen a project reaches major version v1 it is considered stable.
- Learn more about best practices
Repository
Links
Open Source Insights
README¶
archives
Introducingmholt/archives - a cross-platform, multi-format Go library for working with archives and compression formats with a unified API and as virtual file systems compatible withio/fs.
Features
- Stream-oriented APIs
- Automatically identify archive and compression formats:
- By file name
- By stream peeking (headers)
- Traverse directories, archives, and other files uniformly as
io/fsfile systems: - Seamlessly walk into archive files using
DeepFS - Compress and decompress files
- Create and extract archive files
- Walk or traverse into archive files
- Extract only specific files from archives
- Insert into (append to) .tar and .zip archives without re-creating entire archive
- Numerous archive and compression formats supported
- Read from password-protected 7-Zip and RAR files
- Extensible (add more formats just by registering them)
- Cross-platform, static binary
- Pure Go (no cgo)
- Multithreaded Gzip
- Adjustable compression levels
- Super-fast Snappy implementation (viaS2)
Supported compression formats
- brotli (.br)
- bzip2 (.bz2)
- flate (.zip)
- gzip (.gz)
- lz4 (.lz4)
- lzip (.lz)
- minlz (.mz)
- snappy (.sz) and S2 (.s2)
- xz (.xz)
- zlib (.zz)
- zstandard (.zst)
Supported archive formats
- .zip
- .tar (including any compressed variants like .tar.gz)
- .rar (read-only)
- .7z (read-only)
Command line utility
There is an independently-maintained command line tool calledarc currently in development that will expose many of the functions of this library to a shell.
Library use
$ go get github.com/mholt/archivesCreate archive
Creating archives can be done entirely without needing a real disk or storage device. All you need is a list ofFileInfo structs, which can be implemented without a real file system.
However, creating archives from a disk is very common, so you can use theFilesFromDisk() function to help you map filenames on disk to their paths in the archive.
In this example, we add 4 files and a directory (which includes its contents recursively) to a .tar.gz file:
ctx := context.TODO()// map files on disk to their paths in the archive using default settings (second arg)files, err := archives.FilesFromDisk(ctx, nil, map[string]string{"/path/on/disk/file1.txt": "file1.txt","/path/on/disk/file2.txt": "subfolder/file2.txt","/path/on/disk/file3.txt": "", // put in root of archive as file3.txt"/path/on/disk/file4.txt": "subfolder/", // put in subfolder as file4.txt"/path/on/disk/folder": "Custom Folder", // contents added recursively})if err != nil {return err}// create the output file we'll write toout, err := os.Create("example.tar.gz")if err != nil {return err}defer out.Close()// we can use the CompressedArchive type to gzip a tarball// (since we're writing, we only set Archival, but if you're// going to read, set Extraction)format := archives.CompressedArchive{Compression: archives.Gz{},Archival: archives.Tar{},}// create the archiveerr = format.Archive(ctx, out, files)if err != nil {return err}Extract archive
Extracting an archive, extractingfrom an archive, and walking an archive are all the same function.
Simply use your format type (e.g.Zip) to callExtract(). You'll pass in a context (for cancellation), the input stream, and a callback function to handle each file.
// the type that will be used to read the input streamvar format archives.Ziperr := format.Extract(ctx, input, func(ctx context.Context, f archives.FileInfo) error {// do something with the file here; or, if you only want a specific file or directory,// just return until you come across the desired f.NameInArchive value(s)return nil})if err != nil {return err}Identifying formats
When you have an input stream with unknown contents, this package can identify it for you. It will try matching based on filename and/or the header (which peeks at the stream):
// unless your stream is an io.Seeker, use the returned stream value to// ensure you re-read the bytes consumed during Identify()format, stream, err := archives.Identify(ctx, "filename.tar.zst", stream)if err != nil {return err}// you can now type-assert format to whatever you need// want to extract something?if ex, ok := format.(archives.Extractor); ok {// ... proceed to extract}// or maybe it's compressed and you want to decompress it?if decomp, ok := format.(archives.Decompressor); ok {rc, err := decomp.OpenReader(unknownFile)if err != nil {return err}defer rc.Close()// read from rc to get decompressed data}Identify() works by reading an arbitrary number of bytes from the beginning of the stream (just enough to check for file headers). It buffers them and returns a new reader that lets you re-read them anew. If your input stream isio.Seeker however, no buffer is created as it usesSeek() instead, and the returned stream is the same as the input.
Virtual file systems
This is my favorite feature.
Let's say you have a directory on disk, an archive, a compressed archive, any other regular file, or a stream of any of the above! You don't really care; you just want to use it uniformly no matter what it is.
Simply create a file system:
// filename could be:// - a folder ("/home/you/Desktop")// - an archive ("example.zip")// - a compressed archive ("example.tar.gz")// - a regular file ("example.txt")// - a compressed regular file ("example.txt.gz")// and/or the last argument could be a stream of any of the abovefsys, err := archives.FileSystem(ctx, filename, nil)if err != nil {return err}This is a fully-featuredfs.FS, so you can open files and read directories, no matter what kind of file the input was.
For example, to open a specific file:
f, err := fsys.Open("file")if err != nil {return err}defer f.Close()If you opened a regular file or archive, you can read from it. If it's a compressed file, reads are automatically decompressed.
If you opened a directory (either real or in an archive), you can list its contents:
if dir, ok := f.(fs.ReadDirFile); ok {// 0 gets all entries, but you can pass > 0 to paginateentries, err := dir.ReadDir(0)if err != nil {return err}for _, e := range entries {fmt.Println(e.Extension())}}Or get a directory listing this way:
entries, err := fsys.ReadDir("Playlists")if err != nil {return err}for _, e := range entries {fmt.Println(e.Extension())}Or maybe you want to walk all or part of the file system, but skip a folder named.git:
err := fs.WalkDir(fsys, ".", func(path string, d fs.DirEntry, err error) error {if err != nil {return err}if path == ".git" {return fs.SkipDir}fmt.Println("Walking:", path, "Dir?", d.IsDir())return nil})if err != nil {return err}Thearchives package lets you do it all.
Important .tar note: Tar files do not efficiently implement file system semantics due to their historical roots in sequential-access design for tapes. File systems inherently assume some index facilitating random access, but tar files need to be read from the beginning to access something at the end. This is especially slow when the archive is compressed. Optimizations have been implemented to amortizeReadDir() calls so thatfs.WalkDir() only has to scan the archive once, but they use more memory. Open calls require another scan to find the file. It may be more efficient to useTar.Extract() directly if file system semantics are not important to you.
Use withhttp.FileServer
It can be used with http.FileServer to browse archives and directories in a browser. However, due to how http.FileServer works, don't directly use http.FileServer with compressed files; instead wrap it like following:
fileServer := http.FileServer(http.FS(archiveFS))http.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {// disable range requestwriter.Header().Set("Accept-Ranges", "none")request.Header.Del("Range")// disable content-type sniffingctype := mime.TypeByExtension(filepath.Ext(request.URL.Path))writer.Header()["Content-Type"] = nilif ctype != "" {writer.Header().Set("Content-Type", ctype)}fileServer.ServeHTTP(writer, request)})http.FileServer will try to sniff the Content-Type by default if it can't be inferred from file name. To do this, the http package will try to read from the file and then Seek back to file start, which the libray can't achieve currently. The same goes with Range requests. Seeking in archives is not currently supported by this package due to limitations in dependencies.
If Content-Type is desirable, you canregister it yourself.
Compress data
Compression formats let you open writers to compress data:
// wrap underlying writer wcompressor, err := archives.Zstd{}.OpenWriter(w)if err != nil {return err}defer compressor.Close()// writes to compressor will be compressedDecompress data
Similarly, compression formats let you open readers to decompress data:
// wrap underlying reader rdecompressor, err := archives.Snappy{}.OpenReader(r)if err != nil {return err}defer decompressor.Close()// reads from decompressor will be decompressedAppend to tarball and zip archives
Tar and Zip archives can be appended to without creating a whole new archive by callingInsert() on a tar or zip stream. However, for tarballs, this requires that the tarball is not compressed (due to complexities with modifying compression dictionaries).
Here is an example that appends a file to a tarball on disk:
tarball, err := os.OpenFile("example.tar", os.O_RDWR, 0644)if err != nil {return err}defer tarball.Close()// prepare a text file for the root of the archivefiles, err := archives.FilesFromDisk(nil, map[string]string{"/home/you/lastminute.txt": "",})err := archives.Tar{}.Insert(context.Background(), tarball, files)if err != nil {return err}The code is similar for inserting into a Zip archive, except you'll callInsert() on aZip{} value instead.
Traverse into archives while walking
If you are traversing/walking the file system usingfs.WalkDir(), theDeepFS type lets you walk the contents of archives (and compressed archives!) transparently as if the archive file was a regular directory on disk.
Simply root your DeepFS at a real path, then walk away:
fsys := &archives.DeepFS{Root: "/some/dir"}err := fs.WalkDir(fsys, ".", func(fpath string, d fs.DirEntry, err error) error {...})You'll notice that paths within archives look like/some/dir/archive.zip/foo/bar.txt. If you pass a path like that intofsys.Open(), it will split the path at the end of the archive file (/some/dir/archive.zip) and use the remainder of the path (foo/bar.txt) inside the archive.
Documentation¶
Index¶
- Constants
- Variables
- func FileSystem(ctx context.Context, filename string, stream ReaderAtSeeker) (fs.FS, error)
- func PathContainsArchive(path string) bool
- func PathIsArchive(path string) bool
- func RegisterFormat(format Format)
- func TopDirOpen(fsys fs.FS, name string) (fs.File, error)
- func TopDirReadDir(fsys fs.FS, name string) ([]fs.DirEntry, error)
- func TopDirStat(fsys fs.FS, name string) (fs.FileInfo, error)
- type Archival
- type ArchiveAsyncJob
- type ArchiveFS
- type Archiver
- type ArchiverAsync
- type Brotli
- type Bz2
- type CompressedArchive
- func (ca CompressedArchive) Archive(ctx context.Context, output io.Writer, files []FileInfo) error
- func (ca CompressedArchive) ArchiveAsync(ctx context.Context, output io.Writer, jobs <-chan ArchiveAsyncJob) error
- func (ca CompressedArchive) Extension() string
- func (ca CompressedArchive) Extract(ctx context.Context, sourceArchive io.Reader, handleFile FileHandler) error
- func (ca CompressedArchive) Match(ctx context.Context, filename string, stream io.Reader) (MatchResult, error)
- func (ca CompressedArchive) MediaType() string
- type Compression
- type Compressor
- type Decompressor
- type DeepFS
- type DirFS
- type Extraction
- type Extractor
- type FileFS
- type FileHandler
- type FileInfo
- type Format
- type FromDiskOptions
- type Gz
- type Inserter
- type Lz4
- type Lzip
- type MatchResult
- type MinLZ
- type Rar
- type ReaderAtSeeker
- type S2
- type S2Level
- type SevenZip
- type Sz
- type Tar
- func (t Tar) Archive(ctx context.Context, output io.Writer, files []FileInfo) error
- func (t Tar) ArchiveAsync(ctx context.Context, output io.Writer, jobs <-chan ArchiveAsyncJob) error
- func (Tar) Extension() string
- func (t Tar) Extract(ctx context.Context, sourceArchive io.Reader, handleFile FileHandler) error
- func (t Tar) Insert(ctx context.Context, into io.ReadWriteSeeker, files []FileInfo) error
- func (t Tar) Match(_ context.Context, filename string, stream io.Reader) (MatchResult, error)
- func (Tar) MediaType() string
- type Xz
- type Zip
- func (z Zip) Archive(ctx context.Context, output io.Writer, files []FileInfo) error
- func (z Zip) ArchiveAsync(ctx context.Context, output io.Writer, jobs <-chan ArchiveAsyncJob) error
- func (Zip) Extension() string
- func (z Zip) Extract(ctx context.Context, sourceArchive io.Reader, handleFile FileHandler) error
- func (z Zip) Insert(ctx context.Context, into io.ReadWriteSeeker, files []FileInfo) error
- func (z Zip) Match(_ context.Context, filename string, stream io.Reader) (MatchResult, error)
- func (Zip) MediaType() string
- type Zlib
- type Zstd
Constants¶
const (ZipMethodBzip2 = 12// TODO: LZMA: Disabled - because 7z isn't able to unpack ZIP+LZMA ZIP+LZMA2 archives made this way - and vice versa.// ZipMethodLzma = 14ZipMethodZstd = 93ZipMethodXz = 95)
Additional compression methods not offered by archive/zip.Seehttps://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT section 4.4.5.
Variables¶
var NoMatch =fmt.Errorf("no formats matched")NoMatch is a special error returned if there are no matching formats.
Functions¶
funcFileSystem¶
FileSystem identifies the format of the input and returns a read-only file system.The input can be a filename, stream, or both.
If only a filename is specified, it may be a path to a directory, archive file,compressed archive file, compressed regular file, or any other regular file ondisk. If the filename is a directory, its contents are accessed directly fromthe device's file system. If the filename is an archive file, the contents canbe accessed like a normal directory; compressed archive files are transparentlydecompressed as contents are accessed. And if the filename is any other file, itis the only file in the returned file system; if the file is compressed, it istransparently decompressed when read from.
If a stream is specified, the filename (if available) is used as a hint to helpidentify its format. Streams of archive files must be able to be made into anio.SectionReader (for safe concurrency) which requires io.ReaderAt and io.Seeker(to efficiently determine size). The automatic format identification requiresio.Reader and will use io.Seeker if supported to avoid buffering.
Whether the data comes from disk or a stream, it is peeked at to automaticallydetect which format to use.
This function essentially offers uniform read access to various kinds of files:directories, archives, compressed archives, individual files, and file streamsare all treated the same way.
NOTE: The performance of compressed tar archives is not great due to overheadwith decompression. However, the fs.WalkDir() use case has been optimized tocreate an index on first call to ReadDir().
funcPathContainsArchive¶
PathContainsArchive returns true if the path contains an archive file (i.e.whether the path traverses into an archive) solely by lexical analysis (noreading of files or headers is performed). Such a path is not typicallyusable by the OS, but can be used by the DeepFS type. Slash must be thepath component separator. Example: "/foo/example.zip/path/in/archive"
funcPathIsArchive¶added inv0.1.2
PathIsArchive returns true if the path ends with an archive file (i.e.whether the path traverse to an archive) solely by lexical analysis (noreading the files or headers is performed).
funcRegisterFormat¶
func RegisterFormat(formatFormat)
RegisterFormat registers a format. It should be called during init.Duplicate formats by name are not allowed and will panic.
funcTopDirOpen¶
TopDirOpen is a special Open() function that may be useful ifa file system root was created by extracting an archive.
It first tries the file name as given, but if that returns anerror, it tries the name without the first element of the path.In other words, if "a/b/c" returns an error, then "b/c" willbe tried instead.
Consider an archive that contains a file "a/b/c". When thearchive is extracted, the contents may be created without anew parent/root folder to contain them, and the path of thesame file outside the archive may be lacking an exclusive rootor parent container. Thus it is likely for a file systemcreated for the same files extracted to disk to be rooted atone of the top-level files/folders from the archive instead ofa parent folder. For example, the file known as "a/b/c" whenrooted at the archive becomes "b/c" after extraction when rootedat "a" on disk (because no new, exclusive top-level folder wascreated). This difference in paths can make it difficult to usearchives and directories uniformly. Hence these TopDir* functionswhich attempt to smooth over the difference.
Some extraction utilities do create a container folder forarchive contents when extracting, in which case the usermay give that path as the root. In that case, these TopDir*functions are not necessary (but aren't harmful either). Theyare primarily useful if you are not sure whether the root isan archive file or is an extracted archive file, as they willwork with the same filename/path inputs regardless of thepresence of a top-level directory.
EXPERIMENTAL: Subject to change or removal even after stable release.
funcTopDirReadDir¶
TopDirReadDir is like TopDirOpen but for ReadDir.
EXPERIMENTAL: Subject to change or removal even after stable release.
Types¶
typeArchiveAsyncJob¶
ArchiveAsyncJob contains a File to be archived and a channel thatthe result of the archiving should be returned on.EXPERIMENTAL: Subject to change or removal.
typeArchiveFS¶
type ArchiveFS struct {// set one of thesePathstring// path to the archive file on disk, or...Stream *io.SectionReader// ...stream from which to read archiveFormatExtractor// the archive formatPrefixstring// optional subdirectory in which to root the fsContextcontext.Context// optional; mainly for cancellation// contains filtered or unexported fields}ArchiveFS allows reading an archive (or a compressed archive) using aconsistent file system interface. Essentially, it allows traversal andreading of archive contents the same way as any normal directory on disk.The contents of compressed archives are transparently decompressed.
A valid ArchiveFS value must set either Path or Stream, but not both.If Path is set, a literal file will be opened from the disk.If Stream is set, new SectionReaders will be implicitly created toaccess the stream, enabling safe, concurrent access.
NOTE: Due to Go's file system APIs (see package io/fs), the performanceof ArchiveFS can suffer when using fs.WalkDir(). To mitigate this,an optimized fs.ReadDirFS has been implemented that indexes the entirearchive on the first call to ReadDir() (since the entire archive needsto be walked for every call to ReadDir() anyway, as archive contents areoften unordered). The first call to ReadDir(), i.e. near the start of thewalk, will be slow for large archives, but should be instantaneous after.If you don't care about walking a file system in directory order, considercalling Extract() on the underlying archive format type directly, whichwalks the archive in entry order, without needing to do any sorting.
Note that fs.FS implementations, including this one, reject paths startingwith "./". This can be problematic sometimes, as it is not uncommon fortarballs to contain a top-level/root directory literally named ".", whichcan happen if a tarball is created in the same directory it is archiving.The underlying Extract() calls are faithful to entries with this name,but file systems have certain semantics around "." that restrict its use.For example, a file named "." cannot be created on a real file systembecause it is a special name that means "current directory".
We had to decide whether to honor the true name in the archive, or honorfile system semantics. Given that this is a virtual file system and othercode using the fs.FS APIs will trip over a literal directory named ".",we choose to honor file system semantics. Files named "." are ignored;directories with this name are effectively transparent; their contentsget promoted up a directory/level. This means a file at "./x" where "."is a literal directory name, its name will be passed in as "x" inWalkDir callbacks. If you need the raw, uninterpeted values from anarchive, use the formats' Extract() method directly. Seehttps://github.com/golang/go/issues/70155 for a little more background.
This does have one negative edge case... a tar containing contents like[x . ./x] will have a conflict on the file named "x" because "./x" willalso be accessed with the name of "x".
func (ArchiveFS)Open¶
Open opens the named file from within the archive. If name is "." thenthe archive file itself will be opened as a directory file.
func (*ArchiveFS)ReadDir¶
ReadDir reads the named directory from within the archive. If name is "."then the root of the archive content is listed.
typeArchiver¶
type Archiver interface {// Archive writes an archive file to output with the given files.//// Context cancellation must be honored.Archive(ctxcontext.Context, outputio.Writer, files []FileInfo)error}Archiver can create a new archive.
typeArchiverAsync¶
type ArchiverAsync interface {Archiver// Use ArchiveAsync if you can't pre-assemble a list of all// the files for the archive. Close the jobs channel after// all the files have been sent.//// This won't return until the channel is closed.ArchiveAsync(ctxcontext.Context, outputio.Writer, jobs <-chanArchiveAsyncJob)error}ArchiverAsync is an Archiver that can also create archivesasynchronously by pumping files into a channel as they arediscovered.EXPERIMENTAL: Subject to change or removal.
typeBrotli¶
type Brotli struct {Qualityint}Brotli facilitates brotli compression.
func (Brotli)OpenReader¶
func (Brotli)OpenWriter¶
typeBz2¶
type Bz2 struct {CompressionLevelint}Bz2 facilitates bzip2 compression.
func (Bz2)OpenReader¶
func (Bz2)OpenWriter¶
typeCompressedArchive¶
type CompressedArchive struct {ArchivalExtractionCompression}CompressedArchive represents an archive which is compressed externally(for example, a gzipped tar file, .tar.gz.) It combines a compressionformat on top of an archival/extraction format and provides bothfunctionalities in a single type, allowing archival and extractionoperations transparently through compression and decompression. However,compressed archives have some limitations; for example, files cannot beinserted/appended because of complexities with modifying existingcompression state (perhaps this could be overcome, but I'm not about totry it).
func (CompressedArchive)Archive¶
Archive writes an archive to the output stream while compressing the result.
func (CompressedArchive)ArchiveAsync¶
func (caCompressedArchive) ArchiveAsync(ctxcontext.Context, outputio.Writer, jobs <-chanArchiveAsyncJob)error
ArchiveAsync adds files to the output archive while compressing the result asynchronously.
func (CompressedArchive)Extension¶
func (caCompressedArchive) Extension()string
Name returns a concatenation of the archive and compression format extensions.
func (CompressedArchive)Extract¶
func (caCompressedArchive) Extract(ctxcontext.Context, sourceArchiveio.Reader, handleFileFileHandler)error
Extract reads files out of a compressed archive while decompressing the results.
func (CompressedArchive)Match¶
func (caCompressedArchive) Match(ctxcontext.Context, filenamestring, streamio.Reader) (MatchResult,error)
Match matches if the input matches both the compression and archival/extraction format.
func (CompressedArchive)MediaType¶
func (caCompressedArchive) MediaType()string
MediaType returns the compression format's MIME type, sincea compressed archive is fundamentally a compressed file.
typeCompression¶
type Compression interface {FormatCompressorDecompressor}Compression is a compression format with both compress and decompress methods.
typeCompressor¶
type Compressor interface {// OpenWriter wraps w with a new writer that compresses what is written.// The writer must be closed when writing is finished.OpenWriter(wio.Writer) (io.WriteCloser,error)}Compressor can compress data by wrapping a writer.
typeDecompressor¶
type Decompressor interface {// OpenReader wraps r with a new reader that decompresses what is read.// The reader must be closed when reading is finished.OpenReader(rio.Reader) (io.ReadCloser,error)}Decompressor can decompress data by wrapping a reader.
typeDeepFS¶
type DeepFS struct {// The root filepath using OS separator, even if it// traverses into an archive.Rootstring// An optional context, mainly for cancellation.Contextcontext.Context// contains filtered or unexported fields}DeepFS is a fs.FS that represents the real file system, but also hasthe ability to traverse into archive files as if they were part of theregular file system. If a filename component ends with an archiveextension (e.g. .zip, .tar, .tar.gz, etc.), then the remainder of thefilepath will be considered to be inside that archive.
This allows treating archive files transparently as if they were partof the regular file system during a walk, which can be extremely usefulfor accessing data in an "ordinary" walk of the disk, without needing tofirst extract all the archives and use more disk space.
Archives within archives are not supported.
The listing of archive entries is retained for the lifetime of theDeepFS value for efficiency, but this can use more memory if archivescontain a lot of files.
The exported fields may be changed during the lifetime of a DeepFS value(but not concurrently). It is safe to use this type as an FS concurrently.
func (*DeepFS)ReadDir¶
ReadDir returns the directory listing for the given directory name,but for any entries that appear by their file extension to be archivefiles, they are slightly modified to always return true for IsDir(),since we have the unique ability to list the contents of archives asif they were directories.
func (*DeepFS)SplitPath¶added inv0.1.2
SplitPath splits a file path into the "real" path and the "inner" path components,where the split point is the first extension of an archive filetype like ".zip" or".tar.gz" that occurs in the path.
The real path is the path that can be accessed on disk and will be returned withplatform filepath separators. The inner path is the io/fs-compatible path that canbe used within the archive.
If no archive extension is found in the path, only the realPath is returned.If the input path is precisely an archive file (i.e. ends with an archive fileextension), then innerPath is returned as "." which indicates the root of the archive.
typeDirFS¶
type DirFSstring
DirFS is similar to os.dirFS (obtained via os.DirFS()), but it isexported so it can be used with type assertions. It also returnsFileInfo/DirEntry values where Name() always returns the name ofthe directory instead of ".". This type does not guarantee anysort of sandboxing.
typeExtraction¶
Extraction is an archival format that extract from (read) archives.
typeExtractor¶
type Extractor interface {// Extract walks entries in the archive and calls handleFile for each// entry in the archive.//// Any files opened in the FileHandler should be closed when it returns,// as there is no guarantee the files can be read outside the handler// or after the walk has proceeded to the next file.//// Context cancellation must be honored.Extract(ctxcontext.Context, archiveio.Reader, handleFileFileHandler)error}Extractor can extract files from an archive.
typeFileFS¶
type FileFS struct {// The path to the file on disk.Pathstring// If file is compressed, setting this field will// transparently decompress reads.CompressionDecompressor}FileFS allows accessing a file on disk using a consistent file system interface.The value should be the path to a regular file, not a directory. This file willbe the only entry in the file system and will be at its root. It can be accessedwithin the file system by the name of "." or the filename.
If the file is compressed, set the Compression field so that reads from thefile will be transparently decompressed.
typeFileHandler¶
FileHandler is a callback function that is used to handle files as they are readfrom an archive; it is kind of like fs.WalkDirFunc. Handler functions that opentheir files must not overlap or run concurrently, as files may be read from thesame sequential stream; always close the file before returning.
If the special error value fs.SkipDir is returned, the directory of the file(or the file itself if it is a directory) will not be walked. Note that becausearchive contents are not necessarily ordered, skipping directories requiresmemory, and skipping lots of directories may run up your memory bill.
Any other returned error will terminate a walk and be returned to the caller.
typeFileInfo¶
type FileInfo struct {fs.FileInfo// The file header as used/provided by the archive format.// Typically, you do not need to set this field when creating// an archive.Headerany// The path of the file as it appears in the archive.// This is equivalent to Header.Name (for most Header// types). We require it to be specified here because// it is such a common field and we want to preserve// format-agnosticism (no type assertions) for basic// operations.//// When extracting, this name or path may not have// been sanitized; it should not be trusted at face// value. Consider using path.Clean() before using.//// If this is blank when inserting a file into an// archive, the filename's base may be assumed// by default to be the name in the archive.NameInArchivestring// For symbolic and hard links, the target of the link.// Not supported by all archive formats.LinkTargetstring// A callback function that opens the file to read its// contents. The file must be closed when reading is// complete.Open func() (fs.File,error)}FileInfo is a virtualized, generalized file abstraction for interacting with archives.
funcFilesFromDisk¶
func FilesFromDisk(ctxcontext.Context, options *FromDiskOptions, filenames map[string]string) ([]FileInfo,error)
FilesFromDisk is an opinionated function that returns a list of FileInfosby walking the directories in the filenames map. The keys are the names ondisk, and the values become their associated names in the archive.
Map keys that specify directories on disk will be walked and added to thearchive recursively, rooted at the named directory. They should use theplatform's path separator (backslash on Windows; slash on everything else).For convenience, map keys that end in a separator ('/', or '\' on Windows)will enumerate contents only, without adding the folder itself to the archive.
Map values should typically use slash ('/') as the separator regardless ofthe platform, as most archive formats standardize on that rune as thedirectory separator for filenames within an archive. For convenience, mapvalues that are empty string are interpreted as the base name of the file(sans path) in the root of the archive; and map values that end in a slashwill use the base name of the file in that folder of the archive.
File gathering will adhere to the settings specified in options.
This function is used primarily when preparing a list of files to add toan archive.
typeFormat¶
type Format interface {// Extension returns the conventional file extension for this// format.Extension()string// MediaType returns the MIME type ("content type") of this// format (seeRFC 2046).MediaType()string// Match returns true if the given name/stream is recognized.// One of the arguments is optional: filename might be empty// if working with an unnamed stream, or stream might be empty// if only working with a file on disk; but both may also be// specified. The filename should consist only of the base name,// not path components, and is typically used for matching by// file extension. However, matching by reading the stream is// preferred as it is more accurate. Match reads only as many// bytes as needed to determine a match.Match(ctxcontext.Context, filenamestring, streamio.Reader) (MatchResult,error)}Format represents a way of getting data out of something else.A format usually represents compression or an archive (or both).
funcIdentify¶
Identify iterates the registered formats and returns the one thatmatches the given filename and/or stream. It is capable of identifyingcompressed files (.gz, .xz...), archive files (.tar, .zip...), andcompressed archive files (tar.gz, tar.bz2...). The returned Formatvalue can be type-asserted to ascertain its capabilities.
If no matching formats were found, special error NoMatch is returned.
If stream is nil then it will only match on file name and thereturned io.Reader will be nil.
If stream is non-nil, it will be returned in the same read positionas it was before Identify() was called, by virtue of buffering thepeeked bytes. However, if the stream is an io.Seeker, Seek() mustwork, no extra buffering will be performed, and the original inputvalue will be returned at the original position by seeking.
typeFromDiskOptions¶
type FromDiskOptions struct {// If true, symbolic links will be dereferenced, meaning that// the link will not be added as a link, but what the link// points to will be added as a file.FollowSymlinksbool// If true, some file attributes will not be preserved.// Name, size, type, and permissions will still be preserved.ClearAttributesbool}FromDiskOptions specifies various options for gathering files from disk.
typeGz¶
type Gz struct {// Gzip compression level. Seehttps://pkg.go.dev/compress/flate#pkg-constants// for some predefined constants. If 0, DefaultCompression is assumed rather// than no compression.CompressionLevelint// DisableMultistream controls whether the reader supports multistream files.// Seehttps://pkg.go.dev/compress/gzip#example-Reader.MultistreamDisableMultistreambool// Use a fast parallel Gzip implementation. This is only// effective for large streams (about 1 MB or greater).Multithreadedbool}Gz facilitates gzip compression.
func (Gz)OpenReader¶
func (Gz)OpenWriter¶
typeInserter¶
type Inserter interface {// Insert inserts the files into archive.//// Context cancellation must be honored.Insert(ctxcontext.Context, archiveio.ReadWriteSeeker, files []FileInfo)error}Inserter can insert files into an existing archive.EXPERIMENTAL: Subject to change.
typeLz4¶
type Lz4 struct {CompressionLevelint}Lz4 facilitates LZ4 compression.
func (Lz4)OpenReader¶
func (Lz4)OpenWriter¶
typeLzip¶
type Lzip struct{}Lzip facilitates lzip compression.
func (Lzip)OpenReader¶
func (Lzip)OpenWriter¶
typeMatchResult¶
type MatchResult struct {ByName, ByStreambool}MatchResult returns true if the format was matched eitherby name, stream, or both. Name usually refers to matchingby file extension, and stream usually refers to readingthe first few bytes of the stream (its header). A streammatch is generally stronger, as filenames are not alwaysindicative of their contents if they even exist at all.
func (MatchResult)Matched¶
func (mrMatchResult) Matched()bool
Matched returns true if a match was made by either name or stream.
func (MatchResult)String¶
func (mrMatchResult) String()string
typeMinLZ¶added inv0.1.1
type MinLZ struct{}MinLZ facilitates MinLZ compression. Seehttps://github.com/minio/minlz/blob/main/SPEC.mdandhttps://blog.min.io/minlz-compression-algorithm/.
func (MinLZ)OpenReader¶added inv0.1.1
func (MinLZ)OpenWriter¶added inv0.1.1
typeRar¶
type Rar struct {// If true, errors encountered during reading or writing// a file within an archive will be logged and the// operation will continue on remaining files.ContinueOnErrorbool// Password to open archives.Passwordstring// Name for a multi-volume archive. When Name is specified,// the named file is extracted (rather than any io.Reader that// may be passed to Extract). If the archive is a multi-volume// archive, this name will also be used by the decoder to derive// the filename of the next volume in the volume set.Namestring// FS is an fs.FS exposing the files of the archive. Unless Name is// also specified, this does nothing. When Name is also specified,// FS defines the fs.FS that from which the archive will be opened,// and in the case of a multi-volume archive, from where each subsequent// volume of the volume set will be loaded.//// Typically this should be a DirFS pointing at the directory containing// the volumes of the archive.FSfs.FS}typeReaderAtSeeker¶
ReaderAtSeeker is a type that can read, read at, and seek.os.File and io.SectionReader both implement this interface.
typeS2¶
type S2 struct {// reader optionsMaxBlockSizeintAllocBlockintIgnoreStreamIdentifierboolIgnoreCRCbool// writer optionsAddIndexboolCompressionS2LevelBlockSizeintConcurrencyintFlushOnWriteboolPaddingintSnappyIncompatiblebool}S2 is an extension of Snappy that can read Snappystreams and write Snappy-compatible streams, butcan also be configured to write Snappy-incompatiblestreams for greater gains. Seehttps://pkg.go.dev/github.com/klauspost/compress/s2for details and the documentation for each option.
typeS2Level¶
type S2Levelint
Compression level for S2 (Snappy/Sz extension).EXPERIMENTAL: May be changed or removed without a major version bump.
typeSevenZip¶
type SevenZip struct {// If true, errors encountered during reading or writing// a file within an archive will be logged and the// operation will continue on remaining files.ContinueOnErrorbool// The password, if dealing with an encrypted archive.Passwordstring}func (SevenZip)Extract¶
Extract extracts files from z, implementing the Extractor interface. Uniquely, however,sourceArchive must be an io.ReaderAt and io.Seeker, which are oddly disjoint interfacesfrom io.Reader which is what the method signature requires. We chose this signature forthe interface because we figure you can Read() from anything you can ReadAt() or Seek()with. Due to the nature of the zip archive format, if sourceArchive is not an io.Seekerand io.ReaderAt, an error is returned.
typeSz¶
type Sz struct {// Configurable S2 extension.S2S2}Sz facilitates Snappy compression. It uses S2for reading and writing, but by default willwrite Snappy-compatible data.
func (Sz)OpenReader¶
func (Sz)OpenWriter¶
typeTar¶
type Tar struct {// Specify the tar format to use when writing headers.// The default is whichever format is capable of encoding// the header being written, from this ordered list:// USTAR, PAX, GNU.Formattar.Format// DEPRECATED: Use [Tar.Format] instead.FormatGNUbool// If true, preserve only numeric user and group idNumericUIDGIDbool// If true, errors encountered during reading or writing// a file within an archive will be logged and the// operation will continue on remaining files.ContinueOnErrorbool// User ID of the file ownerUidint// Group ID of the file ownerGidint// Username of the file ownerUnamestring// Group name of the file ownerGnamestring}func (Tar)ArchiveAsync¶
typeXz¶
type Xz struct{}Xz facilitates xz compression.
func (Xz)OpenReader¶
func (Xz)OpenWriter¶
typeZip¶
type Zip struct {// Only compress files which are not already in a// compressed format (determined simply by examining// file extension).SelectiveCompressionbool// The method or algorithm for compressing stored files.Compressionuint16// If true, errors encountered during reading or writing// a file within an archive will be logged and the// operation will continue on remaining files.ContinueOnErrorbool// For files in zip archives that do not have UTF-8// encoded filenames and comments, specify the character// encoding here.TextEncodingencoding.Encoding}func (Zip)ArchiveAsync¶
func (Zip)Extract¶
Extract extracts files from z, implementing the Extractor interface. Uniquely, however,sourceArchive must be an io.ReaderAt and io.Seeker, which are oddly disjoint interfacesfrom io.Reader which is what the method signature requires. We chose this signature forthe interface because we figure you can Read() from anything you can ReadAt() or Seek()with. Due to the nature of the zip archive format, if sourceArchive is not an io.Seekerand io.ReaderAt, an error is returned.
func (Zip)Insert¶
Insert appends the listed files into the provided Zip archive stream.If the filename already exists in the archive, it will be replaced.
typeZlib¶
type Zlib struct {CompressionLevelint}Zlib facilitates zlib compression.
func (Zlib)OpenReader¶
func (Zlib)OpenWriter¶
typeZstd¶
Zstd facilitates Zstandard compression.
Source Files