| BGZF (Blocked GNU Zip Format) | |
|---|---|
| Filename extension | .gz |
| Internet media type | application/gzip |
| Magic number | `\x1f\x8b\x08\x04` (initial bytes, standard Gzip magic. BGZF adds specific extra fields in the header of each block.) |
| Developed by | SAMtools project / HTSlib |
| Initial release | c. 2009 (along with SAM/BAM format specification) |
| Type of format | Compressed file format, Indexed file format |
| Container for | Commonly used for bioinformatics data likeSAM,BAM,VCF records |
| Extended from | Gzip |
| Standard | https://samtools.github.io/hts-specs/SAMv1.pdf#page=13.12 |
| Website | www |
Blocked GNU Zip Format (BGZF) is a variant ofgzipfile format that usesblock compression, a method that compresses data in independent blocks of content—each of which is a valid gzip file. This design is utilized widely inbioinformatics for genomic data compression.[1] The block-based design provides efficient storage, random access with indexed queries,[2][3] and parallel processing; allowing large-scale data processing.[4]
The format was developed as part ofSAM/BAM specification andSAMtools.[5] It is a core component of the commonBAM format (the binary version of theSequence Alignment Map format) and is also used to compress and indexVariant Call Format (VCF),FASTA, andBED files.[6] Because each block is a standard gzip block, a BGZF file can be decompressed by any standard gzip-compatible tool, ensuring backward compatibility.[7] A general purpose compression utility for producing BGZF filesbgzip is distributed with HTSlib software library.[6]
BGZF is widely utilized inbioinformatics for the compression of large datasets where efficient random access is a crucial requirement.[1] Due to large sizes ofnext-generation sequencing data formats likeSAM files,[8] they are compressed into binaryBAM format utilizing BGZF compression.[4][9]
For random access, an index file is created for a BGZF-compressed file, typically usingTabix.[10] This index stores the file offsets of the compressed blocks alongside the corresponding genomic coordinates, thus allowing a program to seek directly to the block containing the data queried, decompress only them, and retrieve the requested information, avoiding the need to process the entire file.[10]
The format is also extensively employed for compressingvariant call files (VCF) along with their associatedTabix indexes,[10] and similarly for other substantial genomic data files such asBED, GFF/GTF, and occasionallyFASTQ when indexed access is necessary.[6] A broad range of bioinformatics software packages are equipped to read and write BGZF-compressed files; these include well-known tools likeSAMtools, HTSlib, BCF/VCFtools,[11] Picard tools, the GATK, and libraries such asBiopython.[12][13] The standard command-line utility for creating BGZF-compressed files and their corresponding.gzi indexes isbgzip, which is distributed as part of HTSlib.[7]
BGZF has been adapted for development of more efficient data-specific compression methods and algorithms leveraging its block based design.[14]
A BGZF file consists of a series of concatenated BGZF blocks. Each block, whether in its compressed or uncompressed state, is limited to a maximum size of 64 kilobytes. Each BGZF block is itself a fully compliant gzip archive, adhering to the specifications outlined inRFC 1952.[15]
Each BGZF block contains a standard gzip file header with the following standard-compliant extensions:
F.EXTRA bit in the header is set to indicate that extra fields are present.ID values 66 and 67 (ASCII 'BC').LEN in the gzip specification) is 2 (two bytes of payload).This block design allows use of an associated index file (storing offsets of each BGZF block) to fetch and decompress only the block of data that pertains to the query, thus avoiding the computational overhead of reading and decompressing all BGZF blocks.[10]
End-of-file marker for BGZF enables detection of erroneously truncated files and generate warnings or errors for the user. The EOF marker block is an empty (data block of length zero) BGZF block encoded with the defaultzlib compression level settings, and consists of the following 28 hexadecimal bytes:1f 8b 08 04 00 00 00 00 00 ff 06 00 42 43 02 00 1b 00 03 00 00 00 00 00 00 00 00 00The presence of an EOF marker by itself does not signal an end of the file, however, an EOF marker present at the end of a BGZF file indicates that the immediately following physical EOF is the end of the file as intended by the program that wrote it.[15]
BGZF-block level indexing that is common in standard indexes of genomic file formats
[..] both formats can be either plain (uncompressed) or block-compressed with BGZF for random access and compact size.
Reading, decoding, sorting, encoding, and writing large sequence alignment files (tens or hundreds of GBs) can be time-consuming and resource intensive.