 | This article has multiple issues. Please helpimprove it or discuss these issues on thetalk page.(Learn how and when to remove these messages) (Learn how and when to remove this message)
|
| Developer(s) | IBM |
|---|---|
| Full name | IBM Spectrum Scale |
| Introduced | 1998; 27 years ago (1998) withAIX |
| Limits | |
| Max volume size | 8 YB |
| Max file size | 8 EB |
| Maxno. of files | 264 per file system |
| Features | |
| File system permissions | POSIX |
| Transparent encryption | yes |
| Other | |
| Supported operating systems | AIX,Linux,Windows Server |
GPFS (General Parallel File System, brand nameIBM Storage Scale and previouslyIBM Spectrum Scale)[1] is a high-performanceclustered file system software developed byIBM. It can be deployed inshared-disk orshared-nothing distributed parallel modes, or a combination of these. It is used by many of the world's largest commercial companies, as well as some of thesupercomputers on theTop 500 List.[2]For example, it is the filesystem of theSummit[3]atOak Ridge National Laboratory which was the #1 fastest supercomputer in the world in the November 2019 Top 500 List.[4] Summit is a 200Petaflops system composed of more than 9,000POWER9 processors and 27,000 NVIDIAVoltaGPUs. The storage filesystem is called Alpine.[5]
Like typical cluster filesystems, GPFS provides concurrent high-speed file access to applications executing on multiple nodes of clusters. It can be used withAIX clusters,Linux clusters,[6] on MicrosoftWindows Server, or a heterogeneous cluster of AIX, Linux and Windows nodes running onx86,Power orIBM Z processor architectures.
GPFS began as theTiger Shark file system, a research project at IBM'sAlmaden Research Center as early as 1993. Tiger Shark was initially designed to support high throughput multimedia applications. This design turned out to be well suited to scientific computing.[7]
Another ancestor is IBM'sVesta filesystem, developed as a research project at IBM'sThomas J. Watson Research Center between 1992 and 1995.[8] Vesta introduced the concept of file partitioning to accommodate the needs of parallel applications that run on high-performancemulticomputers withparallel I/O subsystems. With partitioning, a file is not a sequence of bytes, but rather multiple disjoint sequences that may be accessed in parallel. The partitioning is such that it abstracts away the number and type of I/O nodes hosting the filesystem, and it allows a variety of logically partitioned views of files, regardless of the physical distribution of data within the I/O nodes. The disjoint sequences are arranged to correspond to individual processes of a parallel application, allowing for improved scalability.[9][10]
Vesta was commercialized as the PIOFS filesystem around 1994,[11] and was succeeded by GPFS around 1998.[12][13] The main difference between the older and newer filesystems was that GPFS replaced the specialized interface offered by Vesta/PIOFS with the standardUnixAPI: all the features to support high performance parallel I/O were hidden from users and implemented under the hood.[7][13] GPFS also shared many components with the related products IBM Multi-Media Server and IBM Video Charger, which is why many GPFS utilities start with the prefixmm—multi-media.[14]: xi
In 2010, IBM previewed a version of GPFS that included a capability known as GPFS-SNC, where SNC stands for Shared Nothing Cluster. This was officially released with GPFS 3.5 in December 2012, and is now known as FPO[15] (File Placement Optimizer).
 | This sectionneeds additional citations forverification. Please helpimprove this article byadding citations to reliable sources in this section. Unsourced material may be challenged and removed. Find sources: "GPFS" – news ·newspapers ·books ·scholar ·JSTOR(January 2020) (Learn how and when to remove this message) |
It is aclustered file system. It breaks a file into blocks of a configured size, less than 1 megabyte each, which are distributed across multiple cluster nodes.
The system stores data on standard block storage volumes, but includes an internal RAID layer that can virtualize those volumes for redundancy and parallel access much like a RAID block storage system. It also has the ability to replicate across volumes at the higher file level.
Features of the architecture include
Other features include high availability, ability to be used in a heterogeneous cluster, disaster recovery, security,DMAPI,HSM andILM.
Hadoop's HDFS filesystem, is designed to store similar or greater quantities of data on commodity hardware — that is, datacenters withoutRAID disks and astorage area network (SAN).
Storage pools allow for the grouping of disks within a file system. An administrator can create tiers of storage by grouping disks based on performance, locality or reliability characteristics. For example, one pool could be high-performanceFibre Channel disks and another more economical SATA storage.
A fileset is a sub-tree of the file system namespace and provides a way to partition the namespace into smaller, more manageable units. Filesets provide an administrative boundary that can be used to set quotas and be specified in a policy to control initial data placement or data migration. Data in a single fileset can reside in one or more storage pools. Where the file data resides and how it is migrated is based on a set of rules in a user defined policy.
There are two types of user defined policies: file placement and file management. File placement policies direct file data as files are created to the appropriate storage pool. File placement rules are selected by attributes such as file name, the user name or the fileset. File management policies allow the file's data to be moved or replicated or files to be deleted. File management policies can be used to move data from one pool to another without changing the file's location in the directory structure. File management policies are determined by file attributes such as last access time, path name or size of the file.
The policy processing engine is scalable and can be run on many nodes at once. This allows management policies to be applied to a single file system with billions of files and complete in a few hours.[citation needed]
{{cite journal}}: CS1 maint: bot: original URL status unknown (link){{cite journal}}: CS1 maint: bot: original URL status unknown (link){{cite book}}: CS1 maint: bot: original URL status unknown (link)