- Notifications
You must be signed in to change notification settings - Fork57
A vendor-neutral storage library for Golang: Write once, run on every storage service.
License
beyondstorage/go-storage
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Avendor-neutral storage library for Golang.
Write once, run on every storage service.
- Vendor agnostic
- Production ready
- High performance
package mainimport ("log""go.beyondstorage.io/v5/services""go.beyondstorage.io/v5/types"// Add fs support _"go.beyondstorage.io/services/fs/v4"// Add s3 support _"go.beyondstorage.io/services/s3/v3"// Add gcs support _"go.beyondstorage.io/services/gcs/v3"// Add azblob support _"go.beyondstorage.io/services/azblob/v3"// More support could be found under BeyondStorage. _"go.beyondstorage.io/services/xxx" )funcmain() {// Init a Storager from connection string.store,err:=services.NewStoragerFromString("s3://bucket_name/path/to/workdir")iferr!=nil {log.Fatalf("service init failed: %v",err) }// Write data from io.Reader into hello.txtn,err:=store.Write("hello.txt",r,length)// Read data from hello.txt to io.Writern,err:=store.Read("hello.txt",w)// Stat hello.txt to check existence or get its metadatao,err:=store.Stat("hello.txt")// Use object's functions to get metadatalength,ok:=o.GetContentLength()// List will create an iterator of object under path.it,err:=store.List("path")for {// Use iterator.Next to retrieve next object until we meet IteratorDone.o,err:=it.Next()iferrors.Is(err,types.IteraoorDone) {break } }// Delete hello.txterr=store.Delete("hello.txt")}
More examples could be found atgo-storage-example.
16 stable services that have passed allintegration tests.
- azblob:Azure Blob storage
- bos:Baidu Object Storage
- cos:Tencent Cloud Object Storage
- dropbox:Dropbox
- fs: Local file system
- ftp: FTP
- gcs:Google Cloud Storage
- gdrive:Google Drive
- ipfs:InterPlanetary File System
- kodo:qiniu kodo
- memory: data that only in memory
- minio:MinIO
- obs:Huawei Object Storage Service
- oss:Aliyun Object Storage
- qingstor:QingStor Object Storage
- s3:Amazon S3
3 beta services that implemented required functions, but not passedintegration tests.
4 alpha services that still under development.
More service ideas could be found atService Integration Tracking.
Basic operations
- Metadata: get
Storagermetadata
meta:=store.Metadata()_:=meta.GetWorkDir()// Get object WorkDir_,ok:=meta.GetWriteSizeMaximum()// Get the maximum size for write operation
- Read: read
Objectcontent
// Read 2048 byte at the offset 1024 into the io.Writer.n,err:=store.Read("path",w,pairs.WithOffset(1024),pairs.WithSize(2048))
- Write: write content into
Object
// Write 2048 byte from io.Readern,err:=store.Write("path",r,2048)
- Stat: get
Objectmetadata or check existences
o,err:=store.Stat("path")iferrors.Is(err,services.ErrObjectNotExist) {// object is not exist}length,ok:=o.GetContentLength()// get the object content length.
- Delete: delete an
Object
err:=store.Delete("path")// Delete the object "path"
- List: list
Objectin given prefix or dir
it,err:=store.List("path")for {o,err:=it.Next()iferr!=nil&&errors.Is(err,types.IteratorDone) {// the list is over }length,ok:=o.GetContentLength()// get the object content length.}
Extended operations
- Copy: copy a
Objectinside storager
err:=store.(Copier).Copy(src,dst)// Copy an object from src to dst.
- Move: move a
Objectinside storager
err:=store.(Mover).Move(src,dst)// Move an object from src to dst.
- Reach: generate a public accessible url to an
Object
url,err:=store.(Reacher).Reach("path")// Generate an url to the object.
- Dir: Dir
Objectsupport
o,err:=store.(Direr).CreateDir("path")// Create a dir object.
Large file manipulation
- Multipart: allow doing multipart uploads
ms:=store.(Multiparter)// Create a multipart object.o,err:=ms.CreateMultipart("path")// Write 1024 bytes from io.Reader into a multipart at index 1n,part,err:=ms.WriteMultipart(o,r,1024,1)// Complete a multipart object.err:=ms.CompleteMultipart(o, []*Part{part})
- Append: allow appending to an object
as:=store.(Appender)// Create an appendable object.o,err:=as.CreateAppend("path")// Write 1024 bytes from io.Reader.n,err:=as.WriteAppend(o,r,1024)// Commit an append object.err=as.CommitAppend(o)
- Block: allow combining an object with block ids
bs:=store.(Blocker)// Create a block object.o,err:=bs.CreateBlock("path")// Write 1024 bytes from io.Reader with block id "id-abc"n,err:=bs.WriteBlock(o,r,1024,"id-abc")// Combine block via block ids.err:=bs.CombineBlock(o, []string{"id-abc"})
- Page: allow doing random writes
ps:=store.(Pager)// Create a page object.o,err:=ps.CreatePage("path")// Write 1024 bytes from io.Reader at offset 2048n,err:=ps.WritePage(o,r,1024,2048)
Global object metadata
id: unique key in servicename: relative path towards service's work dirmode: object mode can be a combination ofread,dir,partandmoreetag: entity tag as defined inrfc2616content-length: object's content size.content-md5: md5 digest as defined inrfc2616content-type: media type as defined inrfc2616last-modified: object's last updated time.
System object metadata
Service system object metadata likestorage-class and so on.
o,err:=store.Stat("path")// Get service system metadata via API provides by go-service-s3.om:=s3.GetObjectSystemMetadata(o)_=om.StorageClass// this object's storage class_=om.ServerSideEncryptionCustomerAlgorithm// this object's sse algorithm
Self maintained codegendefinitions helps to generate all our APIs, pairs and metadata.
Generated pairs which can be used as API optional arguments.
funcWithContentMd5(vstring)Pair {returnPair{Key:"content_md5",Value:v, }}
Generated object metadata which can be used to get content md5 from object.
func (o*Object)GetContentMd5() (string,bool) {o.stat()ifo.bit&objectIndexContentMd5!=0 {returno.contentMd5,true }return"",false}
Server-Side Encrypt supports via system pair and system metadata, and we can useDefault Pairs to simplify the job.
funcNewS3SseC(key []byte) (types.Storager,error) {defaultPairs:= s3.DefaultStoragePairs{Write: []types.Pair{// Required, must be AES256s3.WithServerSideEncryptionCustomerAlgorithm(s3.ServerSideEncryptionAes256),// Required, your AES-256 key, a 32-byte binary values3.WithServerSideEncryptionCustomerKey(key), },// Now you have to provide customer key to read encrypted dataRead: []types.Pair{// Required, must be AES256s3.WithServerSideEncryptionCustomerAlgorithm(s3.ServerSideEncryptionAes256),// Required, your AES-256 key, a 32-byte binary values3.WithServerSideEncryptionCustomerKey(key), }}return s3.NewStorager(...,s3.WithDefaultStoragePairs(defaultPairs))}
About
A vendor-neutral storage library for Golang: Write once, run on every storage service.
Topics
Resources
License
Code of conduct
Contributing
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Uh oh!
There was an error while loading.Please reload this page.