Movatterモバイル変換


[0]ホーム

URL:


swift

packagemodule
v2.0.4Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 3, 2025 License:MITImports:22Imported by:116

Details

Repository

github.com/ncw/swift

Links

README

Swift

This package provides an easy to use library for interfacing with Swift / Openstack Object Storage / Rackspace cloudfiles from the Go Language

Build StatusGo Reference

Install

Use go to install the library

go get github.com/ncw/swift/v2

Usage

See here for full package docs

Here is a short example from the docs

import "github.com/ncw/swift/v2"// Create a connectionc := swift.Connection{UserName: "user",ApiKey:   "key",AuthUrl:  "auth_url",Domain:   "domain", // Name of the domain (v3 auth only)Tenant:   "tenant", // Name of the tenant (v2 auth only)}// Authenticateerr := c.Authenticate()if err != nil {panic(err)}// List all the containerscontainers, err := c.ContainerNames(nil)fmt.Println(containers)// etc...

Migrating fromv1

The library has current major version v2. If you want to migrate from the first version oflibrarygithub.com/ncw/swift you have to explicitly add the/v2 suffix to the imports.

Most of the exported functions were added a newcontext.Context parameter in thev2, which you will have to providewhen migrating.

Additions

Thers sub project contains a wrapper for the Rackspace specific CDN Management interface.

Testing

To run the tests you can either use an embedded fake Swift server either use a real Openstack Swift server or aRackspace Cloud files account.

When using a real Swift server, you need to set these environment variables before running the tests

export SWIFT_API_USER='user'export SWIFT_API_KEY='key'export SWIFT_AUTH_URL='https://url.of.auth.server/v1.0'

And optionally these if using v2 authentication

export SWIFT_TENANT='TenantName'export SWIFT_TENANT_ID='TenantId'

And optionally these if using v3 authentication

export SWIFT_TENANT='TenantName'export SWIFT_TENANT_ID='TenantId'export SWIFT_API_DOMAIN_ID='domain id'export SWIFT_API_DOMAIN='domain name'

And optionally these if using v3 trust

export SWIFT_TRUST_ID='TrustId'

And optionally this if you want to skip server certificate validation

export SWIFT_AUTH_INSECURE=1

And optionally this to configure the connect channel timeout, in seconds

export SWIFT_CONNECTION_CHANNEL_TIMEOUT=60

And optionally this to configure the data channel timeout, in seconds

export SWIFT_DATA_CHANNEL_TIMEOUT=60

Then run the tests withgo test

License

This is free software under the terms of MIT license (check COPYING file included in this package).

Contact and support

The project website is at:

There you can file bug reports, ask for help or contribute patches.

Authors

Contributors

Documentation

Overview

Package swift provides an easy to use interface to Swift / Openstack Object Storage / Rackspace Cloud Files

Standard Usage

Most of the work is done through the Container*() and Object*() methods.

All methods are safe to use concurrently in multiple go routines.

Object Versioning

As defined byhttp://docs.openstack.org/api/openstack-object-storage/1.0/content/Object_Versioning-e1e3230.html#d6e983 one can create a container which allows for version control of files. The suggested method is to create a version container for holding all non-current files, and a current container for holding the latest version that the file points to. The container and objects inside it can be used in the standard manner, however, pushing a file multiple times will result in it being copied to the version container and the new file put in it's place. If the current file is deleted, the previous file in the version container will replace it. This means that if a file is updated 5 times, it must be deleted 5 times to be completely removed from the system.

Rackspace Sub Module

This module specifically allows the enabling/disabling of Rackspace Cloud File CDN management on a container. This is specific to the Rackspace API and not Swift/Openstack, therefore it has been placed in a submodule. One can easily create a RsConnection and use it like the standard Connection to access and manipulate containers and objects.

Index

Examples

Constants

View Source
const (// Use public URL as storage URLEndpointTypePublic =EndpointType("public")// Use internal URL as storage URLEndpointTypeInternal =EndpointType("internal")// Use admin URL as storage URLEndpointTypeAdmin =EndpointType("admin"))
View Source
const (DefaultUserAgent = "goswift/1.0"// Default user agentDefaultRetries   = 3// Default number of retries on token expiryTimeFormat       = "2006-01-02T15:04:05"// Python date format for json replies parsed as UTCUploadTar        = "tar"// Data format specifier for Connection.BulkUpload().UploadTarGzip    = "tar.gz"// Data format specifier for Connection.BulkUpload().UploadTarBzip2   = "tar.bz2"// Data format specifier for Connection.BulkUpload().)
View Source
const IS_AT_LEAST_GO_16 =true

Variables

View Source
var (// Specific Errors you might want to check for equalityNotModified         = newError(304, "Not Modified")BadRequest          = newError(400, "Bad Request")AuthorizationFailed = newError(401, "Authorization Failed")ContainerNotFound   = newError(404, "Container Not Found")ContainerNotEmpty   = newError(409, "Container Not Empty")ObjectNotFound      = newError(404, "Object Not Found")ObjectCorrupted     = newError(422, "Object Corrupted")TimeoutError        = newError(408, "Timeout when reading or writing data")Forbidden           = newError(403, "Operation forbidden")TooLargeObject      = newError(413, "Too Large Object")RateLimit           = newError(498, "Rate Limit")TooManyRequests     = newError(429, "TooManyRequests")// Mappings for container errorsContainerErrorMap = errorMap{400:BadRequest,403:Forbidden,404:ContainerNotFound,409:ContainerNotEmpty,498:RateLimit,})
View Source
var NotLargeObject =errors.New("not a large object")

NotLargeObject is returned if an operation is performed on an object which isn't large.

View Source
var SLONotSupported =errors.New("SLO not supported")

SLONotSupported is returned as an error when Static Large Objects are not supported.

Functions

funcAddExpectAndTransferEncoding

func AddExpectAndTransferEncoding(req *http.Request, hasContentLengthbool)

funcFloatStringToTime

func FloatStringToTime(sstring) (ttime.Time, errerror)

FloatStringToTime converts a floating point number string to a time.Time

The string is floating point number of seconds since the epoch(Unix time). The number should be in fixed point format (notexponential), eg "1354040105.123456789" which represents the time"2012-11-27T18:15:05.123456789Z"

Some care is taken to preserve all the accuracy in the time.Time(which wouldn't happen with a naive conversion through float64) soa round trip conversion won't change the data.

If an error is returned then time will be returned as the zero time.

funcSetExpectContinueTimeout

func SetExpectContinueTimeout(tr *http.Transport, ttime.Duration)

funcTimeToFloatString

func TimeToFloatString(ttime.Time)string

TimeToFloatString converts a time.Time object to a floating point string

The string is floating point number of seconds since the epoch(Unix time). The number is in fixed point format (notexponential), eg "1354040105.123456789" which represents the time"2012-11-27T18:15:05.123456789Z". Trailing zeros will be droppedfrom the output.

Some care is taken to preserve all the accuracy in the time.Time(which wouldn't happen with a naive conversion through float64) soa round trip conversion won't change the data.

Types

typeAccount

type Account struct {BytesUsedint64// total number of bytes usedContainersint64// total number of containersObjectsint64// total number of objects}

Account contains information about this account.

typeAuthenticator

type Authenticator interface {// Request creates an http.Request for the auth - return nil if not neededRequest(context.Context, *Connection) (*http.Request,error)// Response parses the http.ResponseResponse(ctxcontext.Context, resp *http.Response)error// The public storage URL - set Internal to true to read// internal/service net URLStorageUrl(Internalbool)string// The access tokenToken()string// The CDN url if availableCdnUrl()string}

Auth defines the operations needed to authenticate with swift

This encapsulates the different authentication schemes in use

typeBulkDeleteResult

type BulkDeleteResult struct {NumberNotFoundint64// # of objects not found.NumberDeletedint64// # of deleted objects.Errors         map[string]error// Mapping between object name and an error.HeadersHeaders// Response HTTP headers.}

BulkDeleteResult stores results of BulkDelete().

Individual errors may (or may not) be returned by Errors.Errors is a map whose keys are a full path of where the object wasto be deleted, and whose values are Error objects. A full path ofobject looks like "/API_VERSION/USER_ACCOUNT/CONTAINER/OBJECT_PATH".

typeBulkUploadResult

type BulkUploadResult struct {NumberCreatedint64// # of created objects.Errors        map[string]error// Mapping between object name and an error.HeadersHeaders// Response HTTP headers.}

BulkUploadResult stores results of BulkUpload().

Individual errors may (or may not) be returned by Errors.Errors is a map whose keys are a full path of where an object wasto be created, and whose values are Error objects. A full path ofobject looks like "/API_VERSION/USER_ACCOUNT/CONTAINER/OBJECT_PATH".

typeConnection

type Connection struct {// Parameters - fill these in before calling Authenticate// They are all optional except UserName, ApiKey and AuthUrlDomainstring// User's domain nameDomainIdstring// User's domain IdUserNamestring// UserName for apiUserIdstring// User IdApiKeystring// Key for api accessApplicationCredentialIdstring// Application Credential IDApplicationCredentialNamestring// Application Credential NameApplicationCredentialSecretstring// Application Credential SecretTokenstring// Token used for v3token authenticationAuthUrlstring// Auth URLRetriesint// Retries on error (default is 3)UserAgentstring// Http User agent (default goswift/1.0)ConnectTimeouttime.Duration// Connect channel timeout (default 10s)Timeouttime.Duration// Data channel timeout (default 60s)Regionstring// Region to use eg "LON", "ORD" - default is use first region (v2,v3 auth only)AuthVersionint// Set to 1, 2 or 3 or leave at 0 for autodetectInternalbool// Set this to true to use the the internal / service networkTenantstring// Name of the tenant (v2,v3 auth only)TenantIdstring// Id of the tenant (v2,v3 auth only)EndpointTypeEndpointType// Endpoint type (v2,v3 auth only) (default is public URL unless Internal is set)TenantDomainstring// Name of the tenant's domain (v3 auth only), only needed if it differs from the user domainTenantDomainIdstring// Id of the tenant's domain (v3 auth only), only needed if it differs the from user domainTrustIdstring// Id of the trust (v3 auth only)Transporthttp.RoundTripper `json:"-" xml:"-"`// Optional specialised http.Transport (eg. for Google Appengine)// These are filled in after Authenticate is called as are the defaults for aboveStorageUrlstringAuthTokenstringExpirestime.Time// time the token expires, may be Zero if unknownAuthAuthenticator `json:"-" xml:"-"`// the current authenticator// Workarounds for non-compliant servers that don't always return opts.Limit items per pageFetchUntilEmptyPagebool// Always fetch unless we received an empty pagePartialPageFetchThresholdint// Fetch if the current page is this percentage of opts.Limit// contains filtered or unexported fields}

Connection holds the details of the connection to the swift server.

You need to provide UserName, ApiKey and AuthUrl when you create aconnection then call Authenticate on it.

The auth version in use will be detected from the AuthURL - you canoverride this with the AuthVersion parameter.

If using v2 auth you can also set Region in the Connectionstructure. If you don't set Region you will get the default regionwhich may not be what you want.

For reference some common AuthUrls looks like this:

Rackspace US        https://auth.api.rackspacecloud.com/v1.0Rackspace UK        https://lon.auth.api.rackspacecloud.com/v1.0Rackspace v2        https://identity.api.rackspacecloud.com/v2.0Memset Memstore UK  https://auth.storage.memset.com/v1.0Memstore v2         https://auth.storage.memset.com/v2.0

When using Google Appengine you must provide the Connection with anappengine-specific Transport:

import ("appengine/urlfetch""fmt""github.com/ncw/swift/v2")func handler(w http.ResponseWriter, r *http.Request) {ctx := appengine.NewContext(r)tr := urlfetch.Transport{Context: ctx}c := swift.Connection{UserName:  "user",ApiKey:    "key",AuthUrl:   "auth_url",Transport: tr,}_ := c.Authenticate()containers, _ := c.ContainerNames(nil)fmt.Fprintf(w, "containers: %q", containers)}

If you don't supply a Transport, one is made which relies onhttp.ProxyFromEnvironment (http://golang.org/pkg/net/http/#ProxyFromEnvironment).This means that the connection will respect the HTTP proxy specified by theenvironment variables $HTTP_PROXY and $NO_PROXY.

Example
ctx := context.Background()// Create a v1 auth connectionc := &swift.Connection{// This should be your usernameUserName: "user",// This should be your api keyApiKey: "key",// This should be a v1 auth url, eg//  Rackspace US        https://auth.api.rackspacecloud.com/v1.0//  Rackspace UK        https://lon.auth.api.rackspacecloud.com/v1.0//  Memset Memstore UK  https://auth.storage.memset.com/v1.0AuthUrl: "auth_url",}// Authenticateerr := c.Authenticate(ctx)if err != nil {panic(err)}// List all the containerscontainers, err := c.ContainerNames(ctx, nil)if err != nil {panic(err)}fmt.Println(containers)// etc...// ------ or alternatively create a v2 connection ------// Create a v2 auth connectionc = &swift.Connection{// This is the sub user for the storage - eg "admin"UserName: "user",// This should be your api keyApiKey: "key",// This should be a version2 auth url, eg//  Rackspace v2        https://identity.api.rackspacecloud.com/v2.0//  Memset Memstore v2  https://auth.storage.memset.com/v2.0AuthUrl: "v2_auth_url",// Region to use - default is use first region if unsetRegion: "LON",// Name of the tenant - this is likely your usernameTenant: "jim",}// as above...

func (*Connection)Account

func (c *Connection) Account(ctxcontext.Context) (infoAccount, headersHeaders, errerror)

Account returns info about the account in an Account struct.

func (*Connection)AccountUpdate

func (c *Connection) AccountUpdate(ctxcontext.Context, hHeaders)error

AccountUpdate adds, replaces or remove account metadata.

Add or update keys by mentioning them in the Headers.

Remove keys by setting them to an empty string.

func (*Connection)ApplyEnvironment

func (c *Connection) ApplyEnvironment() (errerror)

ApplyEnvironment reads environment variables and applies them tothe Connection structure. It won't overwrite any parameters whichare already set in the Connection struct.

To make a new Connection object entirely from the environment youwould do:

c := new(Connection)err := c.ApplyEnvironment()if err != nil { log.Fatal(err) }

The naming of these variables follows the official Openstack namingscheme so it should be compatible with OpenStack rc files.

For v1 authentication (obsolete)

ST_AUTH - Auth URLST_USER - UserName for apiST_KEY - Key for api access

For v2 authentication

OS_AUTH_URL - Auth URLOS_USERNAME - UserName for apiOS_PASSWORD - Key for api accessOS_TENANT_NAME - Name of the tenantOS_TENANT_ID   - Id of the tenantOS_REGION_NAME - Region to use - default is use first region

For v3 authentication

OS_AUTH_URL - Auth URLOS_USERNAME - UserName for apiOS_USER_ID - User IdOS_PASSWORD - Key for api accessOS_APPLICATION_CREDENTIAL_ID - Application Credential IDOS_APPLICATION_CREDENTIAL_NAME - Application Credential NameOS_APPLICATION_CREDENTIAL_SECRET - Application Credential SecretOS_USER_DOMAIN_NAME - User's domain nameOS_USER_DOMAIN_ID - User's domain IdOS_PROJECT_NAME - Name of the projectOS_PROJECT_DOMAIN_NAME - Name of the tenant's domain, only needed if it differs from the user domainOS_PROJECT_DOMAIN_ID - Id of the tenant's domain, only needed if it differs the from user domainOS_TRUST_ID - If of the trustOS_REGION_NAME - Region to use - default is use first region

Other

OS_ENDPOINT_TYPE - Endpoint type public, internal or adminST_AUTH_VERSION - Choose auth version - 1, 2 or 3 or leave at 0 for autodetect

For manual authentication

OS_STORAGE_URL - storage URL from alternate authenticationOS_AUTH_TOKEN - Auth Token from alternate authentication

Library specific

GOSWIFT_RETRIES - Retries on error (default is 3)GOSWIFT_USER_AGENT - HTTP User agent (default goswift/1.0)GOSWIFT_CONNECT_TIMEOUT - Connect channel timeout with unit, eg "10s", "100ms" (default "10s")GOSWIFT_TIMEOUT - Data channel timeout with unit, eg "10s", "100ms" (default "60s")GOSWIFT_INTERNAL - Set this to "true" to use the the internal network (obsolete - use OS_ENDPOINT_TYPE)

func (*Connection)Authenticate

func (c *Connection) Authenticate(ctxcontext.Context) (errerror)

Authenticate connects to the Swift server.

If you don't call it before calling one of the connection methodsthen it will be called for you on the first access.

func (*Connection)Authenticated

func (c *Connection) Authenticated()bool

Authenticated returns a boolean to show if the current connectionis authenticated.

Doesn't actually check the credentials against the server.

func (*Connection)BulkDelete

func (c *Connection) BulkDelete(ctxcontext.Context, containerstring, objectNames []string) (resultBulkDeleteResult, errerror)

BulkDelete deletes multiple objectNames from container in one operation.

Some servers may not accept bulk-delete requests since bulk-delete isan optional feature of swift - these will return the Forbidden error.

See also:*http://docs.openstack.org/trunk/openstack-object-storage/admin/content/object-storage-bulk-delete.html*http://docs.rackspace.com/files/api/v1/cf-devguide/content/Bulk_Delete-d1e2338.html

func (*Connection)BulkDeleteHeaders

func (c *Connection) BulkDeleteHeaders(ctxcontext.Context, containerstring, objectNames []string, hHeaders) (resultBulkDeleteResult, errerror)

BulkDeleteHeaders deletes multiple objectNames from container in one operation.

Some servers may not accept bulk-delete requests since bulk-delete isan optional feature of swift - these will return the Forbidden error.

See also:*http://docs.openstack.org/trunk/openstack-object-storage/admin/content/object-storage-bulk-delete.html*http://docs.rackspace.com/files/api/v1/cf-devguide/content/Bulk_Delete-d1e2338.html

func (*Connection)BulkUpload

func (c *Connection) BulkUpload(ctxcontext.Context, uploadPathstring, dataStreamio.Reader, formatstring, hHeaders) (resultBulkUploadResult, errerror)

BulkUpload uploads multiple files in one operation.

uploadPath can be empty, a container name, or a pseudo-directorywithin a container. If uploadPath is empty, new containers may beautomatically created.

Files are read from dataStream. The format of the stream is specifiedby the format parameter. Available formats are:* UploadTar - Plain tar stream.* UploadTarGzip - Gzip compressed tar stream.* UploadTarBzip2 - Bzip2 compressed tar stream.

Some servers may not accept bulk-upload requests since bulk-upload isan optional feature of swift - these will return the Forbidden error.

See also:*http://docs.openstack.org/trunk/openstack-object-storage/admin/content/object-storage-extract-archive.html*http://docs.rackspace.com/files/api/v1/cf-devguide/content/Extract_Archive-d1e2338.html

func (*Connection)Call

func (c *Connection) Call(ctxcontext.Context, targetUrlstring, pRequestOpts) (resp *http.Response, headersHeaders, errerror)

Call runs a remote command on the targetUrl, returns aresponse, headers and possible error.

operation is GET, HEAD etccontainer is the name of a containerAny other parameters (if not None) are added to the targetUrl

Returns a response or an error. If response is returned thenthe resp.Body must be read completely andresp.Body.Close() must be called on it, unless noResponse is set inwhich case the body will be closed in this function

If "Content-Length" is set in p.Headers it will be used - this canbe used to override the default chunked transfer encoding foruploads.

This will Authenticate if necessary, and re-authenticate if itreceives a 401 error which means the token has expired

This method is exported so extensions can call it.

func (*Connection)Container

func (c *Connection) Container(ctxcontext.Context, containerstring) (infoContainer, headersHeaders, errerror)

Container returns info about a single container including anymetadata in the headers.

func (*Connection)ContainerCreate

func (c *Connection) ContainerCreate(ctxcontext.Context, containerstring, hHeaders)error

ContainerCreate creates a container.

If you don't want to add Headers just pass in nil

No error is returned if it already exists but the metadata if any will be updated.

func (*Connection)ContainerDelete

func (c *Connection) ContainerDelete(ctxcontext.Context, containerstring)error

ContainerDelete deletes a container.

May return ContainerDoesNotExist or ContainerNotEmpty

func (*Connection)ContainerNames

func (c *Connection) ContainerNames(ctxcontext.Context, opts *ContainersOpts) ([]string,error)

ContainerNames returns a slice of names of containers in this account.

func (*Connection)ContainerNamesAll

func (c *Connection) ContainerNamesAll(ctxcontext.Context, opts *ContainersOpts) ([]string,error)

ContainerNamesAll is like ContainerNames but it returns all the Containers

It calls ContainerNames multiple times using the Marker parameter

It has a default Limit parameter but you may pass in your own

func (*Connection)ContainerUpdate

func (c *Connection) ContainerUpdate(ctxcontext.Context, containerstring, hHeaders)error

ContainerUpdate adds, replaces or removes container metadata.

Add or update keys by mentioning them in the Metadata.

Remove keys by setting them to an empty string.

Container metadata can only be read with Container() not with Containers().

func (*Connection)Containers

func (c *Connection) Containers(ctxcontext.Context, opts *ContainersOpts) ([]Container,error)

Containers returns a slice of structures with full information asdescribed in Container.

func (*Connection)ContainersAll

func (c *Connection) ContainersAll(ctxcontext.Context, opts *ContainersOpts) ([]Container,error)

ContainersAll is like Containers but it returns all the Containers

It calls Containers multiple times using the Marker parameter

It has a default Limit parameter but you may pass in your own

func (*Connection)DynamicLargeObjectCreate

func (c *Connection) DynamicLargeObjectCreate(ctxcontext.Context, opts *LargeObjectOpts) (LargeObjectFile,error)

DynamicLargeObjectCreate creates or truncates an existing dynamiclarge object returning a writeable object. This sets opts.Flags toan appropriate value before calling DynamicLargeObjectCreateFile

func (*Connection)DynamicLargeObjectCreateFile

func (c *Connection) DynamicLargeObjectCreateFile(ctxcontext.Context, opts *LargeObjectOpts) (LargeObjectFile,error)

DynamicLargeObjectCreateFile creates a dynamic large objectreturning an object which satisfies io.Writer, io.Seeker, io.Closerand io.ReaderFrom. The flags are as passes to thelargeObjectCreate method.

func (*Connection)DynamicLargeObjectDelete

func (c *Connection) DynamicLargeObjectDelete(ctxcontext.Context, containerstring, pathstring)error

DynamicLargeObjectDelete deletes a dynamic large object and all of its segments.

func (*Connection)DynamicLargeObjectMove

func (c *Connection) DynamicLargeObjectMove(ctxcontext.Context, srcContainerstring, srcObjectNamestring, dstContainerstring, dstObjectNamestring)error

DynamicLargeObjectMove moves a dynamic large object from srcContainer, srcObjectName to dstContainer, dstObjectName

func (*Connection)GetStorageUrladded inv2.0.1

func (c *Connection) GetStorageUrl(ctxcontext.Context) (string,error)

GetStorageUrl returns Swift storage URL.

func (*Connection)LargeObjectDelete

func (c *Connection) LargeObjectDelete(ctxcontext.Context, containerstring, objectNamestring)error

LargeObjectDelete deletes the large object named by container, path

func (*Connection)LargeObjectGetSegments

func (c *Connection) LargeObjectGetSegments(ctxcontext.Context, containerstring, pathstring) (string, []Object,error)

LargeObjectGetSegments returns all the segments that compose an objectIf the object is a Dynamic Large Object (DLO), it just returns the objectsthat have the prefix as indicated by the manifest.If the object is a Static Large Object (SLO), it retrieves the JSON contentof the manifest and return all the segments of it.

func (*Connection)Object

func (c *Connection) Object(ctxcontext.Context, containerstring, objectNamestring) (infoObject, headersHeaders, errerror)

Object returns info about a single object including any metadata in the header.

May return ObjectNotFound.

Use headers.ObjectMetadata() to read the metadata in the Headers.

func (*Connection)ObjectCopy

func (c *Connection) ObjectCopy(ctxcontext.Context, srcContainerstring, srcObjectNamestring, dstContainerstring, dstObjectNamestring, hHeaders) (headersHeaders, errerror)

ObjectCopy does a server side copy of an object to a new position

All metadata is preserved. If metadata is set in the headers thenit overrides the old metadata on the copied object.

The destination container must exist before the copy.

You can use this to copy an object to itself - this is the only wayto update the content type of an object.

func (*Connection)ObjectCreate

func (c *Connection) ObjectCreate(ctxcontext.Context, containerstring, objectNamestring, checkHashbool, Hashstring, contentTypestring, hHeaders) (file *ObjectCreateFile, errerror)

ObjectCreate creates or updates the object in the container. Itreturns an io.WriteCloser you should write the contents to. YouMUST call Close() on it and you MUST check the error return fromClose().

If checkHash is True then it will calculate the MD5 Hash of thefile as it is being uploaded and check it against that returnedfrom the server. If it is wrong then it will returnObjectCorrupted on Close()

If you know the MD5 hash of the object ahead of time then set theHash parameter and it will be sent to the server (as an Etagheader) and the server will check the MD5 itself after the upload,and this will return ObjectCorrupted on Close() if it is incorrect.

If you don't want any error protection (not recommended) then setcheckHash to false and Hash to "".

If contentType is set it will be used, otherwise one will beguessed from objectName using mime.TypeByExtension

func (*Connection)ObjectDelete

func (c *Connection) ObjectDelete(ctxcontext.Context, containerstring, objectNamestring)error

ObjectDelete deletes the object.

May return ObjectNotFound if the object isn't found

func (*Connection)ObjectGet

func (c *Connection) ObjectGet(ctxcontext.Context, containerstring, objectNamestring, contentsio.Writer, checkHashbool, hHeaders) (headersHeaders, errerror)

ObjectGet gets the object into the io.Writer contents.

Returns the headers of the response.

If checkHash is true then it will calculate the md5sum of the fileas it is being received and check it against that returned from theserver. If it is wrong then it will return ObjectCorrupted.

headers["Content-Type"] will give the content type if desired.

func (*Connection)ObjectGetBytes

func (c *Connection) ObjectGetBytes(ctxcontext.Context, containerstring, objectNamestring) (contents []byte, errerror)

ObjectGetBytes returns an object as a []byte.

This is a simplified interface which checks the MD5

func (*Connection)ObjectGetString

func (c *Connection) ObjectGetString(ctxcontext.Context, containerstring, objectNamestring) (contentsstring, errerror)

ObjectGetString returns an object as a string.

This is a simplified interface which checks the MD5

func (*Connection)ObjectMove

func (c *Connection) ObjectMove(ctxcontext.Context, srcContainerstring, srcObjectNamestring, dstContainerstring, dstObjectNamestring) (errerror)

ObjectMove does a server side move of an object to a new position

This is a convenience method which calls ObjectCopy then ObjectDelete

All metadata is preserved.

The destination container must exist before the copy.

func (*Connection)ObjectNames

func (c *Connection) ObjectNames(ctxcontext.Context, containerstring, opts *ObjectsOpts) ([]string,error)

ObjectNames returns a slice of names of objects in a given container.

func (*Connection)ObjectNamesAll

func (c *Connection) ObjectNamesAll(ctxcontext.Context, containerstring, opts *ObjectsOpts) ([]string,error)

ObjectNamesAll is like ObjectNames but it returns all the Objects

It calls ObjectNames multiple times using the Marker parameter. Marker isreset unless KeepMarker is set

It has a default Limit parameter but you may pass in your own

func (*Connection)ObjectOpen

func (c *Connection) ObjectOpen(ctxcontext.Context, containerstring, objectNamestring, checkHashbool, hHeaders) (file *ObjectOpenFile, headersHeaders, errerror)

ObjectOpen returns an ObjectOpenFile for reading the contents ofthe object. This satisfies the io.ReadCloser and the io.Seekerinterfaces.

You must call Close() on contents when finished

Returns the headers of the response.

If checkHash is true then it will calculate the md5sum of the fileas it is being received and check it against that returned from theserver. If it is wrong then it will return ObjectCorrupted. Itwill also check the length returned. No checking will be done ifyou don't read all the contents.

Note that objects with X-Object-Manifest or X-Static-Large-Objectset won't ever have their md5sum's checked as the md5sum reportedon the object is actually the md5sum of the md5sums of theparts. This isn't very helpful to detect a corrupted download asthe size of the parts aren't known without doing more operations.If you want to ensure integrity of an object with a manifest thenyou will need to download everything in the manifest separately.

headers["Content-Type"] will give the content type if desired.

func (*Connection)ObjectPut

func (c *Connection) ObjectPut(ctxcontext.Context, containerstring, objectNamestring, contentsio.Reader, checkHashbool, Hashstring, contentTypestring, hHeaders) (headersHeaders, errerror)

ObjectPut creates or updates the path in the container fromcontents. contents should be an open io.Reader which will have allits contents read.

This is a low level interface.

If checkHash is True then it will calculate the MD5 Hash of thefile as it is being uploaded and check it against that returnedfrom the server. If it is wrong then it will returnObjectCorrupted.

If you know the MD5 hash of the object ahead of time then set theHash parameter and it will be sent to the server (as an Etagheader) and the server will check the MD5 itself after the upload,and this will return ObjectCorrupted if it is incorrect.

If you don't want any error protection (not recommended) then setcheckHash to false and Hash to "".

If contentType is set it will be used, otherwise one will beguessed from objectName using mime.TypeByExtension

func (*Connection)ObjectPutBytes

func (c *Connection) ObjectPutBytes(ctxcontext.Context, containerstring, objectNamestring, contents []byte, contentTypestring) (errerror)

ObjectPutBytes creates an object from a []byte in a container.

This is a simplified interface which checks the MD5.

func (*Connection)ObjectPutString

func (c *Connection) ObjectPutString(ctxcontext.Context, containerstring, objectNamestring, contentsstring, contentTypestring) (errerror)

ObjectPutString creates an object from a string in a container.

This is a simplified interface which checks the MD5

func (*Connection)ObjectSymlinkCreate

func (c *Connection) ObjectSymlinkCreate(ctxcontext.Context, containerstring, symlinkstring, targetAccountstring, targetContainerstring, targetObjectstring, targetEtagstring) (headersHeaders, errerror)

func (*Connection)ObjectTempUrl

func (c *Connection) ObjectTempUrl(containerstring, objectNamestring, secretKeystring, methodstring, expirestime.Time)string

ObjectTempUrl returns a temporary URL for an object

func (*Connection)ObjectUpdate

func (c *Connection) ObjectUpdate(ctxcontext.Context, containerstring, objectNamestring, hHeaders)error

ObjectUpdate adds, replaces or removes object metadata.

Add or Update keys by mentioning them in the Metadata. UseMetadata.ObjectHeaders and Headers.ObjectMetadata to convert yourMetadata to and from normal HTTP headers.

This removes all metadata previously added to the object andreplaces it with that passed in so to delete keys, just don'tmention them the headers you pass in.

Object metadata can only be read with Object() not with Objects().

This can also be used to set headers not already assigned such asX-Delete-At or X-Delete-After for expiring objects.

You cannot use this to change any of the object's other headerssuch as Content-Type, ETag, etc.

Refer to copying an object when you need to update metadata orother headers such as Content-Type or CORS headers.

May return ObjectNotFound.

func (*Connection)ObjectUpdateContentType

func (c *Connection) ObjectUpdateContentType(ctxcontext.Context, containerstring, objectNamestring, contentTypestring) (errerror)

ObjectUpdateContentType updates the content type of an object

This is a convenience method which calls ObjectCopy

All other metadata is preserved.

func (*Connection)Objects

func (c *Connection) Objects(ctxcontext.Context, containerstring, opts *ObjectsOpts) ([]Object,error)

Objects returns a slice of Object with information about eachobject in the container.

If Delimiter is set in the opts then PseudoDirectory may be set,with ContentType 'application/directory'. These are not realobjects but represent directories of objects which haven't had anobject created for them.

func (*Connection)ObjectsAll

func (c *Connection) ObjectsAll(ctxcontext.Context, containerstring, opts *ObjectsOpts) ([]Object,error)

ObjectsAll is like Objects but it returns an unlimited number of Objects in a slice

It calls Objects multiple times using the Marker parameter

func (*Connection)ObjectsWalk

func (c *Connection) ObjectsWalk(ctxcontext.Context, containerstring, opts *ObjectsOpts, walkFnObjectsWalkFn)error

ObjectsWalk is uses to iterate through all the objects in chunks asreturned by Objects or ObjectNames using the Marker and Limitparameters in the ObjectsOpts.

Pass in a closure `walkFn` which calls Objects or ObjectNames withthe *ObjectsOpts passed to it and does something with the results.

Errors will be returned from this function

It has a default Limit parameter but you may pass in your own

Example
c, rollback := makeConnection(nil)defer rollback()objects := make([]string, 0)err := c.ObjectsWalk(context.Background(), container, nil, func(ctx context.Context, opts *swift.ObjectsOpts) (interface{}, error) {newObjects, err := c.ObjectNames(ctx, container, opts)if err == nil {objects = append(objects, newObjects...)}return newObjects, err})fmt.Println("Found all the objects", objects, err)

func (*Connection)QueryInfo

func (c *Connection) QueryInfo(ctxcontext.Context) (infosSwiftInfo, errerror)

Discover Swift configuration by doing a request against /info

func (*Connection)StaticLargeObjectCreate

func (c *Connection) StaticLargeObjectCreate(ctxcontext.Context, opts *LargeObjectOpts) (LargeObjectFile,error)

StaticLargeObjectCreate creates or truncates an existing staticlarge object returning a writeable object. This sets opts.Flags toan appropriate value before calling StaticLargeObjectCreateFile

func (*Connection)StaticLargeObjectCreateFile

func (c *Connection) StaticLargeObjectCreateFile(ctxcontext.Context, opts *LargeObjectOpts) (LargeObjectFile,error)

StaticLargeObjectCreateFile creates a static large object returningan object which satisfies io.Writer, io.Seeker, io.Closer andio.ReaderFrom. The flags are as passed to the largeObjectCreatemethod.

func (*Connection)StaticLargeObjectDelete

func (c *Connection) StaticLargeObjectDelete(ctxcontext.Context, containerstring, pathstring)error

StaticLargeObjectDelete deletes a static large object and all of its segments.

func (*Connection)StaticLargeObjectMove

func (c *Connection) StaticLargeObjectMove(ctxcontext.Context, srcContainerstring, srcObjectNamestring, dstContainerstring, dstObjectNamestring)error

StaticLargeObjectMove moves a static large object from srcContainer, srcObjectName to dstContainer, dstObjectName

func (*Connection)UnAuthenticate

func (c *Connection) UnAuthenticate()

UnAuthenticate removes the authentication from the Connection.

func (*Connection)VersionContainerCreate

func (c *Connection) VersionContainerCreate(ctxcontext.Context, current, versionstring)error

VersionContainerCreate is a helper method for creating and enabling version controlled containers.

It builds the current object container, the non-current object version container, and enables versioning.

If the server doesn't support versioning then it will returnForbidden however it will have created both the containers at that point.

Example
c, rollback := makeConnection(nil)defer rollback()// Use the helper method to create the current and versions container.if err := c.VersionContainerCreate(context.Background(), "cds", "cd-versions"); err != nil {fmt.Print(err.Error())}

func (*Connection)VersionDisable

func (c *Connection) VersionDisable(ctxcontext.Context, currentstring)error

VersionDisable disables versioning on the current container.

Example
c, rollback := makeConnection(nil)defer rollback()// Disable versioning on a container.  Note that this does not delete the versioning container.err := c.VersionDisable(context.Background(), "movies")if err != nil {panic(err)}

func (*Connection)VersionEnable

func (c *Connection) VersionEnable(ctxcontext.Context, current, versionstring)error

VersionEnable enables versioning on the current container with version as the tracking container.

May return Forbidden if this isn't supported by the server

Example
ctx := context.Background()c, rollback := makeConnection(nil)defer rollback()// Build the containers manually and enable them.if err := c.ContainerCreate(ctx, "movie-versions", nil); err != nil {fmt.Print(err.Error())}if err := c.ContainerCreate(ctx, "movies", nil); err != nil {fmt.Print(err.Error())}if err := c.VersionEnable(ctx, "movies", "movie-versions"); err != nil {fmt.Print(err.Error())}// Access the primary container as usual with ObjectCreate(), ObjectPut(), etc.// etc...

func (*Connection)VersionObjectList

func (c *Connection) VersionObjectList(ctxcontext.Context, version, objectstring) ([]string,error)

VersionObjectList returns a list of older versions of the object.

Objects are returned in the format <length><object_name>/<timestamp>

typeContainer

type Container struct {Namestring// Name of the containerCountint64// Number of objects in the containerBytesint64// Total number of bytes used in the containerQuotaCountint64// Maximum object count of the container. 0 if not availableQuotaBytesint64// Maximum size of the container, in bytes. 0 if not available}

Container contains information about a container

typeContainersOpts

type ContainersOpts struct {Limitint// For an integer value n, limits the number of results to at most n values.Prefixstring// Given a string value x, return container names matching the specified prefix.Markerstring// Given a string value x, return container names greater in value than the specified marker.EndMarkerstring// Given a string value x, return container names less in value than the specified marker.HeadersHeaders// Any additional HTTP headers - can be nil}

ContainersOpts is options for Containers() and ContainerNames()

typeCustomEndpointAuthenticator

type CustomEndpointAuthenticator interface {StorageUrlForEndpoint(endpointTypeEndpointType)string}

typeDynamicLargeObjectCreateFile

type DynamicLargeObjectCreateFile struct {// contains filtered or unexported fields}

DynamicLargeObjectCreateFile represents an open static large object

func (*DynamicLargeObjectCreateFile)Close

Close satisfies the io.Closer interface

func (*DynamicLargeObjectCreateFile)CloseWithContext

func (file *DynamicLargeObjectCreateFile) CloseWithContext(ctxcontext.Context)error

func (*DynamicLargeObjectCreateFile)Flush

func (*DynamicLargeObjectCreateFile)Seek

func (file *DynamicLargeObjectCreateFile) Seek(offsetint64, whenceint) (int64,error)

Seek sets the offset for the next write operation

func (*DynamicLargeObjectCreateFile)Size

func (file *DynamicLargeObjectCreateFile) Size()int64

func (*DynamicLargeObjectCreateFile)Write

func (file *DynamicLargeObjectCreateFile) Write(buf []byte) (int,error)

func (*DynamicLargeObjectCreateFile)WriteWithContext

func (file *DynamicLargeObjectCreateFile) WriteWithContext(ctxcontext.Context, buf []byte) (int,error)

typeEndpointType

type EndpointTypestring

typeError

type Error struct {StatusCodeint// HTTP status code if relevant or 0 if notTextstring}

Error - all errors generated by this package are of this type. Other errormay be passed on from library functions though.

func (*Error)Error

func (e *Error) Error()string

Error satisfy the error interface.

typeExpireser

type Expireser interface {Expires()time.Time}

Expireser is an optional interface to read the expiration time of the token

typeHeaders

type Headers map[string]string

Headers stores HTTP headers (can only have one of each header like Swift).

func (Headers)AccountMetadata

func (hHeaders) AccountMetadata()Metadata

AccountMetadata converts Headers from account to a Metadata.

The keys in the Metadata will be converted to lower case.

func (Headers)ContainerMetadata

func (hHeaders) ContainerMetadata()Metadata

ContainerMetadata converts Headers from container to a Metadata.

The keys in the Metadata will be converted to lower case.

func (Headers)IsLargeObject

func (headersHeaders) IsLargeObject()bool

func (Headers)IsLargeObjectDLO

func (headersHeaders) IsLargeObjectDLO()bool

func (Headers)IsLargeObjectSLO

func (headersHeaders) IsLargeObjectSLO()bool

func (Headers)Metadata

func (hHeaders) Metadata(metaPrefixstring)Metadata

Metadata gets the Metadata starting with the metaPrefix out of the Headers.

The keys in the Metadata will be converted to lower case

func (Headers)ObjectMetadata

func (hHeaders) ObjectMetadata()Metadata

ObjectMetadata converts Headers from object to a Metadata.

The keys in the Metadata will be converted to lower case.

typeLargeObjectFile

type LargeObjectFile interface {io.Seekerio.Writerio.CloserWriteWithContext(ctxcontext.Context, p []byte) (nint, errerror)CloseWithContext(ctxcontext.Context)errorSize()int64Flush(ctxcontext.Context)error}

typeLargeObjectOpts

type LargeObjectOpts struct {Containerstring// Name of container to place objectObjectNamestring// Name of objectFlagsint// Creation flagsCheckHashbool// If set Check the hashHashstring// If set use this hash to checkContentTypestring// Content-Type of the objectHeadersHeaders// Additional headers to upload the object withChunkSizeint64// Size of chunks of the object, defaults to 10MB if not setMinChunkSizeint64// Minimum chunk size, automatically set for SLO's based on infoSegmentContainerstring// Name of the container to place segmentsSegmentPrefixstring// Prefix to use for the segmentsNoBufferbool// Prevents using a bufio.Writer to write segments}

LargeObjectOpts describes how a large object should be created

typeMetadata

type Metadata map[string]string

Metadata stores account, container or object metadata.

func (Metadata)AccountHeaders

func (mMetadata) AccountHeaders()Headers

AccountHeaders converts the Metadata for the account.

func (Metadata)ContainerHeaders

func (mMetadata) ContainerHeaders()Headers

ContainerHeaders converts the Metadata for the container.

func (Metadata)GetModTime

func (mMetadata) GetModTime() (ttime.Time, errerror)

GetModTime reads a modification time (mtime) from a Metadata object

This is a defacto standard (used in the official python-swiftclientamongst others) for storing the modification time (as read usingos.Stat) for an object. It is stored using the key 'mtime', whichfor example when written to an object will be 'X-Object-Meta-Mtime'.

If an error is returned then time will be returned as the zero time.

func (Metadata)Headers

func (mMetadata) Headers(metaPrefixstring)Headers

Headers convert the Metadata starting with the metaPrefix into aHeaders.

The keys in the Metadata will be converted from lower case to httpCanonical (see http.CanonicalHeaderKey).

func (Metadata)ObjectHeaders

func (mMetadata) ObjectHeaders()Headers

ObjectHeaders converts the Metadata for the object.

func (Metadata)SetModTime

func (mMetadata) SetModTime(ttime.Time)

SetModTime writes an modification time (mtime) to a Metadata object

This is a defacto standard (used in the official python-swiftclientamongst others) for storing the modification time (as read usingos.Stat) for an object. It is stored using the key 'mtime', whichfor example when written to an object will be 'X-Object-Meta-Mtime'.

typeObject

type Object struct {Namestring     `json:"name"`// object nameContentTypestring     `json:"content_type"`// eg application/directoryBytesint64      `json:"bytes"`// size in bytesServerLastModifiedstring     `json:"last_modified"`// Last modified time, eg '2011-06-30T08:20:47.736680' as a string supplied by the serverLastModifiedtime.Time// Last modified time converted to a time.TimeHashstring     `json:"hash"`// MD5 hash, eg "d41d8cd98f00b204e9800998ecf8427e"SLOHashstring     `json:"slo_etag"`// MD5 hash of all segments' MD5 hash, eg "d41d8cd98f00b204e9800998ecf8427e"PseudoDirectorybool// Set when using delimiter to show that this directory object does not really existSubDirstring     `json:"subdir"`// returned only when using delimiter to mark "pseudo directories"ObjectTypeObjectType// type of this object}

Object contains information about an object

typeObjectCreateFile

type ObjectCreateFile struct {// contains filtered or unexported fields}

ObjectCreateFile represents a swift object open for writing

func (*ObjectCreateFile)Close

func (file *ObjectCreateFile) Close()error

Close the object and checks the md5sum if it was required.

Also returns any other errors from the server (eg container notfound) so it is very important to check the errors on this method.

func (*ObjectCreateFile)CloseWithError

func (file *ObjectCreateFile) CloseWithError(errerror)error

CloseWithError closes the object, aborting the upload.

func (*ObjectCreateFile)Headers

func (file *ObjectCreateFile) Headers() (Headers,error)

Headers returns the response headers from the created object if the uploadhas been completed. The Close() method must be called on an ObjectCreateFilebefore this method.

func (*ObjectCreateFile)Write

func (file *ObjectCreateFile) Write(p []byte) (nint, errerror)

Write bytes to the object - see io.Writer

typeObjectOpenFile

type ObjectOpenFile struct {// contains filtered or unexported fields}

ObjectOpenFile represents a swift object open for reading

func (*ObjectOpenFile)Close

func (file *ObjectOpenFile) Close() (errerror)

Close the object and checks the length and md5sum if it wasrequired and all the object was read

func (*ObjectOpenFile)Length

func (file *ObjectOpenFile) Length(ctxcontext.Context) (int64,error)

Length gets the objects content length either from a cached copy orfrom the server.

func (*ObjectOpenFile)Read

func (file *ObjectOpenFile) Read(p []byte) (nint, errerror)

Read bytes from the object - see io.Reader

func (*ObjectOpenFile)Seek

func (file *ObjectOpenFile) Seek(ctxcontext.Context, offsetint64, whenceint) (newPosint64, errerror)

Seek sets the offset for the next Read to offset, interpretedaccording to whence: 0 means relative to the origin of the file, 1means relative to the current offset, and 2 means relative to theend. Seek returns the new offset and an Error, if any.

Seek uses HTTP Range headers which, if the file pointer is moved,will involve reopening the HTTP connection.

Note that you can't seek to the end of a file or beyond; HTTP Rangerequests don't support the file pointer being outside the data,unlike os.File

Seek(0, 1) will return the current file pointer.

typeObjectType

type ObjectTypeint

ObjectType is the type of the swift object, regular, static large,or dynamic large.

const (RegularObjectTypeObjectType =iotaStaticLargeObjectTypeDynamicLargeObjectType)

Values that ObjectType can take

typeObjectsOpts

type ObjectsOpts struct {Limitint// For an integer value n, limits the number of results to at most n values.Markerstring// Given a string value x, return object names greater in value than the  specified marker.EndMarkerstring// Given a string value x, return object names less in value than the specified markerPrefixstring// For a string value x, causes the results to be limited to object names beginning with the substring x.Pathstring// For a string value x, return the object names nested in the pseudo pathDelimiterrune// For a character c, return all the object names nested in the containerHeadersHeaders// Any additional HTTP headers - can be nilKeepMarkerbool// Do not reset Marker when using ObjectsAll or ObjectNamesAll}

ObjectOpts is options for Objects() and ObjectNames()

typeObjectsWalkFn

type ObjectsWalkFn func(context.Context, *ObjectsOpts) (interface{},error)

A closure defined by the caller to iterate through all objects

Call Objects or ObjectNames from here with the context.Context and *ObjectOpts passed in

Do whatever is required with the results then return them

typeRequestOpts

type RequestOpts struct {ContainerstringObjectNamestringOperationstringParametersurl.ValuesHeadersHeadersErrorMap   errorMapNoResponseboolBodyio.ReaderRetriesint// if set this is called on re-authentication to refresh the targetUrlOnReAuth func() (string,error)}

RequestOpts contains parameters for Connection.storage.

typeStaticLargeObjectCreateFile

type StaticLargeObjectCreateFile struct {// contains filtered or unexported fields}

StaticLargeObjectCreateFile represents an open static large object

func (*StaticLargeObjectCreateFile)Close

func (*StaticLargeObjectCreateFile)CloseWithContext

func (file *StaticLargeObjectCreateFile) CloseWithContext(ctxcontext.Context)error

func (*StaticLargeObjectCreateFile)Flush

func (*StaticLargeObjectCreateFile)Seek

func (file *StaticLargeObjectCreateFile) Seek(offsetint64, whenceint) (int64,error)

Seek sets the offset for the next write operation

func (*StaticLargeObjectCreateFile)Size

func (file *StaticLargeObjectCreateFile) Size()int64

func (*StaticLargeObjectCreateFile)Write

func (file *StaticLargeObjectCreateFile) Write(buf []byte) (int,error)

func (*StaticLargeObjectCreateFile)WriteWithContext

func (file *StaticLargeObjectCreateFile) WriteWithContext(ctxcontext.Context, buf []byte) (int,error)

typeSwiftInfo

type SwiftInfo map[string]interface{}

SwiftInfo contains the JSON object returned by Swift when the /inforoute is queried. The object contains, among others, the Swift version,the enabled middlewares and their configuration

func (SwiftInfo)SLOMinSegmentSize

func (iSwiftInfo) SLOMinSegmentSize()int64

func (SwiftInfo)SupportsBulkDelete

func (iSwiftInfo) SupportsBulkDelete()bool

func (SwiftInfo)SupportsSLO

func (iSwiftInfo) SupportsSLO()bool

Source Files

View all Source files

Directories

PathSynopsis
This implements a very basic Swift server Everything is stored in memory
This implements a very basic Swift server Everything is stored in memory

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f orF : Jump to
y orY : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic.Learn more.

[8]ページ先頭

©2009-2025 Movatter.jp