ADLS Gen2 Storage SDK for Go

Service Version: 2021-06-08

Azure Data Lake Storage Gen2 (ADLS Gen2) is Microsoft's hierarchical object storage solution for the cloud with converged capabilities with Azure Blob Storage.For example, Data Lake Storage Gen2 provides file system semantics, file-level security, and scale.Because these capabilities are built on Blob storage, you also get low-cost, tiered storage, with high availability/disaster recovery capabilities.ADLS Gen2 makes Azure Storage the foundation for building enterprise data lakes on Azure.Designed from the start to service multiple petabytes of information while sustaining hundreds of gigabits of throughput, ADLS Gen2 allows you to easily manage massive amounts of data.

Source code |API reference documentation |REST API documentation

Getting started

Install the package

Install the ADLS Gen2 Storage SDK for Go withgo get:

go get github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake

If you're going to authenticate with Azure Active Directory (recommended), install theazidentity module.

go get github.com/Azure/azure-sdk-for-go/sdk/azidentity

Prerequisites

A supportedGo version (the Azure SDK supports the two most recent Go releases).

You need anAzure subscription and aStorage Account to use this package.

To create a new Storage Account, you can use theAzure Portal,Azure PowerShell, or theAzure CLI.Here's an example using the Azure CLI:

az storage account create --name MyStorageAccount --resource-group MyResourceGroup --location westus --sku Standard_LRS

Authenticate the client

In order to interact with the ADLS Gen2 Storage service, you'll need to create an instance of theClient type. Theazidentity module makes it easy to add Azure Active Directory support for authenticating Azure SDK clients with their corresponding Azure services.

// create a credential for authenticating with Azure Active Directorycred, err := azidentity.NewDefaultAzureCredential(nil)// TODO: handle err// create a service.Client for the specified storage account that uses the above credentialclient, err := service.NewClient("https://MYSTORAGEACCOUNT.dfs.core.windows.net/", cred, nil)// TODO: handle err// you can also create filesystem, file and directory clients

Learn more about enabling Azure Active Directory for authentication with Azure Storage inour documentation andour samples.

Key concepts

ADLS Gen2 provides:

• Hadoop-compatible access
• Hierarchical directory structure
• Optimized cost and performance
• Finer grain security model
• Massive scalability

ADLS Gen2 storage is designed for:

• Serving images or documents directly to a browser.
• Storing files for distributed access.
• Streaming video and audio.
• Writing to log files.
• Storing data for backup and restore, disaster recovery, and archiving.
• Storing data for analysis by an on-premises or Azure-hosted service.

ADLS Gen2 storage offers three types of resources:

• Thestorage account
• One or morefilesystems in a storage account
• One or morefiles ordirectories in a filesystem

Instances of theClient type provide methods for manipulating filesystems and paths within a storage account.The storage account is specified when theClient is constructed. The clients available are referenced below.Use the appropriate client constructor function for the authentication mechanism you wish to use.

Goroutine safety

We guarantee that all client instance methods are goroutine-safe and independent of each other (guideline). This ensures that the recommendation of reusing client instances is always safe, even across goroutines.

About metadata

ADLS Gen2 metadata name/value pairs are valid HTTP headers and should adhere to all restrictions governing HTTP headers. Metadata names must be valid HTTP header names, may contain only ASCII characters, and should be treated as case-insensitive. Base64-encode or URL-encode metadata values containing non-ASCII characters.

Additional concepts

Client options |Accessing the response |Handling failures |Logging

Examples

Creating and uploading a file (assuming filesystem exists)

const (path = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs/sample-file")// authenticate with Azure Active Directorycred, err := azidentity.NewDefaultAzureCredential(nil)// TODO: handle error// create a client for the specified storage accountclient, err := file.NewClient(path, cred, nil)// TODO: handle error_, err = client.Create(context.TODO(), nil)// TODO: handle error// open the file for readingfh, err := os.OpenFile(sampleFile, os.O_RDONLY, 0)// TODO: handle errordefer fh.Close()// upload the file to the specified filesystem with the specified file name_, err = client.UploadFile(context.TODO(), fh, nil)// TODO: handle error

Downloading a file

const (    path = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs/cloud.jpg")// authenticate with Azure Active Directorycred, err := azidentity.NewDefaultAzureCredential(nil)// TODO: handle error// create a client for the specified storage accountclient, err := file.NewClient(path, cred, nil)// TODO: handle error// create or open a local file where we can download the filefile, err := os.Create("cloud.jpg")// TODO: handle errordefer file.Close()// download the file_, err = client.DownloadFile(context.TODO(), file, nil)// TODO: handle error

Creating and deleting a filesystem

const (fs = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs")// authenticate with Azure Active Directorycred, err := azidentity.NewDefaultAzureCredential(nil)// TODO: handle error// create a client for the specified storage accountclient, err := filesystem.NewClient(fs, cred, nil)// TODO: handle error_, err = client.Create(context.TODO(), nil)// TODO: handle error_, err = client.Delete(context.TODO(), nil)// TODO: handle error

Enumerating paths (assuming filesystem exists)

const (fs = "https://MYSTORAGEACCOUNT.dfs.core.windows.net/sample-fs")// authenticate with Azure Active Directorycred, err := azidentity.NewDefaultAzureCredential(nil)// TODO: handle error// create a filesystem client for the specified storage accountclient, err := filesystem.NewClient(fs, cred, nil)// TODO: handle error// path listings are returned across multiple pagespager := client.NewListPathsPager(true, nil)// continue fetching pages until no more remainfor pager.More() {  // advance to the next pagepage, err := pager.NextPage(context.TODO())// TODO: handle error// print the path names for this pagefor _, path := range page.PathList.Paths {fmt.Println(*path.Name)        fmt.Println(*path.IsDirectory)}}

Troubleshooting

All Datalake service operations will return an*azcore.ResponseError on failure with apopulatedErrorCode field. Many of these errors are recoverable.Thedatalakeerror package provides the possible Storage error codesalong with various helper facilities for error handling.

Specialized clients

The ADLS Gen2 Storage SDK for Go provides specialized clients in various subpackages.

Thefile package contains APIs related to file path types.

Thedirectory package contains APIs related to directory path types.

Thelease package contains clients for managing leases on paths (paths represent both directory and file paths) and filesystems. Please see thereference docs for general information on leases.

Thefilesystem package contains APIs specific to filesystems. This includes APIs setting access policies or properties, and more.

Theservice package contains APIs specific to Datalake service. This includes APIs for manipulating filesystems, retrieving account information, and more.

Thesas package contains utilities to aid in the creation and manipulation of Shared Access Signature tokens.See the package's documentation for more information.

You can find additional context and examples in our samples for each subpackage (named examples_test.go).

Contributing

See theStorage CONTRIBUTING.md for details on building,testing, and contributing to this library.

This project welcomes contributions and suggestions. Most contributions requireyou to agree to a Contributor License Agreement (CLA) declaring that you havethe right to, and actually do, grant us the rights to use your contribution. Fordetails, visitcla.microsoft.com.

This project has adopted theMicrosoft Open Source Code of Conduct.For more information see theCode of Conduct FAQor contactopencode@microsoft.com with anyadditional questions or comments.

Constants¶

const (// EventUpload is used for logging events related to upload operation.EventUpload =exported.EventUpload// EventError is used for logging errors.EventError =exported.EventError)

Variables¶

Functions¶

Types¶

type DurationType =lease.DurationType

const (DurationTypeInfiniteDurationType =lease.DurationTypeInfiniteDurationTypeFixedDurationType =lease.DurationTypeFixed)

func PossibleDurationTypeValues() []DurationType

type HTTPRange =exported.HTTPRange

type SharedKeyCredential =exported.SharedKeyCredential

func NewSharedKeyCredential(accountName, accountKeystring) (*SharedKeyCredential,error)

type StateType =lease.StateType

const (StateTypeAvailableStateType =lease.StateTypeAvailableStateTypeLeasedStateType =lease.StateTypeLeasedStateTypeExpiredStateType =lease.StateTypeExpiredStateTypeBreakingStateType =lease.StateTypeBreakingStateTypeBrokenStateType =lease.StateTypeBroken)

func PossibleStateTypeValues() []StateType

type StatusType =lease.StatusType

const (StatusTypeLockedStatusType =lease.StatusTypeLockedStatusTypeUnlockedStatusType =lease.StatusTypeUnlocked)

func PossibleStatusTypeValues() []StatusType

type URLParts =sas.URLParts

func ParseURL(ustring) (URLParts,error)