- Notifications
You must be signed in to change notification settings - Fork66
An S3 File System Provider for Java 7
License
Upplication/Amazon-S3-FileSystem-NIO2
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
AnAmazon AWS S3 FileSystem ProviderJSR-203 for Java 7 (NIO2)
Amazon Simple Storage Service provides a fully redundant data storage infrastructure for storing and retrieving any amount of data, at any time.NIO2 is the new file management API, introduced in Java version 7.This project provides a first API implementation, little optimized, but "complete" to manage files and folders directly on Amazon S3.
#### How to use
<dependency><groupId>com.upplication</groupId><artifactId>s3fs</artifactId><version>2.2.2</version></dependency>
And add to your META-INF/services/java.nio.file.spi.FileSystemProvider (create if not exists yet) a new line like this: com.upplication.s3fs.S3FileSystemProvider.
All settings for S3FileSystem and for the underlying AmazonS3 connector library can be set through System properties or environment variables.Possible settings can be found in com.upplication.s3fs.AmazonS3Factory.
#### Using service locator and system vars
Check that s3fs_access_key and s3fs_secret_key system vars are present with the correct values to have full access to your amazon s3 bucket.
Use this code to create the fileSystem and set to a concrete endpoint.
FileSystems.newFileSystem("s3:///",newHashMap<String,Object>(),Thread.currentThread().getContextClassLoader());
Add to your resources folder the file amazon.properties with the content:s3fs_access_key=access-keys3fs_secret_key=secret-key
Use this code to create the fileSystem and set to a concrete endpoint.
FileSystems.newFileSystem("s3:///",newHashMap<String,Object>(),Thread.currentThread().getContextClassLoader());
Create a map with the authentication and use the fileSystem to create the fileSystem and set to a concrete endpoint.
Map<String, ?>env =ImmutableMap.<String,Object>builder().put(com.upplication.s3fs.AmazonS3Factory.ACCESS_KEY,"access key").put(com.upplication.s3fs.AmazonS3Factory.SECRET_KEY,"secret key").build()FileSystems.newFileSystem("s3:///",env,Thread.currentThread().getContextClassLoader());
Complete settings lists:
- s3fs_access_key
- s3fs_secret_key
- s3fs_request_metric_collector_class
- s3fs_connection_timeout
- s3fs_max_connections
- s3fs_max_retry_error
- s3fs_protocol
- s3fs_proxy_domain
- s3fs_proxy_host
- s3fs_proxy_password
- s3fs_proxy_port
- s3fs_proxy_username
- s3fs_proxy_workstation
- s3fs_socket_send_buffer_size_hint
- s3fs_socket_receive_buffer_size_hint
- s3fs_socket_timeout
- s3fs_user_agent
- s3fs_amazon_s3_factory
- s3fs_signer_override
- s3fs_path_style_access
// Northern Virginia or Pacific NorthwestFileSystems.newFileSystem("s3://s3.amazonaws.com/",env,Thread.currentThread().getContextClassLoader());// Northern Virginia onlyFileSystems.newFileSystem("s3://s3-external-1.amazonaws.com/",env,Thread.currentThread().getContextClassLoader());// US West (Oregon) RegionFileSystems.newFileSystem("s3://s3-us-west-2.amazonaws.com/",env,Thread.currentThread().getContextClassLoader());// US West (Northern California) RegionFileSystems.newFileSystem("s3://s3-us-west-1.amazonaws.com/",env,Thread.currentThread().getContextClassLoader());// EU (Ireland) RegionFileSystems.newFileSystem("s3://s3-eu-west-1.amazonaws.com/",env,Thread.currentThread().getContextClassLoader());
For a complete list of available regions look at:http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
publicFileSystemFactorycreateFileSystemFactory(StringbucketName)throwsIOException,URISyntaxException {FileSystemfileSystem =FileSystems.newFileSystem(newURI("s3:///"),env,Thread.currentThread().getContextClassLoader());StringbucketPath =fileSystem.getPath("/" +bucketName);returnnewVirtualFileSystemFactory(bucketPath);}
Add to classpath and configure:
@ConfigurationpublicclassAwsConfig {@Value("${upplication.aws.accessKey}")privateStringaccessKey;@Value("${upplication.aws.secretKey}")privateStringsecretKey;@BeanpublicFileSystems3FileSystem()throwsIOException {Map<String,String>env =newHashMap<>();env.put(com.upplication.s3fs.AmazonS3Factory.ACCESS_KEY,accessKey);env.put(com.upplication.s3fs.AmazonS3Factory.SECRET_KEY,secretKey);returnFileSystems.newFileSystem(URI.create("s3:///"),env,Thread.currentThread().getContextClassLoader()); }}
Now you can inject in any spring component:
@AutowiredprivateFileSystems3FileSystem;
- Preserve URI with end slash #76
- Removed META-INF/services/java.nio.file.spi.FileTypeDetector #78
- Bucket are filestores and root directories for a bucket is the bucket itself.
- getFileName for a root Path is
null - Added new method toURL to S3Path and can be customized with the property s3fs_path_style_access #83
- Improved S3Path Tests
#### Features:
- Copy and create folders and files
- Delete folders and files
- Copy paths between different providers
- Walk file tree
- Works with virtual s3 folders (not really exists and are element's subkeys)
- List buckets for the client
- Multi endpoint fileSystem
- Performance issue (slow querys with virtual folders, add multipart submit...)
- Disallow upload binary files with same name as folders and vice versa
#### Out of Roadmap:
- Watchers
Clone the github repository:
gitclonehttps://github.com/Upplication/Amazon-S3-FileSystem-NIO2.git
To run the tests:
First, you must copy the filesrc/test/resources/amazon-test-sample.properties and paste in the same directory with the name amazon-test.properties. In your copy you must edit all the keys:
bucket_name=/your-bucket-name for test# http://docs.aws.amazon.com/general/latest/gr/rande.html s3fs_secret_key= your secret key for tests3fs_access_key=your access key for testThats all, now you can run the test with the command:mvn test ormvn integration-test -Pintegration-tests
#### LICENSE:
Amazon S3 FileSystem NIO2 is released under the MIT License.
About
An S3 File System Provider for Java 7
Topics
Resources
License
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Packages0
Uh oh!
There was an error while loading.Please reload this page.
Languages
- Java99.9%
- Shell0.1%