Repository files navigation

• S3Mock
• Changelog
• Supported S3 operations
• Usage
  • Usage of AWS S3 SDKs
    • Path-style vs. Domain-style access
    • Presigned URLs
    • Self-signed SSL certificate
  • Usage of AWS CLI
    • Create bucket
    • Put object
    • Get object
    • Get object using HTTPS
  • Usage of plain HTTP / HTTPS with cURL
    • Create bucket
    • Put object
    • Get object
    • Get object using HTTPS
  • S3Mock configuration options
  • S3Mock Docker
    • Start using the command-line
    • Start using the Fabric8 Docker-Maven-Plugin
    • Start using Testcontainers
    • Start using Docker compose
      • Simple example
      • Expanded example
    • Start using a self-signed SSL certificate
  • S3Mock Java
    • Start using the JUnit4 Rule
    • Start using the JUnit5 Extension
    • Start using the TestNG Listener
    • Start programmatically
• File System Structure
  • Root-Folder
  • Buckets
  • Objects
    • Object versions
  • Multipart Uploads
• Build & Run
  • Build
    • With Docker
    • Without Docker
  • Run
    • Spring Application
    • Docker
    • Docker Maven plugin
  • Run integration tests
  • Java
  • Kotlin
• Governance model
• Vulnerability reports
• Security
• Contributing
• Licensing
• Powered by

S3Mock is a lightweight server that implements parts oftheAmazon S3 API.
It has been created to support local integration testing by reducing infrastructure dependencies.

TheS3Mock server can be started as a standaloneDocker container, usingTestcontainers,JUnit4,JUnit5 andTestNG support, or programmatically.

SeeGitHub releases.
See also thechangelog for detailed information about changes in releases and changes planned for futurereleases.

Oftheseoperations of the Amazon S3 API,all marked ✅ are supported by S3Mock:

S3Mock can be used with any of the available AWS S3 SDKs.

TheIntegration Tests and tests intestsupport contain various examples of how to use the S3Mock with the AWS SDK for Javav1 and v2 in Kotlin.

S3Client orS3Presigner instances are created here:

• Kotlin:Integration Test base class
• Java:S3Mock unit test starter base class

AWS S3 SDKs usually use domain-style access by default. Configuration is needed for path-style access.

S3Mock currently only supports path-style access (e.g.,http://localhost:9090/bucket/someKey).

Domain-style access to buckets (e.g.,http://bucket.localhost:9090/someKey) does not work because the domainlocalhost is special and does not allow for subdomain access withoutmodifications to the operating system.

S3 SDKs can be usedtocreate presigned URLs, the S3 APIsupports access through those URLs.

S3Mock will accept presigned URLs, but itignores all parameters.
For instance, S3Mock does not verify the HTTP verb that the presigned uri was created with, and it does not validatewhether the link is expired or not.

S3 SDKs can be used to create presigned URLs pointing to S3Mock if they're configured for path-style access. See the"Usage..." section above for links to examples on how to use the SDK with presigned URLs.

S3Mock supports connections via HTTP and HTTPS. It includes a self-signed SSL certificate which is rejected by most HTTPclients by default.To use HTTPS, the client must accept the self-signed certificate.

On command line, this can be done by setting the--no-verify-ssl option in the AWS CLI or by using the--insecureoption in cURL, see below.

Java and Kotlin SDKs can be configured to trust any SSL certificate, see links toS3Client creation above.

S3Mock can be used with the AWS CLI. Setting the--endpoint-url enables path-style access,--no-verify-ssl is neededfor HTTPS access.

Examples:

aws s3api create-bucket --bucket my-bucket --endpoint-url=http://localhost:9090

aws s3api put-object --bucket my-bucket --key my-file --body ./my-file --endpoint-url=http://localhost:9090

aws s3api get-object --bucket my-bucket --key my-file --endpoint-url=http://localhost:9090 my-file-output

aws s3api get-object --bucket my-bucket --key my-file --no-verify-ssl --endpoint-url=https://localhost:9191 my-file-output

As long as the requests work with the S3 API, they will work with S3Mock as well. Use--insecure to ignore SSL errors.

Examples:

curl --request PUT"http://localhost:9090/my-test-bucket/"

curl --request PUT --upload-file ./my-file http://localhost:9090/my-test-bucket/my-file

curl --request GET http://localhost:9090/my-test-bucket/my-file -O

curl --insecure --request GET https://localhost:9191/my-test-bucket/my-file -O

The mock can be configured with the following environment variables:

• COM_ADOBE_TESTING_S3MOCK_STORE_VALID_KMS_KEYS: list of KMS Key-Refs that are to be treated asvalid.
  • Legacy name:validKmsKeys
  • Default: none
  • KMS keys must be configured as valid ARNs in the format of "arn:aws:kms:region:acct-id:key/key-id", for example "arn:aws:kms:us-east-1:1234567890:key/valid-test-key-id"
  • The list must be comma separated keys likearn-1, arn-2
  • When requesting with KMS encryption, the key ID is passed to the SDK / CLI, in the example above this would be "valid-test-key-id".
  • S3Mock does not implement KMS encryption, if a key ID is passed in a request, S3Mock will just validate if a givenKey was configured during startup and reject the request if the given Key was not configured.
• COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS: list of names for buckets that will be available initially.
  • Legacy name:initialBuckets
  • Default: none
  • The list must be comma separated names likebucketa, bucketb
• COM_ADOBE_TESTING_S3MOCK_STORE_REGION: the region the S3Mock is supposed to mock.
  • Legacy name:COM_ADOBE_TESTING_S3MOCK_REGION
  • Example:eu-west-1
  • Default:us-east-1
• COM_ADOBE_TESTING_S3MOCK_STORE_ROOT: the base directory to place the temporary files exposed by the mock. If S3Mock is started in Docker, a volumemust be mounted as theroot directory, see examples below.
  • Legacy name:root
  • Default: Java temp directory
• COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT: set totrue to let S3Mock keep all files that were created during its lifetime. Default isfalse, all files are removed if S3Mock shuts down.
  • Legacy name:retainFilesOnExit
  • Default: false
• debug: set totrue toenableSpring Boot's debug output.
• trace: set totrue toenableSpring Boot's trace output.

TheS3Mock Docker container is the recommended way to useS3Mock.
It is released toDocker Hub.
The container is lightweight, built on top of the officialLinux Alpine image.

If needed,configurememoryandcpu limits for the S3Mock Docker container.

The JVM will automatically use half the available memory.

Starting on the command-line:

docker run -p 9090:9090 -p 9191:9191 -t adobe/s3mock

The port9090 is for HTTP, port9191 is for HTTPS.

Example with configuration via environment variables:

docker run -p 9090:9090 -p 9191:9191 -e COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=test -e debug=true -t adobe/s3mock

Ourintegration tests are using the Amazon S3 Client to verify the server functionality against theS3Mock. During the Maven build, the Docker image is started using thedocker-maven-plugin andthe corresponding ports are passed to the JUnit test through themaven-failsafe-plugin. SeeBucketITas an example on how it's used in the code.

This way, one can easily switch between calling the S3Mock or the real S3 endpoint, and this doesn't add any additionalJava dependencies to the project.

TheS3MockContaineris aTestcontainer implementation that comes pre-configured exposing HTTP and HTTPS ports. Environment variables canbe set on startup.

The exampleS3MockContainerJupiterTestdemonstrates the usage with JUnit 5. The exampleS3MockContainerManualTestdemonstrates the usage with plain Kotlin. Java will be similar.

Testcontainers provide integrations for JUnit 4, JUnit 5 and Spock.
For more information, visit theTestcontainers website.

To use theS3MockContainer,use the following Maven artifact intest scope:

<dependency>  <groupId>com.adobe.testing</groupId>  <artifactId>s3mock-testcontainers</artifactId>  <version>...</version>  <scope>test</scope></dependency>

Create a filedocker-compose.yml

services:s3mock:image:adobe/s3mock:latestenvironment:      -COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=bucket1ports:      -9090:9090

Start with

docker compose up -d

Stop with

docker compose down

Suppose we want to see what S3Mock is persisting and look at the logs it generates in detail.

A local directory is needed, let's call itlocals3root. This directory must be mounted as a volume into the Dockercontainer when it's started, and that mounted volume must then be configured as theroot for S3Mock. Let's call themounted volume inside the containercontainers3root. S3Mock will delete all files when it shuts down,COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT=true tells it to leave all files instead.

Also, to see debug logs,debug=true must be configured for S3Mock.

Create a filedocker-compose.yml

services:s3mock:image:adobe/s3mock:latestenvironment:      -debug=true      -COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT=true      -COM_ADOBE_TESTING_S3MOCK_STORE_ROOT=containers3rootports:      -9090:9090      -9191:9191volumes:      -./locals3root:/containers3root

Create a directorylocals3root.

Start with

docker compose up -d

Create a bucket "my-test-bucket" with

curl --request PUT"http://localhost:9090/my-test-bucket/"

Stop with

docker compose down

Look into the directorylocals3root where metadata and contents of the bucket are stored.

$ mkdir s3mock-mounttest$cd s3mock-mounttest$ mkdir locals3root$ cat docker-compose.ymlservices:  s3mock:    image: adobe/s3mock:latest    environment:      - debug=true      - COM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT=true      - COM_ADOBE_TESTING_S3MOCK_STORE_ROOT=containers3root    ports:      - 9090:9090      - 9191:9191    volumes:      - ./locals3root:/containers3root$ docker compose up -d[+] Running 2/2 ✔ Network s3mock-mounttest_default     Created ✔ Container s3mock-mounttest-s3mock-1  Started$ curl --request PUT"http://localhost:9090/my-test-bucket/"$ docker compose down[+] Running 2/0 ✔ Container s3mock-mounttest-s3mock-1  Removed ✔ Network s3mock-mounttest_default     Removed $ ls locals3rootmy-test-bucket$ ls locals3root/my-test-bucketbucketMetadata.json

S3Mock includes a self-signed SSL certificate:

$ curl -vvv --insecure --request GET https://localhost:9191/my-test-bucket/my-file -O[...]* Server certificate:*  subject: C=DE; ST=Hamburg; L=Hamburg; O=S3Mock; OU=S3Mock; CN=Adobe S3Mock*  start date: Jul 25 12:28:53 2022 GMT*  expire date: Nov 25 12:28:53 3021 GMT*  issuer: C=DE; ST=Hamburg; L=Hamburg; O=S3Mock; OU=S3Mock; CN=Adobe S3Mock*  SSL certificate verify result: self signed certificate (18), continuing anyway.[...]

To use a custom self-signed SSL certificate, derive your own Docker container from the S3Mock container:

FROM adobe/s3mock:4.2.0ENV server.ssl.key-store=/opt/customcert.jksENV server.ssl.key-store-password=passwordENV server.ssl.key-alias=selfsignedRUN keytool -genkey -keyalg RSA -alias selfsigned \  -validity 360 \  -keystore /opt/customcert.jks \  -dname"cn=Test, ou=Test, o=Docker, l=NY, st=NY, c=US" \  -storepass password -keysize 2048 \  -ext"san=dns:localhost"

$ curl -vvv --insecure --request GET https://localhost:9191/my-test-bucket/my-file -O[...]* Server certificate:*  subject: C=US; ST=NY; L=NY; O=Docker; OU=Test; CN=Test*  start date: May  9 14:33:40 2025 GMT*  expire date: May  4 14:33:40 2026 GMT*  issuer: C=US; ST=NY; L=NY; O=Docker; OU=Test; CN=Test*  SSL certificate verify result: self signed certificate (18), continuing anyway.[...]

S3Mock Java libraries are released and published to the Sonatype Maven Repository and subsequently published tothe officialMaven mirrors.

S3Mock is built using Spring Boot, if projects useS3Mock by adding the dependency to their project and startingtheS3Mock during a JUnit test, classpaths of the tested application and of theS3Mock are merged, leadingto unpredictable and undesired effects such as class conflicts or dependency version conflicts.
This is especially problematic if the tested application itself is a Spring (Boot) application, as both applicationswill load configurations based on the availability of certain classes in the classpath, leading to unpredictable runtimebehavior.

This is the opposite of what software engineers are trying to achieve when thoroughly testing code in continuousintegration...

S3Mock dependencies are updated regularly, any update could break any number of projects.
See alsoissues labeled "dependency-problem".

See alsothe Java section below

The exampleS3MockRuleTestdemonstrates the usage of theS3MockRule, which can be configured through abuilder.

To use the JUnit4 Rule, use the following Maven artifact intest scope:

<dependency>  <groupId>com.adobe.testing</groupId>  <artifactId>s3mock-junit4</artifactId>  <version>...</version>  <scope>test</scope></dependency>

TheS3MockExtension can currently be used in two ways:

1. Declaratively using@ExtendWith(S3MockExtension.class) and by injecting a properly configured instance ofAmazonS3 client and/or the startedS3MockApplication to the tests.See examples:S3MockExtensionDeclarativeTest (for SDKv1)orS3MockExtensionDeclarativeTest (for SDKv2)

2. Programmatically using@RegisterExtension and by creating and configuring theS3MockExtension using abuilder.See examples:S3MockExtensionProgrammaticTest (for SDKv1)orS3MockExtensionProgrammaticTest (for SDKv2)

To use the JUnit5 Extension, use the following Maven artifact intest scope:

<dependency>  <groupId>com.adobe.testing</groupId>  <artifactId>s3mock-junit5</artifactId>  <version>...</version>  <scope>test</scope></dependency>

The exampleS3MockListenerXMLConfigurationTestdemonstrates the usage of theS3MockListener, which can be configured as shown intestng.xml.The listener bootstraps the S3Mock application before TestNG execution starts and shuts down the application just before the execution terminates.Please refer toIExecutionListenerin the TestNG API.

To use the TestNG Listener, use the following Maven artifact intest scope:

<dependency>  <groupId>com.adobe.testing</groupId>  <artifactId>s3mock-testng</artifactId>  <version>...</version>  <scope>test</scope></dependency>

Include the following dependency and use one of thestart methods incom.adobe.testing.s3mock.S3MockApplication:

<dependency>  <groupId>com.adobe.testing</groupId>  <artifactId>s3mock</artifactId>  <version>...</version></dependency>

S3Mock stores Buckets, Objects, Parts and other data on the disk.
This lets users inspect the stored data while the S3Mock is running.
If the environment variableCOM_ADOBE_TESTING_S3MOCK_STORE_RETAIN_FILES_ON_EXIT is set totrue, this data will not be deleted when S3Mock is shut down.

S3Mock stores buckets and objects in a root-folder.

This folder is expected to be empty when S3Mock starts. See also FYI above.

/<root-folder>/

Buckets are stored as a folder with their name as created through the S3 API directly below the root:

/<root-folder>/<bucket-name>/

BucketMetadata is stored in a file in thebucket directory, serialized as JSON.
BucketMetadata contains the "key" -> "uuid"dictionary for all objects uploaded to this bucket among other data.

/<root-folder>/<bucket-name>/bucketMetadata.json

Objects are stored in folders below the bucket they were created in.A folder is created that uses the Object's UUID assigned intheBucketMetadata as a name.

/<root-folder>/<bucket-name>/<uuid>/

Object data is stored below that UUID folder.

Binary data is always stored in a filebinaryData

/<root-folder>/<bucket-name>/<uuid>/binaryData

Object metadata is serialized as JSON andstored asobjectMetadata.json

/<root-folder>/<bucket-name>/<uuid>/objectMetadata.json

If versioning is enabled for a bucket, multiple versions of an object can be stored in S3Mock. The files will be storedalongside each other, prefixed by the versionId.

/<root-folder>/<bucket-name>/<uuid>/<versionId>-binaryData/<root-folder>/<bucket-name>/<uuid>/<versionId>-objectMetadata.json

Multipart Uploads are created in a bucket using object keys and an uploadId.
The object is assigned a UUID within the bucket (storedinBucketMetadata).
TheMultipart upload metadata iscurrently not stored on disk.

The multiparts folder is created below the bucket folder and named with theuploadId:

/<root-folder>/<bucket-name>/multiparts/<uploadId>/

The multiparts metadata file is created below the folder named with theuploadId:

/<root-folder>/<bucket-name>/multiparts/<uploadId>/multipartMetadata.json

Each part is stored in the parts folder with thepartNo as name and.part as a suffix.

/<root-folder>/<bucket-name>/multiparts/<uploadId>/<partNo>.part

Once the MultipartUpload is completed, theuploadId folder is deleted.

To build this project, you need Docker, JDK 17 or higher, and Maven:

./mvnw clean install

If you want to skip the Docker build, pass the optional parameter "skipDocker":

./mvnw clean install -DskipDocker

You can run the S3Mock from the sources by either of the following methods:

Run or Debug the classcom.adobe.testing.s3mock.S3MockApplication in the IDE.

Use Docker:

./mvnw clean package -pl server -am -DskipTestsdocker run -p 9090:9090 -p 9191:9191 -t adobe/s3mock:latest

Use the Docker Maven plugin:

./mvnw clean package docker:start -pl server -am -DskipTests -Ddocker.follow -Dit.s3mock.port_http=9090 -Dit.s3mock.port_https=9191

(stop withctrl-c)

Once the application is started using default ports, you can execute the*IT tests from your IDE.

This repo is built with Java 17, output iscurrently bytecode compatible with Java 17.

TheIntegration Tests and unit tests are built in Kotlin.

The project owner and leads make all final decisions. See thedevelopers section in thepom.xml for a listof leads.

S3Mock uses GitHub actions to produce an SBOM and to check dependencies for vulnerabilities. All vulnerabilities areevaluated and fixed if possible.Vulnerabilities may also be reported through the GitHub issue tracker.

S3Mock is not intended to be used in production environments. It is a mock server meant to be used indevelopment and testing environments only. It does not implement all security features of AWS S3 and should not be usedas a replacement for AWS S3 in production.It is implemented usingSpring Boot, which is a Java framework that isdesigned to be secure by default.

Contributions are welcome! Read theContributing Guide for more information.

This project is licensed under the Apache V2 License. SeeLICENSE for more information.