Repository files navigation

Welcome to AWS for Fluent Bit! Before using this Docker Image, please read this README entirely,especially the section onConsuming AWS for Fluent Bit versions 🫡

• Consuming AWS for Fluent Bit versions
  • AWS Distro for Fluent Bit Release Tags
  • AWS Distro for Fluent Bit release testing
  • Latest stable version
  • CVE scans and latest stable
  • Guidance on consuming versions
• AWS Distro versioning scheme FAQ
• Compliance and Patching
• Debugging Guide
• Use Case Guide
• Public Images
  • arm64 and x86 images
  • Using the init tag
  • Using SSM to find available versions and aws regions
  • Using SSM Parameters in CloudFormation Templates
  • Using image tags
    • Amazon ECR Public Gallery
    • Docker Hub
    • Amazon ECR
  • Using the debug images
• Plugins
• Using the AWS Plugins outside of a container
• Running aws-for-fluent-bit Windows containers
• Development
  • Local build
  • Local integ testing
  • Developing Features in the AWS Plugins
• Fluent Bit Examples
• License

🔥⚠️WARNING⚠️🔥: Please read and understand the following information on how to consume AWS for Fluent Bit. Failure to do so may cause outages to your production environment. 😭💔

Our image repos contain the following types of tags, which are explained in the sections below:

• latest: The most recently released image version. 🔥😵We do not recommend deploying this to production environments ever, seeGuidance on consuming versions.
• Version number tag: Each release has a version number, for example2.28.4.These are the only tags we recommend ✅😍 consuming in production environments:Guidance on consuming versions.
• stable: Some time after a version is released, it may be designated as the latest stable. SeeLatest stable version andGuidance on consuming versions.

Types of tests we run

• Simple integration tests: Short running tests of the AWS output plugins that send log records and verify that all of them were received correctly formatted at the destination.
• Load Tests: Test Fluent Bit AWS output plugins at various throughputs and check for log loss, the results are posted in our release notes:https://github.com/aws/aws-for-fluent-bit/releases
• Long running stability tests: Highly parallel tests run in Amazon ECS for the AWS output plugins using theaws/firelens-datajet project. These tests simulate real Fluent Bit deployments and use cases to test for bugs that crashes.

Latest release testing bar

• Simple integration tests: Must fully pass with all log events received properly formatted at the destination.
• Load Tests: Must pass thethresholds here. Results are posted in our release notes:https://github.com/aws/aws-for-fluent-bit/releases
• Long running stability tests: No crashes observed for at least 1 day.

CVE Patch release testing bar

• Simple integration tests: Must fully pass with all log events received properly formatted at the destination.
• Load Tests: Must pass thethresholds here. Results are posted in our release notes:https://github.com/aws/aws-for-fluent-bit/releases

We do not run our long running stability tests for CVE patches. This is because the goal is to get the CVE patch out as quickly as possible, and because CVE patch releases never include Fluent Bit code changes. CVE patch releases only include base image dependency upgrades.If there is ever a CVE in the Fluent Bit code base itself, the patch for it would be considered a bug fix that might introduce instability and it would undergo the normal latest release testing.

Latest stable release testing bar

For a version to be made the lateststable, it must already have been previously released as the latest release. Thus it will have already passed the testing bar noted above forlatest.

In addition, our stable release undergoes additional testing:

• Long running stability tests: The version undergoes and passes these tests for at least 2 weeks. After the version is promoted to stable we continue to run the long running stability tests, and may roll back the stable designation if issues later surface.

Our latest stable version is the most recent version that we have high confidence is stable for AWS use cases.We recommend using the stable version number in your prod deployments butnot the stable tag itself; seeGuidance on consuming versions

The latest stable version is marked with the tagstable/windowsservercore-stable. The version number that is currently designated as the latest stable can always be found in theAWS_FOR_FLUENT_BIT_STABLE_VERSION file in the root of this repo.

❗There is no guarantee thatstable has no issues- stable simply has a higher testing bar than our latest releases. Thestable tag can be downgraded and rolled back to the previous stable if new test results or customer bug reports surface issues. This has occurredin the past*.Consequently, we recommend locking to aspecific version tag and informing your choice of version using our current stable designation.

Prior to being designated as the latest stable, a version must pass the following criteria:

• It has been out for at least 2 weeks or is a CVE patch with no Fluent Bit changes. Stable designation is based on the Fluent Bit code in the image. A version released for CVE patches can be made stable if the underlying if the underlying Fluent Bit code is already designated as stable.
• No bugs have been reported in Fluent Bit which we expect will have high impact for AWS customers. This means bugs in the components that are most frequently used by AWS customers, such as the AWS outputs or the tail input.
• The version has passed our long running stability tests for at least 2 weeks. The version would have already passed our simple integration and load tests when it was first released as the latest image.

Please read our CVE patching policy.

The stable designation is for the Fluent Bit code contents of the image, not CVE scan results for dependencies installed in the image. We will upgrade a CVE patch to be the latest stable if it contains no Fluent Bit code changes compared to the previous latest stable.

Ourrelease notes call out the key AWS changes in each new version.

We recommend that you only consume non-stable releases in your test/pre-prod stages. Consuming thelatest tag directly is widely considered to be an anti-pattern in the software industry.

⚡We strongly recommend that you always lock deployments to a specific immutable version tag, rather than using ourstable orlatest tags. We recommend you to conduct a gradual rollout of each new version consistent with your deployment rollout strategy as you would for any other code or dependency being deployed: i.e. first to non-production environments first then gradually to your production environments.

Using thestable orlatest tag directly in prod has the following downsides: 🤕

1. 😕Difficulty in determining which version was deployed: If you experience an issue, you will need tocheck the Fluent Bit log output to determine which specific version tag was deployed. This is because thestable andlatest tags are mutable and change over time.
2. 😐Mixed deployments: If you are in the middle of a deployment when we release an update to thestable orlatest immutable tags, some of your deployment may have deployed the previous version, and the rest will deploy the new version.
3. 🤢Difficulty in rolling back: While we take every effort to avoid releasing regressions, there is always a chance a bug might slip out. Explicitly consuming a version helps make it easier to rollback since there would be an existing deployment configuration to rollback to.

The best practice for consuming AWS for Fluent Bit is to check theAWS_FOR_FLUENT_BIT_STABLE_VERSION file and lock your prod deployments to that specific version tag. For example, if the current stable is2.28.4, your deployment should usepublic.ecr.aws/aws-observability/aws-for-fluent-bit:2.28.4 notpublic.ecr.aws/aws-observability/aws-for-fluent-bit:stable.

The version of the AWS for Fluent Bit image is not linked to the version of Fluent Bit which it contains.

What does the version number signify?

We use the standardmajor.minor.patch versioning scheme for our image, AKA Semantic Versioning. The initial release with this versioning scheme is2.0.0. Bug fixes are released in patch version bumps. New features are released in new minor versions. We strive to only release backwards incompatible changes in new major versions.

Please read the below on CVE patches in base images and dependencies. The semantic version number applies to the Fluent Bit code andAWS Go plugin code compiled and installed in the image.

Image Versions and CVE Patches

The AWS for Fluent Bit image includes the following contents:

• A base image (currently Amazon Linux or Windows Server Core 2019 or Windows Server Core 2022)
• Runtime dependencies installed on top of the base image
• Fluent Bit binary
• Several Fluent BitGo Plugin binaries

The process for pushing out new builds with CVE patches in the base image or installed dependencies is different for Windows vs Linux.

For Windows, every month after theB release date/"patch tuesday", we re-build and update all Windows images currently found in thewindows.versions file in this repo with the newest base images from Microsoft. The Fluent Bit and go plugin binaries are copied into the newly released base windows image. Thus, the windows image tags are not immutable images; only the Fluent Bit and Go plugin binaries are immutable over time.

For Linux, each image tag is immutable. When there is a report of high or critical CVEs reported in the base amazon linux image or installed linux packages, we will work to push out a new imageper our patching policy. However, we will not increment the semantic version number to simply re-build to pull in new linux dependencies. Instead, we will add a 4th version number signifying the date the image was built.

For example, a series of releases in time might look like:

1. 2.31.12: New Patch release with changes in Fluent Bit code compared to2.31.11. This release will have standard release notes and will have images for both linux and windows.
2. 2.31.12-20230629: Re-build of2.31.12 just for Linux CVEs found in the base image or installed dependencies. The Fluent Bit code contents are the same as2.31.12. There only be linux images with this version tag, and no windows images. Thelatest tag for linux will be updated to point to this new image. There will be short release notes that call out it is simply a re-build for linux.
3. 2.31.12-20230711: Another re-build of2.31.12 for Linux CVEs on a subsequent date. This release is special as explained above in the way same as2.31.12-20230629.
4. 2.31.13: New Patch release with changes in Fluent Bit code compared to2.31.12. This might be for bugs found in the Fluent Bit code. It could also be for a CVE found in the Fluent Bit code. This release has standard release notes and linux and windows images.

Why do some image tags contain 4 version numbers?

Please see the above explanation on our Linux image re-build process for CVEs found in dependencies.

Are there edge cases to the rules on breaking backwards compatibility?

One edge case for the above semantic versioning rules is changes to configuration validation. Between Fluent Bit upstream versions 1.8 and 1.9, validation of config options was fixed/improved. Previous to this distro's upgrade to Fluent Bit upstream 1.9, configurations that included certain invalid options would run without error (the invalid options were ignored). After we released Fluent Bit usptream 1.9 support, these invalid options were validated and Fluent Bit would exit with an error. See theissue discussion here.

Another edge case to the above rules are bug fixes that require removing a change. We have and will continue to occasionally remove new changes in a patch version if they were found to be buggy. We do this to unblock customers who do not depend on the recent change. Please always check our release notes for the changes in a specific version. A past example of a patch release that removed something is2.31.4. A prior release had fixed how S3 handles the timestamps in S3 keys and theRetry_Limit configuration option. Those changes were considered to be bug fixes. However, they introduced instability so we subsequently removed them in a patch.

What about the 1.x image tags in your repositories?

The AWS for Fluent Bit image was launched in July 2019. Between July and October of 2019 we simply versioned the image based on the version of Fluent Bit that it contained. During this time we released1.2.0,1.2.2 and1.3.2.

The old versioning scheme was simple and it made it clear which version of Fluent Bit our image contained. However, it had a serious problem- how could we signify that we had changed the other parts of the image? If we did not update Fluent Bit, but updated one of the plugins, how would we signify this in a new release? There was no answer- we could only release an update when Fluent Bit released a new version. We ultimately realized this was unacceptable- bug fixes or new features in our plugins should not be tied to the Fluent Bit release cadence.

Thus, we moved to the a new versioning scheme. Because customers already are relying on the1.x tags, we have left them in our repositories. The first version with the new scheme is2.0.0. From now on we will follow semantic versioning- but the move from1.3.2 did not follow semantic versioning. There are no backwards incompatible changes betweenaws-for-fluent-bit:1.3.2 andaws-for-fluent-bit:2.0.0. Our release notes for2.0.0 clearly explain the change.

Does this mean you are diverging from fluent/fluent-bit?

No. We continue to consume Fluent Bit from its main repository. We are not forking Fluent Bit.

Q: Is AWS for Fluent Bit HIPAA Compliant?

Fluent Bit can be used in a HIPAA compliant matter to send logs to AWS, even if the logs contain PHI. Please see the call outs in theAWS HIPAA white paper for ECS.

Q: What is the policy for patching AWS for Fluent Bit for vulnerabilities, CVEs and image scan findings?

AWS for Fluent Bit uses ECR image scanning in its release pipeline and any scan that finds high or critical vulnerabilities will block a release:scripts/publish.sh

If you find an issue from a scan on our latest images please follow the reporting guidelines below and we will work quickly to introduce a new release. To be clear, we do not patch existing images, we just will release a new image without the issue. The team usesAmazon ECR Basic image scanning andAmazon ECR Enhanced scanning powered by AWS Inspector as the primary source of truth for whether or not the image contains a vulnerability in a dependency.

If your concern is about a vulnerability in the Fluent Bit upstream (github.com/fluent/fluent-bit open source code), please let us know as well. However, fixing upstream issues requires additional work and time because we must work closely with upstream maintainers to commit a fix and cut an upstream release, and then we can cut an AWS for Fluent Bit release.

Q: How do I report security disclosures?

If you think you’ve found a potentially sensitive security issue, please do not post it in the Issues on GitHub. Instead, please follow the instructionshere or email AWS security directly ataws-security@amazon.com.

Please read the debugging.md

A set of tutorials on use cases that Fluent Bit can solve.

Each release updates thelatest tag and adds a tag for the version of the image. Thestable tag is also available which marks a release as the latest stable version.

For Windows images, we update thewindowsservercore-latest tag and add a tag as<VERSION>-windowsservercore. The stable tag is available aswindowsservercore-stable. We update all the supported versions each month whenMicrosoft releases the latest securitypatches for Windows.

Note: Deployinglatest/windowsservercore-latest to prod without going through a test stage first is not recommended.

AWS for Fluent Bit currently distributes container images for arm64 and amd64 CPU architectures. Our images all usemutli-archictecture tags. For example, this means that if you pull thelatest tag on a Graviton instance, you would get the arm64 image build.

For Windows,we release images only for amd64 CPU architecture of the following Windows releases-

• Windows Server 2019
• Windows Server 2022

Theinit tags indicate that an image contains init process and supports multi-config. Init tag is used in addition to our other tags, e.g.aws-for-fluent-bit:init-latest means this is a latest released image supports multi-config. For more information about the usage of multi-config please see ouruse case guide andFireLens example.

Note: Windows images with init tag are not available at the moment.

As of 2.0.0, there are SSM Public Parameters which allow you to see available versions. These parameters are available in every region that the image is available in. Any AWS account can query these parameters.

To see a list of available version tags, run the following command:

aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/ --query 'Parameters[*].Name'

Example output:

[    "/aws/service/aws-for-fluent-bit/latest"    "/aws/service/aws-for-fluent-bit/windowsservercore-latest"    "/aws/service/aws-for-fluent-bit/2.0.0"    "/aws/service/aws-for-fluent-bit/2.0.0-windowsservercore"]

If there is no output, it means the aws for fluent bit image is not available in current region.

To see the ECR repository ID for a given image tag, run the following:

$ aws ssm get-parameter --name /aws/service/aws-for-fluent-bit/2.0.0{    "Parameter": {        "Name": "/aws/service/aws-for-fluent-bit/2.0.0",        "Type": "String",        "Value": "906394416424.dkr.ecr.us-east-1.amazonaws.com/aws-for-fluent-bit:2.0.0",        "Version": 1,        "LastModifiedDate": 1539908129.759,        "ARN": "arn:aws:ssm:us-west-2::parameter/aws/service/aws-for-fluent-bit/2.0.0"    }}

You can use these SSM Parameters as parameters in your CloudFormation templates.

Parameters:  FireLensImage:    Description: Fluent Bit image for the FireLens Container    Type: AWS::SSM::Parameter::Value<String>    Default: /aws/service/aws-for-fluent-bit/latest

You should lock your deployments to a specific version tag. We guarantee that these tags will be immutable- once they are released the will not change.Windows images will be updated each month to include the latest security patches in the base layers but the contents of the image will not change in a tag.

aws-for-fluent-bit

Our images are available in Amazon ECR Public Gallery. We recommend our customers to download images from this public repo. You can get images with different tags by following command:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:<tag>

For example, you can pull the image with latest version by:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest

If you see errors for image pull limits, or get the following error:

Error response from daemon: pull access denied for public.ecr.aws/amazonlinux/amazonlinux, repository does not exist or may require 'docker login': denied: Your authorization token has expired. Reauthenticate and try again.

Then try log into public ECR with your AWS credentials:

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws

You can check theAmazon ECR Public official doc for more details.

amazon/aws-for-fluent-bit

We also provide images in Amazon ECR for high availability. These images are available in almost every AWS region, included AWS Gov Cloud.

The official way to find the ECR image URIs for your region is to use the SSM Parameters. In your region, run the following command:

aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/

Deploying AWS for Fluent Bit debug images can help the AWS team troubleshoot an issue. If you experience a bug, especially acrash/SIGSEGV issue, then please consider deploying the debug version of the image. After a crash, the debug image can print out a stacktrace and upload a core dump to S3. See ourdebugging guide for more info on using debug images.

For debug images, we update thedebug-latest tag and add a tag asdebug-<Version>.

We currently bundle the following projects in this image:

• amazon-kinesis-firehose-for-fluent-bit
• amazon-cloudwatch-logs-for-fluent-bit
• amazon-kinesis-streams-for-fluent-bit

You can use the AWS Fluent Bit plugins withtd-agent-bit.

We provide atutorial on using SSM to configure instances with td-agent-bit and the plugins.

You can runaws-for-fluent-bit Windows containers using the image tags as specified underWindows Images section. These are distributed as multi-arch images with the manifests for the supported Windows releases as specified above.

For more details about running Fluent Bit Windows containers in Amazon EKS, please visit ourblog post.

For more details about running Fluent Bit Windows containers in Amazon ECS, please visit ourblog post. For running Fluent Bit as a Amazon ECS Service usingdaemon scheduling strategy, please visit our Amazon ECStutorial. For more details about using the AWS provided default configurations for Amazon ECS, please visit ourdocumentation.

Note: There is a known issue with networking failure when running Fluent Bit in Windows containers ondefault container network. Check out the guidance in our debugging guide for aworkaround to this issue.

AWS for Fluent Bit can be built locally using the following commands:

• make release: Builds the image with the--no-cache option, ensuring a clean build
• make dev: Builds the image with Docker caching enabled for faster development iterations

You can customize which Amazon Linux base image version is used in the Docker builds. This is useful for testing with different Amazon Linux versions or when you need to match a specific base image version.

The default value is:

• AL_TAG: "2" (Amazon Linux 2)

There are two ways to customize this value:

Method 1: Using environment variables

Set the environment variable when running the make command:

# Use a specific Amazon Linux 2 versionAL_TAG="2.0.20250623.0" make release# For development buildsAL_TAG="2.0.20250623.0" make dev

Method 2: Updating the Makefile directly

You can also modify the default value in the Makefile:

1. Open the Makefile in your editor
2. Locate this line (near the top):

   # Amazon Linux Tag to use for images, will use value if not setAL_TAG ?= "2"

3. Update the value as needed (e.g., change"2" to"2.0.20250623.0")
4. Save the file and runmake release ormake dev

TheAL_TAG variable affects all Docker builds in the Makefile, including:

• make release - Release image build
• make dev - Development image build
• make build - Main build target
• make build-init - Init container build
• make main-debug-base - Debug image build
• Plugin builds (linux-plugins,cloudwatch-plugins, etc.)

You can customize which version of Fluent Bit is built and which repository it's sourced from. The default values are:

• FLB_VERSION: "1.9.10" (can be either a branch name or tag name within the repository)
• FLB_REPOSITORY: "https://github.com/amazon-contributing/upstream-to-fluent-bit.git"

There are two ways to customize these values:

Method 1: Using environment variables

Set the environment variables when running the make command:

# Build with a specific Fluent Bit version tagFLB_VERSION="2.0.8" make release# Build with a specific branch nameFLB_VERSION="feature-branch" make release# Build from a different repositoryFLB_VERSION="your-branch" FLB_REPOSITORY="https://github.com/your-username/fluent-bit.git" make release# Combine with dev mode for faster builds with cachingFLB_VERSION="your-branch" FLB_REPOSITORY="https://github.com/your-username/fluent-bit.git" make dev

Method 2: Updating the Makefile directly

You can also modify the default values in the Makefile:

1. Open the Makefile in your editor
2. Locate these lines (near the compile stage):

   # Fluent Bit version (branch or tag) to checkout, will use value if not set FLB_VERSION ?= "1.9.10"# Fluent Bit repository to checkout, will use value if not setFLB_REPOSITORY ?= "https://github.com/amazon-contributing/upstream-to-fluent-bit.git"

3. Update the values as needed
4. Save the file and runmake release ormake dev

Usemake dev to build the image.

Themake integ-dev command will run the integration tests for all of our plugins-kinesis streams, kinesis firehose, and cloudwatch.

Note that these steps rely on creating Cfn stacks in an AWS account in region us-west-2,so AWS credentials must be setup before they are run.

Instructions:

1. Setup AWS access via EC2 instance role or AWS_* env vars
2. Install dependent packages:docker awscli
3. Install docker-compose:

sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-composesudo chmod +x /usr/local/bin/docker-compose

4. Build validator images:

pushd integ/validate_cloudwatch && docker build -t flbcwinteg . && popdpushd integ/s3 && docker build -t flbs3integ . && popdexport CW_INTEG_VALIDATOR_IMAGE="flbcwinteg"export S3_INTEG_VALIDATOR_IMAGE="flbs3integ"

5. Runmake integ-dev

To run integration tests separately, executemake integ-cloudwatch ormake integ-kinesis ormake integ-firehose.

Documentation on GitHub steps for releases.

You can build a version of the image with code in your GitHub fork. To do so, you must need to set the following environment variables.Otherwise, you will see an error message like the following one:fatal: repository '/kinesis-streams' or '/kinesis-firehose' or '/cloudwatch' does not exist.

Set the following environment variables for CloudWatch:

export CLOUDWATCH_PLUGIN_CLONE_URL="Your GitHub fork clone URL"export CLOUDWATCH_PLUGIN_BRANCH="Your branch on your fork"

Or for Kinesis Streams:

export KINESIS_PLUGIN_CLONE_URL="Your GitHub fork clone URL"export KINESIS_PLUGIN_BRANCH="Your branch on your fork"

Or for Kinesis Firehose:

export FIREHOSE_PLUGIN_CLONE_URL="Your GitHub fork clone URL"export FIREHOSE_PLUGIN_BRANCH="Your branch on your fork"

Then runmake cloudwatch-dev ormake kinesis-dev ormake firehose-dev to build the image with your changes.

To run the integration tests on your code, executemake integ-cloudwatch-dev ormake integ-kinesis-dev ormake integ-firehose-dev.

Check out Fluent Bit examples from ouramazon-ecs-firelens-examples repo.

This project is licensed under the Apache-2.0 License.