Movatterモバイル変換


[0]ホーム

URL:


Try our new Snowflake Connector — the simplest way to stream data to Snowflake.Learn more.
    Was this helpful?
    🎉 Thanks for your feedback!

    Get Started with Redpanda Connect usingrpk

    This guide explains how to get started with Redpanda Connect usingrpk, the Redpanda command-line interface (CLI). You can also install and runrpk inFIPS compliance mode.

    Install

    Therpk CLI allows you to create and manage data pipelines with Redpanda Connect as well asinteract with Redpanda clusters.

    Therpk CLI also includes anrpk connect plugin, which manages installations and upgrades of Redpanda Connect. This plugin is automatically installed when you first runrpk connect commands, unless you runrpk connect --version, which prompts you to install the plugin.
    Also interacting with a Redpanda cluster?

    If you want to userpk to also communicate with a Redpanda cluster, ensure the version ofrpk that you install matches the version of Redpanda running in your cluster.

    Linux

    To install, or update to, the latest version ofrpk for Linux, run:

    • amd64

    • arm64

    curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip &&  mkdir -p ~/.local/bin &&  export PATH="~/.local/bin:$PATH" &&  unzip rpk-linux-amd64.zip -d ~/.local/bin/
    curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-arm64.zip &&  mkdir -p ~/.local/bin &&  export PATH="~/.local/bin:$PATH" &&  unzip rpk-linux-arm64.zip -d ~/.local/bin/
    You can userpk on Windows only withWSL. However, commands that require Redpanda to be installed on your machine are not supported, such asrpk container commands,rpk iotune, andrpk redpanda commands.

    To install, or update to, a version other than the latest, run:

    • amd64

    • arm64

    curl -LO https://github.com/redpanda-data/redpanda/releases/download/v<version>/rpk-linux-amd64.zip &&  mkdir -p ~/.local/bin &&  export PATH="~/.local/bin:$PATH" &&  unzip rpk-linux-amd64.zip -d ~/.local/bin/
    curl -LO https://github.com/redpanda-data/redpanda/releases/download/v<version>/rpk-linux-arm64.zip &&  mkdir -p ~/.local/bin &&  export PATH="~/.local/bin:$PATH" &&  unzip rpk-linux-arm64.zip -d ~/.local/bin/

    FIPS compliance

    This feature requires anenterprise license. You can eitherupgrade to an Enterprise Edition license, orgenerate a trial license key that’s valid for 30 days.

    To installrpk to run the latest version of Redpanda Connect in FIPS-compliant mode, you must install theredpanda-rpk-fips andredpanda-connect-fips packages. Both packages are built using theMicrosoft GoLang compiler and theMicrosoft’s Go Crypto OpenSSL package, which uses the FIPS-approved version of OpenSSL.

    The packages for FIPS-compliantrpk (redpanda-rpk-fips) and Redpandarpk (redpanda-rpk) are mutually exclusive, and so cannot be installed in the same environment.
    • RHEL

    • Debian/Ubuntu

    1. To make sure your distribution is up to date, run:

      sudo dnf upgrade
    2. Addredpanda to yourdnf list of repositories.

      curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.rpm.sh' | \sudo -E bash
    3. Install Redpanda packages for FIPS compliance.

      sudo dnf install -y redpanda-rpk-fips redpanda-connect-fips
    4. Verify your installation.

      rpk connect --version

    To keep up-to-date with Redpanda Connect releases, run the following command:

    sudo dnf update
    1. To make sure your distribution is up to date, run:

      sudo apt upgrade
    2. Addredpanda to yourapt list of repositories.

      curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | sudo -E bash
    3. Install Redpanda packages for FIPS compliance.

      sudo apt install -y redpanda-rpk-fips redpanda-connect-fips
    4. Verify your installation.

      rpk connect --version

    To keep up-to-date with the Redpanda Connect releases, run the following command:

    sudo apt update

    MacOS

    • Homebrew

    • Manual Download

    1. If you don’t have Homebrew installed,install it.

    2. To install or updaterpk, run:

      brew install redpanda-data/tap/redpanda

    To install or updaterpk through a manual download, choose the option for your system architecture. For example, if you have an M1 or newer chip, selectApple Silicon.

    • Intel macOS

    • Apple Silicon

    To install, or update to, the latest version ofrpk for Intel macOS, run:

    curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-darwin-amd64.zip &&  mkdir -p ~/.local/bin &&  export PATH="~/.local/bin:$PATH" &&  unzip rpk-darwin-amd64.zip -d ~/.local/bin/

    To install, or update to, a version other than the latest, run:

    curl -LO https://github.com/redpanda-data/redpanda/releases/download/v<version>/rpk-darwin-amd64.zip &&  mkdir -p ~/.local/bin &&  export PATH="~/.local/bin:$PATH" &&  unzip rpk-darwin-amd64.zip -d ~/.local/bin/

    To install, or update to, the latest version ofrpk for Apple Silicon, run:

    curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-darwin-arm64.zip &&  mkdir -p ~/.local/bin &&  export PATH="~/.local/bin:$PATH" &&  unzip rpk-darwin-arm64.zip -d ~/.local/bin/

    To install, or update to, a version other than the latest, run:

    curl -LO https://github.com/redpanda-data/redpanda/releases/download/v<version>/rpk-darwin-arm64.zip &&  mkdir -p ~/.local/bin &&  export PATH="~/.local/bin:$PATH" &&  unzip rpk-darwin-arm64.zip -d ~/.local/bin/

    Docker

    To userpk in Docker, you can use the Redpanda Docker image, which includes both Redpanda andrpk as part of the installation.

    docker pull docker.redpanda.com/redpandadata/connect

    Run a pipeline

    A Redpanda Connect stream pipeline is configured with a singleconfig file, you can generate a fresh one with:

    rpk connect create > connect.yaml
    This command may take a few seconds to run. If this is the firstrpk connect command you have run, therpk connect plugin is automatically installed.

    For Docker installations:

    docker run --rm docker.redpanda.com/redpandadata/connect create > ./connect.yaml

    The main sections that make up a config areinput,pipeline andoutput. When you generate a fresh config it’ll simply pipestdin tostdout like this:

    input:  stdin: {}pipeline:  processors: []output:  stdout: {}

    Eventually we’ll want to configure a more usefulinput andoutput, but for now this is useful for quickly testing processors. You can execute this config with:

    rpk connect run connect.yaml

    For Docker installations:

    docker run --rm -it -v $(pwd)/connect.yaml:/connect.yaml docker.redpanda.com/redpandadata/connect run

    Anything you write to stdin will get written unchanged to stdout, cool! Resist the temptation to play with this for hours, there’s more stuff to try out.

    Next, let’s add some processing steps in order to mutate messages. The most powerful one is themapping processor which allows us to perform mappings, let’s add a mapping to uppercase our messages:

    input:  stdin: {}pipeline:  processors:    - mapping: root = content().uppercase()output:  stdout: {}

    Now your messages should come out in all caps.

    You can add as manyprocessing steps as you like, and since processors are what make Redpanda Connect powerful they are worth experimenting with. Let’s create a more advanced pipeline that works with JSON documents:

    input:  stdin: {}pipeline:  processors:    - sleep:        duration: 500ms    - mapping: |        root.doc = this        root.first_name = this.names.index(0).uppercase()        root.last_name = this.names.index(-1).hash("sha256").encode("base64")output:  stdout: {}

    First, we sleep for 500 milliseconds just to keep the suspense going. Next, we restructure our input JSON document by nesting it within a fielddoc, we map the upper-cased first element ofnames to a new fieldfirst_name. Finally, we map the hashed and base64 encoded value of the last element ofnames to a new fieldlast_name.

    Try running that config with some sample documents:

    echo '{"id":"1","names":["celine","dion"]}{"id":"2","names":["chad","robert","kroeger"]}' | rpk connect run connect.yaml

    For Docker installations:

    echo '{"id":"1","names":["celine","dion"]}{"id":"2","names":["chad","robert","kroeger"]}' | docker run --rm -i -v $(pwd)/connect.yaml:/connect.yaml docker.redpanda.com/redpandadata/connect run

    You should see this output in the logs:

    {"doc":{"id":"1","names":["celine","dion"]},"first_name":"CELINE","last_name":"1VvPgCW9sityz5XAMGdI2BTA7/44Wb3cANKxqhiCo50="}{"doc":{"id":"2","names":["chad","robert","kroeger"]},"first_name":"CHAD","last_name":"uXXg5wCKPjpyj/qbivPbD9H9CZ5DH/F0Q1Twytnt2hQ="}

    See also:

    Back to top
    ×

    Simple online edits

    For simple changes, such as fixing a typo, you can edit the content directly on GitHub.

    Or, open an issue to let us know about something that you want us to change.

    Contribution guide

    For extensive content updates, or if you prefer to work locally, read ourcontribution guide .

    Was this helpful?
    group Ask in the community
    mail Share your feedback
    group_addMake a contribution
    🎉 Thanks for your feedback!
    Expand navigation tree

    [8]ページ先頭

    ©2009-2025 Movatter.jp