Repository files navigation

Fluvio is a lean and mean distributed data streaming engine written in Rust. Combined withStateful DataFlow distributed stream processing framework, Fluvio provides aunifiedcomposabledistributed streaming andstream processing paradigm for developers. It is the foundation ofInfinyOn Cloud.

Fluvio is installed via theFluvio Version Manager, shortened tofvm.

To installfvm, run the following command:

curl -fsS https://hub.infinyon.cloud/install/install.sh| bash

As part of the initial setup,fvm will also install the Fluvio CLI available in the stable channel as of the moment of installation.

Fluvio is stored in$HOME/.fluvio, with the executable binaries stored in$HOME/.fluvio/bin.

For the best compatibliity on Windows, InfinyOn recommends WSL2

Start cluster on you local machine with the following command:

fluvio cluster start

Stateful dataflows are managed viasdf cli that we install it usingfvm.

fvm install sdf-beta8

Create a dataflow file in the directorysplit-sentence directory:

mkdir -p split-sentence-inlinecd split-sentence-inline

Create thedataflow.yaml and add the following content:

apiVersion:0.5.0meta:name:split-sentence-inlineversion:0.1.0namespace:exampleconfig:converter:rawtopics:sentence:schema:value:type:stringconverter:rawwords:schema:value:type:stringconverter:rawservices:sentence-words:sources:      -type:topicid:sentencetransforms:      -operator:flat-maprun:|          fn sentence_to_words(sentence: String) -> Result<Vec<String>> {            Ok(sentence.split_whitespace().map(String::from).collect())          }      -operator:maprun:|          pub fn augment_count(word: String) -> Result<String> {            Ok(format!("{}({})", word, word.chars().count()))          }sinks:      -type:topicid:words

Use sdf command line tool to run the dataflow:

sdf run --ui

The --ui flag serves the graphical representation of the dataflow on SDF Studio.

Produce sentences to insentence topic:

fluvio produce sentence

Input some text, for example:

Hello worldHi there

Consume fromwords to retrieve the result:

fluvio consume words -Bd

See the results, for example:

Hello(1)world(1)Hi(1)there(1)

The dataflow collects runtime metrics that you can inspect in the runtime terminal.

Check the sentence-to-words counters:

show state sentence-words/sentence-to-words/metrics

See results, for example:

 Key    Window  succeeded  failed stats*       2          0

Check the augment-count counters:

show state sentence-words/augment-count/metrics

See results, for example:

Key    Window  succeeded  failedstats*       4          0

Congratulations! You've successfully built and run a composable dataflow!

More examples of Stateful DataFlow are on GitHub -https://github.com/infinyon/stateful-dataflows-examples/.

Fluvio documentation will provide additional context on how to use the Fluvio clusters, CLI, clients, a development kits.

• Fluvio overview
• Fluvio CLI docs home
• Fluvio Architecture

Stateful DataFlow designed to handle complex data processing workflows, allowing for customization and scalability through various programming languages and system primitives.

• SDF overview
• SDF Architecture

Fluvio can connect to practically any system that you can think of.

• For first party systems, fluvio clients can integrate with the edge system or application to source data.
• For third party systems fluvio connectors connect at the protocol level and collects data into fluvio topics.

Out of the box Fluvio has native http, webhook, mqtt, kafka inbound connectors. In terms of outbound connectors out of the box Fluvio supports http, SQL, kafka, and experimental builds of DuckDB, Redis, S3, Graphite etc.

Using Connector Development Kit, its intuitive to build connectors to any system fast.

Check out the docs and let us know if you need help building any connector.

• Connector docs
• Connector Development Kit (cdk) docs

Fluvio applies wasm based stream processing and data transformations. We call these reusable transformation functions smart modules. Reusable Smart modules are built using Smart Module Development Kit and can be distributed using InfinyOn Cloud hub.

• Smart Modules docs
• Smart Modules Development Kit (smdk) docs

InfinyOn Cloud is Fluvio on the cloud as a managed service.

• Check InfinyOn Cloud Guides
• Check out experimental data flows on InfinyOn Labs Repo

• Fluvio Client API docs home

Language Specifc API docs:

• Rust API docs
• Python API docs
• Javascript API docs

Community Maintained:

• Go API docs
• Java API docs
• Elixir API docs

If you'd like to contribute to the project, please read ourContributing guide.

Many fluvio users and developers have made projects to share with the community.Here a a few listed below:

• Swiftide Project: a Rust native library for building LLM applications
• Real Time Stock Charts: See how Fluvio is used to update real time stock charts

• Qdrant Connector
• Google Sheets Connector: Send data from Fluvio to Google Sheets
• Elastic Connector: Send data from Fluvio to Elastic Search

More projects and utilities are available in theFluvio Community Github Org

• Hello World Fluvio Connector: Sample Fluvio connector template to build your own connector
• Gurubase: Third-party AI/LLM Docs query

Made withcontrib.rocks.

This project is licensed under theApache license.