Repository files navigation

Ultra performant data transformation framework for AI, with core engine written in Rust. Support incremental processing and data lineage out-of-box. Exceptional developer velocity. Production-ready at day 0.

⭐ Drop a star to help us grow!

CocoIndex makes it super easy to transform data with AI workloads, and keep source data and target in sync effortlessly.

Either creating embedding, building knowledge graphs, or any data transformations - beyond traditional SQL.

Just declare transformation in dataflow with ~100 lines of python

# importdata['content']=flow_builder.add_source(...)# transformdata['out']=data['content']    .transform(...)    .transform(...)# collect datacollector.collect(...)# export to db, vector db, graph db ...collector.export(...)

CocoIndex follows the idea ofDataflow programming model. Each transformation creates a new field solely based on input fields, without hidden states and value mutation. All data before/after each transformation is observable, with lineage out of the box.

Particularly, developers don't explicitly mutate data by creating, updating and deleting. They just need to define transformation/formula for a set of source data.

Native builtins for different source, targets and transformations. Standardize interface, make it 1-line code switch between different components.

CocoIndex keep source data and target in sync effortlessly.

It has out-of-box support for incremental indexing:

• minimal recomputation on source or logic change.
• (re-)processing necessary portions; reuse cache when possible

If you're new to CocoIndex, we recommend checking out

• 📖Documentation
• ⚡Quick Start Guide
• 🎬Quick Start Video Tutorial

1. Install CocoIndex Python library

pip install -U cocoindex

2. Install Postgres if you don't have one. CocoIndex uses it for incremental processing.

FollowQuick Start Guide to define your first indexing flow. An example flow looks like:

@cocoindex.flow_def(name="TextEmbedding")deftext_embedding_flow(flow_builder:cocoindex.FlowBuilder,data_scope:cocoindex.DataScope):# Add a data source to read files from a directorydata_scope["documents"]=flow_builder.add_source(cocoindex.sources.LocalFile(path="markdown_files"))# Add a collector for data to be exported to the vector indexdoc_embeddings=data_scope.add_collector()# Transform data of each documentwithdata_scope["documents"].row()asdoc:# Split the document into chunks, put into `chunks` fielddoc["chunks"]=doc["content"].transform(cocoindex.functions.SplitRecursively(),language="markdown",chunk_size=2000,chunk_overlap=500)# Transform data of each chunkwithdoc["chunks"].row()aschunk:# Embed the chunk, put into `embedding` fieldchunk["embedding"]=chunk["text"].transform(cocoindex.functions.SentenceTransformerEmbed(model="sentence-transformers/all-MiniLM-L6-v2"))# Collect the chunk into the collector.doc_embeddings.collect(filename=doc["filename"],location=chunk["location"],text=chunk["text"],embedding=chunk["embedding"])# Export collected data to a vector index.doc_embeddings.export("doc_embeddings",cocoindex.targets.Postgres(),primary_key_fields=["filename","location"],vector_indexes=[cocoindex.VectorIndexDef(field_name="embedding",metric=cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY)])

It defines an index flow like this:

More coming and stay tuned 👀!

For detailed documentation, visitCocoIndex Documentation, including aQuickstart guide.

We love contributions from our community ❤️. For details on contributing or running the project for development, check out ourcontributing guide.

Welcome with a huge coconut hug 🥥⋆｡˚🤗. We are super excited for community contributions of all kinds - whether it's code improvements, documentation updates, issue reports, feature requests, and discussions in our Discord.

Join our community here:

• 🌟Star us on GitHub
• 👋Join our Discord community
• ▶️Subscribe to our YouTube channel
• 📜Read our blog posts

We are constantly improving, and more features and examples are coming soon. If you love this project, please drop us a star ⭐ at GitHub repo to stay tuned and help us grow.

CocoIndex is Apache 2.0 licensed.