Repository files navigation

Ultra performant data transformation framework for AI, with core engine written in Rust. Support incremental processing and data lineage out-of-box. Exceptional developer velocity. Production-ready at day 0.

⭐ Drop a star to help us grow!

CocoIndex makes it effortless to transform data with AI, and keep source data and target in sync. Whether you’re building a vector index, creating knowledge graphs for context engineering or performing any custom data transformations — goes beyond SQL.

Just declare transformation in dataflow with ~100 lines of python

# importdata['content']=flow_builder.add_source(...)# transformdata['out']=data['content']    .transform(...)    .transform(...)# collect datacollector.collect(...)# export to db, vector db, graph db ...collector.export(...)

CocoIndex follows the idea ofDataflow programming model. Each transformation creates a new field solely based on input fields, without hidden states and value mutation. All data before/after each transformation is observable, with lineage out of the box.

Particularly, developers don't explicitly mutate data by creating, updating and deleting. They just need to define transformation/formula for a set of source data.

Native builtins for different source, targets and transformations. Standardize interface, make it 1-line code switch between different components - as easy as assembling building blocks.

CocoIndex keep source data and target in sync effortlessly.

It has out-of-box support for incremental indexing:

• minimal recomputation on source or logic change.
• (re-)processing necessary portions; reuse cache when possible

If you're new to CocoIndex, we recommend checking out

• 📖Documentation
• ⚡Quick Start Guide
• 🎬Quick Start Video Tutorial

1. Install CocoIndex Python library

pip install -U cocoindex

2. Install Postgres if you don't have one. CocoIndex uses it for incremental processing.

3. (Optional) Install Claude Code skill for enhanced development experience. Run these commands inClaude Code:

/plugin marketplace add cocoindex-io/cocoindex-claude/plugin install cocoindex-skills@cocoindex

FollowQuick Start Guide to define your first indexing flow. An example flow looks like:

@cocoindex.flow_def(name="TextEmbedding")deftext_embedding_flow(flow_builder:cocoindex.FlowBuilder,data_scope:cocoindex.DataScope):# Add a data source to read files from a directorydata_scope["documents"]=flow_builder.add_source(cocoindex.sources.LocalFile(path="markdown_files"))# Add a collector for data to be exported to the vector indexdoc_embeddings=data_scope.add_collector()# Transform data of each documentwithdata_scope["documents"].row()asdoc:# Split the document into chunks, put into `chunks` fielddoc["chunks"]=doc["content"].transform(cocoindex.functions.SplitRecursively(),language="markdown",chunk_size=2000,chunk_overlap=500)# Transform data of each chunkwithdoc["chunks"].row()aschunk:# Embed the chunk, put into `embedding` fieldchunk["embedding"]=chunk["text"].transform(cocoindex.functions.SentenceTransformerEmbed(model="sentence-transformers/all-MiniLM-L6-v2"))# Collect the chunk into the collector.doc_embeddings.collect(filename=doc["filename"],location=chunk["location"],text=chunk["text"],embedding=chunk["embedding"])# Export collected data to a vector index.doc_embeddings.export("doc_embeddings",cocoindex.targets.Postgres(),primary_key_fields=["filename","location"],vector_indexes=[cocoindex.VectorIndexDef(field_name="embedding",metric=cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY)])

It defines an index flow like this:

More coming and stay tuned 👀!

For detailed documentation, visitCocoIndex Documentation, including aQuickstart guide.

We love contributions from our community ❤️. For details on contributing or running the project for development, check out ourcontributing guide.

Welcome with a huge coconut hug 🥥⋆｡˚🤗. We are super excited for community contributions of all kinds - whether it's code improvements, documentation updates, issue reports, feature requests, and discussions in our Discord.

Join our community here:

• 🌟Star us on GitHub
• 👋Join our Discord community
• ▶️Subscribe to our YouTube channel
• 📜Read our blog posts

We are constantly improving, and more features and examples are coming soon. If you love this project, please drop us a star ⭐ at GitHub repo to stay tuned and help us grow.

CocoIndex is Apache 2.0 licensed.