Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Represent, send, store and search multimodal data

License

NotificationsYou must be signed in to change notification settings

docarray/docarray

DocArray logo: The data structure for unstructured data
The data structure for multimodal data

PyPICodecov branchPyPI - Downloads from official pypistats

NoteThe README you're currently viewing is for DocArray>0.30, which introduces some significant changes from DocArray 0.21. If you wish to continue using the older DocArray <=0.21, ensure you install it viapip install docarray==0.21. Refer to itscodebase,documentation, andits hot-fixes branch for more information.

DocArray is a Python library expertly crafted for therepresentation,transmission,storage, andretrieval of multimodal data. Tailored for the development of multimodal AI applications, its design guarantees seamless integration with the extensive Python and machine learning ecosystems. As of January 2022, DocArray is openly distributed under theApache License 2.0 and currently enjoys the status of a sandbox project within theLF AI & Data Foundation.

Installation

To install DocArray from the CLI, run the following command:

pip install -U docarray

NoteTo use DocArray <=0.21, make sure you install viapip install docarray==0.21 and check out itscodebase anddocs andits hot-fixes branch.

Get Started

New to DocArray? Depending on your use case and background, there are multiple ways to learn about DocArray:

Represent

DocArray empowers you torepresent your data in a manner that is inherently attuned to machine learning.

This is particularly beneficial for various scenarios:

  • 🏃 You aretraining a model: You're dealing with tensors of varying shapes and sizes, each signifying different elements. You desire a method to logically organize them.
  • ☁️ You areserving a model: Let's say through FastAPI, and you wish to define your API endpoints precisely.
  • 🗂️ You areparsing data: Perhaps for future deployment in your machine learning or data science projects.

💡Familiar with Pydantic? You'll be pleased to learnthat DocArray is not only constructed atop Pydantic but also maintains complete compatibility with it!Furthermore, we have aspecific section dedicated to your needs!

In essence, DocArray facilitates data representation in a way that mirrors Python dataclasses, with machine learning being an integral component:

fromdocarrayimportBaseDocfromdocarray.typingimportTorchTensor,ImageUrlimporttorch# Define your data modelclassMyDocument(BaseDoc):description:strimage_url:ImageUrl# could also be VideoUrl, AudioUrl, etc.image_tensor:TorchTensor[1704,2272,3]# you can express tensor shapes!# Stack multiple documents in a Document VectorfromdocarrayimportDocVecvec=DocVec[MyDocument](    [MyDocument(description="A cat",image_url="https://example.com/cat.jpg",image_tensor=torch.rand(1704,2272,3),        ),    ]*10)print(vec.image_tensor.shape)# (10, 1704, 2272, 3)
Click for more details

Let's take a closer look at how you can represent your data with DocArray:

fromdocarrayimportBaseDocfromdocarray.typingimportTorchTensor,ImageUrlfromtypingimportOptionalimporttorch# Define your data modelclassMyDocument(BaseDoc):description:strimage_url:ImageUrl# could also be VideoUrl, AudioUrl, etc.image_tensor:Optional[TorchTensor[1704,2272,3]    ]=None# could also be NdArray or TensorflowTensorembedding:Optional[TorchTensor]=None

So not only can you define the types of your data, you can evenspecify the shape of your tensors!

# Create a documentdoc=MyDocument(description="This is a photo of a mountain",image_url="https://upload.wikimedia.org/wikipedia/commons/2/2f/Alpamayo.jpg",)# Load image tensor from URLdoc.image_tensor=doc.image_url.load()# Compute embedding with any model of your choicedefclip_image_encoder(image_tensor:TorchTensor)->TorchTensor:# dummy functionreturntorch.rand(512)doc.embedding=clip_image_encoder(doc.image_tensor)print(doc.embedding.shape)# torch.Size([512])

Compose nested Documents

Of course, you can compose Documents into a nested structure:

fromdocarrayimportBaseDocfromdocarray.documentsimportImageDoc,TextDocimportnumpyasnpclassMultiModalDocument(BaseDoc):image_doc:ImageDoctext_doc:TextDocdoc=MultiModalDocument(image_doc=ImageDoc(tensor=np.zeros((3,224,224))),text_doc=TextDoc(text='hi!'))

You rarely work with a single data point at a time, especially in machine learning applications. That's why you can easily collect multipleDocuments:

Collect multipleDocuments

When building or interacting with an ML system, usually you want to process multiple Documents (data points) at once.

DocArray offers two data structures for this:

  • DocVec: A vector ofDocuments. All tensors in the documents are stacked into a single tensor.Perfect for batch processing and use inside of ML models.
  • DocList: A list ofDocuments. All tensors in the documents are kept as-is.Perfect for streaming, re-ranking, and shuffling of data.

Let's take a look at them, starting withDocVec:

fromdocarrayimportDocVec,BaseDocfromdocarray.typingimportAnyTensor,ImageUrlimportnumpyasnpclassImage(BaseDoc):url:ImageUrltensor:AnyTensor# this allows torch, numpy, and tensor flow tensorsvec=DocVec[Image](# the DocVec is parametrized by your personal schema!    [Image(url="https://upload.wikimedia.org/wikipedia/commons/2/2f/Alpamayo.jpg",tensor=np.zeros((3,224,224)),        )for_inrange(100)    ])

In the code snippet above,DocVec isparametrized by the type of document you want to use with it:DocVec[Image].

This may look weird at first, but we're confident that you'll get used to it quickly!Besides, it lets us do some cool things, like havingbulk access to the fields that you defined in your document:

tensor=vec.tensor# gets all the tensors in the DocVecprint(tensor.shape)# which are stacked up into a single tensor!print(vec.url)# you can bulk access any other field, too

The second data structure,DocList, works in a similar way:

fromdocarrayimportDocListdl=DocList[Image](# the DocList is parametrized by your personal schema!    [Image(url="https://upload.wikimedia.org/wikipedia/commons/2/2f/Alpamayo.jpg",tensor=np.zeros((3,224,224)),        )for_inrange(100)    ])

You can still bulk access the fields of your document:

tensors=dl.tensor# gets all the tensors in the DocListprint(type(tensors))# as a list of tensorsprint(dl.url)# you can bulk access any other field, too

And you can insert, remove, and append documents to yourDocList:

# appenddl.append(Image(url="https://upload.wikimedia.org/wikipedia/commons/2/2f/Alpamayo.jpg",tensor=np.zeros((3,224,224)),    ))# deletedeldl[0]# insertdl.insert(0,Image(url="https://upload.wikimedia.org/wikipedia/commons/2/2f/Alpamayo.jpg",tensor=np.zeros((3,224,224)),    ),)

And you can seamlessly switch betweenDocVec andDocList:

vec_2=dl.to_doc_vec()assertisinstance(vec_2,DocVec)dl_2=vec_2.to_doc_list()assertisinstance(dl_2,DocList)

Send

DocArray facilitates thetransmission of your data in a manner inherently compatible with machine learning.

This includes native support forProtobuf and gRPC, along withHTTP and serialization to JSON, JSONSchema, Base64, and Bytes.

This feature proves beneficial for several scenarios:

  • ☁️ You areserving a model, perhaps through frameworks likeJina orFastAPI
  • 🕸️ You aredistributing your model across multiple machines and need an efficient means of transmitting your data between nodes
  • ⚙️ You are architecting amicroservice environment and require a method for data transmission between microservices

💡Are you familiar with FastAPI? You'll be delighted to learnthat DocArray maintains full compatibility with FastAPI!Plus, we have adedicated section specifically for you!

When it comes to data transmission, serialization is a crucial step. Let's delve into how DocArray streamlines this process:

fromdocarrayimportBaseDocfromdocarray.typingimportImageTorchTensorimporttorch# model your dataclassMyDocument(BaseDoc):description:strimage:ImageTorchTensor[3,224,224]# create a Documentdoc=MyDocument(description="This is a description",image=torch.zeros((3,224,224)),)# serialize it!proto=doc.to_protobuf()bytes_=doc.to_bytes()json=doc.json()# deserialize it!doc_2=MyDocument.from_protobuf(proto)doc_4=MyDocument.from_bytes(bytes_)doc_5=MyDocument.parse_raw(json)

Of course, serialization is not all you need. So check out how DocArray integrates withJina andFastAPI.

Store

After modeling and possibly distributing your data, you'll typically want tostore it somewhere. That's where DocArray steps in!

Document Stores provide a seamless way to, as the name suggests, store your Documents. Be it locally or remotely, you can do it all through the same user interface:

The Document Store interface lets you push and pull Documents to and from multiple data sources, all with the same user interface.

For example, let's see how that works with on-disk storage:

fromdocarrayimportBaseDoc,DocListclassSimpleDoc(BaseDoc):text:strdocs=DocList[SimpleDoc]([SimpleDoc(text=f'doc{i}')foriinrange(8)])docs.push('file://simple_docs')docs_pull=DocList[SimpleDoc].pull('file://simple_docs')

Retrieve

Document Indexes let you index your Documents in avector database for efficient similarity-based retrieval.

This is useful for:

Currently, Document Indexes supportWeaviate,Qdrant,ElasticSearch,Redis,Mongo Atlas, andHNSWLib, with more to come!

The Document Index interface lets you index and retrieve Documents from multiple vector databases, all with the same user interface.

It supports ANN vector search, text search, filtering, and hybrid search.

fromdocarrayimportDocList,BaseDocfromdocarray.indeximportHnswDocumentIndeximportnumpyasnpfromdocarray.typingimportImageUrl,ImageTensor,NdArrayclassImageDoc(BaseDoc):url:ImageUrltensor:ImageTensorembedding:NdArray[128]# create some datadl=DocList[ImageDoc](    [ImageDoc(url="https://upload.wikimedia.org/wikipedia/commons/2/2f/Alpamayo.jpg",tensor=np.zeros((3,224,224)),embedding=np.random.random((128,)),        )for_inrange(100)    ])# create a Document Indexindex=HnswDocumentIndex[ImageDoc](work_dir='/tmp/test_index')# index your dataindex.index(dl)# find similar Documentsquery=dl[0]results,scores=index.find(query,limit=10,search_field='embedding')

Learn DocArray

Depending on your background and use case, there are different ways for you to understand DocArray.

Coming from DocArray <=0.21

Click to expand

If you are using DocArray version 0.30.0 or lower, you will be familiar with itsdataclass API.

DocArray >=0.30 is that idea, taken seriously. Every document is created through a dataclass-like interface,courtesy ofPydantic.

This gives the following advantages:

  • Flexibility: No need to conform to a fixed set of fields -- your data defines the schema
  • Multimodality: At their core, documents are just dictionaries. This makes it easy to create and send them from any language, not just Python.

You may also be familiar with our old Document Stores for vector DB integration.They are now calledDocument Indexes and offer the following improvements (seehere for the new API):

  • Hybrid search: You can now combine vector search with text search, and even filter by arbitrary fields
  • Production-ready: The new Document Indexes are a much thinner wrapper around the various vector DB libraries, making them more robust and easier to maintain
  • Increased flexibility: We strive to support any configuration or setting that you could perform through the DB's first-party client

For now, Document Indexes supportWeaviate,Qdrant,ElasticSearch,Redis,Mongo Atlas, Exact Nearest Neighbour search andHNSWLib, with more to come.

Coming from Pydantic

Click to expand

If you come from Pydantic, you can see DocArray documents as juiced up Pydantic models, and DocArray as a collection of goodies around them.

More specifically, we set out tomake Pydantic fit for the ML world - not by replacing it, but by building on top of it!

This means you get the following benefits:

  • ML-focused types: Tensor, TorchTensor, Embedding, ..., includingtensor shape validation
  • Full compatibility withFastAPI
  • DocList andDocVec generalize the idea of a model to asequence orbatch of models. Perfect foruse in ML models and other batch processing tasks.
  • Types that are alive: ImageUrl can.load() a URL to image tensor, TextUrl can load and tokenize text documents, etc.
  • Cloud-ready: Serialization toProtobuf for use with microservices andgRPC
  • Pre-built multimodal documents for different data modalities: Image, Text, 3DMesh, Video, Audio and more. Note that all of these are valid Pydantic models!
  • Document Stores andDocument Indexes let you store your data and retrieve it usingvector search

The most obvious advantage here isfirst-class support for ML centric data, such as{Torch, TF, ...}Tensor,Embedding, etc.

This includes handy features such as validating the shape of a tensor:

fromdocarrayimportBaseDocfromdocarray.typingimportTorchTensorimporttorchclassMyDoc(BaseDoc):tensor:TorchTensor[3,224,224]doc=MyDoc(tensor=torch.zeros(3,224,224))# worksdoc=MyDoc(tensor=torch.zeros(224,224,3))# works by reshapingtry:doc=MyDoc(tensor=torch.zeros(224))# fails validationexceptExceptionase:print(e)# tensor# Cannot reshape tensor of shape (224,) to shape (3, 224, 224) (type=value_error)classImage(BaseDoc):tensor:TorchTensor[3,'x','x']Image(tensor=torch.zeros(3,224,224))# workstry:Image(tensor=torch.zeros(3,64,128)    )# fails validation because second dimension does not match thirdexceptExceptionase:print()try:Image(tensor=torch.zeros(4,224,224)    )# fails validation because of the first dimensionexceptExceptionase:print(e)# Tensor shape mismatch. Expected(3, 'x', 'x'), got(4, 224, 224)(type=value_error)try:Image(tensor=torch.zeros(3,64)    )# fails validation because it does not have enough dimensionsexceptExceptionase:print(e)# Tensor shape mismatch. Expected (3, 'x', 'x'), got (3, 64) (type=value_error)

Coming from PyTorch

Click to expand

If you come from PyTorch, you can see DocArray mainly as a way oforganizing your data as it flows through your model.

It offers you several advantages:

  • Expresstensor shapes in type hints
  • Group tensors that belong to the same object, e.g. an audio track and an image
  • Go directly to deployment, by re-using your data model as aFastAPI orJina API schema
  • Connect model components betweenmicroservices, using Protobuf and gRPC

DocArray can be used directly inside ML models to handle and represent multimodaldata.This allows you to reason about your data using DocArray's abstractions deep inside ofnn.Module,and provides a FastAPI-compatible schema that eases the transition between model training and model serving.

To see the effect of this, let's first observe a vanilla PyTorch implementation of a tri-modal ML model:

importtorchfromtorchimportnndefencoder(x):returntorch.rand(512)classMyMultiModalModel(nn.Module):def__init__(self):super().__init__()self.audio_encoder=encoder()self.image_encoder=encoder()self.text_encoder=encoder()defforward(self,text_1,text_2,image_1,image_2,audio_1,audio_2):embedding_text_1=self.text_encoder(text_1)embedding_text_2=self.text_encoder(text_2)embedding_image_1=self.image_encoder(image_1)embedding_image_2=self.image_encoder(image_2)embedding_audio_1=self.image_encoder(audio_1)embedding_audio_2=self.image_encoder(audio_2)return (embedding_text_1,embedding_text_2,embedding_image_1,embedding_image_2,embedding_audio_1,embedding_audio_2,        )

Not very easy on the eyes if you ask us. And even worse, if you need to add one more modality you have to touch every part of your code base, changing theforward() return type and making a whole lot of changes downstream from that.

So, now let's see what the same code looks like with DocArray:

fromdocarrayimportDocList,BaseDocfromdocarray.documentsimportImageDoc,TextDoc,AudioDocfromdocarray.typingimportTorchTensorfromtorchimportnnimporttorchdefencoder(x):returntorch.rand(512)classPodcast(BaseDoc):text:TextDocimage:ImageDocaudio:AudioDocclassPairPodcast(BaseDoc):left:Podcastright:PodcastclassMyPodcastModel(nn.Module):def__init__(self):super().__init__()self.audio_encoder=encoder()self.image_encoder=encoder()self.text_encoder=encoder()defforward_podcast(self,docs:DocList[Podcast])->DocList[Podcast]:docs.audio.embedding=self.audio_encoder(docs.audio.tensor)docs.text.embedding=self.text_encoder(docs.text.tensor)docs.image.embedding=self.image_encoder(docs.image.tensor)returndocsdefforward(self,docs:DocList[PairPodcast])->DocList[PairPodcast]:docs.left=self.forward_podcast(docs.left)docs.right=self.forward_podcast(docs.right)returndocs

Looks much better, doesn't it?You instantly win in code readability and maintainability. And for the same price you can turn your PyTorch model into a FastAPI app and reuse your Documentschema definition (seebelow). Everything is handled in a pythonic manner by relying on type hints.

Coming from TensorFlow

Click to expand

Like thePyTorch approach, you can also use DocArray with TensorFlow to handle and represent multimodal data inside your ML model.

First off, to use DocArray with TensorFlow we first need to install it as follows:

pip install tensorflow==2.12.0pip install protobuf==3.19.0

Compared to using DocArray with PyTorch, there is one main difference when using it with TensorFlow:While DocArray'sTorchTensor is a subclass oftorch.Tensor, this is not the case for theTensorFlowTensor: Due to some technical limitations oftf.Tensor, DocArray'sTensorFlowTensor is not a subclass oftf.Tensor but rather stores atf.Tensor in its.tensor attribute.

How does this affect you? Whenever you want to access the tensor data to, let's say, do operations with it or hand it to your ML model, instead of handing over yourTensorFlowTensor instance, you need to access its.tensor attribute.

This would look like the following:

fromtypingimportOptionalfromdocarrayimportDocList,BaseDocimporttensorflowastfclassPodcast(BaseDoc):audio_tensor:Optional[AudioTensorFlowTensor]=Noneembedding:Optional[AudioTensorFlowTensor]=NoneclassMyPodcastModel(tf.keras.Model):def__init__(self):super().__init__()self.audio_encoder=AudioEncoder()defcall(self,inputs:DocList[Podcast])->DocList[Podcast]:inputs.audio_tensor.embedding=self.audio_encoder(inputs.audio_tensor.tensor        )# access audio_tensor's .tensor attributereturninputs

Coming from FastAPI

Click to expand

Documents are Pydantic Models (with a twist), and as such they are fully compatible with FastAPI!

But why should you use them, and not the Pydantic models you already know and love?Good question!

  • Because of the ML-first features, types and validations,here
  • Because DocArray can act as anORM for vector databases, similar to what SQLModel does for SQL databases

And to seal the deal, let us show you how easily documents slot into your FastAPI app:

importnumpyasnpfromfastapiimportFastAPIfromdocarray.base_docimportDocArrayResponsefromdocarrayimportBaseDocfromdocarray.documentsimportImageDocfromdocarray.typingimportNdArray,ImageTensorclassInputDoc(BaseDoc):img:ImageDoctext:strclassOutputDoc(BaseDoc):embedding_clip:NdArrayembedding_bert:NdArrayapp=FastAPI()defmodel_img(img:ImageTensor)->NdArray:returnnp.zeros((100,1))defmodel_text(text:str)->NdArray:returnnp.zeros((100,1))@app.post("/embed/",response_model=OutputDoc,response_class=DocArrayResponse)asyncdefcreate_item(doc:InputDoc)->OutputDoc:doc=OutputDoc(embedding_clip=model_img(doc.img.tensor),embedding_bert=model_text(doc.text)    )returndocinput_doc=InputDoc(text='',img=ImageDoc(tensor=np.random.random((3,224,224))))asyncwithAsyncClient(app=app,base_url="http://test")asac:response=awaitac.post("/embed/",data=input_doc.json())

Just like a vanilla Pydantic model!

Coming from Jina

Click to expand

Jina has adopted docarray as their library for representing and serializing Documents.

Jina allows to serve models and services that are built with DocArray allowing you to serve and scale these applicationsmaking full use of DocArray's serialization capabilites.

importnumpyasnpfromjinaimportDeployment,Executor,requestsfromdocarrayimportBaseDoc,DocListfromdocarray.documentsimportImageDocfromdocarray.typingimportNdArray,ImageTensorclassInputDoc(BaseDoc):img:ImageDoctext:strclassOutputDoc(BaseDoc):embedding_clip:NdArrayembedding_bert:NdArraydefmodel_img(img:ImageTensor)->NdArray:returnnp.zeros((100,1))defmodel_text(text:str)->NdArray:returnnp.zeros((100,1))classMyEmbeddingExecutor(Executor):@requests(on='/embed')defencode(self,docs:DocList[InputDoc],**kwargs)->DocList[OutputDoc]:ret=DocList[OutputDoc]()fordocindocs:output=OutputDoc(embedding_clip=model_img(doc.img.tensor),embedding_bert=model_text(doc.text),            )ret.append(output)returnretwithDeployment(protocols=['grpc','http'],ports=[12345,12346],uses=MyEmbeddingExecutor)asdep:resp=dep.post(on='/embed',inputs=DocList[InputDoc](            [InputDoc(text='',img=ImageDoc(tensor=np.random.random((3,224,224))))]        ),return_type=DocList[OutputDoc],    )print(resp)

Coming from a vector database

Click to expand

If you came across DocArray as a universal vector database client, you can best think of it asa new kind of ORM for vector databases.DocArray's job is to take multimodal, nested and domain-specific data and to map it to a vector database,store it there, and thus make it searchable:

fromdocarrayimportDocList,BaseDocfromdocarray.indeximportHnswDocumentIndeximportnumpyasnpfromdocarray.typingimportImageUrl,ImageTensor,NdArrayclassImageDoc(BaseDoc):url:ImageUrltensor:ImageTensorembedding:NdArray[128]# create some datadl=DocList[ImageDoc](    [ImageDoc(url="https://upload.wikimedia.org/wikipedia/commons/2/2f/Alpamayo.jpg",tensor=np.zeros((3,224,224)),embedding=np.random.random((128,)),        )for_inrange(100)    ])# create a Document Indexindex=HnswDocumentIndex[ImageDoc](work_dir='/tmp/test_index2')# index your dataindex.index(dl)# find similar Documentsquery=dl[0]results,scores=index.find(query,limit=10,search_field='embedding')

Currently, DocArray supports the following vector databases:

An integration ofOpenSearch is currently in progress.

Of course this is only one of the things that DocArray can do, so we encourage you to check out the rest of this readme!

Coming from Langchain

Click to expand

With DocArray, you can connect external data to LLMs through Langchain. DocArray gives you the freedom to establishflexible document schemas and choose from different backends for document storage.After creating your document index, you can connect it to your Langchain app usingDocArrayRetriever.

Install Langchain via:

pip install langchain
  1. Define a schema and create documents:
fromdocarrayimportBaseDoc,DocListfromdocarray.typingimportNdArrayfromlangchain.embeddings.openaiimportOpenAIEmbeddingsembeddings=OpenAIEmbeddings()# Define a document schemaclassMovieDoc(BaseDoc):title:strdescription:stryear:intembedding:NdArray[1536]movies= [    {"title":"#1 title","description":"#1 description","year":1999},    {"title":"#2 title","description":"#2 description","year":2001},]# Embed `description` and create documentsdocs=DocList[MovieDoc](MovieDoc(embedding=embeddings.embed_query(movie["description"]),**movie)formovieinmovies)
  1. Initialize a document index using any supported backend:
fromdocarray.indeximport (InMemoryExactNNIndex,HnswDocumentIndex,WeaviateDocumentIndex,QdrantDocumentIndex,ElasticDocIndex,RedisDocumentIndex,MongoDBAtlasDocumentIndex,)# Select a suitable backend and initialize it with datadb=InMemoryExactNNIndex[MovieDoc](docs)
  1. Finally, initialize a retriever and integrate it into your chain!
fromlangchain.chat_modelsimportChatOpenAIfromlangchain.chainsimportConversationalRetrievalChainfromlangchain.retrieversimportDocArrayRetriever# Create a retrieverretriever=DocArrayRetriever(index=db,embeddings=embeddings,search_field="embedding",content_field="description",)# Use the retriever in your chainmodel=ChatOpenAI()qa=ConversationalRetrievalChain.from_llm(model,retriever=retriever)

Alternatively, you can use built-in vector stores. Langchain supports two vector stores:DocArrayInMemorySearch andDocArrayHnswSearch.Both are user-friendly and are best suited to small to medium-sized datasets.

See also

DocArray is a trademark of LF AI Projects, LLC

Contributors76

Languages

[8]ページ先頭
©2009-2025 Movatter.jp