Async API¶

We demonstrate the following functionalities suppored by LanceDB using our asynchonous APIs:

• Automatic versioning
• Instant rollback
• Appends, updates, deletions
• Schema evolution

Let's first prepare the data. We will be using a CSV file with a bunch of quotes from Rick and Morty

Let's load this into a pandas dataframe.

It's got 3 columns, a quote id, the quote string, and the first name of the author of the quote:

Creating a LanceDB table from a pandas dataframe is straightforward usingcreate_table

We'll start with a local LanceDB connection

Updates¶

Now, since Rick is the smartest man in the multiverse, he deserves to have his quotes attributed to his full name: Richard Daniel Sanchez.

This can be done viaLanceTable.update. It needs two arguments:

1. Awhere string filter (sql syntax) to determine the rows to update
2. A dict ofupdates where the keys are the column names to update and the values are the new values

Schema evolution¶

Let's add anew_id column to the table, where each value is the originalid plus 1.

If we look at the schema, we see that a new int64 column was added

Rollback¶

Suppose we used the table and found that the new column should be a different value. How do we use another new column without losing the change history?

First, major operations are automatically versioned in LanceDB.Version 1 is the table creation, with the initial insertion of data.Versions 2 and 3 represents the update (deletion + append)Version 4 is adding the new column.

We can restore version 3, before we added thenew_id vector column

Notice that we now have one more, not less versions. When we restore an old version, we're not deleting the version history, we're just creating a new version where the schema and data is equivalent to the restored old version. In this way, we can keep track of all of the changes and always rollback to a previous state.

Add another new column¶

Now we'll change the value of thenew_id column and add it to the restored dataset again

Deletion¶

What if the whole show was just Rick-isms?Let's delete any quote not said by Rick

We can see that the number of rows has been reduced to 30

Ok we had our fun, let's get back to the full quote set

History¶

We now have 9 versions in the data. We can review the operations that corresponds to each version below:

Versions:

• 1 - Create
• 2 - Update
• 3 - Add a new column
• 4 - Restore (2)
• 5 - Add a new column
• 6 - Delete
• 7 - Restore

Summary¶

We never had to explicitly manage the versioning. And we never had to create expensive and slow snapshots. LanceDB automatically tracks the full history of operations I created and supports fast rollbacks. In production this is critical for debugging issues and minimizing downtime by rolling back to a previously successful state in seconds.