Repository files navigation

This repository exists to test our integrations in Third-Party AI/ML libraries.

With the public release of$vectorSearch, we have needed to integrate into these AI/ML sponsored libraries.(LangChain,LlamaIndex,Semantic Kernel... etc) This repository runs continuous testing against each of these repos.

NOTE All tests run against this repo are now required to work against a local Atlas deployment. See details below to ensure proper setup.

Each AI/ML pipeline is sorted by the composite of the name of the library, and the driver language the library is implemented in. This comes out in the format{pipeline}-{language} -->semantic-kernel-python. All tests should be scoped within the bounds of these subdirectories.

Each subdirectory is scoped to run only one AI/ML integration's suite of tests for one language within that cloned repository. For example, if an AI/ML integration has both a Python and C# implementation of Atlas Vector Search, two subdirectories need to be made: one for Python, titled{repo}-python, and one for C#, titled{repo}-csharp. Seesemantic-kernel-* subdirectories in the layout example below.

Within each subdirectory you should expect to have:

• run.sh -- A script that should handle any additional library installations and steps for executing the test suite. This script should not populate the Atlas database with any required test data.
• config.env - A file that defines the following environment variables:
  • REPO_NAME -- The name of the AI/ML framework repository that will get cloned
  • REPO_ORG -- The Github org of the repository
  • REPO_BRANCH -- The optional branch to clone
  • DATABASE -- The optional database where the Atlas CLI will load your index configs
• database/ -- An optional directory used by.evergreen/scaffold_atlas.py to populate a MongoDB database with test data. Only provide this if your tests require pre-populated data.
• database/{collection}.json -- An optional JSON file containing one or more MongoDB documents that will be uploaded to$DATABASE.{collection} in the local Atlas instance. Only provide this if your tests require pre-populated data.
• indexConfig.json -- An optional file containing configuration for a specified Atlas Search Index.
• Additionally, you can add other useful files, including.env files, if required by your tests.

The general layout of this repo looks like this:

├── LICENSE# License Agreement├── README.md# This Document├── langchain-python# Folder scoped for one Integration│   └── run.sh# Script that executes test├── semantic-kernel-csharp# Folder scoped for one Integration│   ├── database# Optional database definition directory│   │   └── nearestSearch.json# Populates $DATABASE.nearestSearch│   │   └── furthestSearch.json# Populates $DATABASE.furthestSearch│   ├── indexes# Optional Index definitions directory│   │   └── indexConfig.json# Optional Search index definition|   ├── config.env# Configuration file│   └── run.sh# Script that executes test|├── semantic-kernel-python# Folder scoped for one Integration│   ├── database# Optional database definition│   │   └── nearestSearch.json# Populates $DATABASE.nearestSearch│   │   └── furthestSearch.json# Populates $DATABASE.furthestSearch│   ├── indexConfig.json# Creates Search Index on $DATABASE|   ├── config.env# Configuration file│   └── run.sh# Script that executes test

Each test subdirectory will automatically have its own local Atlas deployment. As a result, database and collection names will not conflict between different AI/ML integrations. To connect to your local Atlas using a connection string,utils.sh has afetch_local_atlas_uri that you can call from therun.sh script within your subdirectory. For example:

. .evergreen/utils.shCONN_STRING=$(fetch_local_atlas_uri)

Stores the local Atlas URI within theCONN_STRING var. The script can then passCONN_STRING as an environment variable to the test suite.

We can run the tests with a local checkout of the repo.

For example, to run thedocarray tests using local atlas:

export DIR=docarraybash .evergreen/fetch-secrets.shbash .evergreen/fetch-repo.shbash .evergreen/provision-atlas.shbash .evergreen/execute-tests.sh

Use.evergreen/setup-remote.sh instead of.evergreen/provision-atlas.sh to test against the remote cluster.

You can pre-populate a test's local Atlas deployment before running therun.sh script by providing JSON files in the optionaldatabase directory of the created subdirectory. The.evergreen/scaffold_atlas.py file will search for every JSON file within this database directory and upload the documents to the database provided by theDATABASE expansion provided in the build variant of the.evergreen/config.yml setup. The collection the script uploads to is based on the name of your JSON file:

• critical_search.json withDATABASE: app will upload all of its contents toapp.critical_search collection.

To create a search index, provide the search index configuration in theindexes dir; the file needs to be a json and for each individual index you wish to create, you must make a separate json file; for easier readability, please name each index configuration after the collection the index will be defined on or the name of the index itself. If multiple indexes will define one collection or multiples collections within a database share the name index name, append a name "purpose" (test_collection_vectorsearch_index) to favor readability. The evergreen script will then create the search index before therun.sh script is executed. See the"Create an Atlas Search Index and Run a Query" documentation instructions on how to create a search index using Atlas CLI.

If you need more customized behavior when populating your database or configuring your local Atlas deployment, include that behavior in yourrun.sh script.

Test execution flow is defined in.evergreen/config.yml. The test pipeline's config is structured as follows:

Build variants -- This is the highest granularity we will use to define how and when a test pipeline will run. A build variant should only ever be scoped to service one test pipeline. There can be multiple tasks run within a build variant, but they should all only scope themselves to a singular test pipeline in order to maintain an ease of traceability for testing.

• name -- This should be in the formattest-{pipeline}-{language}-{os}

• display_name -- This can be named however you see fit. Ensure it is easy to understand. See.evergreen/config.yml for examples

• expansions -- Build variant specific variables. Expansions that need to be maintained as secrets should be stored inthe Evergreen project settings usingvariables. Some common expansions needed are:

  • DIR -- The subdirectory where the tasks will run

• run_on -- Specified platform to run on.rhel8.9-small orubuntu2204-small should be used by default. Any other distro may fail Atlas CLI setup.

• tasks -- Tasks to run. See below for more details

• cron -- The tests are run via a cron job on a nightly cadence. This can be modified by setting a different cadence. Cron jobs can be scheduled usingcron syntax

• tags -- This should include the language where the AI/ML is run. i.e.[python, csharp, golang, javascript] Any tagged language will populate theappropriate language-specific slack channel.

Tasks -- These are the "building blocks" of our runs. Here is where we consolidate the specific set of functions. The basic parameters to add are shown below

• name -- This should be in the formattest-{pipeline}-{language}
• commands -- See below.

Functions -- We've defined some common functions that will be used. See the.evergreen/config.yml for example cases. The standard procedure is to fetch the repository, provision Atlas as needed, and then execute the tests specified in therun.sh script you create. Ensure that the expansions are provided for these functions, otherwise the tests will run improperly and most likely fail.

• fetch repo -- Clones the library's git repository; make sure to provide the expansion REPO_ORG/REPO_NAME and REPO_BRANCH (optional)
• execute tests -- Usessubprocess.exec to run the providedrun.sh file.run.sh must be within the specifiedDIR path.
• fetch source -- Retrieves the current (ai-ml-pipeline-testing) repo
• setup atlas cli -- Sets up the local Atlas deployment

For better or worse, we do not maintain AI/ML libraries with which we integrate.We provide workarounds for a few common issues that we encounter.

As we develop a testing infrastructure, we commonly make changes to our integrations with the third-party library.This is the case, in particular, when we add a new integration.Over time, we may make bug fixes, add new features, and update the API.At the start, we will hopefully add the integration tests themselves.

The bad news is that the maintainers of the AI/ML packages may take considerabletime to review and merge our changes. The good news is that we can begin testingwithout pointing to the main branch of the upstream repo.We can useREPO_ORG,REPO_NAME, and an optionalREPO_BRANCH to define which repo to clone.As such, we can point to an arbitrary branch on an arbitrary repo.While developing, we encourage developers to point to a feature branchon their own fork, and add a TODO with the JIRA ticket to update the urlonce the pull-request has been merged.

We provide a simple mechanism to make changes to the third-party packageswithout requiring a pull-request (and acceptance by the upstream maintainers).This is done via Git Patch files.

Patch files are created very simply:git diff > mypatch.patch.If you can believe it, this was the primary mechanism to share code with another maintainerbefore pull-requests existed!To apply patches, add them to apatches directory within the$DIR of your build variant.As of this writing, thechatgpt-retrieval-plugin contains an example that you may use as a reference.You can create a number of different patch files, which will be applied recursively.This is useful to describe rationale, or to separate out ones that will be removedupon a merged pull-request to the upstream repo.

During ChatGPT Retrieval Plugin integration, we ran into build issues on Evergreen hosts.In this case, the package failed to build from source.It required a library that wasn't available on the host and had no wheel on PyPI.As it turned out, the package was actually an optional requirement,and so a one-line change topyproject.toml solved our problem.

We realized that we could easily get this working without changing the upstreamsimply by applying a git patch file.This is a standard practice used byconda package maintainers,as they often have to build for a more broad set of scenarios than the original authors intended.

Rather than making a new branch and modifying aconfig.env file, you can run a patch build as follows:

evergreen patch -p ai-ml-pipeline-testing --param REPO_ORG="<my-org>" --param REPO_BRANCH="<my-branch>" -y -d"<my-message>"

For example

evergreen patch -p ai-ml-pipeline-testing --param REPO_ORG=caseyclements --param REPO_NAME="langchain-mongodb" --param REPO_BRANCH="INTPYTHON-629" -y -d"Increased retries to 4."

Tests are run periodically (nightly) and any failures will propagate into both thedbx-ai-ml-testing-pipline-notifications anddbx-ai-ml-testing-pipeline-notifications-{language} channel. Repo owners of thisai-ml-testing-pipeline library are required to join thedbx-ai-ml-testing-pipeline-notifications. Pipeline specific implementers mustat least joindbx-ai-ml-testing-pipline-notifications-{language} (e.g. whomever implementedlangchain-js must at least be a member ofdbx-ai-ml-testing-pipeline-notifications-js).

If tests are found to be failing, and cannot be addressed quickly, the responsible team MUST create a JIRA ticket within their team's project (e.g. a python failure should generate anINTPYTHON ticket), and disable the relevant testsin theconfig.yml file, with a comment about the JIRA ticket that will address it.

This policy will help ensure that a single failing integration does not cause noise in thedbx-ai-ml-testing-pipeline-notifications ordbx-ai-ml-testing-pipeline-notifications-{language} that would mask otherfailures.