Repository files navigation

UsingPinecone,LangChain +OpenAI for Generative Q&A withRetrieval Augmented Generation (RAG).

1. Setup the knowledge base (inPinecone)

• Chunk the content
• Create vector embeddings from the chunks
• Load embeddings into a Pinecone index

2. Ask a question

• Create vector embedding of the question
• Find relevant context in Pinecone, looking for embeddings similar to the question
• Ask a question of OpenAI, using the relevant context from Pinecone

[TODO - add diagram]

pip install -r ./setup/requirements.txt

cp dotenv .envvi .env

[TODO - non-splade example is working, update the splade example]

[TODO - show sample output]

[TODO - clean up requirements.txt or pipenv]

streamlit run streamlit-app.py

[TODO - show example screenshot]

[TODO - Update to support multiple PDFs]

[TODO - add customization of look & feel]

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., … Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, & H. Lin (Eds.),Advances in Neural Information Processing Systems (Vol. 33, pp. 9459–9474). Retrieved fromhttps://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf