An index created by GPT to organize external information and answer queries!

🗂️ ️GPT Index

GPT Index is a project consisting of a set of data structures that are created using GPT-3 and can be traversed using GPT-3 in order to answer queries.

🚀 Overview

Context

• GPT-3 is a phenomenonal piece of technology for knowledge generation and reasoning.
• A big limitation of GPT-3 is context size (e.g. Davinci's limit is 4096 tokens. Large, but not infinite).
• The ability to feed "knowledge" to GPT-3 is restricted to this limited prompt size and model weights.
• Thought: What if GPT-3 can have access to potentially a much larger database of knowledge without retraining/finetuning?

Proposed Solution

That's where the GPT Index data structures come in. Instead of relying on world knowledge encoded in the model weights, a GPT Index data structure does the following:

• Uses a pre-trained GPT-3 model primarily for reasoning/summarization instead of prior knowledge.
• Takes as input a large corpus of text data and build a structured index over it (using GPT-3 or heuristics).
• Allow users to query the index in order to synthesize an answer to the question - this requires both traversal of the index as well as a synthesis of the answer.

The high-level design exercise of this project is to test the capability of GPT-3 as a general-purpose processor to organize and retrieve data. From our current understanding, related works have used GPT-3 to reason with external db sources (see below); this work links reasoning with knowledge building.

💻 Example Usage

pip install gpt-index

Examples are in the examples folder. Indices are in the indices folder (see list of indices below).

To build a tree index do the following:

from gpt_index import GPTTreeIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader('data').load_data()
index = GPTTreeIndex(documents)

To save to disk and load from disk, do

# save to disk
index.save_to_disk('index.json')
# load from disk
index = GPTTreeIndex.load_from_disk('index.json')

To query,

index.query("<question_text>?", child_branch_factor=1)

🔧 Dependencies

The main third-party package requirements are transformers, openai, and langchain.

All requirements should be contained within the setup.py file. To run the package locally without building the wheel, simply do pip install -r requirements.txt.

Index Data Structures

• Tree Index: Tree data structures
  • Creation: with GPT hierarchical summarization over sub-documents
  • Query: with GPT recursive querying over multiple choice problems
• Keyword Table Index: a keyword-based table
  • Creation: with GPT keyword extraction over each sub-document
  • Query: with GPT keyword extraction over question, match to sub-documents. Create and refine an answer over candidate sub-documents.
• List Index: a simple list-based data structure
  • Creation: by splitting documents into a list of text chunks
  • Query: use GPT with a create and refine prompt iterately over the list of sub-documents

Data Connectors

We currently offer connectors into the following data sources. External data sources are retrieved through their APIs + corresponding authentication token.

• Notion (NotionPageReader)
• Google Drive (GoogleDocsReader)
• Slack (SlackReader)
• MongoDB (local) (SimpleMongoReader)
• Wikipedia (WikipediaReader)
• local file directory (SimpleDirectoryReader)

Example notebooks of how to use data connectors are found in the Data Connector Example Notebooks.

🔬 Related Work [WIP]

Measuring and Narrowing the Compositionality Gap in Language Models, by Press et al.

• Introduces a self-ask paradigm, which forces the model to ask and answer followup questions before answering the original question. Similar to GPT Index in that it uses GPT to reason through subproblems; the difference is that the GPT Index also tries to organize the external information as opposed to being trained on it.
• Example (from Langchain)

ReAct: Synergizing Reasoning and Acting in Language Models, by Yao et al.

• Introduces a joint reasoning and action framework in an interleaved manner. This approach of connecting to external knowledge sources is similar to our approach of having GPT traverse an externally stored index of data. ReAct has much more fluid/sophisticated ways of traversal (e.g. search, lookup, finish), whereas this project just tries building an index with simple tree-based traversal.

Please let me know if there are other related works - I am not up-to-date on the latest NLP/LLM ArXiv papers or Github projects. I am happy to give references/credit below.