Nomic Embeddings and Their Integration with LlamaIndex

Understanding textual content is essential in the field of artificial intelligence. Text embeddings, numerical representations that capture semantic relationships, are crucial to do this. Text retrieval may be fully realized with the help of Nomic Embeddings, which Nomic AI invented, and LlamaIndex, a potent indexing tool. These embeddings capture the semantics and meanings of data, making them indispensable for search, classification, and clustering tasks. It supports text and image data, Nomic Embeddings offers a unified approach to handling diverse data types. Their flexibility, which includes local and dynamic inference modes, ensures optimal performance and reduced latency for various applications. In this article, we will go through Nomic Embeddings and how to integrate it with LlamaIndex to retrieve a document.

Table of Contents

1. Understanding Nomic Embeddings
2. Key Features of Nomic Embeddings
3. Nomic Embeddings and LlamaIndex
4. Using Nomic Embeddings in Retrieval Process with LlamaIndex
5. Benefits of Integration

Let us now understand Nomic Embeddings and its Key Features. Later, we will also look into its integration with LlamaIndex.

Understanding Nomic Embeddings

Nomic Embeddings is a powerful open-source. This model is designed by Nomic to handle large language models. These are advanced vector representations of data for tasks requiring semantic understanding, such as search, classification, and clustering. These embeddings can be generated for text and images, enabling applications in various AI-driven domains. This offers high performance comparable to OpenAI’s embedding models. Unlike simpler models, Nomic Embeddings handle complex language structures with impressive accuracy. Their adaptability ensures context-sensitive embeddings, leading to more precise retrieval results.

Key Features of Nomic Embeddings

Multimodal Capabilities

Nomic Embedding can be used for text and image data, providing a unified embedding space for different applications​.

Local and Dynamic Inference

By leveraging GPT4All, Nomic allows embeddings to be generated locally on user machines. This reduces dependency on remote servers and improves efficiency. The dynamic inference mode intelligently chooses between local and remote processing based on input size and complexity​​.

Task-Specific Embeddings

Nomic offers specialized embeddings for various tasks such as document retrieval, search queries, classification, and clustering, ensuring high performance tailored to specific use cases​.

Adjustable Dimensionality

Users can customize the dimensionality of embeddings, balancing computational requirements with performance needs. This feature is for applications with different resource constraints​​.

Integration with Nomic Atlas 

Nomic Embeddings are integrated into Nomic Atlas, a platform for managing and exploring unstructured data at scale, aiding in tasks like anomaly detection and refinement​​.

Nomic Embeddings and LlamaIndex

LlamaIndex, formerly known as GPT Index, is a framework designed to interface with Large Language Models (LLMs) for efficient data indexing and retrieval. Integrating Nomic Embeddings with LlamaIndex can significantly enhance the capabilities of LLM-based applications by providing robust semantic understanding and efficient data handling.

The synergy between Nomic Embeddings and LlamaIndex empowers a variety of applications:

Semantic Search

It is like searching a massive document corpus for a specific concept, not just keywords, and Nomic embeddings capture the essence of our query, while LlamaIndex swiftly retrieves documents that align semantically.

Question Answering Systems

Nomic Embeddings ensures that the system grasps the meaning and intention behind a question, while LlamaIndex delivers the most relevant answers from the knowledge base. This helps to achieve high accuracy in building a question-answering system. 

Chatbots and Virtual Assistants

These interactive agents rely on understanding user queries. Nomic Embeddings empower them to interpret the true meaning of a user’s request, and LlamaIndex facilitates the retrieval of the most helpful response. 

Using Nomic Embeddings in Retrieval Process with LlamaIndex

In this section, we will use Nomic Embeddings to embed text data and retrieve a response to a query using LlamaIndex. The retrieval process includes a knowledge base, retriever, and query. The knowledge base is the vector-embedded vessel of an input data or document. The vector embeddings are formed using Nomic Embeddings. A retrieval agent is built using LlamaIndex which retrieves a suitable and most efficient response.

To begin with, install all the required packages and libraries. 

pip install -U llama-index llama-index-embeddings-nomic

import nest_asyncio
nest_asyncio.apply()

from llama_index.embeddings.nomic import NomicEmbedding
from llama_index.core import settings
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
from llama_index.llms.openai import OpenAI
import os

Next, call the Nomic API key and OpenAI API Key to the environment.

nomic_api_key = "********"
os.environ["OPENAI_API_KEY"] = "sk-*******"

Build an embedding model using NomicEmbedding(). Here we will be using “nomic-embed-text-v1.5” model of Nomic. We will also build our LLM using “gpt-3.5-turbo”.

embed_model = NomicEmbedding(
   api_key=nomic_api_key,
   dimensionality=768,
   model_name="nomic-embed-text-v1.5",
)

llm = OpenAI(model="gpt-3.5-turbo")

Now, read the input document you are using. 

documents = SimpleDirectoryReader("./Document").load_data()

And, we are going to need the service_context attribute.

service_context = ServiceContext.from_defaults(
   embed_model=embed_model, chunk_size=1024,
)

Now, store the vectors in a vessel using VectorStoreIndex. 

index = VectorStoreIndex.from_documents(
   documents=documents, service_context=service_context, show_progress=True
)

Next, we will build the retrieval agent – a query engine that will retrieve a suitable context based on the query given by the agent. 

query_engine = index.as_query_engine(documents=documents,service_context=service_context, similarity_top_k=1)

Give a query to the query engine. This will convert the query to vector embedding and search for the most related response for this query in the knowledge base.

query = query_engine.retrieve(
   "Describe the great hall of Hogwarts?"
)
query

The output of this query will be something like this:

Thus, by using Nomic Embedding we can retrieve a very suitable response for the query given. Nomic Embeddings with its capabilities of embedding text and image data and high performance in local and dynamic interfaces has made retrieval process, classification, and clustering more efficient.

Benefits of Integration

Integrating Nomic Embeddings with LlamaIndex brings several advantages:

Enhanced Semantic Search

Leverage the high-quality embeddings from Nomic to improve the accuracy and relevance of search results.

Efficient Data Handling

Benefit from LlamaIndex’s optimized indexing and retrieval mechanisms, especially when dealing with large datasets.

Scalability

The combination of local and dynamic inference modes ensures that your applications can scale efficiently, handling varying loads and data sizes seamlessly.

Flexibility

Adjustable dimensionality and task-specific embeddings allow for fine-tuning based on the specific needs of your application.

Conclusion

Nomic Embeddings is a powerful open-source text embedding model that offers excellent performance and flexibility. Its integration with LlamaIndex through the llama-index-embeddings-nomic package makes it easy to incorporate into your knowledge management workflows. With support for variable-sized embeddings, matryoshka learning, and long context lengths, Nomic Embeddings is a valuable tool for working with large language models and complex data.

References

1. LlamaIndex-Nomic Embeddings Documentation
2. Nomic-Atlas
3. Link to Code

Join the following course to learn more about Vector Database, Vector search and Purple Llama.