The integration of vector search into ElasticSearch has bridged the gap between conventional search engines and AI-powered semantic understanding. ElasticSearch now enables users to implement search solutions with new possibilities for creating more intelligent, context-aware applications. ElasticSearch’s vector search capabilities, combined with Kibana’s visualisation provide a comprehensive approach to vector search capabilities under information retrieval. This article explores ElasticSearch through a practical approach.
Table of Contents
- Understanding ElasticSearch
- Key functionalities of ElasticSearch
- Introduction to Kibana data monitoring and analytics
- Hands-on implementation of ElasticSearch through Llama-Index
Understanding ElasticSearch
ElasticSearch is a powerful and versatile search and analytics engine built on top of Apache Lucene and is designed to handle massive amounts of data while providing near real-time search capabilities. Unlike traditional search, Elasticsearch specialises in making data easily searchable and analysable. It includes a vector database for vector search enabling users to build their own vector search engines.
ElasticSearch is the heart of the Elastic stack, combined with Kibana, it powers different Elastic solutions – Search, Observability, Security and Analytics. The system operates by storing data as JSON documents, which offers great flexibility since there is no need for a predefined schema. These documents are organised into indices. When a user stores a document in ElasticSearch, the system automatically distributes it across multiple containers called shards, which can be spread across different servers or nodes. This distributed nature allows ElasticSearch to scale horizontally and handle large volumes of data efficiently.
ElasticSearch utilises the concept of clusters which is a collection of one or more nodes, working together to store data and perform search and indexing operations. An index is a collection of documents that share similar properties where each document is stored as a JSON object.
The document in a node is a single instance of data stored in an index. Each document includes one or more fields that contain data. Fields can be of different data types, such as numeric or text. An index is a fundamental unit of storage in ElasticSearch, a logical namespace for storing data that share similar properties. These indexes are uniquely identified by a name or an alias. This unique alias is used in search queries and other relevant operations under ElasticSearch.
ElasticSearch allows the users two different options for mapping indices – dynamic and explicit. In dynamic mapping, ElasticSearch automatically detects the data types and creates the mappings whereas, in explicit mapping, the user has to define the mappings by specifying data types for each field. ElasticSearch clusters can be managed, indexed and searched using REST API, making it relatively easy for users to integrate ElasticSearch into their applications.
Query languages such as Query DSL, and ES|QL can be used to interact with user data. Query DSL is the primary language whereas ES|QL is a piped query language and compute engine which was added in version 8.11. Apart from Query DSL and ES|QL, EQL, ElasticSearch SQL and Kibana Query Language can also be used as per the user’s requirement.
ElasticSearch provides an AI Playground where the users can interact with their data and explore building RAG systems. They can test different LLMs from providers like OpenAI, Amazon Bedrock, Anthropic and more with ease and efficiency. The multimodal search capabilities of ElasticSearch can be utilised to perform similarity searches based on similar images, video clips and audio samples.
ElasticSearch supports a huge set of Gen AI and LLM features to ensure high-quality search performance. It supports traditional keyword and text-based search (BM25) and an AI-ready vector search with exact match and approximate kNN search capabilities. These features allow users to implement RAG as well on ElasticSearch through various integrations and orchestration platforms such as LlamaIndex.
Key functionalities of ElasticSearch
ElasticSearch exhibits the following key functionalities, as shown in the image below:
Introduction to Kibana for data monitoring and analytics
Kibana is the data visualisation and management platform built for ElasticSearch. It serves as a window into the Elastic Stack, providing a user-friendly interface to explore, visualise and manage the ElasticSearch data. The core purpose of Kibana is to make data visualisation accessible and intuitive. Users can create complex visualisations without writing code.
Kibana enables users to build charts and combine them into comprehensive dashboards that update in real time as the new data flows into ElasticSearch. Kibana offers powerful tools for managing ElasticSearch clusters. Users can write and test ElasticSearch queries, manage indices, and monitor cluster performance using Kibana’s web UI and console.
One of the most important use cases of Kibana is log analysis and monitoring. When combined with ElasticSearch and Logstash, Kibana excels at visualising log data, helping users to identify patterns, anomalies and potential issues in their systems. Users can also create dashboards showing error rates, response times, and system metrics based on crucial operational data.
Kibana also consists of machine learning capabilities and its platform is extensible through plugins, allowing users to create and manage machine learning jobs directly from the interface, enabling automated anomaly detection and forecasting without requiring high technical expertise. Users can make use of plugins to add custom functionality or integrate other tools or services into the Kibana platform making it adaptable to a wide range of use cases beyond its core capabilities.
Hands-on implementation of ElasticSearch through Llama-Index
Step 1: Set up and run ElasticSearch and Kibana locally using a start-local script, This script creates an elastic-start-local folder containing configuration files and starts both Elasticsearch and Kibana using Docker (Docker installation is a pre-requisite for this execution) –
curl -fsSL https://elastic.co/start-local | sh
This will generate a username and password for accessing ElasticSearch and Kibana UI’s on localhost based on the following endpoints.
- Elasticsearch: http://localhost:9200
- Kibana: http://localhost:5601
Output (Terminal):
ElasticSearch (Localhost Endpoint):
Kibana (Localhost Endpoint):
Step 2: The next step is to implement Llama-Index ElasticSearch Vector Store for document loading and response generation, let’s install and import the required libraries –
%pip install -qU llama-index-vector-stores-elasticsearch llama-index llama-index-llms-groq llama-index-embeddings-huggingface
from llama_index.core import (
Settings,
VectorStoreIndex,
SimpleDirectoryReader,
StorageContext
)
from llama_index.vector_stores.elasticsearch import ElasticsearchStore
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.groq import Groq
from dotenv import load_dotenv
import os
Step 3: Initialise the Groq API and setting LlamaIndex configuration parameters –
load_dotenv()
llm = Groq(model = "llama3-8b-8192")
embed_model = HuggingFaceEmbedding(model_name = "BAAI/bge-small-en-v1.5")
Settings.llm = llm
Settings.embed_model = embed_model
Step 4: Load the documents in our ElasticSearch vector store using the generated endpoint and API Key from Step 1 –
documents = SimpleDirectoryReader("data/").load_data()
vector_store = ElasticsearchStore(
es_url="http://localhost:9200", # see Elasticsearch Vector Store for more authentication options
index_name="got_data",
es_api_key="UV************************************************"
)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
documents, storage_context=storage_context
)
Step 5: Query the data to generate a response –
query_engine = index.as_query_engine()
response = query_engine.query("What is happening as per the data?")
print(response)
Output
A Game of Thrones takes place over the course of one year on or near the fictional continent of Westeros. The story begins when King Robert visits the northern castle Winterfell to ask Ned Stark to be his right-hand assistant, or Hand of the King.
Step 6: Visit the Web UI of Kibana (http://localhost:5601) to check the index created as per your loaded document –
Step 7: For implementing analytics, we will now create a Data View using Analytics Pane –
Step 8: Experiment and add more data to the vector store through LlamaIndex and create vector data-based visualisations to understand the data well. Users can also check Data Drift through the ML pane.
Final Words
The emergence of vector search capabilities in ElasticSearch is an important upgrade in the field of Gen AI and LLMs. By supporting vector embeddings and similarity-based queries, ElasticSearch is currently leading the vector database rankings on DBEngines. Vector search in ElasticSearch opens up remarkable possibilities for applications ranging from recommendation systems to semantic document retrieval. This integration of vector search in the ElasticSearch-Kibana ecosystem represents more than a technical feature and over time it will continue to evolve in the field of Gen AI.