How to Build a Multi-Agent System With AutoGen?

Large Language Models (LLMs) are big players in AI, excelling at tasks like writing, translating, and creating content. They’re especially useful for solving tough problems that need special knowledge and skills. Enter AutoGen-powered Multi-Agent LLM systems, which use different methods and feedback to help AI assistants understand text better and make it sound more like how humans talk. This article talks about Multi-Agent LLMs, what AutoGen can do, and how to build systems that boost human productivity with AutoGen.

Table of Contents

1. Understanding Multi-Agent LLM System
2. Introducing AutoGen
3. Orchestrating Multi-Agent LLM Conversations
4. Applications of Multi-Agent LLMs with AutoGen
5. Building a Sequential Chat Using AutoGen

Let’s start with an introduction to the Multi-Agent LLM system.

Understanding Multi-Agent LLM System

Multi-Agent Systems (MAS) is a part of AI that studies systems that consist of many intelligent agents interacting among themselves. These agents can be represented by robots, software programs, or even human beings working together. Every one of them possesses its own goals, perceptions, and decision-making abilities. The aim of each of these agents through interaction and communication is either to reach a common goal or to adjust to changes in the environment.

The core aspects of Multi-Agent LLMs:

Diversity is Key

Machine Learning ensembles usually use similar models to perform tasks with slight differences, whereas Multi-Agent LLMs use agents with different skills. This difference helps them solve complex problems that require a range of expertise.

Specialized Agents

In a Multi-Agent LLM system, each agent is trained for a specific task, which makes them highly efficient in their work. When these agents are combined, they contribute their individual expertise to solve complex problems.

Communication is Key

Effective collaboration among agents requires good communication. This is achieved through natural language processing capabilities embedded within each agent. They can share information, ask for help from other agents with specific expertise, and collectively work towards a solution.

Source: University of Adelaide

Introducing AutoGen

AutoGen is the controller of LLM agents working together in a Multi-Agent LLM system. It’s an Open-Source Framework designed to support communication, collaboration, and task execution within this collaborative environment. 

Here are some of the elements that make AutoGen powerful:

Conversational Agents

AutoGen’s Conversational Agents use natural language dialogues, which helps them exchange information, seek help from other expert agents, and engage in shared reasoning. AutoGen ensures a smooth flow of information and a good task-based conversation.

Integrating External Tools

We can integrate external tools through AutoGen, which allows external tools and APIs to be incorporated into the conversation flows. This is one reason for the well-working Multi-Agent LLM system. Because of this, these MAS can not only communicate amongst themselves but also use external resources such as databases, code repositories, or specific software applications.

Human-in-the-Loop Integration: 

AutoGen also allows human input into the conversation flow. In this, humans can provide guidance, impart domain knowledge (that might not be readily available to LLM agents), or interfere when the agents reach saturation. This builds a mutual understanding between humans and agents, and human guidance is ensured at all stages of the process, especially during critical decision-making scenarios.

Flexibility and Customization

AutoGen provides developers with a great deal of flexibility. They can define conversation patterns, customize agent behaviors, and even specify how different stages of a task should be solved by different agents. This enables developers to create very specific LLM applications. Additionally, AutoGen allows for reinforcement learning techniques integration where they learn from their interactions and become better over time in collaborative problem-solving skills.

Orchestrating Multi-Agent LLM Conversations

AutoGen is more than a framework for helping LLM agents negotiate. It really comes down to managing conversations, coordinating tasks, and completing tasks, making it a powerful platform for collaborative problem-solving.

Let’s take a closer look at how it works:

Agent Communication Protocols

AutoGen can set clear communication rules for agents. These rules ensure that information is exchanged efficiently, avoid misunderstandings, and keep the conversation flowing smoothly.

Source: AutoGen

Conversation Patterns

AutoGen has several different conversation patterns that developers can use to customize how agents interact for specific tasks. These patterns can be as simple as a two-agent chat or as complex as a Multi-Agent workflow that involves sharing information, assigning tasks, and working together to solve problems. 

Some important conversation patterns to consider are:

• Two-Agent Chat: This basic pattern allows two agents to exchange information and collaboratively complete tasks. For example, one agent might provide factual information while another analyzes it to draw conclusions.

• Sequence of Two-Agent Chats: This pattern extends the two-agent chat by allowing the conversation to be passed on to other agents sequentially. Each agent contributes its expertise, building upon the previous interactions. This is useful for tasks requiring multiple stages or diverse skill sets.

• Group Chat: This pattern allows communication between multiple agents within a single conversation thread, which allows for analyzing and collaborating on complex problems.

Source: AutoGen

Agent Management and Coordination

AutoGen provides mechanisms for managing and coordinating the actions of various agents within the system.

This includes:

• Agent Selection: AutoGen can determine which agent is best suited to participate in a conversation based on the current context of the task and the expertise of each agent.
• Turn-Taking: AutoGen ensures a smooth flow of conversation by establishing turn-taking protocols. These protocols prevent agents from talking over each other and maintain order within the Multi-Agent dialogue.
• Conflict Resolution: AutoGen can identify potential conflicts arising from agent disagreements and employ strategies to resolve them, such as voting mechanisms or escalation to a human supervisor.

Integration with External Resources

AutoGen allows developers to integrate external tools and APIs into the conversation flow. This empowers agents to access and leverage resources beyond their individual knowledge base. Examples include:

• Database Access: Agents can query databases to retrieve relevant information for the task at hand.
• Code Execution: Agents can interact with code repositories and execute specific code snippets for calculations or data analysis.
• API Calls: Agents can increase their capacity by using APIs of various services, such as translation and weather data feeds.

AutoGen provides a powerful framework for orchestrating complex conversations between LLM agents, enabling them to collaborate, share knowledge, and solve challenges beyond the scope of individual models. Its diverse conversation patterns, agent control properties, and integration with external resources make it a versatile tool for exploring the full potential of Multi-Agent LLMs. As this technology continues to evolve, we can expect even more innovative applications and advancements in the field of Multi-Agent LLMs.

Applications of Multi-Agent LLMs with AutoGen

The applications of Multi-Agent LLMs with AutoGen are vast. 

Here are a few exciting applications:

Scientific Discovery

Imagine a group of LLM agents who are experts in reviewing the literature, skilled in data analysis, and have some skills in creating hypotheses. This group can increase the pace, efficiency, and accuracy of scientific discovery by reviewing many research papers, analyzing them, and considering all the possibilities. 

Personalized Learning

Multi-agent LLMs create intelligent tutoring systems. One agent might pinpoint gaps in students’ knowledge, another would tailor explanations based on learning style, and a person could intervene for personalized guidance.

Complex Problem-Solving

Multi-agent LLMs address difficult problems in engineering or design. One agent may generate designs, while the other can assess whether the concepts are feasible, and another could evaluate economic viability. In this way, collaboration can lead to better ways of doing things, which are more efficient in terms of time and innovation.

Building a Sequential Chat Using AutoGen

Building a sequence chat using AutoGen is a great way to explore the capabilities of Multi-Agent LLMs.

In this experiment, we will build three AssistantAgents and one User Agent.  We will give each agent a math problem and ask them to solve it in their own way. The chat will happen in a sequence.

First of all, begin with installing pyAutoGen in your environment and import AutoGen

import autoGen

Now, create a config_list.json file and add the LLM configurations to it.

[
   {
       "model": "gpt-3.5-turbo",
       "api_key": "sk-***"
   }
]

Now, call this config_list.json file to our Python notebook. config_list_from_json is an AutoGen attribute that we use to call a JSON file.

config_list = autoGen.config_list_from_json(
   "confi_list.json"
)

Create a llm_config variable where we can store all the configurations for LLM.

llm_config = {
   "timeout": 600,
   "cache_seed": 42,
   "config_list": "config_list",
   "temperature": 0,
}

To start the conversation, we have to create our agents first.

Create three Assistant Agents, 

1. Assistant1 is the first agent to solve a mathematical problem that we will give it. 
2. Assistant2 is the second agent, who will solve the same problem in a different way. 
3. Assistant3 is the third agent, who will try to find a new solution by combining the solutions of the other two agents.

assistant1 = AutoGen.AssistantAgent(
   name="assistant1",
   system_message="You are an assistant agent who gives solution.  Return 'TERMINATE' when the task is done.",
   llm_config={"config_list":config_list},
)


assistant2 = AutoGen.AssistantAgent(
   name="assistant2",
   system_message="You are another assistant agent who gives solution.  Return 'TERMINATE' when the task is done.",
   llm_config={"config_list":config_list},
   max_consecutive_auto_reply=1
)


assistant3 = AutoGen.AssistantAgent(
   name="assistant3",
   system_message="You will create a new solution based on others.  Return 'TERMINATE' when the task is done.",
   llm_config={"config_list":config_list},
   max_consecutive_auto_reply=1
)

Once all the agents are set, we can initiate the chat. The chat will be sequential, and to achieve it, we have to pair the user agent with each of the assistant agents. 

AutoGen’s sequence chat allows you to structure conversations between multiple LLMs in a specific order. Imagine a relay race. Each LLM agent takes a turn, receiving information from the previous agent, processing it, and then passing it on to the next agent in the sequence. This continues until the final agent completes the task or reaches a point where human intervention is needed.

user_proxy = AutoGen.UserProxyAgent(
   name="user_proxy",
   is_termination_msg=lambda msg: msg.get("content") is not None and "TERMINATE" in msg["content"],
   human_input_mode="NEVER",
   max_consecutive_auto_reply=10,
   code_execution_config=False
)

The output will be something like this,

The Future of AI Collaboration

Multi-Agent LLMs with AutoGen have contributed to a significant amount of development in the field of AI. These Multi-Agents can solve any complex challenges that are beyond the reach of individual LLMs because of their collaborative property and diverse skill sets. As the field continues to evolve, we can expect even more innovative applications to emerge, ushering in a new era of human-AI collaboration.

References

1. AutoGen
2. Link to the above code

For further studies on Building Multi-Agent LLMs with AutoGen, join the below course.