Automated Cyclic Logic
On August 19th, three researchers from the University of British Columbia published a paper titled "Automated Design of Agent Systems". In this paper, they designed a system that allows AI to discover and build agents on its own, and even iterate itself.
Recall OpenAI's classic definition of an agent: a product that can store knowledge, plan, and apply tools.
When we build agents using workflows, we also utilize existing knowledge (knowledge about agent forms), plan ourselves (build processes), and use tools (connect APIs) to execute and produce output, without exceeding the capabilities of the agent itself.
So why not build an agent that can automatically discover and design other agents?
The paper's authors followed this line of thinking, calling the designer a meta-agent, and having it design new agents. The designed agents are added to a database as resources, continuously iterating to produce new, stronger versions of agents.
They call this entire method ADAS (Automated Design of Agent Systems).
Setting the Chain in Motion
The process of generating new agents in the ADAS system can be divided into three parts:
The first part establishes the search space, which can be understood as designing potential new agents through some basic tools and rules.
The second part is running the search algorithm, which specifies how the meta-agent uses the search space to specifically build new agents using elements within it.
The final part is running the evaluation function, which assesses the built agents based on performance and other objectives.
The researchers explained step-by-step in the paper how to construct these three core parts.
First, they needed to determine the basic elements for constructing the search space. The researchers believed that code was the best method.
This is because code has Turing completeness and can express all possibilities. Therefore, theoretically, the meta-agent can discover any possible building blocks (such as prompts, tool usage, control processes) and agent systems that combine these building blocks in any way.
More importantly, various workflows that already exist in websites like Langchain for building agents have already been coded. Therefore, relevant data is readily available without needing conversion. And tool calls, such as capability components like RAG (retrieval augmented generation), already have a very sufficient code base.
Using code to construct the search space also means that agents generated by ADAS can be run directly for error correction and scoring, without the need for human intervention.
After defining the search space, the researchers began to design the search algorithm, that is, letting the meta-agent explore possible methods to complete tasks. This process is basically completed through prompt engineering.
First, it is given a series of system prompts.
Then, the materials mentioned in the basic prompt are given to the meta-agent, including:
-
Basic description of the task.
-
The most basic framework code, such as formatting prompts, encapsulation and operation names, as well as the ability to call other foundation models (FM) and APIs.
-
Format and examples of task input and output.
-
An example library formed by some agents generated in previous iterations, including their baseline test results.
Based on these basic prompts, ADAS can start operating and generate an agent to solve specific problems.
In this process, the meta-agent will conduct two rounds of reflection to ensure the novelty and correctness of the generated agent. It will check whether the newly generated code framework has errors and whether it has sufficient innovation compared to past results in the agent example library.
The meta-agent also needs to conduct a preliminary evaluation of the generated agent based on its "understanding" of practical performance capabilities, judging whether it performs better than previous agents.
When some of these conditions are not met, the meta-agent must either modify the generated agent or completely remake it.
After two rounds of reflection and modification, the meta-agent will send the newly generated agent that it considers both new and good to the third step evaluation system.
The evaluation system will give an objective score to this agent's ability based on baseline tests, and then return it to the agent example library.
The meta-agent will then continue the next iteration optimization based on the previous scoring results and past examples in the example library, to achieve a higher task performance score.
A fully automated chain is born in this process. And to generate an excellent agent may require more than ten iterations.
Surpassing Handcrafted Agents
How complex can agents obtained through the ADAS method automated process be? The figure below shows the framework of an agent generated after 14 iterations.
In the structure of this agent, there are five chain-of-thought processes giving preliminary answers. After three expert models and one model imitating human evaluation give feedback on these answers, these answers will undergo three modifications and enhancements in the optimization process stage. Finally, three results are selected through evaluation and merged to give the final answer.