NVIDIA Agent Intelligence toolkit is an open-source library for efficiently connecting and optimizing teams of AI agents. It focuses on enabling developers to quickly build, evaluate, profile, and accelerate complex agentic AI workflows?—?systems in which multiple AI agents collaborate to perform tasks.
The Agent Intelligence toolkit acts as a unifying framework that integrates existing agents, tools, and workflows across various platforms, such as LangChain, LlamaIndex, Semantic Kernel, and CrewAI. The toolkit treats these components as function calls, making them composable and reusable. Think of the Agent toolkit as a “conductor” for orchestrating a team of AI agents. In addition, it provides tools for profiling (tracking latency and token usage, for example), optimization, scaling, and observability, ensuring efficient performance of enterprise-grade agentic systems.
The core strength of Agent Intelligence toolkit is its extensibility. This post explains how to extend the toolkit by adding integration with an additional agentic framework, Agno.
Agno, formerly Phidata, is a lightweight library for building agents, particularly agents with multimodal capabilities. It exposes large language models (LLMs) as a unified API and gives them superpowers like memory, knowledge, tools, and reasoning. Agno has a growing developer ecosystem with over 26,000 GitHub stars, at the time of writing. Agno is model agnostic, lightning fast, and natively multimodal. For a full list of features, check out agno-agi/agno on GitHub.
Agent toolkit key features
Common features and terms in the Agent toolkit ecosystem include the following:
- Workflow: A workflow is the main entry point of an Agent toolkit application. It’s fully configured by a YAML configuration file, which specifies which entities (functions, LLMs, or embedders, for example) to use in the workflow, along with general configuration settings.
- Package: Modular components or plugins that extend the functionality of the toolkit, allowing it to integrate with various frameworks, tools, and workflows. These packages can be installed individually based on specific use cases or requirements, and they enable Agent toolkit to support diverse agentic workflows. For instance, a workflow like CrewAI’s multi-agent collaboration would use
agentiq-crewaipackage. - Plugin: The Agent toolkit features an extensible plugin system that enables you to add new tools, agents, workflows, and more to the library. The Agent toolkit plugin system is designed around two main concepts: entry points and decorators. Entry points allow the toolkit to discover plugins from any installed distribution package in a Python environment. Decorators enable developers to register their plugins with the library.
Figure 2 shows the plugins at the different levels of the toolkit.
Integrating Agno into the Agent Intelligence Toolkit
This section walks you through the steps to integrate Agno into the Agent toolkit. The same steps can be applied to integrate other agentic frameworks into the toolkit.
Step 0: Prerequisites
There are no specific GPU requirements for running Agent toolkit workflows when accessing NVIDIA NIM microservices. However, if you self-host a NIM, you will need the specific GPU or GPUs required for the chosen NIM.?For more details about the prerequisites for Agent toolkit and its installation steps, see NVIDIA/AIQToolkit on GitHub.
Step 1: Create a new Agent Toolkit package
Agent toolkit uses packages (located in the packages directory) to integrate with external frameworks. Create a new package, agentiq_agno, to connect Agno to the Agent toolkit. First, create a new folder named agentiq_agno under AgentIQ/packages/. Studying an existing package structure such as agentiq_crewai leads to a basic structure shown in Figure 3.
agentiq_agnoSecond, configure pyproject.toml, which defines the package and its dependencies:
[build-system]build-backend = "setuptools.build_meta"requires = ["setuptools >= 64", "setuptools-scm>=8"][tool.setuptools.packages.find]where = ["src"]include = ["aiq.*"][tool.setuptools_scm]root = "../.."[project]name = "agentiq-agno"dynamic = ["version"]dependencies = [ "agentiq", "agno~=1.2.3",]requires-python = ">=3.12"readme = "src/aiq/meta/pypi.md"description = "Subpackage for Agno integration in Agent toolkit"keywords = ["ai", "rag", "agents"]classifiers = ["Programming Language :: Python"][tool.uv]config-settings = { editable_mode = "compat" }[tool.uv.sources]agentiq = { workspace = true }[project.entry-points.'aiq.components']aiq_agno = "aiq.plugins.agno.register" |
Note the following in this pyproject.toml:
- Dependencies: A list of the dependency libraries such as
agentiq, andagno. - Project entry points: You scan the Python environment for entry points that have the name
aiq.components. The value of the entry point is a Python module that will be imported when the entry point is loaded. For this example, it points toaiq.plugins.agno.register, which means when theaiq-agnodistribution is installed, theaiq.plugins.agno.registermodule will be imported when the entry point is loaded. This module must contain all the@register_<plugin_type>decorators which need to be loaded when the library is initialized.
Then define the LLM client for integration using NVIDIA NIM. In the following code sample, the @register_llm_client decorator registers the LLM client for Agno integration, ready to be used by agentic workflows using Agno.
@register_llm_client(config_type=NIMModelConfig, wrapper_type=LLMFrameworkEnum.AGNO)async def nim_agno(llm_config: NIMModelConfig, builder: Builder): from agno.models.nvidia import Nvidia config_obj = { **llm_config.model_dump(exclude={"type", "model_name"}, by_alias=True), "id": f"{llm_config.model_name}", } if ("api_key" not in config_obj or config_obj["api_key"] is None): if ("NVIDIA_API_KEY" in os.environ): pass else: nvidai_api_key = os.getenv("NVIDIA_API_KEY") if (nvidai_api_key is not None): os.environ["NVIDIA_API_KEY"] = nvidai_api_key nvidia_args = {"id": config_obj.get("id")} if "base_url" in config_obj and config_obj.get("base_url") is not None: nvidia_args["base_url"] = config_obj.get("base_url") yield Nvidia(**nvidia_args) |
Next, register a tool wrapper for Agno. Tool wrappers are used to wrap functions in a way that is specific to an LLM framework. For this example, when using the Agno framework, Agent toolkit functions need to be wrapped in Agno’s Tool class to be compatible with Agno. Check out the source code for? agno_tool_wrapper.
Step 2: Install the new toolkit package
You can now install the newly created Agno package, along with its new plugins, by running the following command:
uv pip install -e '.[agno]' |
Step 3: Create a custom workflow using the newly supported agentic framework
The next step is to create a custom workflow of an agent using Agno. This example will use a personal finance agent that generates personalized financial plans using NVIDIA LLM NIM. It automates the process of researching, planning, and creating tailored budgets, investment strategies, and savings goals, empowering a user to take control of their financial future.
The Agent toolkit comes with a set of command line interfaces to manage the workflows. You can run the following command at the agentiq project root to automate the setup process by generating the necessary files and directory structure for our new workflow agno_personal_finance:
aiq workflow create --workflow-dir examples agno_personal_finance |
This command generates the following files and folders under the examples directory of agentiq:
agno_personal_financeThis structure is a starter template. Next, fill in the details.
pyproject.toml
[build-system]build-backend = "setuptools.build_meta"requires = ["setuptools >= 64", "setuptools-scm>=8"][tool.setuptools_scm]root = "../.."[project]name = "aiq_agno_personal_finance"dynamic = ["version"]dependencies = [ "agentiq[agno]", "openai~=1.66", "litellm~=1.63.14"]requires-python = ">=3.12"description = "Custom Workflow using Agno for personal finance"classifiers = ["Programming Language :: Python"][tool.uv.sources]agentiq = { path = "../..", editable = true }[project.entry-points.'aiq.components']aiq_agno_personal_finance = "aiq_agno_personal_finance.register" |
agno_personal_finance_function.py
This file contains the main function of the personal finance agent. The AgnoPersonalFinanceFunctionConfig class defines the parameters used to create runtime instances of the function agno_personal_finance. The function takes in a user’s financial goals and current situation, calls a researcher agent to search for financial advice, investment opportunities, and savings strategies based on the user preference, and the planner agent will generate a personalized financial plan. To view the source of this file, see wenqiglantz/AgentIQ on GitHub.
config.yml
The workflow configuration file specifies the functions and LLMs, and composes a workflow based on the defined functions and LLMs, along with the general configuration settings. This example uses Llama 3.3 70B Instruct as the LLM. You can customize it to use any LLM that supports function calling from build.nvidia.com or a self-hosted NIM.
general: use_uvloop: truefunctions: agno_personal_finance: _type: agno_personal_finance llm_name: nim_llmllms: nim_llm: _type: nim model_name: meta/llama-3.3-70b-instruct temperature: 0.0workflow: _type: agno_personal_finance tool_names: [agno_personal_finance] llm_name: nim_llm verbose: true retry_parsing_errors: true max_retries: 3 |
Step 4: Refine the workflow with reusable functions
One of the distinct features of the Agent toolkit is its ability to use its builder to get or create tools in a framework agnostic manner. This means that the tools can be reused by any workflows that require the features defined in the tools.
In the current implementation of agno_personal_finance, the research agent calls a Serp API search tool to search for financial advice, investment opportunities, or other information, based on user input. The search tool is initialized by directly calling Agno’s SerpApiTools in agno_personal_finance_function.py:
search_tool = SerpApiTools(api_key=os.getenv("SERP_API_KEY")) |
Serp API search is a good candidate for a reusable function in Agent toolkit. The next section explains how to add a reusable function for Serp API search in the toolkit.
First, study the Agent toolkit plugins diagram (Figure 2) and determine that the best place for this reusable function is at the package level. Why? Because SerpApiTools is a class defined by the Agno framework. Then add the config class SerpApiToolConfig, implement the serp_api_tool function, and register this new function serp_api_tool into the toolkit. See the complete source code for the serp_api_tool function.
Once defined as a new function and have it registered with the Agent toolkit, this new function can be discovered by Agent toolkit’s builder. You can then revise the original line of constructing the Serp API search tool in the workflow from:
search_tool = SerpApiTools(api_key=os.getenv("SERP_API_KEY")) |
to:
search_tool = builder.get_tool(fn_name=config.serp_api_tool, wrapper_type=LLMFrameworkEnum.AGNO) |
See the finalized source code for agno_personal_finance_function.py.
With the new function developed, it’s necessary to make some adjustments to the workflow config file, config.yml. Declare the serp_api_tool under the functions section, and also inject it to the agno_personal_finance function. The workflow section stitches together the llm_name and serp_api_tool to compose it into an agno_personal_finance workflow. See the following sample snippet of config.yml:
general: use_uvloop: truefunctions: serp_api_tool: _type: serp_api_tool # You can add your SerpAPI key here or use environment variables # api_key: your_serp_api_key agno_personal_finance: _type: agno_personal_finance llm_name: nim_llm serp_api_tool: serp_api_tool llms: nim_llm: _type: nim model_name: meta/llama-3.3-70b-instruct temperature: 0.0workflow: _type: agno_personal_finance llm_name: nim_llm serp_api_tool: serp_api_tool verbose: true retry_parsing_errors: true max_retries: 3 |
Last but not least, with the new function serp_api_tool added in the agentiq_agno package, it’s necessary to modify that package pyproject.toml entry points. Notice the new entry point for aiq_agno_tools:
[project.entry-points.'aiq.components']aiq_agno = "aiq.plugins.agno.register"aiq_agno_tools = "aiq.plugins.agno.tools.register" |
Step 5: Install and run the new workflow
Next, install the new workflow by running the following command:
uv pip install -e examples/agno_personal_finance |
Before kicking off the workflow, set two environment variables:
NVIDIA_API_KEY, used for accessing the NVIDIA-hosted NIM endpointSERP_API_KEY, used for the search tool?
Then call aiq run to run the workflow:
export NVIDIA_API_KEY=<YOUR_NVIDIA_API_KEY>export SERP_API_KEY=<YOUR_SERP_API_KEY>aiq run --config_file examples/agno_personal_finance/src/aiq_agno_personal_finance/configs/config.yml --input "My financial goal is to retire at age 60. I am currently 40 years old, working as a Machine Learning engineer at NVIDIA." |
A sample response looks like the following:
Workflow Result:["To create a personalized financial plan for a 40-year-old Machine Learning engineer at NVIDIA with a goal to retire at 60, we'll need to consider several factors, including their current income, expenses, savings, and investment goals.\n\nAssuming the engineer has a stable income and a decent savings rate, here's a possible plan:\n\n1. **Retirement Savings**: Contribute at least 10% to 15% of their income towards retirement accounts such as 401(k) or IRA. NVIDIA may offer a matching program, so it's essential to contribute enough to maximize the match.\n2. **Investment Strategy**: Allocate a significant portion of their portfolio towards low-cost index funds or ETFs, which provide broad diversification and tend to be less volatile. A possible allocation could be:\n\t* 60% Stocks (40% US, 20% International)\n\t* 30% Bonds (20% Government, 10% Corporate)\n\t* 10% Alternatives (Real Estate, Commodities, etc.)\n3. **Savings Rate**: Aim to save at least 20% to 25% of their income towards short-term and long-term goals, including retirement, emergencies, and large purchases.\n4. **Expense Management**: Create a budget that accounts for all necessary expenses, including housing, food, transportation, and insurance. Aim to keep discretionary spending in check and allocate any excess funds towards savings and investments.\n5. **Tax Optimization**: Consider contributing to tax"] |
Conclusion
NVIDIA Agent Intelligence toolkit enables developers to quickly build, evaluate, profile, and accelerate complex agentic AI workflows. It is open source and remarkably extendable, a feature that empowers developers to tailor its capabilities to diverse needs.
This post provides a detailed walkthrough of enhancing the Agent toolkit by integrating agentic frameworks, such as Agno, and composing a custom workflow. This process shows the toolkit’s flexibility and highlights its potential as a foundation for innovative AI agent-driven solutions within an enterprise environment.
By seamlessly incorporating new frameworks and workflows, the Agent toolkit enables you to push the boundaries of what’s possible in agent-based systems, adapting to specific use cases with precision and efficiency. For more details, including features, configuration options, and advanced applications, see the official documentation.
Ready to show us what you can do? Register for the NVIDIA Agent Toolkit Hackathon for a chance to win an NVIDIA GeForce RTX 5090.
