LLM Inference Benchmarking Guide: NVIDIA GenAI-Perf and NIM

This is the second post in the LLM Benchmarking series, which shows how to use GenAI-Perf to benchmark the Meta Llama 3 model when deployed with NVIDIA NIM.?

• Part 1: LLM Inference Benchmarking: Fundamental Concepts

When building LLM-based applications, it is critical to understand the performance characteristics of these models on a given hardware. This serves multiple purposes: 

• Identifying the bottleneck and potential optimization opportunities
• Identifying the quality of service and throughput tradeoff
• Infrastructure provisioning

As a client-side LLM-focused benchmarking tool, NVIDIA GenAI-Perf provides key metrics: 

• Time to first token (TTFT)
• Inter-token latency (ITL)
• Tokens per second (TPS)
• Requests per second (RPS)
• …and more

GenAI-Perf also supports any LLM inference service conforming to the OpenAI API specification, a widely accepted default standard in the industry. 

For this benchmarking guide, we used NVIDIA NIM, a collection of inference microservices that offer high-throughput and low-latency inference for both base and fine-tuned LLMs. NIM features ease-of-use and enterprise-grade security and manageability. 

To optimize your AI application, this post walks through the process of setting up a NIM inference microservice for Llama 3, using GenAI-Perf to measure the performance, and analyzing the outputs.

As NIM and GenAI-Perf evolve, see the Using GenAI-Perf to Benchmark documentation.

Why use GenAI-Perf for benchmarking model performance?

NVIDIA NIM microservices are prepackaged, optimized containers that make it easy to deploy and run AI models across various platforms, including clouds, data centers, and NVIDIA RTX AI PCs, accelerating generative AI development for any domain. NIM LLMs are powered by industry-leading backends, including TensorRT-LLM and vLLM. NIM is continually improved to provide optimal performance even on the same hardware generation. 

GenAI-Perf enables you to not only reproduce and validate these performance results but to do benchmarking using your own hardware. For more information about this process, see NVIDIA NIM Performance. 

Steps for benchmarking with NIM

All the performance figures on the NIM Performance page were collected with the NVIDIA GenAI-Perf tool. Following the step-by-step recipe presented in the rest of this post, you can reproduce these performance numbers, as well as benchmark models on your own hardware. 

Setting up an OpenAI-compatible Llama-3 inference service with NIM

NIM provides the easiest and quickest way to put LLMs and other AI foundation models into production. For more information, see A Simple Guide to Deploying Generative AI with NVIDIA NIM or the latest NVIDIA NIM for Large Language Models documentation, which walks you through the hardware requirements and prerequisites, including NVIDIA NGC API keys.  

Use the following commands to deploy NIM and execute inference:   

In this code example, we specified the Meta Llama-3.1-8b-Instruct model, which is also the name of the container. We also provided and mounted a local directory as a cache directory where downloaded model assets are persisted. 

During startup, the NIM container downloads the required resources and begins serving the model behind an API endpoint. The following message indicates a successful startup:

When running, NIM provides an OpenAI-compatible API that you can query:

Setting up GenAI-Perf and benchmarking a single use case

Now that you have the NIM Llama-3 inference service running, next set up the benchmarking tool. 

The easiest way to do this is using a prebuilt Docker container. We recommend starting a GenAI-Perf container on the same server as NIM to avoid network latency, unless you specifically want to factor in the network latency as part of the measurement.

For more information about getting started, see the GenAI-Perf documentation. 

Run the following commands to use the prebuilt container.

You should mount a directory from the host machine into the container, so that benchmarking assets can be persisted. In the earlier example, we mounted the current directory.

Inside the container, you can fire up the GenAI-Perf evaluation harness as follows, which carries out a warming load test on the NIM backend:

In this minimal example, we specified the input and output sequence length and a concurrency to test. We also asked the backend to ignore the special EOS tokens, so that the output reaches the intended length. 

This test uses the Llama-3 tokenizer from Hugging Face, which is a guarded repository. You must apply for access and then log in with your Hugging Face credential.

For more information about the full set of options and parameters, see the GenAI-Perf documentation.

Upon successful execution, you should see results similar to the following in the terminal:

Sweeping through several use cases

With benchmarking, a test would typically be set up to sweep over, or execute, several use cases (input and output length combinations) and load scenarios (different concurrency values). Using the following BASH script, define the parameters so that GenAI-Perf executes through all the combinations. 

Before doing a benchmarking sweep, we recommend running a warm-up test. In our case, we did this while setting up GenAI-Perf and a single use case. 

In the GenAI-Perf container, run the following command:

Here’s the benchmark.sh script:

The --measurement-interval 30000 parameter is the time interval used for each measurement in milliseconds. GenAI-Perf measures the requests that finish in a specified time interval. Choose a value big enough for several requests to finish. For larger networks, such as Llama-3 70B, and more concurrency, such as 250, choose a higher value. For example, you could use 100000 ms, which is 100 sec.

Analyzing the output

When the tests are complete, GenAI-Perf generates the structured outputs in a default directory named \artifacts, organized by model name, concurrency, and input/output length.

The *genai_perf.csv files are the main benchmarking results. You can read all the request-per-second (RPS) and time-to-first-token (TTFT) metrics across all concurrencies for a given use case, such as ISL/OSL of 200/5, with the following code:

Finally, you can plot and analyze the latency-throughput curve using the collected data with the following code. Here, each data point corresponds to a concurrency value. 

Figure 2 shows the resulting plot using GenAI-Perf measurement data. 

Interpreting the results

Figure 2 shows TTFT on the x-axis, total system throughput on the y-axis, and concurrencies on each dot. There are several ways to use the plot:

• An LLM application owner who has a latency budget, for example, the maximum TTFT that is acceptable, can use that value for x, and look for the matching y value and the concurrencies. That shows the highest throughput that can be achieved with that latency budget and corresponding concurrency value.
• An LLM application owner can use the concurrency values to locate corresponding dots on the graph. The x and y values at the dots show the latency and throughput for that concurrency level.

Figure 2 also shows the concurrencies where latency grows quickly with little or no throughput gain. For example, concurrency=50 is one such value.

Similar plots can use ITL, e2e_latency, or TPS_per_user as the x-axis, showing the trade-off between total system throughput and individual user latency.

Benchmarking customized LLMs

LLMs nowadays can cater well to many common tasks, such as open QA or meeting summarization. 

However, there are tasks that can benefit hugely from customizing LLMs, such as training them on the internal corporate knowledge base so that they get used to the internal work cultures, know-how and protocol, specific product portfolios, and the acronyms, or even using internal productivity tools.   

NIM enables customized models to be served. NIM supports low-rank adaptation (LoRA), which is a simple and efficient way to tailor LLMs for specific domains and use cases. To customize LLM with LoRA, you can make use of NVIDIA NeMo,  an open-source platform for training generative AI models. When you’re done, you can load and deploy multiple LoRA adapters using NIM. 

First, put the NVIDIA NeMo-trained LoRA adapters into a certain directory structure and then pass the adapter folder name to NIM as an environment variable, following the instructions at Parameter-Efficient Fine-Tuning. NIM also supports LoRA adapters trained with the HuggingFace PEFT library. 

After it’s added, the LoRA model can be queried similar to the base model by replacing the base model ID with the LoRA model name:

With GenAI-Perf, the deployment metrics for LoRA models can be benchmarked by passing the IDs of the LoRA models through the -m argument:

In this case, three LoRA models are being tested:

• llama-3-8b-lora_1
• llama-3-8b-lora_2
• llama-3-8b-lora_3

The --model-selection-strategy {round_robin,random} parameter specifies whether these adapters should be called in a round-robin or random fashion.

Conclusion

The NVIDIA GenAI-Perf tool was created to meet the urgent need for benchmarking solutions powered by LLM serving at scale. It supports NVIDIA NIM, as well as other OpenAI-compatible LLM serving solutions. 

This post provided you with the most important, popular metrics and parameters to help standardize how model benchmarking takes place across the industry.

For more information and expert guidance on how to select the right metrics and optimize your AI application’s performance, see the LLM Inference Sizing: Benchmarking End-to-End Inference Systems GTC session.