Joe DeLaere – NVIDIA Technical Blog News and tutorials for developers, data scientists, and IT admins 2025-05-29T17:31:00Z http://www.open-lab.net/blog/feed/ Joe DeLaere <![CDATA[Integrating Semi-Custom Compute into Rack-Scale Architecture with NVIDIA NVLink Fusion]]> http://www.open-lab.net/blog/?p=100146 2025-05-29T17:31:00Z 2025-05-19T04:54:31Z Data centers are being re-architected for efficient delivery of AI workloads. This is a hugely complicated endeavor, and NVIDIA is now delivering AI factories...]]>

Data centers are being re-architected for efficient delivery of AI workloads. This is a hugely complicated endeavor, and NVIDIA is now delivering AI factories based on the NVIDIA rack-scale architecture. To deliver the best performance for the AI factory, many accelerators need to work together at rack-scale with maximal bandwidth and minimal latency to support the largest number of users in the…

Source

]]> Joe DeLaere <![CDATA[Low Latency Inference Chapter 2: Blackwell is Coming. NVIDIA GH200 NVL32 with NVLink Switch Gives Signs of Big Leap in Time to First Token Performance]]> http://www.open-lab.net/blog/?p=88938 2024-11-29T21:06:06Z 2024-09-26T21:44:00Z Many of the most exciting applications of large language models (LLMs), such as interactive speech bots, coding co-pilots, and search, need to begin responding...]]>

Many of the most exciting applications of large language models (LLMs), such as interactive speech bots, coding co-pilots, and search, need to begin responding to user queries quickly to deliver positive user experiences. The time that it takes for an LLM to ingest a user prompt (and context, which can be sizable) and begin outputting a response is called time to first token (TTFT).

Source

]]> Joe DeLaere <![CDATA[NVIDIA NVLink and NVIDIA NVSwitch Supercharge Large Language Model Inference]]> http://www.open-lab.net/blog/?p=87063 2024-08-22T18:25:32Z 2024-08-12T14:00:00Z Large language models (LLM) are getting larger, increasing the amount of compute required to process inference requests. To meet real-time latency requirements...]]>

Large language models (LLM) are getting larger, increasing the amount of compute required to process inference requests. To meet real-time latency requirements for serving today’s LLMs and do so for as many users as possible, multi-GPU compute is a must. Low latency improves the user experience. High throughput reduces the cost of service. Both are simultaneously important. Even if a large…

Source

]]> Joe DeLaere <![CDATA[New Architecture: NVIDIA Blackwell]]> http://www.open-lab.net/blog/?p=80556 2024-04-09T23:45:09Z 2024-03-25T17:17:20Z Learn how the NVIDIA Blackwell GPU architecture is revolutionizing AI and accelerated computing.]]>

Learn how the NVIDIA Blackwell GPU architecture is revolutionizing AI and accelerated computing.

Source

]]> Joe DeLaere <![CDATA[NVIDIA TensorRT-LLM Supercharges Large Language Model Inference on NVIDIA H100 GPUs]]> http://www.open-lab.net/blog/?p=70549 2023-11-07T22:27:14Z 2023-09-09T17:00:00Z Large language models (LLMs) offer incredible new capabilities, expanding the frontier of what is possible with AI. However, their large size and unique...]]>

Large language models (LLMs) offer incredible new capabilities, expanding the frontier of what is possible with AI. However, their large size and unique execution characteristics can make them difficult to use in cost-effective ways. NVIDIA has been working closely with leading companies, including Meta, Anyscale, Cohere, Deci, Grammarly, Mistral AI, MosaicML (now a part of Databricks)…

Source

]]> 5 Joe DeLaere <![CDATA[Dividing NVIDIA A30 GPUs and Conquering Multiple Workloads]]> http://www.open-lab.net/blog/?p=50380 2023-04-04T16:58:51Z 2022-08-30T19:00:35Z Multi-Instance GPU (MIG) is an important feature of NVIDIA H100, A100, and A30 Tensor Core GPUs, as it can partition a GPU into multiple instances. Each...]]>

Multi-Instance GPU (MIG) is an important feature of NVIDIA H100, A100, and A30 Tensor Core GPUs, as it can partition a GPU into multiple instances. Each instance has its own compute cores, high-bandwidth memory, L2 cache, DRAM bandwidth, and media engines such as decoders. This enables multiple workloads or multiple users to run workloads simultaneously on one GPU to maximize the GPU…

Source

]]> 0 ���˳���97caoporen����