Join the NVIDIA Triton and NVIDIA TensorRT community and stay current on the latest product updates, bug fixes, content, best practices, and more. Register Free

NVIDIA Triton Inference Server

NVIDIA Triton? Inference Server? ?? ?? ? ??? ???? ? ??? ????, ???? ??? ??? ?? ??? AI? ???? ?? ?? ?? ?? ????????.

????

NVIDIA Triton? ?? ???????

NVIDIA AI ???? ?? ??? Triton Inference Server? ?? GPU ?? CPU ?? ???? ??????? ??? AI ??? ??, ?? ? ??? ? ??? ?????? AI ??? ????? ??????. AI ???? ??? ???? Triton? ?? ???? ??? ??? ??? ?? ????? ??? ?????? ???? ??? ? ???, ???? ????, ?????, ?? ? ???? ???? ???? ??? ??? ??? ? ????.

?? ????

Triton supports multiple inference frameworks

??? ????? ??

Triton? TensorFlow, NVIDIA? TensorRT?, PyTorch, MXNet, Python, ONNX, XGBoost, scikit-learn, RandomForest, OpenVINO, ??? C++ ? ?? ?? ? ?? ?????? ?? ?????.

Triton offers high-performance inference

??? ??

Triton? NVIDIA GPU, x86, Arm? CPU ? AWS Inferentia ?? ??? ?? ????, ???? ??(batching), ?? ??, ??? ?? ??, ?? ??? ? ???? ???/??? ??? ??? ???? ???? ??????.

Triton is designed for DevOps and MLOps

DevOps? MLOps? ?? ??

Triton? ???????? ??? ?? Kubernetes? ????, ????? ?? Prometheus ???? ????, ??? ?? ????? ????, ?? ?? ??? ???? AI? Kubernetes ???? ?????. ?? ???? MLOps ????? ????? ?????.

Triton is an integral part of NVIDIA AI platform

NVIDIA AI? ?? ??

Triton? ??? NVIDIA AI ???? AI? ????? ???? ? ??? ??? ??? ? ? ????? ?????? ?? ??? ?? ??????? ?? ??? ?? ??? ????? ??? ?? ????? ?????.

?? ???????? ??? ?? ??? AI? ??????

??? ?? ??

Triton? ? ??????? ?? ??, ?? ?? ?? ??????? ?? ?? ?? ??? ?? GPU ?? CPU?? ??? ?????. ?? GPU ??? ??, Triton? ? GPU?? ? ??? ?? ????? ???? ???? ???? ????.

?? ???? ??? ?? ?? ??? ??? ??? ???? ??? ?? ??? ?????, ??(batch) ??? ??? GPU ? CPU ???? ?????, ??? ? ??? ???? ??? ????? ?????. Triton? ??? AI? ?? ?? ? ?? ??? ???? ?? ?? ??? ?????? ??? ?? ???? ?? ???? ?????.

Triton?? ??????? ?? ???? ??? ???? ???? ????? ??? ????? ? ????. Triton? ?? GPU? ???? ???? ?? ?? ???? ?? GPU/?? ?? ??? ?????.

Top 5 Reasons Why NVIDIA Triton is Simplifying Inference

??? ?? ??

Docker ???? ??? ???? Triton? Kubernetes? ???? ???????? ??? ? ?? ????? ?????. ?? Kubeflow ? KServe? ???? ???? AI ?????? ???? Prometheus ???? ???? GPU ???, ?? ??, ??? ??? ? ?? ???? ???????. ?? ??? ?? ?? ??????? ??? ? ??? ?? HTTP/gRPC ?????? ????, ??? ??? ??? ??? ???? ?? ????? ???? ?? ??? ??? ? ????.

Triton? ?? ?? API? ?? ?? ? ?? ?? ?? ??? ??? ? ????. GPU ?? CPU ???? ?? ?? ??? ??? ?? ??? ??? ??/???? ? ????. Triton??? GPU? CPU? ?? ??? ??? ????? ???? ??? ??? ???? ??? ?????, CPU ?? GPU?? ?? ??? ?? ?? ??? ??? ? ????.

NVIDIA Triton can scale inference with ease

Triton ?? ??? ????

?? ???? ?? ?? ???????

Triton? ??? ?? ??????? ??? ???? ?? ?? ??? ???? ?????. ???? ??? ??? ???? ? ??? ?? ? ??? ???? ???? ?? ?? ??? ??????. ?? ?? GPU ??? ??? ??? ?? ???? GPU ???? ????? ????, ?? ?? ??????? ?? ??? ????? ??? ?? ??? ?????. ?? ??????? ??? ??? ?? ???(EA)? ?? ?????.

EA ???? ????
TBD
TBD

??? ?? ?? ??

????? ??(?: GPT-3 - 175B, Megatron 530B ??)? ???? ??? ??? ???? ??? ????. GPU? ??? ?? ??? ??? ??? ?????, ??? ??? ?? ?? ?? GPU ??? ????? ?????. Triton? ??? ?? ?? ?? ?? ???? ?? ???, ?? ?? ??? ?? ??? GPU?? ? ??? ??? ? ????. ??? ?? GPU/?? ?? ??? ???? Triton? FasterTransformer ???? ??? GPT ???? T5, OPT ? UL2 ??? ???? ?? ??? ??? ?????.

????? ??? ????

?? ???? ??? ??? ?? ??

Triton? Model Analyzer? ?? ?????? ?? ??, ???, ?? ?? ???? ?? Triton ?? ??? ???? ???? ????. ?? ??, ???, ??? ?? ?? ?? ?????? Quality-of-Service(QoS) ?? ??? ???? ??? ??? ???? ? ????, ??? ??? ?? ? ???? ??? ? ??? ? ???? ?????.

??? ????
TBD
TBD

Forest Inference Library(FIL) ???? ?? ?? ?? ?? ??

Triton? ??? FIL ???? CPU? GPU?? ?? ???(SHAP ?)? ?? ?? ?? ??? ??? ??? ?????. XGBoost, LightGBM, scikit-learn RandomForest, RAPIDS? cuML RandomForest ? ?? Treelite ??? ??? ?????.

??? ????

????? ?? ?? ????

AI? ??? ??? ???? ?? ??? ??? ???? ???, ??? ??? ???? NVIDIA AI? ????. ?? ?? ????? ???? Triton? AI ?? ? ?? ??? ?? ??? ?????. Triton? Alibaba Cloud, Amazon Elastic Kubernetes Service (EKS), Amazon Elastic Container Service (ECS), Amazon SageMaker, Google Kubernetes Engine (GKE), Google Vertex AI, HPE Ezmeral, Microsoft Azure Kubernetes Service (AKS) , and Azure Machine Learning . ?? ?????. ???? Triton? ???? ??? ?????.

NVIDIA LaunchPad? ?? ???? ???? ?? ????? ??? ?? ?? ?? Triton Inference Server? ??????.




?? ????

?? ?? ??

Learn how NVIDIA AI improves Amazon customer satisfaction
amazon logo

Amazon? NVIDIA AI? ?? ?? ??? 5? ?? ?? ???? ??? ??? ?????.

??? ????
Learn how NVIDIA AI improves AMEX fraud detection
american express logo

American Express? ??? ?? ??? ?? ?? ??? 50? ?? ??? ???? ?? ??? ??? ??? ?????.

??? ????
Learn how NVIDIA AI improves Siemens Energy physical inspections
Siemens Energy

Siemens Energy? ??, ???? ?? ?? ?? AI ??? ?? ????? ???? ??? ??? ??? ??? ?????.

??? ????

? ?? ??? ????

Learn how NVIDIA Triton can simplify AI deployment at scale.

AI ?? ????

NVIDIA Triton? ?? ?? ??? ???? ?? AI ?? ??? ????? ??? ?????.

??? ??
Explore the latest NVIDIA Triton on-demand sessions.

Triton GTC ???? ?? ????

NVIDIA GTC?? Triton Inference Server? ?? ?? ???? ??? ?????.

??? ????
Deploy AI deep learning models.

AI ?? ????

NVIDIA Triton? ?? ?? ??? ???? ?? AI ?? ??? ????? ??? ?????.

??? ??

????? ?? NVIDIA ????

NVIDIA Inception? ????? NVIDIA Triton, NVIDIA ???, ?? ???, ?? ??? ?? ?? ??? ??? ????? ??? ????? ?? ?? ??? ?? ???????.

??? ????

Learn more about NVIDIA Inception program for startups.

Triton ????? ???? ?? ?? ????, ?? ?? ?? ?? ?? ??? ?????.

????

??????? AI ?? ????

???? ???? AI? ?? ?? ????? ??? ? ?? ?? ???? ?? ?????. NVIDIA Triton? ?? ??? NVIDIA ?????? ?? ???? ?? ?? ??, ?? ?? ??, ?? ????, NVIDIA AI ??? ?? ?? ??? NVIDIA AI Enterprise? ?? ?????.

??? ????

NVIDIA H100? ???? ??? NVIDIA AI Enterprise ?????? ????? ??? ?????.