Torch-TensorRT for Orin AGX (Python 3.12)

Pre-built torch_tensorrt 2.8.0 for NVIDIA Jetson Orin AGX.

• Device: NVIDIA Jetson Orin AGX
• JetPack: 6.x with TensorRT 10.7.x
• CUDA: 12.6
• Python: 3.12
• PyTorch: 2.9.x

torch_tensorrt_dist/
├── torch_tensorrt-2.8.0.dev0+xxx-cp312-cp312-linux_aarch64.whl
├── tensorrt_py312/          # TensorRT Python bindings for Python 3.12
│   ├── __init__.py
│   └── tensorrt.so
├── install.sh               # Installation script
└── README.md

cd torch_tensorrt_dist
./install.sh

1. Install tensorrt Python module:

SITE_PACKAGES=$(python3 -c "import site; print(site.getsitepackages()[0])")
cp -r tensorrt_py312 $SITE_PACKAGES/tensorrt

2. Install torch_tensorrt wheel:

pip install --no-deps torch_tensorrt-*.whl

import torch
import torch_tensorrt

model = YourModel().eval().cuda()

# Method 1: torch.compile backend
compiled = torch.compile(model, backend="torch_tensorrt",
    options={"enabled_precisions": {torch.float16}})

# Method 2: Direct compilation
trt_model = torch_tensorrt.compile(model,
    inputs=[torch_tensorrt.Input(shape=[1, 3, 224, 224])],
    enabled_precisions={torch.float16})

• The tensorrt Python bindings were compiled from TensorRT OSS v10.7.0
• This is needed because JetPack only provides Python 3.10 bindings
• For LLM workloads, consider using TensorRT-LLM for better performance