Support for Nvidia Jetson

[hornej]

Are there any plans for support the Nvidia Jetson hardware?
I am building a small home automation cluster using the Turing Pi 2 and have a Jetson Orin Nano that I would love to try this with.

[kristiankielhofner]

I have a Jetson Orin Nano with WIS running on it. I'm conflicted about whether or not to include support for it in tree:

• Jetpack software and releases are a mess compared to CUDA on x86_64.
• Quite a few components need to be compiled - ctranslate2, various wheels, etc. This is a very slow and memory intensive process (you need a lot of extra swap) but due to variations in Jetpack releases, CUDA versions, etc distributing binaries is probably a non-starter.
• It's not very fast. With our sample 3.8s audio file it is slightly slower than realtime with our defaults using the medium model and beam size 1. For reference a used, six year old $100 GTX 1070 is roughly 9x realtime with the same parameters. The difference in response time of 200ms vs 2s is the difference between a great user experience and an unacceptable one.
• Value. I understand you already have one and it can make sense for others that do but for a $500 development kit you can get a complete x86_64 system with a discrete Nvidia GPU that will run circles around this hardware while not presenting any additional software support issues.

I'd be interested to hear more about your intended use case and feedback on my points above but we implement support for hardware and functionality with the long-term in mind. My concern is that WIS support with Jetson may work today but may not (for a variety of reasons) in the future...

I was personally burned pretty badly with prior Jetson releases that Nvidia essentially gave up support for: using ancient base distributions (they STILL use Ubuntu 20.04) and generations old CUDA versions (latest Jetpack release includes CUDA 11.4, which is 13 months old already). Even for WIS today with the Jetson Orin Nano you need to upgrade docker and a couple of other system components.

It's a mess.

[hornej]

Those are all good points.

The main motivations for using the Jetson Orin Nano are (1) that I have one (2) it only draws 15W (3) its small and quiet and could fit in a self contained product and (4) I am a hardware designer so I'm intrigued by something that I could build a system around and also I like seeing how much I can get out of constrained systems 😄

That being said, I am pretty new to all of this and I have tried doing some ONNX Runtime optimizations for Whisper on the Jetson Orin Nano but haven't had much luck yet.

For my use case of smart home automations I have plenty of room for an x86_64 machine with an Nvidia GPU. I am currently testing with a Quadro P4000 and it is working great (about 3-11x realtime) for relatively short recordings.

[kristiankielhofner]

Those are fair points as well :)!

ONNX runtime optimizations (especially with TRT runtime) can be promising but for Whisper nothing beats the ctranslate2 implementation we currently use.

Glad to hear your Quadro P4000 is working well - those speeds are roughly what I would expect from that hardware. FWIW you will notice that the realtime multiple increases dramatically with longer recordings. Of course for Willow we emphasize short recording performance and benchmarks because the vast majority of voice commands are a few seconds or less. See the benchmarks if you already haven't.

I will review my local changes for Jetson support. If I can minimize them and somewhat streamline the current (convoluted) install process we may add unofficial/unsupported support for them. I know that doesn't sound promising but like I said I've been using Jetson hardware since the TK1 and I'm unfortunately very familiar with how difficult long term support for these platforms can be.

[hornej]

Yes I have seen the benchmarks they are impressive. I have also tested it on my colleague's RTX 3090.

Sounds fair! I'm kind of just learning right now so anything you want to share is helpful.
It's unfortunate the Jetson lineup is kind of a pain to deal with. I'm seeing a lot of SBCs and modules come out recently using the RK3588 which I think has a lot of promise for low-power edge AI applications and hoping it gets more adoption and support.

[kristiankielhofner]

After roughly a day of casual use and automated testing with our platform I'm even less optimistic about this implementation. The inference times vary quite a bit (possibly do to clock scaling) to the point where user experience and interactivity is completely broken. When issuing commands blindly (without viewing the display) the speech to text response times vary so significantly you end up repeating yourself, etc.

Additionally, due to the low combined RAM/VRAM (even with additional swapfile) TTS won't fit in memory, which is another significant usability concern.

Unfortunately I think we'll have to go with my original instinct and not support Jetson Orin with WIS.

[mikey60]

How about WIS on a NVIDIA Jetson AGX Orin 64GB Developer Kit. Does it have the same software issues? The kit has the latest NVIDIA Ampere GPU architecture. It is pricey at $2000. I did some more research and answered my own question. It looks like it might have the same issues. The NVIDIA Jetson Linux 35.4.1 is still based on Linux Kernel 5.10 and Ubuntu 20.04. It looks like it is still using CUDA 11.4.

[kristiankielhofner]

Unfortunately the Jetson line just doesn't make sense unless you really need the extremely low power usage, form factor, etc. It is extremely slow and limited for the price point. For $2000 you can get an RTX 3090 and a really nice machine and have money left over to cover the power costs for well beyond the life of the hardware and then some. This configuration from GPU, to CPU, to I/O, etc would make the AGX Orin look like a Raspberry Pi. It also supports the latest and greatest in terms of whatever kernel, distro, Nvidia driver release, CUDA 12, etc.

It being based on Ampere doesn't mean much in the end when the core count, clocks, and memory bandwidth have been slashed to meet the form factor and extremely low power usage targets. The memory bandwidth alone is at least 10% slower than a $100 seven year old GTX 1070 and 5x slower than an RTX 3090.

Sorry another edit: these are dev kits for the compute modules. What the platform is intended for is commercial and industrial embedded applications - things like putting CUDA in an MRI machine.

[mikey60]

Thank you! I am almost ready to purchase a RTX 3090.