onnx inference is slow

[oaifaye]

I compared onnx with PPOCR on CPU and found that using python was 9 times faster than tract. I wonder if this is normal

[cospectrum]

I compared onnx with PPOCR on CPU and found that using python was 9 times faster than tract. I wonder if this is normal

onnx uses multiple threads by default. The tract is single-threaded and will of course be slower on one input. However, since tract is thread safe, I still use it in some cases.

If the task is by its nature divided into batches, for example, processing several images from a video stream, in combination with rayon there will be good speed; in my area, tract will very often be faster than onnx.
Also, if, for example, the task is to write a web server, tract can process each client independently, while the onnx model will have to be wrapped in a mutex and in this case the clients will wait for each other, which will be slow under constant load.

It all depends on the subject area.