Failed analyse for node #678 "/Expand" MultiBroadcastTo

[genericalexacc]

Trying to run a quantized model of the decoder of SAM.

If I run it without into_optimized(), it does run but the inference time is around 1min 50sec on my M1 Pro Macbook.
Whereas I expect this to run under a second(which is the performance I see in python)

I have an ONNX file and here is the code I use to load and run inference:

use tract_onnx::prelude::*;

let model_bytes = include_bytes!("../assets/mobilesam.decoder.quant.onnx");
let mut file = Cursor::new(model_bytes);

let model = tract_onnx::onnx()
    .model_for_read(&mut file)?
    .with_output_fact(
        0,
        InferenceFact::dt_shape(bool::datum_type(), tvec!(3, 1040, 1456)),
    )?
    .with_output_fact(
        1,
        InferenceFact::dt_shape(f32::datum_type(), tvec!(3,)),
    )?
    .with_output_fact(
        2,
        InferenceFact::dt_shape(f32::datum_type(), tvec!(3, 256, 256)),
    )?
    // .into_optimized()?
   .into_runnable()?;

let embeddings: Tensor =
    tract_ndarray::Array4::from_shape_vec((1, 256, 64, 64), embeddings)?.into();

let point_coords_vec: Vec<f32> = vec![1150.0, 375.0];
let point_coords: Tensor =
    tract_ndarray::Array3::from_shape_vec((1, 1, 2), point_coords_vec)?.into();

let point_labels_vec: Vec<f32> = vec![1.0];
let point_labels: Tensor =
    tract_ndarray::Array2::from_shape_vec((1, 1), point_labels_vec)?.into();

let mask_input_vec: Vec<f32> = vec![0.0; 256 * 256];
let mask_input: Tensor =
    tract_ndarray::Array4::from_shape_vec((1, 1, 256, 256), mask_input_vec)?.into();

let has_mask_input_vec: Vec<f32> = vec![1.0];
let has_mask_input: Tensor =
    tract_ndarray::Array1::from_shape_vec(1, has_mask_input_vec)?.into();

let orig_im_size_vec: Vec<f32> = vec![1040.0, 1456.0];
let orig_im_size: Tensor = tract_ndarray::Array::from_shape_vec(2, orig_im_size_vec)?.into();

let result = model.run(tvec!(
    embeddings.into(),
    point_coords.into(),
    point_labels.into(),
    mask_input.into(),
    has_mask_input.into(),
    orig_im_size.into(),
))?;

When I try running with into_optimized() it gives me the following error:

res: Err(Failed analyse for node #678 "/Expand" MultiBroadcastTo

Caused by:
    0: Infering facts
    1: Applying rule outputs[0].shape == 1,num_points,256
    2: Unifying shapes unk__0,unk__1,unk__2 and 1,num_points,256
    3: Impossible to unify Sym(unk__0) with Val(1).)

Is there a way to fix this, or alternatively is there a way to make the inference faster?

Thank you in advance :)

[kali]

Hey, thanks for your interest in tract.

Performance without the call to into_optimized() is often ridiculously bad. It is meant to be used for debugging only.

Your problem look like a typical case of ONNX problematic symbol usage in dimensions. Its semantics and tract's are different. It's usually pretty easy to fix.

You can try tract main branch: I have relaxed how tract interprets symbols coming from ONNX (ignoring the output shapes altogether by default). If it does not fix, I'll need a link to the ONNX model to have a look.

[genericalexacc]

Hey,

Thanks for the fast response!
It doesn't seem to work on main.

Here is the link to the model:
https://github.com/akbartus/MobileSAM-in-the-Browser/blob/main/models/mobilesam.decoder.quant.onnx

Is there a link to any troubleshooting guide or any tools you use, so I can troubleshoot models myself in the future?

Thanks :)

[kali]

A good starting point is the command line interface, there are some recipes in https://github.com/sonos/tract/blob/main/doc/cli-recipe.md . It does help, but the documentation is a bit lacking so it takes a bit of trial and error. There are some examples of usage for debugging in some discussions too.

[kali]

I'm surprised by the fact you're using a lot of .with_output_fact() calls in fixing the ONNX interface. Most of the time, people end up overwritting the inputs (.with_input_fact(...)) and cancelling the output (using .with_output_fact(..., Default::default()) but you seem to have somehow landed some place else.

[genericalexacc]

I tried using onnx-simplifier and setting num_points to 1 and now I'm getting a panic from unwrapping a none in the /onnx/src/ops/resize.rs.rs:225:52 file.
I'm not sure if this is a step forward or backwards haha.
This seems to happen regardless of whether I'm overwriting inputs and outputs, only inputs, only outputs or neither.

Model:
https://www.dropbox.com/scl/fi/dsn5dzru4z6rj3spn12fp/mobile-simplified-sam-prepared.onnx?rlkey=b2ktw9he6vpyti9kwyb99s33p&dl=0

[2023-09-26T03:12:06Z INFO  stroll::app] Some info
[2023-09-26T03:12:06Z INFO  stroll::app] resp: "start"
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /Users/alexandershevchenko/.cargo/git/checkouts/tract-01a3138c7a4b58a3/2b71121/onnx/src/ops/resize.rs:225:52
stack backtrace:
   0:        0x10160ab88 - std::backtrace_rs::backtrace::libunwind::trace::h77b39d3188241b9b
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:        0x10160ab88 - std::backtrace_rs::backtrace::trace_unsynchronized::h255fc0d6f6a27160
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x10160ab88 - std::sys_common::backtrace::_print_fmt::hc63b2fe172b28820
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:65:5
   3:        0x10160ab88 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h523fafbfdb8f0857
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:44:22
   4:        0x10162b6b4 - core::fmt::rt::Argument::fmt::h4c8d9e4aebabcbc2
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/fmt/rt.rs:138:9
   5:        0x10162b6b4 - core::fmt::write::hf94a55b5a3abd106
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/fmt/mod.rs:1094:21
   6:        0x1016074d0 - std::io::Write::write_fmt::hc7c6bf1da111b052
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/io/mod.rs:1714:15
   7:        0x10160a9e0 - std::sys_common::backtrace::_print::h12835c9b28903edc
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:47:5
   8:        0x10160a9e0 - std::sys_common::backtrace::print::h68ede8fb1e716cba
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:34:9
   9:        0x10160bf08 - std::panicking::default_hook::{{closure}}::hba2205c2705c60bb
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:269:22
  10:        0x10160bc98 - std::panicking::default_hook::h9d927e01472bad1a
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:288:9
  11:        0x10160c438 - std::panicking::rust_panic_with_hook::h8654c51ef9980a29
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:705:13
  12:        0x10160c308 - std::panicking::begin_panic_handler::{{closure}}::hd188a636b3b90298
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:595:13
  13:        0x10160af68 - std::sys_common::backtrace::__rust_end_short_backtrace::hc331d455ac21f427
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:151:18
  14:        0x10160c0b0 - rust_begin_unwind
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:593:5
  15:        0x10166dbc4 - core::panicking::panic_fmt::h4f2054f72ff905b1
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:67:14
  16:        0x10166dc34 - core::panicking::panic::h4d2ee9cac0d6e9c1
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:117:5
  17:        0x100be391c - <tract_onnx::ops::resize::Resize as tract_hir::infer::rules::InferenceRulesOp>::rules::hbd6530944df4d5bc
  18:        0x100bdb600 - tract_hir::infer::rules::<impl tract_hir::infer::ops::InferenceOp for O>::infer_facts::hc0a461a48d5cad1f
  19:        0x100bdb078 - tract_hir::infer::ops::InferenceOp::infer::h39eaf2f605fdb875
  20:        0x100d618ac - tract_hir::infer::analyser::Analyser<M>::analyse_obstinate::h08c2ea7b9c58c04e
  21:        0x100db7894 - <tract_core::model::graph::Graph<tract_hir::infer::fact::InferenceFact,alloc::boxed::Box<dyn tract_hir::infer::ops::InferenceOp>> as tract_hir::infer::model::InferenceModelExt>::into_typed::h258d1015f03c77de
  22:        0x100db79a4 - <tract_core::model::graph::Graph<tract_hir::infer::fact::InferenceFact,alloc::boxed::Box<dyn tract_hir::infer::ops::InferenceOp>> as tract_hir::infer::model::InferenceModelExt>::into_optimized::h8afc628f0dd01415
  23:        0x100a9fdfc - stroll::app::Stroll::new::h33ad5db1d66da618
  24:        0x100a06484 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h54448f9aae7cfa28
  25:        0x1013e3c60 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::he672e60c972977ba
  26:        0x1013c3ec4 - <eframe::native::run::glow_integration::GlowWinitApp as eframe::native::run::WinitApp>::on_event::heb49b2abd34ed8f3
  27:        0x1013eeabc - eframe::native::run::run_and_return::{{closure}}::h94d1e1d3e719671e
  28:        0x10140242c - <winit::platform_impl::platform::app_state::EventLoopHandler<T> as winit::platform_impl::platform::app_state::EventHandler>::handle_nonuser_event::hb2a02b5a68d3f5b1
  29:        0x10153e1e0 - winit::platform_impl::platform::app_state::Handler::handle_nonuser_event::h1dfb649771f8c68d
  30:        0x1015406cc - winit::platform_impl::platform::app_delegate::ApplicationDelegate::did_finish_launching::h7d1ca4e6c333acc4
  31:        0x181197180 - <unknown>
  32:        0x181232eec - <unknown>
  33:        0x181232e34 - <unknown>
  34:        0x1811684cc - <unknown>
  35:        0x1820c68f4 - <unknown>
  36:        0x1843c9158 - <unknown>
  37:        0x1843c8f08 - <unknown>
  38:        0x1843c6fa4 - <unknown>
  39:        0x1843c6ba0 - <unknown>
  40:        0x1820f0b64 - <unknown>
  41:        0x1820f09c4 - <unknown>

[kali]

Hello, would you give a shot to the branch https://github.com/sonos/tract/tree/resize-op ?