how to use examples/tensorflow-mobilenet-v2 with mobilenet-v3?

[112x112]

i tried replacing

wget https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.4_224.tgz
tar zxf mobilenet_v2_1.4_224.tgz

with

$  wget https://storage.googleapis.com/mobilenet_v3/checkpoints/mobilenet_v3_1.4_224.tgz
--2024-08-07 12:51:55--  https://storage.googleapis.com/mobilenet_v3/checkpoints/mobilenet_v3_1.4_224.tgz
Resolving storage.googleapis.com (storage.googleapis.com)... 142.250.115.207, 142.250.113.207, 142.250.138.207, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|142.250.115.207|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-08-07 12:51:55 ERROR 404: Not Found.

$  wget https://storage.googleapis.com/mobilenet_v3/checkpoints/mobilenet_v3_224.tgz
--2024-08-07 12:52:03--  https://storage.googleapis.com/mobilenet_v3/checkpoints/mobilenet_v3_224.tgz
Resolving storage.googleapis.com (storage.googleapis.com)... 142.250.114.207, 142.251.116.207, 142.250.113.207, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|142.250.114.207|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-08-07 12:52:03 ERROR 404: Not Found.

$  wget https://storage.googleapis.com/mobilenet_v3/checkpoints/mobilenet_v3.tgz
--2024-08-07 12:52:07--  https://storage.googleapis.com/mobilenet_v3/checkpoints/mobilenet_v3.tgz
Resolving storage.googleapis.com (storage.googleapis.com)... 142.250.115.207, 142.251.116.207, 142.250.114.207, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|142.250.115.207|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-08-07 12:52:07 ERROR 404: Not Found.

but none worked.

i also tried generating a Tensorflow mobilenet-v3 model with Python, but it seemed like i couldn't just "plug and play" the generated model with the code in https://github.com/sonos/tract/tree/main/examples/tensorflow-mobilenet-v2.

do you know where I can get a Tensorflow mobilenet-v3 model that will easily work with https://github.com/sonos/tract/tree/main/examples/tensorflow-mobilenet-v2?

    let model = tract_tflite::tflite()
        // load the model
        .model_for_path("./mobilenet_v3_small_100_224.tflite")?

from https://github.com/sonos/tract/tree/main/examples/tflite-mobilenet-v3 seems to give less accurate results, so I wanted to try a tensorflow-mobilenet-v3 example.

[banitalebi]

Please follow these steps: First, run the commands below,

wget -q https://storage.googleapis.com/mobilenet_v3/checkpoints/v3-small_224_1.0_float.tgz tar zxf v3-small_224_1.0_float.tgz
then find,
v3-small_224_1.0_float.pb

and finally use examples/tensorflow-mobilenet-v2/src/main.rs with slight change as below,

`use tract_tensorflow::prelude::*;

fn main() -> TractResult<()> {
let model = tract_tensorflow::tensorflow()
// load the model
.model_for_path("v3-small_224_1.0_float.pb")?
// specify input type and shape
.with_input_fact(0, f32::fact([1, 224, 224, 3]).into())?
// optimize the model
.into_optimized()?
// make the model runnable and fix its inputs and outputs
.into_runnable()?;

// open image, resize it and make a Tensor out of it
let image = image::open("grace_hopper.jpg").unwrap().to_rgb8();
let resized =
    image::imageops::resize(&image, 224, 224, ::image::imageops::FilterType::Triangle);
let image: Tensor = tract_ndarray::Array4::from_shape_fn((1, 224, 224, 3), |(_, y, x, c)| {
    resized[(x as _, y as _)][c] as f32 / 255.0
})
.into();

// run the model on the input
let result = model.run(tvec!(image.into()))?;

// find and display the max value with its index
let best = result[0]
    .to_array_view::<f32>()?
    .iter()
    .cloned()
    .zip(1..)
    .max_by(|a, b| a.0.partial_cmp(&b.0).unwrap());
println!("result: {best:?}");
Ok(())

}`
I would be happy to submit a pull request if needed.

[112x112]

Thank you very much for responding.

I got some mobilenet_v3 model using your code, and I'm using

pub fn load_model() -> TractResult<LoadModelResult> {
    tract_tensorflow::tensorflow()
        // load the model
        .model_for_path("v3-small_224_1.0_float.pb")?
        // specify input type and shape
        .with_input_fact(0, f32::fact([1, 224, 224, 3]).into())?
        // optimize the model
        .into_optimized()?
        .into_runnable()
}

but my image recognition task is failing, even with the exact same images in a similar workflow in a Python code base. I thought it was because the Python code base is using tensorflow, and the Rust code base was using tflite, but even after switching to the above code with tensorflow mobilenet_v3, my Rust code base can't match images as well as the Python code base.

Do you know how i can use the same model from the Python codebase in the Rust codebase with Sonos tract?

Thank you again for your response earlier.

[112x112]

from tensorflow.keras.applications import MobileNetV3Small  # type: ignore

# Load pre-trained MobileNetV3Small model.
MODEL = MobileNetV3Small(
    weights="imagenet", include_top=False, pooling="avg", input_shape=(224, 224, 3)
)

This is the Python code. How would I grab the model from here and use it with tract?

[banitalebi]

Hello,

I tried this, and it works for me.

First, on the Python side, using TensorFlow and Keras, you can develop and train or fine-tune any model you like. Then, at the last line of your training code, export the model as follows:

python
model.export('savedmodel')

Next, following the documentation at ONNX Runtime, use this command:

bash
python -m tf2onnx.convert --saved-model path/to/savedmodel --output dst/path/model.onnx --opset 13

This will produce a file that can be used in the Rust tract inference part.

I hope this is useful for you.

[kali]

@banitalebi thanks a lot for helping, I'm pretty weak on the training side of affairs...

I noticed you proposed a PR, but I'm not sure what you intended. That said, I think an update of the current example (tf/mobilenet v2) to mobilenet v3 could be a good option to make the example more readily usable...

[banitalebi]

@kali Thank you for your message! I'm happy to assist and would love to contribute by adding a new example for MobileNet V3, as you suggested.

[banitalebi]

Hello everyone,

I spent some time fine-tuning MobileNet-V3 for classifying violence versus non-violence using a dataset from Kaggle. The model fine-tuning and the generation of the 'mobilenetv3_model.onnx' file have been completed. Additionally, the model evaluation using Python libraries has been conducted, as described [here] (https://github.com/banitalebi/MobileNetV3-Violence-Detection), where the results using onnxruntime and python tract are shown.

The same mobilenetv3_model.onnx file and the same test [dataset] (https://drive.google.com/file/d/1sDwIpnQJD8iaRm8M322pITrD0esdMZZ6/view?usp=sharing) were utilized in this [repository] (https://github.com/banitalebi/MobileNetV3-Inference). But, apart from the differences in runtime, the results themselves also not the same. In python side the accuracy is 82.78% and in rust the accuracy: 77.28%.

[112x112]

I was just about to post something similar.

I followed your process at #1504 (comment) and the mobilenet v3 model I recreated for tract, from Python code, incorrectly shows an image with black columns on either side of it being very, very close to a completely black image, even though there's a lot of detail in the input image.

It even shows the completely black image as being a closer match to the input image than the input image with the black columns on either side of it removed.

[banitalebi]

I think it is no longer a discussion about how to use MobileNet V3, but it can be considered as a new issue. So, I will post about it.