Set camera perspective so that projected ellipses on image plane fit on ellipsoids

[ttsesm]

Hi @marcomusy,

I am trying to set the camera perspective so that the projected ellipses on image plane fit on ellipsoids that I am manually creating. For example I have the following set of ellipsoids and ellipses (projected on a random image plane):

The image plane:

and now I want to set my camera perspective to a fixed position so that I can get the contours of the projected ellipses
to fit the silhouettes of the ellipsoids as you can see bellow (tried to set it manually with the mouse, without great success):

I have the camera settings, check the code snippet bellow:

# pip install 'git+https://gitlab.inria.fr/tangram/pyellcv.git'
from ellcv.types import Ellipsoid, Ellipse
from ellcv.algo.cpp import solveP3P_ransac
from ellcv.visu import draw_ellipse, draw_bbox, sample_ellipse_points
import ellcv
from ellcv.utils import (
    generate_random_scene,
    generate_random_camera,
    generate_K,
    bbox_from_ellipse,
)

from vedo import Points, Plotter
import vedo as vd
import matplotlib
import numpy as np
import cv2

# matplotlib.use("Qt5Agg")
np.set_printoptions(suppress=True)

# Create a scene of ellipsoids
N = 10
scene = generate_random_scene(N, axes_range=[0.3, 1.5], position_range=[-13.0, 13.0])

# Define the camera
W, H = 640, 480
o, p = generate_random_camera(dist_range=[20.0, 30.0], target_range=[-0.001, 0.001])
c2w = np.eye(4)
c2w[:3, :3] = o  # .transpose()
c2w[:3, -1] = p
w2c = np.linalg.inv(c2w)

fx = 500.0 / float(max(W, H))
# K = generate_K(fx, 0.0, 0.0)
K = np.array([[500.0, 0.0, float(W) / 2], [0.0, 500.0, float(H) / 2], [0.0, 0.0, 1.0]])
print(K)
P = K @ w2c[:3, :]

pose = c2w
axes = []
rot = []
center = []
psoid_pcds = []
ellipse_pcds = []
ellipses = []
img = np.full((H, W, 3), 180, dtype=np.uint8)

for i, psoid in enumerate(scene):
    axes_, rot_, center_ = psoid.decompose()
    axes.append(axes_)
    rot.append(rot_)
    center.append(center_)
    ellipse = psoid.project(P)
    ellipses.append(ellipse)
    psoid_pcds.append(
        vd.Points(ellcv.visu.draw_3d.generate_ellipsoid_pointcloud(psoid)).c(i)
    )
    ellipse_pnts = sample_ellipse_points(ellipse)
    ellipse_pnts[:, 0] *= 1
    ellipse_pcds.append(
        vd.Points(
            np.insert(ellipse_pnts, ellipse_pnts.shape[1], 0.5, axis=1), c=i
        ).apply_transform(pose)
    )
    #     bbox = bbox_from_ellipse(ellipse)
    #     draw_bbox(img, bbox, color=(255, 0, 0))
    draw_ellipse(img, ellipse, color=ellipse_pcds[-1].color() * 255)

import matplotlib.pyplot as plt

plt.imshow(img)
plt.draw()
plt.pause(0.001)
# cv2.imshow("viz", img)
# v2.waitKey(0)

pic = vd.image.Image(np.flip(img, axis=0)).scale(2 / W).apply_transform(pose).alpha(0.9) # image needs to be flipped, due to opengl coordinate system?? also scaled down (not sure if scale ration is correct though)
vd.show(psoid_pcds, pic, axes=1, interactive=True).close()

[marcomusy]

hi @ttsesm
unfortunately i cannot run pip install 'git+https://gitlab.inria.fr/tangram/pyellcv.git'
as it fails compiling...
Maybe you just miss the orthogonal view? Try:
vd.settings.use_parallel_projection = True

[ttsesm]

@marcomusy what python version are you using? maybe you could try with v3.10 (I think I had similar issues installing the lib with a newer python version)

The vd.settings.use_parallel_projection = True will not work (and actually it didn't), since my camera is each time randomly set and in practice I need to be always behind the camera plane in order to be able to see the ellipsoids through the projected plane (I do not know if I explain it well).

[marcomusy]

..it still fails compiling..
anyways i'm not sure I get what you are trying to do ... maybe you ca use the vedo equivalent pca_ellipsoid()?

[ttsesm]

Hhhmmm... interesting...

How I can save the ellipsoids and ellipses points... 🤔
I am trying the following command vd.show(ellipsoid_pcds, ellipses_pcds, axes=1, interactive=True).export("scene.npz").close() but when I am loading it with plt = vd.load("scene.npz") I am getting the following error: [vedo.file_io:268] ERROR: in load(), cannot load scene.npz both ellipsoid_pcds and ellipses_pcds are lists of vd.Points() objects.

I've tried also with pickle, based on the recent support of the vtkmodules but this didn't work as well.

Also can I use somehow the pca_ellipsoid() and pca_ellipse() functions to create my ellipsoids and ellipses from my mean and covariance matrices?

[marcomusy]

thanks for the link! if you have many such ellipsoids you can merge and save them normally with

my_list_of_ellis
...
ellis = vedo.merge(my_list_of_ellis, flag=1)
ellis.write("merged_ellis.vtk")

another option is to save an Assembly, see examples/basic/slider_browser.py

[ttsesm]

The vedo.merge() is not a good solution since doesn't maintain the colors of the individual meshes/pcds which is importance in this case scenario.

The vedo.Assembly() seems to be fine, it saves the objects, but again when I am trying to load them (vedo.load("data_scene.npz")) I am getting the error I've pointed earlier above: [vedo.file_io:268] ERROR: in load(), cannot load data_scene.npz though the file seems to be valid.

I am saving the Assembly object with the following command: vd.show(vd.Assembly(ellipsoid_pcds, ellipses_pcds), axes=1, interactive=True).export("data_scene.npz").close() if I use vedo.write(vd.Assembly(ellipsoid_pcds, ellipses_pcds), "data_scene.npz") I get the error {AttributeError}AttributeError("'Assembly' object has no attribute 'dataset'").

[marcomusy]

I just pushed a fix.. try:

from vedo import *

# test write
ellis = []
for i in range(200):
    elli = Ellipsoid().pos(5*np.random.randn(3)).rotate_z(i)
    elli.c(i).alpha(0.1)
    elli.metadata['my_name'] = f'ellipsoid_{i}'
    ellis.append(elli)
asse = Assembly(ellis).shift([100,0,0])
asse.write("ellipses.npy")

# test read
asse2 = Assembly("ellipses.npy")
meshes = asse2.unpack()
meshes[0].alpha(1).print()
show(asse2, axes=1)

[ttsesm]

Thanks Marco, indeed now everything seems to work fine.
Please find the script bellow to load the scene.
I am also giving you the means and covariance matrices of both the 2D (mt, ct) and 3D (Ms, Cs) gaussians in case that there is a way to create the ellipses and ellipsoids respectively in vedo from such data (though checking on the pca_ellipsoid() and pca_ellipse() it doesn't seem so).

import vedo as vd
import numpy as np
import pickle

asse = vd.Assembly("gaussians.npy")
meshes = asse.unpack()
ellipsoids = meshes[:100]
ellipses = meshes[100:]

with open('data.pkl', 'rb') as inp:
    img = pickle.load(inp)
    pose = pickle.load(inp)
    mt = pickle.load(inp)
    ct = pickle.load(inp)
    Ms = pickle.load(inp)
    Cs = pickle.load(inp)

import matplotlib.pyplot as plt
# image plane where ellipsoids are projected as ellipses
plt.imshow(img)
plt.draw()
plt.pause(0.001)

W, H = 640, 480
K = np.array([[500.0, 0.0, float(W) / 2], [0.0, 500.0, float(H) / 2], [0.0, 0.0, 1.0]]) # camera intrinsics
pic = vd.image.Image(np.flip(img, axis=0)).scale(2 / max(W,H)).apply_transform(pose).alpha(0.9) # image needs to be flipped, due to opengl coordinate system?? also scaled down to be compatible with the NDC coordinates (not sure if scale ratio is correct though, but it should be fine)
vd.show(ellipsoids, ellipses, pic, axes=1, interactive=True).close()

So, now the initial question is how I can set the camera (if possible) position in 3D so that I look from behind and overlay the projected ellipses on the image plane to the ellipsoids 🤔

btw, the projected ellipsoids that are outside the plane are obviously not visible that's why you also have some ellipses being cut out fully or partially on the image plane since not all of them fit on the image.

data.zip