Panorama 360 RGBD to point cloud

[sctrueew]

Checklist

• I have searched for similar issues.
• For Python issues, I have tested with the latest development wheel.
• I have checked the release documentation and the latest documentation (for main branch).

My Question

Hi,

I have a 360° panorama image and a depth map, but when I try to extract the point cloud and display it, the output doesn’t appear correctly. Is it possible to fix this or provide some information?

I need the display to look like a typical 360° panorama viewer, not the current incorrect output.

Thanks!

I'm using C++ and here is my code:

std::shared_ptr<open3d::geometry::PointCloud> my_depthTo3DPointCloud_panorama(
	const cv::Mat& depthMap, const cv::Mat& rawImage, const cv::Mat& mask = cv::Mat(), float depth_scale = 1.0f) {

	auto point_cloud = std::make_shared<open3d::geometry::PointCloud>();

	// Get dimensions of the depth map
	int height = depthMap.rows;
	int width = depthMap.cols;

	// Panorama field of view (adjust as necessary for your specific image)
	const float fov_x = 2.0f * M_PI;  // 360 degrees horizontally
	const float fov_y = M_PI;         // 180 degrees vertically

	for (int v = 0; v < height; ++v) {
		for (int u = 0; u < width; ++u) {
			if (mask.empty() || mask.at<uchar>(v, u) > 0) {
				// Retrieve depth value (assuming uchar depth map, scale it for precision)
				float depth = static_cast<float>(depthMap.at<uchar>(v, u)) * depth_scale;

				// Calculate angles for spherical mapping
				float theta = (static_cast<float>(u) / width - 0.5f) * fov_x; // Azimuth angle
				float phi = (0.5f - static_cast<float>(v) / height) * fov_y;  // Elevation angle

				// Convert spherical coordinates to Cartesian coordinates
				float x = depth * cos(phi) * sin(theta);
				float y = depth * sin(phi);
				float z = depth * cos(phi) * cos(theta);

				// Add the 3D point
				point_cloud->points_.emplace_back(x, y, z);

				// Extract color information from the original image
				cv::Vec3b color = rawImage.at<cv::Vec3b>(v, u);
				point_cloud->colors_.emplace_back(color[2] / 255.0, color[1] / 255.0, color[0] / 255.0);
			}
		}
	}

	return point_cloud;
}

point cloud