ARCore Depth Lab - Depth API Samples for Unity

Copyright 2020 Google LLC

Depth Lab is a set of ARCore Depth API samples that provides assets using depth for advanced geometry-aware features in AR interaction and rendering. Some of these features have been used in this Depth API overview video.

ARCore Depth API is enabled on a subset of ARCore-certified Android devices. iOS devices (iPhone, iPad) are not supported. Find the list of devices with Depth API support (marked with Supports Depth API) here: https://developers.google.com/ar/devices. See the ARCore developer documentation for more information.

Download the pre-built ARCore Depth Lab app on Google Play Store today.

Branches

ARCore Depth Lab has two branches: master and arcore_unity_sdk.

The master branch contains a subset of Depth Lab features in v1.1.0 and is built upon the recommended AR Foundation 4.2.0 (preview 7) or newer. The master branch supports features including oriented 3D reticles, depth map visualization, collider with depth mesh, avatar locomotion, raw point cloud visualization, recording and playback.

The arcore_unity_sdk branch contains the full features of Depth Lab and is built upon ARCore SDK for Unity v1.24.0 or newer. We recommend using the master branch to build new projects with the AR Foundation SDK and refer to this branch when necessary.

Getting started

These samples target Unity 2020.3.6f1 and require AR Foundation 4.2.0-pre.7 or newer, ARCore Extensions 1.24 or newer. Download arcore-unity-extensions-1.24.0.tgz from the ARCore Extensions for AR Foundation releases page on GitHub.

Choose one of the following approaches to import the ARCore Extensions Package:

1. Manually update dependencies before launching the project:

   • Locate arcore-depth-lab/Packages/manifest.json.
   • Add the dependency of ARCore Extensions SDK with the loca path: "com.google.ar.core.arfoundation.extensions": "file:{path/to}/arcore-unity-extensions-1.24.0.tgz",
   • Launch the project with Unity Editor 2020.3.

2. Use Package Manager to import dependent package:

   • Use Unity 2020.3 to open the project.
   • It's expected that a window pops up and shows there are compile errors in this project due to missing Extensions SDK, select Continue.
   • In Unity editor, navigate to Window -> Package Manager
   • Click the plus button, and choose the Add package from tarball... option from the drop-down menu.
   • Locate arcore-unity-extensions-1.24.0.tgz and click Open.

This project only builds with the Build Platform Android. Build the project to an Android device instead of using the Play button in the Unity editor.

Sample features

The sample scenes demonstrate three different ways to access depth. Supported features in the master branch is labeled with ⭐, while the rest features can be found in the arcore_unity_sdk branch.

1. Localized depth: Sample single depth values at certain texture coordinates (CPU).
   • Oriented 3D reticles ⭐
   • Character locomotion on uneven terrain ⭐
   • Collision checking for AR object placement
   • Laser beam reflections
   • Rain and snow particle collision
2. Surface depth: Create a connected mesh representation of the depth data (CPU/GPU).
   • Point cloud fusion ⭐
   • AR shadow receiver
   • Paint splat
   • Physics simulation
   • Surface retexturing
3. Dense depth: Process depth data at every screen pixel (GPU).
   • False-color depth map ⭐
   • AR fog
   • Occlusions
   • Depth-of-field blur
   • Environment relighting
   • 3D photo

Building samples

Individual scenes can be built and run by enabling a particular scene (e.g., OrientedReticle to try out the oriented 3D reticle.) and the ARFDepthComponents object in the scene. Remember to disable the ARFDepthComponents object in individual scenes when building all demos with the DemoCarousel scene.

We also provide a demo user interface that allows users to seamlessly switch between examples. Please make sure to set the Build Platform to Android and verify that the main DemoCarousel scene is the first enabled scene in the Scenes In Build list under Build Settings. Enable all scenes that are part of the demo user interface.

Assets/ARRealismDemos/DemoCarousel/Scenes/DemoCarousel.unity Assets/ARRealismDemos/OrientedReticle/Scenes/OrientedReticle.unity Assets/ARRealismDemos/DepthEffects/Scenes/DepthEffects.unity Assets/ARRealismDemos/Collider/Scenes/Collider.unity Assets/ARRealismDemos/AvatarLocomotion/Scenes/AvatarLocomotion.unity Assets/ARRealismDemos/PointCloud/Scenes/RawPointClouds.unity

The following scenes can be found in the arcore_unity_sdk branch, but are not yet available with the AR Foundation SDK.

Assets/ARRealismDemos/MaterialWrap/Scenes/MaterialWrap.unity Assets/ARRealismDemos/Splat/Scenes/OrientedSplat.unity Assets/ARRealismDemos/LaserBeam/Scenes/LaserBeam.unity Assets/ARRealismDemos/Relighting/Scenes/PointsRelighting.unity Assets/ARRealismDemos/DepthEffects/Scenes/FogEffect.unity Assets/ARRealismDemos/SnowParticles/Scenes/ArCoreSnowParticles.unity Assets/ARRealismDemos/RainParticles/Scenes/RainParticlesScene.unity Assets/ARRealismDemos/DepthEffects/Scenes/DepthOfFieldEffect.unity Assets/ARRealismDemos/Water/Scenes/Water.unity Assets/ARRealismDemos/CollisionDetection/Scenes/CollisionAwareObjectPlacement.unity Assets/ARRealismDemos/ScreenSpaceDepthMesh/Scenes/ScreenSpaceDepthMesh.unity Assets/ARRealismDemos/ScreenSpaceDepthMesh/Scenes/StereoPhoto.unity

Sample project structure

The main sample assets are placed inside the Assets/ARRealismDemos folder. Each subfolder contains sample features or helper components.

AvatarLocomotion

The AR character in this scene follows user-set waypoints while staying close to the surface of an uneven terrain. This scene uses raycasting and depth lookups on the CPU to calculate a 3D point on the surface of the terrain.

Collider

This physics simulation playground uses screen-space depth meshes to enable collisions between Unity's rigid-body objects and the physical environment.

After pressing an on-screen button, a Mesh object is procedurally generated from the latest depth map. This is used to update the sharedMesh parameter of the MeshCollider object. A randomly selected primitive rigid-body object is then thrown into the environment.

CollisionDetection

This AR object placement scene uses depth lookups on the CPU to test collisions between the vertices of virtual objects and the physical environment.

Common

This folder contains scripts and prefabs that are shared between the feature samples. For more details, see the Helper Classes section below.

DemoCarousel

This folder contains the main scene, which provides a carousel user interface. This scene allows the user to seamlessly switch between different features. A scene can be selected by directly touching a preview thumbnail or dragging the carousel UI to the desired position.

DepthEffects

This folder contains three dense depth shader processing examples.

The DepthEffects scene contains a fragment-shader effect that can transition from the AR camera view to a false-color depth map. Warm colors indicate closer regions in the depth map. Cold colors indicate further regions.

The DepthOfFieldEffect scene contains a simulated Bokeh fragment-shader effect. This blurs the regions of the AR view that are not at the user-defined focus distance. The focus anchor is set in the physical environment by touching the screen. The focus anchor is a 3D point that is locked to the environment and always in focus.

The FogEffect scene contains a fragment-shader effect that adds a virtual fog layer on the physical environment. Close objects will be more visible than objects further away. A slider controls the density of the fog.

LaserBeam

This laser reflection scene allows the user to shoot a slowly moving laser beam by touching anywhere on the screen.

This uses:

• The DepthSource.GetVertexInWorldSpaceFromScreenXY(..) function to look up a raycasted 3D point
• The ComputeNormalMapFromDepthWeightedMeanGradient(..) function to look up the surface normal based on a provided 2D screen position.

MaterialWrap

This experience allows the user to change the material of real-world surfaces through touch. This uses depth meshes.

OrientedReticle

This sample uses depth hit testing to obtain the raycasted 3D position and surface normal of a raycasted screen point.

PointCloud

This sample computes a point cloud on the CPU using the depth array. Press the Update button to compute a point cloud based on the latest depth data.

RawPointClouds

This sample fuses point clouds with the raw depth maps on the CPU using the depth array. Drag the confidence slider to change the visibility of each point based on the confidence value of the corresponding raw depth.

RainParticles

This sample uses the GPU depth texture to compute collisions between rain particles and the physical environment.

Relighting

This sample uses the GPU depth texture to computationally re-light the physical environment through the AR camera. Areas of the physical environment close to the artificial light sources are lit, while areas farther away are darkened.

ScreenSpaceDepthMesh

This sample uses depth meshes. A template mesh containing a regular grid of triangles is created once on the CPU. The GPU shader displaces each vertex of the regular grid based on the reprojection of the depth values provided by the GPU depth texture. Press Freeze to take a snapshot of the mesh and press Unfreeze to revert back to the live updating mesh.

StereoPhoto

This sample uses depth meshes and ScreenSpaceDepthMesh. After freezing the mesh, we cache the current camera's projection and view matrices, circulate the camera around a circle, and perform projection mapping onto the depth mesh with the cached camera image. Press Capture to create the animated 3D photo and press Preview to go back to camera preview mode.

SnowParticles

This sample uses the GPU depth texture to compute collisions between snow particles, the physical environment, and the orientation of each snowflake.

Splat

This sample uses the Oriented Reticle and the depth mesh in placing a surface-aligned texture decal within the physical environment.

Water

This sample uses a modified GPU occlusion shader to create a flooding effect with artificial water in the physical environment.

Helper classes

DepthSource

A singleton instance of this class contains references to the CPU array and GPU texture of the depth map, camera intrinsics, and many other depth look up and coordinate transformation utilities. This class acts as a high-level wrapper for the MotionStereoDepthDataSource class.

DepthTarget

Each GameObject containing a DepthTarget becomes a subscriber to the GPU depth data. DepthSource will automatically update the depth data for each DepthTarget. At least one instance of DepthTarget has to be present in the scene in order for DepthSource to provide depth data.

MotionStereoDepthDataSource

This class contains low-level operations and direct access to the depth data. It should only be use by advanced developers.

User privacy requirements

You must prominently disclose the use of Google Play Services for AR (ARCore) and how it collects and processes data in your application. This information must be easily accessible to end users. You can do this by adding the following text on your main menu or notice screen: "This application runs on Google Play Services for AR (ARCore), which is provided by Google LLC and governed by the Google Privacy Policy".

Related publication

Please refer to https://augmentedperception.github.io/depthlab/ for our paper, supplementary material, and presentation published in ACM UIST 2020: "DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality".

References

If you use ARCore Depth Lab in your research, please reference it as:

@inproceedings{Du2020DepthLab,
  title = {{DepthLab: Real-time 3D Interaction with Depth Maps for Mobile Augmented Reality}},
  author = {Du, Ruofei and Turner, Eric and Dzitsiuk, Maksym and Prasso, Luca and Duarte, Ivo and Dourgarian, Jason and Afonso, Joao and Pascoal, Jose and Gladstone, Josh and Cruces, Nuno and Izadi, Shahram and Kowdle, Adarsh and Tsotsos, Konstantine and Kim, David},
  booktitle = {Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology},
  year = {2020},
  publisher = {ACM},
  pages = {829--843},
  series = {UIST '20}
  doi = {10.1145/3379337.3415881}
}

or

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim. 2020. DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality. Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST '20), 829-843. DOI: http://dx.doi.org/10.1145/3379337.3415881.

We would like to also thank Levana Chen, Xinyun Huang, and Ted Bisson for integrating DepthLab with AR Foundation.

Additional information

You may use this software under the Apache 2.0 License.