DistanceAssistant

Introduction

DistanceAssistant is a social distancing monitor which visually alerts people when they are not observing proper social distancing guidelines. The application is packaged and deployed as a docker container.

Inside of the Docker container, this application uses the Robot Operating System (ROS) as the underlying framework for runtime, launch, and configuration. For more details, see: https://www.ros.org/

Host Setup

This project has been run/tested on a machine with the following specifications/software installed:

• Ubuntu 18.04
  • Other operating systems may also work, but exposing the RealSense camera USB to VMs is incredibly tricky. Thus, a Linux OS with native Docker support is recomended.
• An Nvidia GPU
  • The GPU must support CUDA as mentioned below
  • We recommend a GTX 1070 or better
• CUDA 10.0 or greater: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html
• An Intel DS435i Realsense camera
  • Other models may work, but this is the only model that was tested
• docker-ce
  • Tested on Docker 19.03, but older/newer versions may work
• nvidia container toolkit
• Intel RealSense SDK
  • Intel realsense drivers can be installed by following the directions here: https://github.com/IntelRealSense/librealsense/blob/master/doc/installation.md.
  • Note: the pre-built dkms drivers provided by Intel did not work for us. We compile the drivers using the instructions in the librealsense README for compilation.
  • Note: if compiling for Ubuntu 20.04 and kernel version 5.4, some scripts need to be patched. Further information can be found in the librealsense directory.

NOTE: It is recommended that the RealSense camera be attached to the PC via a USB 3.1 port with a powered USB hub or an "active" (powered) USB cable. During testing on various PCs, we have seen intermittent issues due to lack of power on the USB port. This can happen across two PCs with the same model number. A powered USB3 hub greatly improves the reliability of the RealSense camera

Example Host Setup

To follow along with an example host setup from a fresh installation of Ubuntu 18.04, see the Example Host Setup Instructions.

Docker configuration

The following instructions use the --gpus=all flag to enable the NVIDIA GPU inside of the Docker container. However, for older versions of Docker (pre-19.03), the flag would be --runtime=nvidia. See the NVIDIA Container Toolkit instructions above for more.

Ubuntu 18.04 Alternative Method

An alternative method to passing the --gpus flag would be to modify the /etc/docker/daemon.json file to use NVIDIA as the default runtime.

To do so, install the nvidia-container-runtime:

sudo apt install nvidia-container-runtime

Then, add the following to your /etc/docker/daemon.json:

"default-runtime": "nvidia",
"runtimes": {
    "nvidia": {
        "path": "/usr/bin/nvidia-container-runtime",
        "runtimeArgs": []
    }
}

Finally, restart Docker:

sudo service docker restart

Now, you can omit the --gpus flag from the runtime Docker command.

Build Instructions

To build the docker image, first download the yolo model weights and build the docker file. The weights should be stored in the distance_assistant/src folder.

# cd <distance-assistant folder>
# The below command assumes that it's run in the root folder of the
# repository
wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.weights  -P distance_assistant/src/
docker build --network=host . -t distance_assistant/prototype

# The source tree should look like this:
# <root>/
#    distance_assistant/
#      scripts/
#      msg/
#      src/
#        yolov4.weights
#    Dockerfile
#    README.md

Camera Mount Instructions

We suggest to mount the camera between 1.7 and 2.1 meters above the ground and slightly tilted down (~10 degrees) in order to maximize the usable depth sensor FOV.

During the application startup we perform an automated calibration step which computes the camera pitch & roll as well as it's height from the ground plane. Therefore it's important that the camera is stationary and there are no large objects in front of the camera blocking the ground when you launch the application.

Local Execution Instructions

To execute a locally built docker image,

# IMPORTANT!!
# allow connections to X server
xhost +local:root

# NOTE: The realsense camera should be connected to the host.
docker run \
    --gpus=all \
    --net=host \
    --privileged \
    --device=/dev/usb \
    --device=/dev/media0 \
    --device=/dev/media1 \
    --device=/dev/video0 \
    --device=/dev/video1 \
    --device=/dev/video2 \
    --device=/dev/video3 \
    --device=/dev/video4 \
    --device=/dev/video5 \
    --env="DISPLAY" \
    --env="QT_X11_NO_MITSHM=1" \
    --volume=/tmp/.X11-unix:/tmp/.X11-unix:rw \
    -it \
    distance_assistant/prototype

Visualization

Distance Assistant should automatically open a video window for you. If it does not, or if you are working on the code and need to connect to the Distance Assistant, you can use the rviz tool.

On Ubuntu, install it by running

sudo apt-get install rviz

Then, run rviz and open the config_file.rviz file in the distance_assistant/rviz directory.

Troubleshooting

Display Failures

You might see errors like this:

[ INFO] [1592853388.649425112]: Using transport "raw"
No protocol specified
Unable to init server: Could not connect: Connection refused

(vis_img:105): Gtk-WARNING **: 19:16:28.652: cannot open display: :1

[image_view-5] process has died [pid 105, exit code -11

These errors mean that Docker does not have access to the X11 display server. To fix this issue, ensure that you run the following command before starting the Docker container via docker run:

xhost +local:root