Haoquan/rtp video stream control #308

hq286 · 2022-11-23T20:34:26Z

Built a new remote python controller with low latency RTP video stream for mobile network connection. New python controller supports autopilot, recording, without crashing remote video stream.

izivkov · 2022-11-28T20:29:44Z

I am trying to run the new branch. One thing I noticed running the main app:

after setting the Stream Mode to "RTP" in settings, and then selecting "Free Roam" fragment, the program crashes:

D/AndroidRuntime: Shutting down VM
E/AndroidRuntime: FATAL EXCEPTION: main
Process: org.openbot, PID: 26454
java.lang.ClassCastException: org.openbot.main.MainActivity cannot be cast to android.media.ImageReader$OnImageAvailableListener
at org.openbot.env.RtpServer.init(RtpServer.java:49)
at org.openbot.env.PhoneController.init(PhoneController.java:64)
at org.openbot.env.PhoneController.getInstance(PhoneController.java:33)
at org.openbot.common.ControlsFragment.onViewCreated(ControlsFragment.java:76)
at org.openbot.robot.FreeRoamFragment.onViewCreated(FreeRoamFragment.java:44)
at androidx.fragment.app.Fragment.performViewCreated(Fragment.java:2987)
at androidx.fragment.app.FragmentStateManager.createView(FragmentStateManager.java:546)
at androidx.fragment.app.FragmentStateManager.moveToExpectedState(FragmentStateManager.java:282)
at androidx.fragment.app.FragmentManager.executeOpsTogether(FragmentManager.java:2189)
at androidx.fragment.app.FragmentManager.removeRedundantOperationsAndExecute(FragmentManager.java:2106)
at androidx.fragment.app.FragmentManager.execPendingActions(FragmentManager.java:2002)
at androidx.fragment.app.FragmentManager$5.run(FragmentManager.java:524)
at android.os.Handler.handleCallback(Handler.java:938)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loop(Looper.java:250)
at android.app.ActivityThread.main(ActivityThread.java:7755)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:958)
V/FA: Recording user engagement, ms: 5450

izivkov · 2022-11-28T20:32:17Z

Is the new python script called "joystick_controller.py"? I cannot test it since I do not have a joystick. Is it a particular joystick model that works with it? I am wondering if the RTP stream would still be useful with keyboard-only controller, since most people do not have joystick.

thias15 · 2022-11-29T22:28:15Z

In my opinion, having also a keyboard controller would be useful.

thias15 · 2022-11-29T22:30:20Z

android/app/src/main/java/org/openbot/env/ControllerConfig.java

+  }
+
+  public String[] getVideoServerAddress() {
+    String ip = get("ip", "172.217.22.14");


I think we should have some neutral IP as default or just something like N/A if no server is setup.

I changed it to "127.0.0.1"

thias15 · 2022-11-29T22:32:33Z

android/app/build.gradle

@@ -80,6 +80,9 @@ dependencies {
    implementation 'com.github.pedroSG94:RTSP-Server:1.0.8'
    implementation 'com.github.pedroSG94.rtmp-rtsp-stream-client-java:rtplibrary:2.0.2'

+    //RTP Client
+    implementation 'com.github.ar-android:libstreaming:1.0.0'


We should use the original library.
https://github.com/fyhertz/libstreaming

I tried the original library in the beginning. However, the official libstreaming published on jitpack is the very old version, and they didn't publish their latest version. I'm not sure if there is another way around. Do you have any idea?

thias15 · 2022-11-29T22:36:55Z

controller/python/README.md

+
+Controller mapping
+```
+▲: autodrive


On the BT joystick, autopilot is on R1, data logging on X. Let's be consistent. Triangle, square and circle are for the indicators.

thias15 · 2022-12-03T17:29:49Z

@haoquan on what OS did you test this? On Mac I'm unable to install approxeng.input. Is this essential or could it be done differently?

error: legacy-install-failure

thias15 · 2022-12-03T17:31:41Z

Also, what kind of latency are you getting with RTP for the video stream?

thias15 · 2022-12-17T13:40:09Z

@haoquan, after further testing I also encountered frequent app crashes.

java.lang.ClassCastException: org.openbot.main.MainActivity cannot be cast to android.media.ImageReader$OnImageAvailableListener

Did you not face these app crashes?

hq286 · 2022-12-20T15:45:05Z

I am trying to run the new branch. One thing I noticed running the main app:

after setting the Stream Mode to "RTP" in settings, and then selecting "Free Roam" fragment, the program crashes:

D/AndroidRuntime: Shutting down VM E/AndroidRuntime: FATAL EXCEPTION: main Process: org.openbot, PID: 26454 java.lang.ClassCastException: org.openbot.main.MainActivity cannot be cast to android.media.ImageReader$OnImageAvailableListener at org.openbot.env.RtpServer.init(RtpServer.java:49) at org.openbot.env.PhoneController.init(PhoneController.java:64) at org.openbot.env.PhoneController.getInstance(PhoneController.java:33) at org.openbot.common.ControlsFragment.onViewCreated(ControlsFragment.java:76) at org.openbot.robot.FreeRoamFragment.onViewCreated(FreeRoamFragment.java:44) at androidx.fragment.app.Fragment.performViewCreated(Fragment.java:2987) at androidx.fragment.app.FragmentStateManager.createView(FragmentStateManager.java:546) at androidx.fragment.app.FragmentStateManager.moveToExpectedState(FragmentStateManager.java:282) at androidx.fragment.app.FragmentManager.executeOpsTogether(FragmentManager.java:2189) at androidx.fragment.app.FragmentManager.removeRedundantOperationsAndExecute(FragmentManager.java:2106) at androidx.fragment.app.FragmentManager.execPendingActions(FragmentManager.java:2002) at androidx.fragment.app.FragmentManager$5.run(FragmentManager.java:524) at android.os.Handler.handleCallback(Handler.java:938) at android.os.Handler.dispatchMessage(Handler.java:99) at android.os.Looper.loop(Looper.java:250) at android.app.ActivityThread.main(ActivityThread.java:7755) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:958) V/FA: Recording user engagement, ms: 5450

I have fixed this issue. Check out the latest commit

hq286 · 2022-12-20T15:46:01Z

@haoquan, after further testing I also encountered frequent app crashes.
java.lang.ClassCastException: org.openbot.main.MainActivity cannot be cast to android.media.ImageReader$OnImageAvailableListener
Did you not face these app crashes?

I fixed the issue. Check out the latest commit

hq286 · 2022-12-20T15:50:45Z

Is the new python script called "joystick_controller.py"? I cannot test it since I do not have a joystick. Is it a particular joystick model that works with it? I am wondering if the RTP stream would still be useful with keyboard-only controller, since most people do not have joystick.

Yes. It should work with major bluetooth joystick controllers like PS4, and xbox and etc.
I agree that a keyboard controller would be very convenient. The problem now is that I am using cv2 as the video displayer, which will cause conflict between keyboard control and cv2 video display (waitkey). It turns out that PyGame is introducing a lot of latency when displaying the video. I will think about how to resolve this issue.

hq286 · 2022-12-20T15:57:14Z

@haoquan on what OS did you test this? On Mac I'm unable to install approxeng.input. Is this essential or could it be done differently?
error: legacy-install-failure

I developed and tested everything on a Linux computer. Unfortunately approxeng.input only supports Linux system. I'm stilly working on finding a solution for Mac.

hq286 · 2022-12-20T15:59:19Z

Also, what kind of latency are you getting with RTP for the video stream?

I tested it on a Linux computer. The latency is about 150ms on average. It could vary depending on your network condition.

izivkov · 2022-12-20T16:41:15Z

That is pretty good response time. I will check it out. Can I run the app and see the video without a Joystick?

izivkov · 2022-12-20T17:10:31Z

I am trying to run the new branch. One thing I noticed running the main app:

after setting the Stream Mode to "RTP" in settings, and then selecting "Free Roam" fragment, the program crashes:

D/AndroidRuntime: Shutting down VM E/AndroidRuntime: FATAL EXCEPTION: main Process: org.openbot, PID: 26454 java.lang.ClassCastException: org.openbot.main.MainActivity cannot be cast to android.media.ImageReader$OnImageAvailableListener at org.openbot.env.RtpServer.init(RtpServer.java:49) at org.openbot.env.PhoneController.init(PhoneController.java:64) at org.openbot.env.PhoneController.getInstance(PhoneController.java:33) at org.openbot.common.ControlsFragment.onViewCreated(ControlsFragment.java:76) at org.openbot.robot.FreeRoamFragment.onViewCreated(FreeRoamFragment.java:44) at androidx.fragment.app.Fragment.performViewCreated(Fragment.java:2987) at androidx.fragment.app.FragmentStateManager.createView(FragmentStateManager.java:546) at androidx.fragment.app.FragmentStateManager.moveToExpectedState(FragmentStateManager.java:282) at androidx.fragment.app.FragmentManager.executeOpsTogether(FragmentManager.java:2189) at androidx.fragment.app.FragmentManager.removeRedundantOperationsAndExecute(FragmentManager.java:2106) at androidx.fragment.app.FragmentManager.execPendingActions(FragmentManager.java:2002) at androidx.fragment.app.FragmentManager$5.run(FragmentManager.java:524) at android.os.Handler.handleCallback(Handler.java:938) at android.os.Handler.dispatchMessage(Handler.java:99) at android.os.Looper.loop(Looper.java:250) at android.app.ActivityThread.main(ActivityThread.java:7755) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:958) V/FA: Recording user engagement, ms: 5450

I have fixed this issue. Check out the latest commit

OK, the code works with RTP selected, but it crashes if we select WebRTC, and then select Free Roam fragment. RTSP does not crash.

hq286 · 2022-12-20T20:30:43Z

Fixed it. There was a type with webrtc.

hq286 · 2022-12-20T20:32:49Z

That is pretty good response time. I will check it out. Can I run the app and see the video without a Joystick?

Yes. You can run keyboard-pygame.py without the video flag. Then run the camera.py. Make sure you are in the RTP mode, and match the ports for control and stream

hq286 · 2022-12-20T20:34:23Z

This works at least in Linux. I'm not sure about Mac. Getting GStreamer, opencv and python to work together on Mac needs some work

thias15 · 2022-12-28T10:47:08Z

I compiled OpenCV with GStreamer support on Mac, but when I try to run the camera, I get the following error.

Failed to open camera
Src opened, 0x0 @ 0 fps

@izivkov did you manage to get this running under Linux?

izivkov · 2022-12-28T13:47:29Z

No, I got the same error. I never got video to work with the new Python script. I thought it was something wrong with my configuration.

…

On Wed., Dec. 28, 2022, 5:47 a.m. thias15, ***@***.***> wrote: I compiled OpenCV with GStreamer support on Mac, but when I try to run the camera, I get the following error. Failed to open camera Src opened, 0x0 @ 0 fps @izivkov <https://github.com/izivkov> did you manage to get this running under Linux? — Reply to this email directly, view it on GitHub <#308 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7M37RZMKFEDBNHXTSFSO3WPQLDNANCNFSM6AAAAAASJLNUYQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

thias15 · 2022-12-31T15:42:18Z

@hq286 any idea what we are doing wrong?

hq286 · 2023-01-01T03:17:52Z

No, I got the same error. I never got video to work with the new Python script. I thought it was something wrong with my configuration.
…
On Wed., Dec. 28, 2022, 5:47 a.m. thias15, @.> wrote: I compiled OpenCV with GStreamer support on Mac, but when I try to run the camera, I get the following error. Failed to open camera Src opened, 0x0 @ 0 fps @izivkov https://github.com/izivkov did you manage to get this running under Linux? — Reply to this email directly, view it on GitHub <#308 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7M37RZMKFEDBNHXTSFSO3WPQLDNANCNFSM6AAAAAASJLNUYQ . You are receiving this because you were mentioned.Message ID: @.>

It's the problem of GStreamer pipeline configuration. This configuration works in Ubuntu 20.04. I need to investigate the configuration on MacBook

thias15 · 2023-01-03T13:23:34Z

No, I got the same error. I never got video to work with the new Python script. I thought it was something wrong with my configuration.
…
On Wed., Dec. 28, 2022, 5:47 a.m. thias15, @.> wrote: I compiled OpenCV with GStreamer support on Mac, but when I try to run the camera, I get the following error. Failed to open camera Src opened, 0x0 @ 0 fps @izivkov https://github.com/izivkov did you manage to get this running under Linux? — Reply to this email directly, view it on GitHub <#308 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7M37RZMKFEDBNHXTSFSO3WPQLDNANCNFSM6AAAAAASJLNUYQ . You are receiving this because you were mentioned.Message ID: @.>

How did you install OpenCV? Can you try the script I added to the repo? Make sure to install GStreamer before.

izivkov · 2023-01-03T15:00:56Z

@thias15 What is the name of your "install OpenCV" script and what branch is it in?

thias15 · 2023-01-04T09:31:07Z

I pushed it to this PR, it's called install_opencv.sh

thias15 · 2023-01-11T12:49:51Z

@izivkov any update on your end?

thias15 · 2023-01-11T12:51:33Z

By the way, I think we should also unify the port setting, at least for the controller across the different scripts.

izivkov · 2023-01-11T14:49:39Z

@izivkov any update on your end?

I installed GStreamer and OpenCV using your installation script on my Linux PC but did not see video. Basically, the Video Window never appears. I used the following command:

python3 joystick_controller.py -video

Maybe it is my PC - only a Celaron CPU. I'm not sure if it needs a more powerful GPU.

Also, I'm not that familiar with GStreamer and OpenCV, so I may be doing some incorrectly.

haoquan and others added 8 commits November 18, 2022 14:46

added libstreaming library in gradle file; added resource xml files

1f281f9

added a mobile network connection for vehicle control message

bfe6488

updated controller config, and phone controller, and camera activity

9ff45e7

added a new RTP video server type

d260dc5

added a RTP video stream displayer, and a python joystick controller

75627c3

updated README and root preference file

7498931

updated README

4425b08

fixed code styles for python and java

6810eaf

thias15 reviewed Nov 29, 2022

View reviewed changes

fixed free roam bug, and changed the default ip in settings

99f6c5c

fixed code style

83195ee

fixed webrtc init

43e8d30

OpenCV install script

49c1205

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Haoquan/rtp video stream control #308

Haoquan/rtp video stream control #308

hq286 commented Nov 23, 2022

izivkov commented Nov 28, 2022

izivkov commented Nov 28, 2022 •

edited

Loading

thias15 commented Nov 29, 2022

thias15 Nov 29, 2022

hq286 Dec 20, 2022

thias15 Nov 29, 2022

hq286 Dec 20, 2022

thias15 Nov 29, 2022

thias15 commented Dec 3, 2022

thias15 commented Dec 3, 2022

thias15 commented Dec 17, 2022

hq286 commented Dec 20, 2022

hq286 commented Dec 20, 2022

hq286 commented Dec 20, 2022

hq286 commented Dec 20, 2022

hq286 commented Dec 20, 2022

izivkov commented Dec 20, 2022

izivkov commented Dec 20, 2022

hq286 commented Dec 20, 2022

hq286 commented Dec 20, 2022

hq286 commented Dec 20, 2022

thias15 commented Dec 28, 2022

izivkov commented Dec 28, 2022 via email

thias15 commented Dec 31, 2022

hq286 commented Jan 1, 2023

thias15 commented Jan 3, 2023

izivkov commented Jan 3, 2023

thias15 commented Jan 4, 2023

thias15 commented Jan 11, 2023

thias15 commented Jan 11, 2023

izivkov commented Jan 11, 2023 •

edited

Loading

Haoquan/rtp video stream control #308

Are you sure you want to change the base?

Haoquan/rtp video stream control #308

Conversation

hq286 commented Nov 23, 2022

izivkov commented Nov 28, 2022

izivkov commented Nov 28, 2022 • edited Loading

thias15 commented Nov 29, 2022

thias15 Nov 29, 2022

Choose a reason for hiding this comment

hq286 Dec 20, 2022

Choose a reason for hiding this comment

thias15 Nov 29, 2022

Choose a reason for hiding this comment

hq286 Dec 20, 2022

Choose a reason for hiding this comment

thias15 Nov 29, 2022

Choose a reason for hiding this comment

thias15 commented Dec 3, 2022

thias15 commented Dec 3, 2022

thias15 commented Dec 17, 2022

hq286 commented Dec 20, 2022

hq286 commented Dec 20, 2022

hq286 commented Dec 20, 2022

hq286 commented Dec 20, 2022

hq286 commented Dec 20, 2022

izivkov commented Dec 20, 2022

izivkov commented Dec 20, 2022

hq286 commented Dec 20, 2022

hq286 commented Dec 20, 2022

hq286 commented Dec 20, 2022

thias15 commented Dec 28, 2022

izivkov commented Dec 28, 2022 via email

thias15 commented Dec 31, 2022

hq286 commented Jan 1, 2023

thias15 commented Jan 3, 2023

izivkov commented Jan 3, 2023

thias15 commented Jan 4, 2023

thias15 commented Jan 11, 2023

thias15 commented Jan 11, 2023

izivkov commented Jan 11, 2023 • edited Loading

izivkov commented Nov 28, 2022 •

edited

Loading

izivkov commented Jan 11, 2023 •

edited

Loading