Bluetooth support and amazon fire stick

[phant0mias]

Hi,

I managed to control a amazon fire stick with omote.

I opened the discussion to share it with more people. Some features I still cannot use with bluetooth:

• controlling the volume up/down/mute
• on/off

These features work with IR but might be interesting if you have only bluetooth.

I needed some time to get it stable, I realized that it was working really good with libraries that use NIMBLE.

Now how it can be implemented.

Use the BLE Gamepad Library:

https://github.com/lemmingDev/ESP32-BLE-Gamepad

When using the library you should use the newest platformio.ini file to get more flash memory (#43)

A small problem is that the boot time is getting higher.

After connecting the device in the fire stick settings as gamecontroller you can send the remote commands. Automatic reconnect after power down the TV and remote is working too.

You have to experiment with it and check the examples of this class.

I had an effort with the followings commands:

#include <BleGamepad.h>

BleGamepad bleGamepad

void setup()
{
.....
BleGamepadConfiguration bleGamepadConfig;
bleGamepadConfig.setAutoReport(true);
bleGamepadConfig.setControllerType(CONTROLLER_TYPE_GAMEPAD);
bleGamepadConfig.setIncludeStart(true);
bleGamepadConfig.setIncludeSelect(true);
bleGamepadConfig.setIncludeMenu(true);
bleGamepadConfig.setIncludeHome(true);
bleGamepadConfig.setIncludeBack(true);

bleGamepad.begin(&bleGamepadConfig);
....
}

as commands you can send:

// Back
bleGamepad.press(BUTTON_2);
delay( 20 );
bleGamepad.release(BUTTON_2);

// Down
bleGamepad.setHat1(HAT_DOWN);
delay( 20 );
bleGamepad.setHat1(HAT_CENTERED);

// Home
bleGamepad.press(BUTTON_12);
delay( 20 );
bleGamepad.release(BUTTON_12);

// Left
bleGamepad.setHat1(HAT_LEFT);
delay( 20 );
bleGamepad.setHat1(HAT_CENTERED);

// Menu
bleGamepad.press(BUTTON_11);
delay( 20 );
bleGamepad.release(BUTTON_11);

// OK
bleGamepad.press(BUTTON_1);
delay( 20 );
bleGamepad.release(BUTTON_1);

// Right
bleGamepad.setHat1(HAT_RIGHT);
delay( 20 );
bleGamepad.setHat1(HAT_CENTERED);

// Up
bleGamepad.setHat1(HAT_UP);
delay( 20 );
bleGamepad.setHat1(HAT_CENTERED);

[Dark1886]

Is there a typo in this? I see some m_bleGamepad and some without the "m_".

[phant0mias]

Hi,
there was a typo, I corrected it....
thanks

[Dark1886]

Have you had any issues with DRAM usage when using this library? When I try to compile the main project and adding this in I get a dram0 overflow problem.

[phant0mias]

I'm not sure what do you mean with dram0 overflow. I create own devices and integrate the remote control in this environment. So I'm not using the original project.
The problem I got was if you use the omote project settings that the flash memory is too small for the program size. But the error message was different, it was like:

Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
Error: The program size (1380125 bytes) is greater than maximum allowed (1310720 bytes)
RAM: [====*** [checkprogsize] Explicit exit, status 1
] 36.9% (used 120900 bytes from 327680 bytes)
Flash: [==========] 105.3% (used 1380125 bytes from 1310720 bytes)

This problem should be solved if you use the latest platformio.ini file. There is an entry which changes the flash file size:
board_build.partitions = huge_app.csv

[Dark1886]

This was a problem with me implementing ArduinoOTA and this BT library, it caused some dRam problems that was solved by disabling OTA.

Did you manage to get the back/home buttons to work correctly? I'm seeing that mine work, but sometimes not within netflix/youtube.

[phant0mias]

I haven't tested the bluetooth usage too much, just stopped when it was working. I wanted to share the information, because I found no working arduino implementation to control the fire stick.
You have to test it a little but and see what is the correct usage. One thing might be changing the the delay time between press and release.
Please tell when you found any useful information. Also interseting would be how to change the volume and switch on/off.

[Dark1886]

Your example helped me a lot so thanks for that. This is likely see weird android bug that I need to find a solution to.

As for volume, you may not be able to directly do that with Bluetooth since it is sending the volume command to the fire stick, but the stick isn't actually making the sound. You would either need to direct the sound to the TV instead or send HDMI-CEC commands.

Now all of this seems odd since obviously the default remote manages those functions, but there could be a middle-man service running to do the Bluetooth<->HDMI-Cec conversion.

[phant0mias]

There must be a way sending the volume command with bluetooth without using IR or anything else. The BLE GamePad has a volume command, but it is not working. Maybe it is a different command ID.

You can connect Bluettoth HID devices to the amazon fire stick. So a BLE HID device can control the stick. Maybe somebody has a documentation which commands of a keyboard, mouse, gamepad are used to control the stick. Then it is easy.

[phant0mias]

At the beginning I tested with a samsung Bluetooth keyboard if it would work. I worked but some commands not.
Maybe somebody has a bluetooth gamepad to test if all controls (like the fire remote control) are working. Nobody of my friends has one, perhaps to have the possibilty?

[Dark1886]

I have a non-fully-functional version of it. I had to mess with a different library because some buttons wouldn't work within netflix when using the gamepad library. I don't have all buttons quite figured out yet, but I will grab a snippet of my code later tonight for you.

[phant0mias]

You wrote:

I have a non-fully-functional version of it

The example I send at the beginning of the discussion is "fully" working with amazon prime, I`m not using any apps on the fire stick. For the volume I send the IR commands of the TV. I am able to control amazon prime and the TV with omote.

If you manage to get it fully running with another library or as you wrote capture the bluetooth commands it would be very interesting.

[Dark1886]

When using the example you shared it worked for almost everything except for netflix and i believe youtube too. I think maybe messing with the exact codes it was sending that could have been fixed, but by that point I was already using the BLE Keyboard library instead of the gamepad.

I also experimented with capturing the bluetooth commands received by the android device using an ADB connection, but that was also confusing me. I think there is a little more to the bluetooth protocol when used with android that I would need to understand before truly "fixing" my problems.

I then also got sidetracked by trying to get OTA updates working with my remote (without disabling other libraries and overconsuming the memory) so it was easier to try different button combinations, but that is where i'm currently at and I haven't progressed a whole lot.

[phant0mias]

Would be nice if you found a fully working solution that you send us the solution.

By the way, I saw in your omote repository that you use a lot of delay() calls, some of them 1/4 second long. It is not a good idea to use delays in programs. You block the program nothing is done. There are other possibilities to wait without blocking. It is a little more complicated to handle. If you want to know how it is working I think there is a basic arduino example called something like blinkwithoutdelay. In my bluetooth example I just have delays to show that are needed.

[Dark1886]

I agree with the 'not using delay() calls', That was mainly just because it is easy to drop in place for debugging. I would change that to a better delay in the final product once I figured everything out button-wise.

[KlausMu]

Have you had any issues with DRAM usage when using this library? When I try to compile the main project and adding this in I get a dram0 overflow problem.

Yes, this can happen. To solve this problem, you need to change

static lv_color_t bufA[ screenWidth * screenHeight / 10 ];
static lv_color_t bufB[ screenWidth * screenHeight / 10 ];

which would go into DRAM, to

lv_color_t * bufA = (lv_color_t *) malloc(sizeof(lv_color_t) * screenWidth * screenHeight / 10);
lv_color_t * bufB = (lv_color_t *) malloc(sizeof(lv_color_t) * screenWidth * screenHeight / 10);

which goes into the flash.

But take care, you cannot use BLE and WiFi at the same time, at least not in a straightforward way. I'm still working on this.

[phant0mias]

If you use the newest platform.ini file oder add

board_build.partitions = huge_app.csv

into the platform.ini file there is enough memory.

But take care, you cannot use BLE and WiFi at the same time, at least not in a straightforward way. I'm still working on this.

Sorry, but this is not true. You can use both together, I already use it. After changing the partition configuration it should work

[KlausMu]

No, DRAM has nothing to do with what you can define in the partition table. You might get the dram0 overflow, depending on all the static variables you have, no matter which partition table you are using. See #54

And about WiFi and bluetooth: did you use it regularly together? Since the ESP32 has only one radio, you need to take care that the hardware is only used by one of them at the same time. It can work for some time, and suddenly you get problems. See
https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/coexist.html
https://espressif-docs.readthedocs-hosted.com/projects/esp-faq/en/latest/software-framework/coexistence.html
https://esp32.com/viewtopic.php?t=6707&start=30
https://esp32.com/viewtopic.php?t=16747

[phant0mias]

Interesting to read the links. I will have a closer look to this issues and check if everything is working well.

And about WiFi and bluetooth: did you use it regularly together?

Yes I use the together. I control the amazon fire stick and with WiFI I have a weather display and display of my solar energy usage.

[KlausMu]

I think I found an answer to the problem on using WiFi and bluetooth at the same time.
It indeed does not work out of the box if you are using the standard Arduino libraries:

c:\Users\<yourusername>\.platformio\packages\framework-arduinoespressif32\libraries\BLE 
c:\Users\<yourusername>\.platformio\packages\framework-arduinoespressif32\libraries\WiFi 

If you want to use them at the same time, it seems you have to use at least one of them in a separate thread, like here:
https://www.instructables.com/ESP32C3-BLE-to-WiFi-Bridge-BLE-WiFi-Running-Togeth/

But there exists an alternative BLE implementation called "NimBLE-Arduino" which seems to be better in many ways:
https://github.com/h2zero/NimBLE-Arduino/
As far as I understand this library does not have this issue. At least it is using threads extensively.

So it depends on what your BLE-Keyboard, BLE-Mouse or BLE-Gamepad library is using.

Libraries always using NimBLE:
https://github.com/lemmingDev/ESP32-BLE-Gamepad
https://github.com/wakwak-koba/ESP32-NimBLE-Keyboard
https://github.com/wakwak-koba/ESP32-NimBLE-Mouse

Libraries which can use both standard BLE and NimBLE, but you have to explicitely activate NimBLE:
https://github.com/T-vK/ESP32-BLE-Keyboard
https://github.com/T-vK/ESP32-BLE-Mouse

I used "T-vK/ESP32-BLE-Keyboard" with standard BLE, that caused my problem.
@phant0mias , I think you used "lemmingDev/ESP32-BLE-Gamepad", which uses NimBLE by default.

If I activate NimBLE in "T-vK/ESP32-BLE-Keyboard", problems are gone.

I did not try "wakwak-koba/ESP32-NimBLE-Keyboard". I think this library is obsolete. It was only necessary as long as "T-vK/ESP32-BLE-Keyboard" did not support NimBLE.

[phant0mias]

Yes you a right, I use NimBle and I can remember having problems when using the keypad library without NimBle. I think I wrote it in the top.
I will check you suggestions with the keypad, but need some time. I am currently working on a different device

[KlausMu]

BTW, @phant0mias did you get REWIND and FAST_FORWARD to work with https://github.com/lemmingDev/ESP32-BLE-Gamepad ?
I do not find them in the library.

HOME, BACK, MENU do work?
VOLUME_UP, VOLUME_DOWN, MUTE: do they work?
REWIND, FAST_FORWARD: not even available in the library, right?

[KlausMu]

Now I was able change the keycodes for "next/previous track" to "rewind/fastforward" in the HID descriptor.
But they only work when sending

  bleKeyboard.press(value);
  bleKeyboard.release(value);

bleKeyboard.releaseAll() does not work. The key did not get released. At least in my tests with Windows.

[Dark1886]

If you're changing the keycodes I would expect it to still work. I'm using bleKeyboard.write(KEY_MEDIA_WWW_BACK) for my call, but I'm not sure how that works with a FF/Rewind keys.

[KlausMu]

The following change works (the most you already know, but this is also for someone not too familiar with what to do):

Folder "lib"
Put source from
https://github.com/T-vK/ESP32-BLE-Keyboard
to your lib folder.

BLEKeyboard.h

// original from BLE-Keyboard
// const MediaKeyReport KEY_MEDIA_CALCULATOR = {0, 2};
// changed for usage in OMOTE
const MediaKeyReport KEY_MEDIA_FASTFORWARD = {0, 2};
...
// original from BLE-Keyboard
// const MediaKeyReport KEY_MEDIA_EMAIL_READER = {0, 128};
// changed for usage in OMOTE
const MediaKeyReport KEY_MEDIA_REWIND = {0, 128};

BLEKeyboard.cpp

  // original from BLE-Keyboard
  // USAGE(2),           0x92, 0x01,    //   Usage (Calculator)  ; bit 1: 2
  // changed for usage in OMOTE
  USAGE(1),           0xB3,          //   USAGE (Fast Forward); bit 1: 2
...
  // original from BLE-Keyboard
  // USAGE(2),           0x8A, 0x01,    //   Usage (Mail)        ; bit 7: 128
  // changed for usage in OMOTE
  USAGE(1),           0xB4,          //   USAGE (Rewind)      ; bit 7: 128

platformio.ini

lib_deps = 
	h2zero/NimBLE-Arduino@^1.4.1
build_flags = 
	-D USE_NIMBLE=1 ; for BLE Keyboard

Usage

  bleKeyboard.write(KEY_MEDIA_REWIND);

or

  bleKeyboard.press(KEY_MEDIA_REWIND);
  delay(1000);
  bleKeyboard.release(KEY_MEDIA_REWIND);

[Dark1886]

Thanks for documenting this. I went through similar stuff earlier but never really implemented it into this library because I just ended up using the keyboard library. But knowing this works and the update is relatively easy I may switch back since this library seems to use a lot less memory than the keyboard library.

[KlausMu]

And one more comment.
I saw that BACK and HOME, like I did it (with the ESC key), do not work exactly like the Fire TV remote does. E.g. in YouTube HOME did not work if a video was played.

So

bleKeyboard.write(KEY_ESC);

should be replaced with

bleKeyboard.write(KEY_MEDIA_WWW_BACK);

and

bleKeyboard.press(KEY_LEFT_ALT);
bleKeyboard.press(KEY_ESC);
bleKeyboard.releaseAll();

should be replaced with

bleKeyboard.write(KEY_MEDIA_WWW_HOME);

That works better with a Fire TV.

If you are interested in the whole code:
https://github.com/KlausMu/OMOTE/tree/modular-approach/Platformio/src
files "device_keyboard_ble.h" and "device_keyboard_ble.cpp"