Tibber Pulse IR data format (only Germany?)

[jsphuebner]

I have sort of misused issue #1 for discussing the pulse data format. The goal is to use your own meter reading setup and "spoof" tibber into thinking you're sending data from the Pulse. This is useful if your meter is not supported by tibber or the port is already blocked by your own reader. Some say the Pulses data rate is not usable for running zero export controllers.

Thought it might be a good idea to recollect this in a new issue.
At first @Kristian-R found you need to empty the encryption keys by nulling the respective parameters, e.g.
param_set ca_cert \
Then we got some messages. The topic pattern is
$aws/rules/ingest_tibber_bridge_data/tibber-bridge/xxx/publish/TJH01/yyy where xxx is a long "Tibber ID" and yyy is the device ID. There are two device IDs, one for the bridge, one for the Pulse IR.
So far we have seen 3 topics:

• event (various events, like boot up, no data, ota related)
• metric (very verbose dump of environment, wifi ssid, uptimes etc.)
• imp (impulse reader i.e. pulses/kWh message)

An event message looks like this:

{
  "$type" : "event",
  "id" : 10001,
  "class" : "efr32_node_event",
  "name" : "NODE_EVENT_METER_PROBING",
  "args" : {
    "meter_mode" : 2,
    "node" : {
      "$type" : "node",
      "id" : 1,
      "eui" : "dc8e95fffeb82bfe",
      "model" : "TFD01",
      "version" : "986-49df5891",
      "rssi" : -27.396474838256836,
      "seen" : 23,
      "available" : true,
      "pubcnt" : 0,
      "ota_state" : "idle",
      "ota_status" : "up2date",
      "manifest_version" : "986-49df5891",
      "up2date" : true
    }
  },
  "metrics" : {
    "$type" : "metrics",
    "ssid" : "tach",
    "rssi" : -59,
    "bssid" : "3481c4e31c3f",
    "mqttcon" : 0,
    "wificon" : 0,
    "id" : "3494545e8400",
    "uptime" : 189,
    "efr_uptime" : 188,
    "vcc" : 3.497999906539917,
    "vacrms" : 225.68521118164062,
    "heap" : 116228,
    "coredump_available" : true,
    "model" : "TJH01",
    "esp_version" : "1268-f95e5239",
    "efr_version" : "557-dee09095",
    "nodes" : [ {
      "$type" : "node",
      "id" : 1,
      "eui" : "dc8e95fffeb82bfe",
      "model" : "TFD01",
      "version" : "986-49df5891",
      "rssi" : -27.396474838256836,
      "seen" : 23,
      "available" : true,
      "pubcnt" : 0,
      "ota_state" : "idle",
      "ota_status" : "up2date",
      "manifest_version" : "986-49df5891",
      "up2date" : true
    } ],
    "ota" : [ {
      "model" : "tibber-pulse-ir-hub-esp32",
      "manifest_version" : "1268-f95e5239",
      "current_version" : "1268-f95e5239",
      "state" : "up2date"
    }, {
      "model" : "tibber-pulse-ir-hub-css",
      "manifest_version" : "1268-f95e5239",
      "current_version" : "1268-f95e5239",
      "state" : "up2date"
    }, {
      "model" : "tibber-pulse-ir-hub-js",
      "manifest_version" : "1268-f95e5239",
      "current_version" : "1268-f95e5239",
      "state" : "up2date"
    }, {
      "model" : "tibber-pulse-ir-hub-efr32-fg1",
      "manifest_version" : "557-dee09095",
      "current_version" : "557-dee09095",
      "state" : "up2date"
    } ],
    "hub_ota_status" : "up2date",
    "manifest_state" : "synced",
    "netflags" : 25,
    "ip" : "192.168.188.31",
    "gateway" : "192.168.188.1",
    "dns0" : "192.168.188.1",
    "dns1" : "0.0.0.0",
    "dns2" : "0.0.0.0",
    "mac" : "34:94:54:5e:84:0",
    "ap_mac" : "34:94:54:5e:84:1"
  }
}

A metric dump looks like this (Bridge):

{
  "$type" : "status",
  "status" : {
    "$type" : "metrics",
    "ssid" : "tach",
    "rssi" : -58,
    "bssid" : "3481c4e31c3f",
    "mqttcon" : 0,
    "wificon" : 0,
    "id" : "3494545e8400",
    "uptime" : 250,
    "efr_uptime" : 249,
    "vcc" : 3.49399995803833,
    "vacrms" : 224.85430908203125,
    "heap" : 118048,
    "coredump_available" : true,
    "model" : "TJH01",
    "esp_version" : "1268-f95e5239",
    "efr_version" : "557-dee09095",
    "nodes" : [ {
      "$type" : "node",
      "id" : 1,
      "eui" : "dc8e95fffeb82bfe",
      "model" : "TFD01",
      "version" : "986-49df5891",
      "rssi" : -26.945465087890625,
      "seen" : 0,
      "available" : true,
      "pubcnt" : 0,
      "ota_state" : "idle",
      "ota_status" : "up2date",
      "manifest_version" : "986-49df5891",
      "up2date" : true
    } ],
    "ota" : [ {
      "model" : "tibber-pulse-ir-hub-esp32",
      "manifest_version" : "1268-f95e5239",
      "current_version" : "1268-f95e5239",
      "state" : "up2date"
    }, {
      "model" : "tibber-pulse-ir-hub-css",
      "manifest_version" : "1268-f95e5239",
      "current_version" : "1268-f95e5239",
      "state" : "up2date"
    }, {
      "model" : "tibber-pulse-ir-hub-js",
      "manifest_version" : "1268-f95e5239",
      "current_version" : "1268-f95e5239",
      "state" : "up2date"
    }, {
      "model" : "tibber-pulse-ir-hub-efr32-fg1",
      "manifest_version" : "557-dee09095",
      "current_version" : "557-dee09095",
      "state" : "up2date"
    } ],
    "hub_ota_status" : "up2date",
    "manifest_state" : "synced",
    "netflags" : 25,
    "ip" : "192.168.188.31",
    "gateway" : "192.168.188.1",
    "dns0" : "192.168.188.1",
    "dns1" : "0.0.0.0",
    "dns2" : "0.0.0.0",
    "mac" : "34:94:54:5e:84:0",
    "ap_mac" : "34:94:54:5e:84:1"
  }
}

or that (Pulse IR):

{
  "$type" : "node_status",
  "node_status" : {
    "product_id" : 49344,
    "bootloader_version" : 17563650,
    "node_battery_voltage" : 3.464355,
    "node_temperature" : 27.833693,
    "node_avg_rssi" : -38.267334,
    "node_avg_lqi" : 195.028442,
    "acmp_rx_autolevel_300" : 146,
    "acmp_rx_autolevel_9600" : 175,
    "radio_tx_power" : 200,
    "node_uptime_ms" : 5523747,
    "meter_msg_count_sent" : 0,
    "meter_pkg_count_sent" : 0,
    "time_in_em0_ms" : 373,
    "time_in_em1_ms" : 0,
    "time_in_em2_ms" : 299780
  },
  "hub_attachments" : {
    "meter_pkg_count_recv" : 0,
    "node_version" : "986-49df5891"
  }
}

And finally an impulse message looks like this:
{"$type": "imp_data", "timestamp_ms": 34652489,"delta_ms": 11782,"kw":0.030555, "kwh": 0.8297}

A test run with pulses generated from the actual meter power yielded no display in the app though. Maybe tibber has disabled this method.
So if someone could intercept real data from a meter talking either in the binary SML or the ASCII IEC-62056-21 format that would greatly help in writing a tibber export script

[jsphuebner]

I got a reply from Tibber today that the ebz DD3 ODZ1 meter is definitely supported. So I hope to see data soon. A firmware update was installed on the Pulse IR today, version 1001-b6715572

[Kristian-R]

Hi Johannes, nice to hear that your meter is supported. If you still don't get any data make sure you did not "misconfigure" your pulse by setting it up as "1000 impulse/kWh" with your LED-experiment. Under the node-tab there is a configure-link. Usually it is set to autoprobe.

Concerning your issue with not blocking the meter with the pulse this project might be interesting:
evcc-io/evcc#7070

And this ist what I use to receive the mqtt-data and ingest in home assistant
https://github.com/micw/tibber-pulse-reader

By the way: if someone knows how to decode the SML-MQTT directly in home assistant - pleas leave a note.

Regards
Kristian-R

[jsphuebner]

Yes I've set it back to autoprobe
I also send the raw string read from the meter via MQTT so I can replicate it anywhere. I want it to work on the actual meter first before experimenting with that.

[disaster123]

Any news?

[christiankratzer]

Stumbled upon this here.

I have another variation of a script that pulls data from the bridge via http and publishes decoded values over mqtt.

https://github.com/christiankratzer/pulse2mqtt

I use this to poll the tibber bridge every 10s for the latest value. I do get the occasional corrupted sml packet from the bridge that I throw away.

Also you could poll every second using the http interface but would have frequent collisions with duplicate data. I use the transactionid to dedup.

[jsphuebner]

Finally got the pulse to like my eBZ :) It has to sit on the top port, the front port doesn't work.
Anyway, got the stream and it is super simple.
Topic: $aws/rules/ingest_tibber_bridge_data/tibber-bridge/xxx/publish/TJH01/yyy/obis_stream
yyy in this case is the eui (see metric)
It just contains the raw data from the meter, in my case

/EBZ5DD3BZ06ETA_107
1-0:0.0.0*255(1EBZzzzz)
1-0:96.1.0*255(1EBZzzzz)
1-0:1.8.0*255(009094.27916715*kWh)
1-0:16.7.0*255(000169.25*W)
1-0:36.7.0*255(000133.51*W)
1-0:56.7.0*255(000035.74*W)
1-0:76.7.0*255(000000.00*W)
1-0:32.7.0*255(233.4*V)
1-0:52.7.0*255(234.5*V)
1-0:72.7.0*255(234.6*V)
1-0:96.5.0*255(001C0104)
0-0:96.8.0*255(08F0EAB3)
!

The Tibber app shows my current power consumption. I'll now try to generate the message myself along with the metrics and see how that goes

[jsphuebner]

The metrics now look like this.
IR reader (TFD01) sends every 5 minutes:

{
  "$type" : "node_status",
  "node_status" : {
    "product_id" : 49344,
    "bootloader_version" : 17563650,
    "meter_mode" : 4,
    "node_battery_voltage" : 3.435,
    "node_temperature" : 28.166,
    "node_avg_rssi" : -43.557,
    "node_avg_lqi" : 193.705,
    "radio_tx_power" : 0,
    "node_uptime_ms" : 2435139570,
    "meter_msg_count_sent" : 99,
    "meter_pkg_count_sent" : 133,
    "time_in_em0_ms" : 4351,
    "time_in_em1_ms" : 33,
    "time_in_em2_ms" : 295767,
    "acmp_rx_autolevel_300" : 146,
    "acmp_rx_autolevel_9600" : 139
  },
  "hub_attachments" : {
    "meter_pkg_count_recv" : 133,
    "meter_reading_count_recv" : 99,
    "node_version" : "1001-b6715572"
  }
}

So now meter_pkg_count_recv and meter_reading_count_recv are >0

And the wifi bridge TJH01 sends every 2 minutes:

{
  "$type" : "status",
  "device_id" : "yyy",
  "ts" : 1692217927,
  "status" : {
    "$type" : "metrics",
    "ssid" : "tach",
    "rssi" : -73,
    "bssid" : "3481c4e31c3f",
    "channel" : 6,
    "mqttcon" : 0,
    "wificon" : 0,
    "id" : "yyy",
    "uptime" : 1771259,
    "efr_uptime" : 1771152,
    "vcc" : 3.49,
    "vacrms" : 227.22,
    "heap" : 124808,
    "coredump_available" : true,
    "model" : "TJH01",
    "nodes" : [ {
      "$type" : "node",
      "id" : 1,
      "eui" : "dc8e95fffeb82bfe",
      "model" : "TFD01",
      "version" : "1001-b6715572",
      "rssi" : -64.5310287475586,
      "seen" : 2,
      "available" : true,
      "pubcnt" : 93,
      "ota_state" : "idle",
      "ota_status" : "up2date",
      "manifest_version" : "1001-b6715572",
      "up2date" : true
    } ],
    "ota" : [ {
      "model" : "tibber-pulse-ir-hub-esp32",
      "manifest_version" : "1357-05c0c2f6",
      "current_version" : "1357-05c0c2f6",
      "state" : "up2date"
    }, {
      "model" : "tibber-pulse-ir-hub-css",
      "manifest_version" : "1357-05c0c2f6",
      "current_version" : "1357-05c0c2f6",
      "state" : "up2date"
    }, {
      "model" : "tibber-pulse-ir-hub-js",
      "manifest_version" : "1357-05c0c2f6",
      "current_version" : "1357-05c0c2f6",
      "state" : "up2date"
    }, {
      "model" : "tibber-pulse-ir-hub-efr32-fg23",
      "manifest_version" : "584-9be5169b",
      "current_version" : "584-9be5169b",
      "state" : "up2date"
    }, {
      "model" : "tibber-pulse-ir-hub-efr32-fg1",
      "manifest_version" : "584-9be5169b",
      "current_version" : "584-9be5169b",
      "state" : "up2date"
    } ],
    "hub_ota_status" : "up2date",
    "manifest_state" : "synced",
    "manifest_version" : "2306.4",
    "manifest_group" : "",
    "netflags" : 25,
    "ip" : "192.168.188.31",
    "gateway" : "192.168.188.1",
    "dns0" : "192.168.188.1",
    "dns1" : "0.0.0.0",
    "dns2" : "0.0.0.0",
    "dns3" : "0.0.0.0",
    "dns4" : "0.0.0.0",
    "dns5" : "0.0.0.0",
    "dns6" : "0.0.0.0",
    "dns7" : "0.0.0.0",
    "dns8" : "0.0.0.0",
    "mac" : "34:94:54:5e:84:0",
    "ap_mac" : "34:94:54:5e:84:1"
  }
}

Here pubcnt > 0

I reckon the various uptimes should also be plausible. They are in 1s resolution.

[jsphuebner]

I have successfully sent data to tibber! I found the topic you need to send to contains the eui, dc8e95fffeb82bfe in my case. So the full topic is $aws/rules/ingest_tibber_bridge_data/tibber-bridge/xxx/publish/TFD01/dc8e95fffeb82bfe/obis_stream

Due to my guerilla solar I have negative power and tibber doesn't like that. But I don't want to modify the meter string. I wonder if the original pulse somehow handles this. It is very unreliable, if it just moves by 1mm the data is already destroyed, i.e. you only get part of the string.

UPDATE: never mind this. While in the Apps home screen the data seems to freeze on negative values, they are being correctly recorded in the historic data

[jsphuebner]

And more updates. After like 15 minutes the joy was over. The data froze and the icon disappeared in the app. I plugged in the original bridge and the icon reappeared. I'm thinking there is some kind of hand shake that I failed to do

Update2: it is now running stable since 20h. In one metric I still send "yyy" as id, that was not appreciated ;)

[jsphuebner]

I have published my code and instructions here: https://github.com/jsphuebner/esp-egycounter/tree/main/battery-control/tibber (tibbersend.py is in parent directory)

[reinhard-brandstaedter]

Hi @jsphuebner
I went through your code and tried to replicate that tibbersender setup. I do have my own ESP IR reader built which is now decoding SML (extracting current power value) and reporting to MQTT.
In my journey to replace the tibber pulse IR header I wanted to use your tibbersender.py script to read the raw SML from MQTT and then send it to tibber's MQTT, along with the other metrics/heartbeats.

However whatever I try that script won't connect successfully to tibber's MQTT (double checked all settings).
I feel there is some extra added check on tibber side to identify real devices. At one point my real pulse stopped working - well it worked according to dmseg command on console - but no data was shown in the App.
Tibber support told me that suddenly there were two Pulse devices registered on their side and they had to remove one to get it back working.

Have you experienced anything like this? Maybe there is an additional check on what the pulse reports to tibber during pairing/first setup? Maybe any additional (hardware) info that is cross-checked?

I also tried having the pulse report to my local MQTT but it seems that clearing the ca_cert via console isn't successful and replacing it with my own CA seems to always truncate/garbage the cert value stored on the bridge...

I have now built a hardware proxy that is palced between the smartmeter IR interface and the pulse. It is an ESP that reads the IR signal via a photodiode and puts out plain SML via an IR led (which the pulse then receives). At the same time it also decodes SML and reports current power for my own usage. Works but that is not really a elegant solution...

[jsphuebner]

I never had that issue. I did go through the entire setup procedure in the Tibber app and verified the original pulse was (somewhat) working. Then I switched over to tibbersender
You may also need to check the content of the static json files (metric and event) and get them close to what your bridge produces.

Indeed, once you have removed the cert info it is impossible to get it back in (probably by resetting the bridge and starting over).

[reinhard-brandstaedter]

Interesting, I tried to connect to tibber's MQTT broker with all the details (CA, Cert, key), client ID. And I don't even get a successful connection with mqtt client (cancels even before any subscription)...

[christiankratzer]

Who said you can subscribe to the topics on their broker. I assume you can only publish with those credentials. Also you should not have the pulse connected and try to connect at the same time with the same clientId.

If people keep messing with their broker without knowing what they are doing this will lead to tibber eventually closing this up for everybody.

Just let the pulse submit and consume from the pulses http server.

[reinhard-brandstaedter]

@christiankratzer
First: a MQTT connection (establishing the SSL connection) should work. As I wrote I'm not even subscribing to anything nor publishing, just connecting.
Second: I know that the pulse is only publishing SML data and the app is subscribing to read the updates that have been processed on tibber side.
Third: I'm well aware that the same clientid is only allowed once, but that is irrelevant for the SSL connection, plus the pulse was disconnected at the same time
Fourth: consuming via API from the tibber http server is a rather problematic solution if you want to control anyting in realtime locally or if various other things already have been set up in a home automation scenario. A few things in my household depend on the data from the smartmeter which I have configured before the tibber pulse and I'd like to keep them that way, so the pulse is for me just another consumer/reporter and should not be a replacement for the data collection.

I understand your concern regarding "messing" with the broker. But I also gave tibber that feedback that they should offically support the custom submit of data (without the need of an expensive, battery-operated IR reader) for people that are already getting the data in another way. (like it is possible in Norway i.e).

[niklasvieth]

Finally got the pulse to like my eBZ :) It has to sit on the top port, the front port doesn't work. Anyway, got the stream and it is super simple. Topic: $aws/rules/ingest_tibber_bridge_data/tibber-bridge/xxx/publish/TJH01/yyy/obis_stream yyy in this case is the eui (see metric) It just contains the raw data from the meter, in my case

/EBZ5DD3BZ06ETA_107
1-0:0.0.0*255(1EBZzzzz)
1-0:96.1.0*255(1EBZzzzz)
1-0:1.8.0*255(009094.27916715*kWh)
1-0:16.7.0*255(000169.25*W)
1-0:36.7.0*255(000133.51*W)
1-0:56.7.0*255(000035.74*W)
1-0:76.7.0*255(000000.00*W)
1-0:32.7.0*255(233.4*V)
1-0:52.7.0*255(234.5*V)
1-0:72.7.0*255(234.6*V)
1-0:96.5.0*255(001C0104)
0-0:96.8.0*255(08F0EAB3)
!

The Tibber app shows my current power consumption. I'll now try to generate the message myself along with the metrics and see how that goes

Hi @jsphuebner I am struggling as well with my EBZ - ODZ1 as well, I set the INF=ON PIN=OFF via the 4 digit pin. But I get not get the Pulse to work in the front and also not on the top either. I even have the Tibber adapter for the eBZ ... I can see the infrarot light pulsing via my smartphone camera, but the pulse does not read any data. Do you have any tips how you placed the pulse at the top? I even got a replacement hardware already but still not working ...

[jsphuebner]

It's a long time ago. I think I omitted the mechanical adapter and placed the head on top free-style until it worked. The final missing piece of a software update of the Pulse. Before it didn't seem to support the data format above

[RobinBeismann]

Hi @jsphuebner I went through your code and tried to replicate that tibbersender setup. I do have my own ESP IR reader built which is now decoding SML (extracting current power value) and reporting to MQTT. In my journey to replace the tibber pulse IR header I wanted to use your tibbersender.py script to read the raw SML from MQTT and then send it to tibber's MQTT, along with the other metrics/heartbeats.

However whatever I try that script won't connect successfully to tibber's MQTT (double checked all settings). I feel there is some extra added check on tibber side to identify real devices. At one point my real pulse stopped working - well it worked according to dmseg command on console - but no data was shown in the App. Tibber support told me that suddenly there were two Pulse devices registered on their side and they had to remove one to get it back working.

Have you experienced anything like this? Maybe there is an additional check on what the pulse reports to tibber during pairing/first setup? Maybe any additional (hardware) info that is cross-checked?

I also tried having the pulse report to my local MQTT but it seems that clearing the ca_cert via console isn't successful and replacing it with my own CA seems to always truncate/garbage the cert value stored on the bridge...

I have now built a hardware proxy that is palced between the smartmeter IR interface and the pulse. It is an ESP that reads the IR signal via a photodiode and puts out plain SML via an IR led (which the pulse then receives). At the same time it also decodes SML and reports current power for my own usage. Works but that is not really a elegant solution...

Hi @reinhard-brandstaedter - I'm in a similiar but worse situation, I'd like to use tibber as well but my power meter is locked in a sealed cage by the meter supplier. The last time we had a certified electrician here, he did me a favor and mounted my Hichi Wifi (based on tasmota 12.3.1) onto the IR Port of my DVS7612.2 energy meter and unlocked it via the PIN for me.
I regret that I didn't use the TTL version, that way I could use your workaround.
Can you imagine a way to stream the raw SML data into a 2nd Hichi (or any ESP) and output it to an IR LED which I could attach the tibber to?
I've just ordered an Solarflow Hyper 2000 with two AB2000 and could benefit from charging them while the sun doesn't satisfy it with 4 panels attached (exchanging two existing inverters with the Hyper 2000).