Multi breakers per CT and multi CTs per breaker

[segdy]

1.) I have 24 circuits to monitor but only the 16 clamps. Is it OK to wrap a CT over multiple wires (if their phase is the same) and monitoring them together?

2.) The manual explicitly says to only use one CT for multi breakers and use multiplier of 2. But why is that? I don't think that's accurate since it's not guaranteed that the same current flows (that would only be the case is both wires are connected to the same load, basically having two thinner wires instead of one thick one). Can someone explain the background?

Now I do have multi breakers and most of them use both phases (e.g. branch circuits going to another sub panel). Especially in this case I must monitor their individual wires with separate CTs, right?

3.) For the multi breakers in (2), is there a config option to automatically together the contributions of two circuits? Example: One tandem breaker goes to the dryer. Of course, it includes both phases. I have one CT over each. But I am really interested in the dryer power/energy as a whole and not per phase. While I can add them outside ESPhome (e.g. in Home Assistant) it would be much cleaner if I could define those in ESPhome directly.

[ajacques]

2.) The manual explicitly says to only use one CT for multi breakers and use multiplier of 2. But why is that? I don't think that's accurate since it's not guaranteed that the same current flows (that would only be the case is both wires are connected to the same load, basically having two thinner wires instead of one thick one). Can someone explain the background?

It depends on if the load is balanced or unbalanced (does it also have a neutral.) I had a few appliances that were dual breakers (Heat Pump, Dryer, Oven) and I looked through and identified which were unbalanced and which were balanced. If it's unbalanced (i.e. there's a neutral) then the total power may not be equal to one CT * 2.

To solve this, you can sum two phases internally like this:

sensor:
  - platform: emporia_vue
    i2c_id: i2c_a
    ct_clamps:
      - { phase_id: phase_a, input:  "2", power: { id:  cir2, internal: true, filters: [ *pos ] } }
      - { phase_id: phase_b, input:  "4", power: { id:  cir4, internal: true, filters: [ *pos ] } }

  - platform: template
    name: "Oven Power"
    lambda: return id(cir2).state + id(cir4).state;
    update_interval: 3s
    id: oven_power_raw
    device_class: power
    state_class: measurement
    unit_of_measurement: "W"

  - { power_id:  oven_power_raw,   platform: total_daily_energy, method: "left", accuracy_decimals: 0, restore: false, unit_of_measurement: "Wh", state_class: "total_increasing", device_class: "energy", filters: [ *throttle_energy ], name: "Oven Energy" }

I did notice some incorrect outputs for spiky loads because the update interval caused sampling bias. To counteract this for some appliances, I actually created two internal total_daily_energy sensors to sum up the two phases, then created a sum template over the two integrals.

[segdy]

This is great stuff, thanks!

Indeed, sadly this does not seem to be accurate for me either:

circuit_2_power: 1.7
circuit_3: power: 22.8

total_power_upper_floor: 23.5

In fact, the sum is always less then when I add the two manually.

I am not sure if this has to do with sampling bias because it's consistently always wrong even though the individual powers stay fairly constant.

Do you have any advice on this?

EDIT: Here is the history:

I used the code exactly as you described.

[ajacques]

Is it supposed to be circuit_15_power + circuit_16_power = upper_floor_power? If so those graphs really don't make sense.

Can you post more of your config? In my post, I marked my underlying sensors as internal: true which means they won't be transmitted to MQTT. Depending on if you changed that to be false or had a duplicate CT sensor which change what I think is happening.

Keep in mind that there is a timing biases because the total is computed every 3 seconds whereas the underlying circuits are computed at a different rate (depending on if you're using the moving average or transmitting raw values as is.)
I don't expect them to be perfectly accurate and instead have the vue do continuous integration for the daily Wh usage. I went through what I did in detail here.

[segdy]

(EDIT: Seems it was the retain flag — please see my follow up below)

Thank you very very much for helping me out here!

Yes, it is summing circuits 15 and 16. Below is my full config.

The internal property I was actually wondering already what it does. I thought it would not do exactly this but circuits 15 and 16 still show up in MQTT (see also my screenshots).

Note that here I am not even talking about energy yet, only power ... this should really match. Very strange...

esphome:
  name: emporiavue2
  friendly_name: vue2

external_components:
  - source: github://emporia-vue-local/esphome@dev
    components:
      - emporia_vue

esp32:
  board: esp32dev
  framework:
    type: esp-idf
    version: recommended

# Enable Home Assistant API
api:
  encryption:
    # Encryption key is generated by the new device wizard.
    key: "******"

  services:
    - service: play_rtttl
      variables:
        song_str: string
      then:
        - rtttl.play:
            rtttl: !lambda 'return song_str;'

ota:
  # Create a secure password for pushing OTA updates.
  password: "******"

# Enable logging
logger:

wifi:
  ssid: ******
  password: ******
  manual_ip:
    static_ip: 192.168.222.6
    gateway: 192.168.222.1
    subnet: 255.255.255.0

mqtt:
  broker: 10.227.79.10
  id: vue2
  topic_prefix: vue2
  username: vue2
  password: ******
  discovery: True # Only if you use the HA API usually

preferences:
  # the default of 1min is far too short--flash chip is rated
  # for approx 100k writes.
  flash_write_interval: "48h"

output:
  - platform: ledc
    pin: GPIO12
    id: buzzer

rtttl:
  output: buzzer
  on_finished_playback:
    - logger.log: 'Song ended!'

button:
  - platform: template
    name: "Two Beeps"
    on_press:
      - rtttl.play: "two short:d=4,o=5,b=100:16e6,16e6"

light:
  - platform: status_led
    name: "D3_LED"
    pin: 23
    restore_mode: ALWAYS_ON

i2c:
  sda: 21
  scl: 22
  scan: false
  frequency: 200kHz  # recommended range is 50-200kHz
  id: i2c_a

time:
  - platform: sntp
    id: my_time

# these are called references in YAML. They allow you to reuse
# this configuration in each sensor, while only defining it once
.defaultfilters:
  - &moving_avg
    # we capture a new sample every 0.24 seconds, so the time can
    # be calculated from the number of samples as n * 0.24.
    sliding_window_moving_average:
      # we average over the past 2.88 seconds
      window_size: 12
      # we push a new value every 1.44 seconds
      send_every: 6
  - &invert
    # invert and filter out any values below 0.
    lambda: 'return max(-x, 0.0f);'
  - &pos
    # filter out any values below 0.
    lambda: 'return max(x, 0.0f);'
  - &abs
    # take the absolute value of the value
    lambda: 'return abs(x);'

sensor:
  - platform: emporia_vue
    i2c_id: i2c_a
    phases:
      - id: phase_a  # Verify that this specific phase/leg is connected to correct input wire color on device listed below
        input: BLACK  # Vue device wire color
        #calibration: 0.022  # 0.022 is used as the default as starting point but may need adjusted to ensure accuracy
        # To calculate new calibration value use the formula <in-use calibration value> * <accurate voltage> / <reporting voltage>
        # Meas: 121.5V
        # Disp: 116.8V
        calibration: 0.02288527397
        voltage:
          name: "Phase A Voltage"
          filters: [*moving_avg, *pos]
        frequency:
          name: "Phase A Frequency"
          filters: [*moving_avg, *pos]
      - id: phase_b  # Verify that this specific phase/leg is connected to correct input wire color on device listed below
        input: RED  # Vue device wire color
        #calibration: 0.022  # 0.022 is used as the default as starting point but may need adjusted to ensure accuracy
        # To calculate new calibration value use the formula <in-use calibration value> * <accurate voltage> / <reporting voltage>
        # Meas: 121.4
        # Disp: 121.9
        #calibration: 0.0219097621
        #calibration: 0.0218917442
        #calibration:  0.02187380015
        calibration: 0.02192772077
        voltage:
          name: "Phase B Voltage"
          filters: [*moving_avg, *pos]
        phase_angle:
          name: "Phase B Phase Angle"
          filters: [*moving_avg, *pos]
    ct_clamps:
      - phase_id: phase_a
        input: "A"  # Verify the CT going to this device input also matches the phase/leg
        power:
          name: "Phase A Power"
          id: phase_a_power
          device_class: power
          filters: [*moving_avg, *pos]
      - phase_id: phase_b
        input: "B"  # Verify the CT going to this device input also matches the phase/leg
        power:
          name: "Phase B Power"
          id: phase_b_power
          device_class: power
          filters: [*moving_avg, *pos]
      # Pay close attention to set the phase_id for each breaker by matching it to the phase/leg it connects to in the panel
      - { phase_id: phase_a, input:  "1", power: { name:  "Circuit 1 Power", id:  cir1, filters: [ *moving_avg, *pos, calibrate_linear:    [0->0,121.5->119] ] } }
      - { phase_id: phase_a, input:  "2", power: { name:  "Circuit 2 Power", id:  cir2, filters: [ *moving_avg, *pos, calibrate_linear:    [0->0,120.25->119] ] } }
      - { phase_id: phase_b, input:  "3", power: { name:  "Circuit 3 Power", id:  cir3, filters: [ *moving_avg, *pos, calibrate_linear: [0->0,121->119.5] ] } }
      - { phase_id: phase_b, input:  "4", power: { name:  "Circuit 4 Power", id:  cir4, filters: [ *moving_avg, *pos, calibrate_linear: [0->0,121.5->119.5] ] } }
      - { phase_id: phase_a, input:  "5", power: { name:  "Circuit 5 Power", id:  cir5, filters: [ *moving_avg, *pos, calibrate_linear:    [0->0,121.75->119] ] } }
      - { phase_id: phase_a, input:  "6", power: { name:  "Circuit 6 Power", id:  cir6, filters: [ *moving_avg, *pos, calibrate_linear:    [0->0,122.6->119] ] } }
      - { phase_id: phase_b, input:  "7", power: { name:  "Circuit 7 Power", id:  cir7, filters: [ *moving_avg, *pos, calibrate_linear: [0->0,124->119.5] ] } }
      - { phase_id: phase_b, input:  "8", power: { name:  "Circuit 8 Power", id:  cir8, filters: [ *moving_avg, *pos, calibrate_linear: [0->0,121.5->119.5] ] } }
      - { phase_id: phase_a, input:  "9", power: { name:  "Circuit 9 Power", id:  cir9, filters: [ *moving_avg, *pos, calibrate_linear:    [0->0,121.6->119] ] } }
      - { phase_id: phase_b, input: "10", power: { name: "Circuit 10 Power", id: cir10, filters: [ *moving_avg, *pos, calibrate_linear: [0->0,122.1->119.5] ] } }
      - { phase_id: phase_a, input: "11", power: { name: "Circuit 11 Power", id: cir11, filters: [ *moving_avg, *pos, calibrate_linear:    [0->0,122.5->119] ] } }
      - { phase_id: phase_a, input: "12", power: { name: "Circuit 12 Power", id: cir12, filters: [ *moving_avg, *pos, calibrate_linear:    [0->0,120.75->119] ] } }
      - { phase_id: phase_b, input: "13", power: { name: "Circuit 13 Power", id: cir13, filters: [ *moving_avg, *pos, calibrate_linear: [0->0,122.4->119.5] ] } }
      - { phase_id: phase_b, input: "14", power: { name: "Circuit 14 Power", id: cir14, filters: [ *moving_avg, *pos, calibrate_linear: [0->0,120.6->119.5] ] } }
      - { phase_id: phase_a, input: "15", power: { name: "Circuit 15 Power", id: cir15, internal: true, filters: [ *pos, calibrate_linear:    [0->0,121.8->119] ] } }
      - { phase_id: phase_b, input: "16", power: { name: "Circuit 16 Power", id: cir16, internal: true, filters: [ *pos, calibrate_linear: [0->0,121.8->119.5] ] } }
  - platform: template
    name: "Total Power"
    lambda: return id(phase_a_power).state + id(phase_b_power).state;
    update_interval: 1s
    id: total_power
    unit_of_measurement: "W"

  - platform: template
    name: "Upper Floor Power"
    lambda: return id(cir15).state + id(cir16).state;
    update_interval: 1s
    id: power_upper_floor
    device_class: power
    state_class: measurement
    unit_of_measurement: "W"

  - platform: total_daily_energy
    name: "Total Daily Energy"
    power_id: total_power
    accuracy_decimals: 0
  - { power_id:  cir1, platform: total_daily_energy, accuracy_decimals: 0, name:  "Circuit 1 Daily Energy" }
  - { power_id:  cir2, platform: total_daily_energy, accuracy_decimals: 0, name:  "Circuit 2 Daily Energy" }
  - { power_id:  cir3, platform: total_daily_energy, accuracy_decimals: 0, name:  "Circuit 3 Daily Energy" }
  - { power_id:  cir4, platform: total_daily_energy, accuracy_decimals: 0, name:  "Circuit 4 Daily Energy" }
  - { power_id:  cir5, platform: total_daily_energy, accuracy_decimals: 0, name:  "Circuit 5 Daily Energy" }
  - { power_id:  cir6, platform: total_daily_energy, accuracy_decimals: 0, name:  "Circuit 6 Daily Energy" }
  - { power_id:  cir7, platform: total_daily_energy, accuracy_decimals: 0, name:  "Circuit 7 Daily Energy" }
  - { power_id:  cir8, platform: total_daily_energy, accuracy_decimals: 0, name:  "Circuit 8 Daily Energy" }
  - { power_id:  cir9, platform: total_daily_energy, accuracy_decimals: 0, name:  "Circuit 9 Daily Energy" }
  - { power_id: cir10, platform: total_daily_energy, accuracy_decimals: 0, name: "Circuit 10 Daily Energy" }
  - { power_id: cir11, platform: total_daily_energy, accuracy_decimals: 0, name: "Circuit 11 Daily Energy" }
  - { power_id: cir12, platform: total_daily_energy, accuracy_decimals: 0, name: "Circuit 12 Daily Energy" }
  - { power_id: cir13, platform: total_daily_energy, accuracy_decimals: 0, name: "Circuit 13 Daily Energy" }
  - { power_id: cir14, platform: total_daily_energy, accuracy_decimals: 0, name: "Circuit 14 Daily Energy" }
  - { power_id: cir15, platform: total_daily_energy, accuracy_decimals: 0, name: "Circuit 15 Daily Energy" }
  - { power_id: cir16, platform: total_daily_energy, accuracy_decimals: 0, name: "Circuit 16 Daily Energy" }
  - { power_id: power_upper_floor, platform: total_daily_energy, accuracy_decimals: 0, name: "Upper Floor Daily Energy" }


 

Thanks for sharing the link, that looks like a great description. Will be my night article today.

[segdy]

Ok let me add, I’m just stupid I believe : I think it was because of retained messages. Why on earth does esphome set the retain flag?

Anyway the sun seems to be much closer now.

i got a few follow up questions though:

1. internal: true is purely optical right? No mechanical differences …
2. What is the reason you removed the moving average filter?
3. The default configuration (total power) uses 1s update. What is the reason you change to 3s?
4. I also see you added device_class and state_class. Is this just for nicer style or does it have a mechanical reason?
5. on:

I did notice some incorrect outputs for spiky loads because the update interval caused sampling bias.

by this you mean for example if a fridge draws a large current for <3s so it’s not captured with the total power sensor? I am trying to understand how this is different from an ordinary circuit. That problem exists there too, right?

To counteract this for some appliances, I actually created two internal total_daily_energy sensors to sum up the two phases, then created a sum template over the two integrals.

For power you are still using the same code as above? … but then, instead of creating an energy sensor from the total_power sensor, you just sun up the individual energy sensors? Would you mind sharing the yaml fragment for this too?

[agoode]

Thanks for these config ideas. One thing I liked to do was to avoid setting update_interval on the template sensors but instead use component.update as described here: https://community.home-assistant.io/t/how-to-make-template-sensor-update-immediately-and-not-wait-x-seconds/451874

Example:

      - phase_id: phase_a
        input: "5"
        power:
          internal: true
          id: oven_a_power
          filters: [*moving_avg, *pos]
          on_value:
            then:
              component.update: oven_power
      - phase_id: phase_b
        input: "6"
        power:
          internal: true
          id: oven_b_power
          filters: [*moving_avg, *pos]
          on_value:
            then:
              component.update: oven_power
...
  - platform: template
    name: "Oven Power"
    lambda: return id(oven_a_power).state + id(oven_b_power).state;
    id: oven_power
    device_class: power
    state_class: measurement
    unit_of_measurement: "W"

It is a little more verbose but it makes the summed sensor update in sync with the underling sensor updates.

[ajacques]

ESPHome uses retain for in frequently changing values in case Home Assistant needs to be restarted, but with the Emporia Vue, these values change frequently enough that they shouldn't be retained. This partially why ESPHome introduced the Native API mechanism.

1. internal: true tells ESPHome not to transmit these values to MQTT or the Native API, but they are still calculated on device. This means you can chain together on-device processing to improve accuracy and reduce MQTT traffic.
2. Not sure if you read the article I posted earlier which showed detailed pictures, but effectively the moving average was causing some accuracy issues on some appliances I tested that were spiky like my washing machine. The moving average averaged over the spikes and dips and caused the total kWh usage to be different than I expected. Since the device collects data every 240ms, I don't want to transmit an MQTT message every 240ms, but I still want accuracy, so my ESPHome config actually looks something like this:

.defaultfilters:
  - &moving_avg
    # we capture a new sample every 0.24 seconds, so the time can
    # be calculated from the number of samples as n * 0.24.
    sliding_window_moving_average:
      # we average over the past 2.88 seconds
      window_size: 12
      # we push a new value every 1.44 seconds
      send_every: 6
  - &pos
    # filter out any values below 0.
    lambda: 'return max(x, 0.0f);'
  - &abs
    # take the absolute value of the value
    lambda: 'return abs(x);'
  # Reduce noise in the power class
  - &power_max
    or:
      - delta: 5
      - throttle: 60s
  - &power_min
    throttle: 5s
  # Energy doesn't need high throughput. every 60 seconds
  - &throttle_energy
    throttle: 60s

sensor:
  - platform: emporia_vue
    i2c_id: i2c_a
    phases:
      - id: phase_a  # Verify that this specific phase/leg is connected to correct input wire color on device listed below
        input: BLACK  # Vue device wire color
        # 0.0226  # 0.022 is used as the default as starting point but may need adjusted to ensure accuracy
        # To calculate new calibration value use the formula <in-use calibration value> * <accurate voltage> / <reporting voltage>
        calibration: 0.022737
        voltage:
          name: "Phase A Voltage"
          filters: [*moving_avg, *pos]
        frequency:
          name: "Phase A Frequency"
          filters: [*moving_avg, *pos]
      - id: phase_b  # Verify that this specific phase/leg is connected to correct input wire color on device listed below
        input: RED  # Vue device wire color
        calibration: 0.021295  # 0.022 is used as the default as starting point but may need adjusted to ensure accuracy
        # To calculate new calibration value use the formula <in-use calibration value> * <accurate voltage> / <reporting voltage>
        voltage:
          name: "Phase B Voltage"
          filters: [*moving_avg, *pos]
        phase_angle:
          name: "Phase B Phase Angle"
          filters: [*moving_avg, *pos]
    ct_clamps:
      - phase_id: phase_a
        input: "A"  # Verify the CT going to this device input also matches the phase/leg
        power:
          name: "Phase A Power"
          id: phase_a_power
          device_class: power
          internal: true
          filters: [*pos]
        current:
          name: "Phase A Current"
          device_class: current
          filters: [ *moving_avg, *pos ]
      - phase_id: phase_b
        input: "B"  # Verify the CT going to this device input also matches the phase/leg
        power:
          name: "Phase B Power"
          id: phase_b_power
          device_class: power
          internal: true
          filters: [*pos]
        current:
          name: "Phase B Current"
          device_class: current
          filters: [ *moving_avg, *pos ]
      # Pay close attention to set the phase_id for each breaker by matching it to the phase/leg it connects to in the panel
      # Some circuits are commented out because they don't have a CT connected yet
      - { phase_id: phase_a, input:  "1", power: { name: "Heat Pump Power #1", internal: true, id:  cir1, filters: [ *pos, multiply: 2 ] } }
      - { phase_id: phase_a, input:  "2", power: { name: "Oven Power #2", id:  cir2, internal: true, filters: [ *pos ] } }
      # CT 3 is the opposite side of Heat Pump #1. Heat Pump is connected with a NEMA 6 which does not include a neutral, thus only one CT is needed
      - { phase_id: phase_b, input:  "4", power: { name: "Oven Power #4", id:  cir4, internal: true, filters: [ *pos ] } }
      - { phase_id: phase_a, input:  "5", power: { name: "Dryer Power #5", internal: true, id:  cir5, filters: [ *pos ] } }
      - { phase_id: phase_a, input:  "6", power: { name: "Dishwasher / Disposal Power #6", internal: true, id:  cir6, filters: [ *pos ] } }
      - { phase_id: phase_b, input:  "7", power: { name: "Dryer Power #7", internal: true, id:  cir7, filters: [ *pos ] } }
      - { phase_id: phase_b, input:  "8", power: { name: "Kitchen Power #8", id: cir8, internal: true, filters: [ *pos ] } }
      - { phase_id: phase_a, input:  "9", power: { name: "Washer Power #9", id: cir9, internal: true, filters: [ *pos ] } }
      - { phase_id: phase_a, input: "10", power: { name: "Kitchen Power #10", id: cir10, internal: true, filters: [ *pos ] } }
      - { phase_id: phase_a, input: "13", power: { internal: true, id: cir13, filters: [ *pos ] } }
      - { phase_id: phase_a, input: "14", power: { name: "Bedroom Power #14", internal: true, id: cir14, filters: [ *pos ] } }
      - { phase_id: phase_b, input: "15", power: { name: "General Power #15", internal: true, id: cir15, filters: [ *pos ] } }
      - { phase_id: phase_b, input: "16", power: { name: "General Power #16", internal: true, id: cir16, filters: [ *pos ] } }

  - { platform: copy, id: phase_a_power_b, device_class: power, state_class: "measurement", source_id: phase_a_power, filters: [ *power_min, *power_max ], name: "Phase A Power" }
  - { platform: copy, id: phase_b_power_b, device_class: power, state_class: "measurement", source_id: phase_b_power, filters: [ *power_min, *power_max ], name: "Phase B Power" }

  - { platform: copy, id:  cir1_b, source_id: cir1, filters: [ *power_min, *power_max ], name: "Heat Pump Power" }
  - { platform: copy, id:  cir2_b, source_id: oven_power_raw, filters: [ *power_min, *power_max ], name: "Oven Power" }
  - { platform: copy, id:  cir5_b, source_id: dryer_power, filters: [ *power_min, *power_max ], name: "Dryer Power" }
  - { platform: copy, id:  cir6_b, source_id: cir6, filters: [ *power_min, *power_max ], name: "Dishwasher / Disposal Power #6" }
  - { platform: copy, id:  cir8_b, source_id: cir8, filters: [ *power_min, *power_max ], name: "Kitchen Power #8" }
  - { platform: copy, id:  cir9_b, source_id: cir9, filters: [ *power_min, *power_max ], name: "Washer Power #9" }
  - { platform: copy, id:  cir10_b, source_id: cir10, filters: [ *power_min, *power_max ], name: "Kitchen Power #10" }
  - { platform: copy, id:  cir13_b, source_id: cir13, filters: [ *power_min, *power_max ], name: "Microwave Power #13" }
  - { platform: copy, id:  cir14_b, source_id: cir14, filters: [ *power_min, *power_max ], name: "Bedroom Power #14" }
  - { platform: copy, id:  cir15_b, source_id: cir15, filters: [ *power_min, *power_max ], name: "General Power #15" }
  - { platform: copy, id:  cir16_b, source_id: cir16, filters: [ *power_min, *power_max ], name: "General Power #16" }



  - platform: template
    name: "Dryer Power"
    lambda: return id(cir5).state + id(cir7).state;
    update_interval: 1s
    id: dryer_power
    internal: true
    device_class: power
    state_class: measurement
    unit_of_measurement: "W"
    #filters: [ *power_min, *power_max ]

  - platform: template
    name: "Oven Power"
    lambda: return id(cir2).state + id(cir4).state;
    update_interval: 1s
    id: oven_power_raw
    internal: true
    device_class: power
    state_class: measurement
    unit_of_measurement: "W"
    #filters: [ *power_min, *power_max ]

  ## Total Power and Energy aggregation
  - platform: template
    name: "Total Power"
    lambda: return id(phase_a_power).state + id(phase_b_power).state;
    update_interval: 3s
    id: total_power_raw
    device_class: power
    state_class: measurement
    unit_of_measurement: "W"

  # Total Energy
  # In the default template, we bucketized every 1 second, but this means we compute the integral every 240ms (the fastest we can)
  - id: phase_a_energy_raw
    power_id: phase_a_power
    platform: total_daily_energy
    restore: false
    internal: true

  - id: phase_b_energy_raw
    power_id: phase_b_power
    platform: total_daily_energy
    restore: false
    internal: true

  - name: "Total Daily Energy"
    platform: template
    lambda: return id(phase_a_energy_raw).state + id(phase_b_energy_raw).state;
    update_interval: 60s
    accuracy_decimals: 0
    unit_of_measurement: Wh
    state_class: total_increasing
    device_class: energy

  - { power_id:  cir1,  platform: total_daily_energy, method: "left", accuracy_decimals: 0, restore: false, unit_of_measurement: "Wh", state_class: "total_increasing", device_class: "energy", filters: [ *throttle_energy ], name: "Heat Pump Energy" }
  - { power_id:  oven_power_raw,   platform: total_daily_energy, method: "left", accuracy_decimals: 0, restore: false, unit_of_measurement: "Wh", state_class: "total_increasing", device_class: "energy", filters: [ *throttle_energy ], name: "Oven Energy" }

It's kind of complex I realize, but the key parts are for total power I actually have a few different sensors that are calculated. Total Energy is calculated as a sum of two continuous integrals instead of an integral of a 1s recalculated integral to ensure there's no sampling biases. Then, because I have decoupled the energy integral from the 1s-3s interval, I'm free to transmit the power sensor at a slower rate to reduce MQTT traffic and the size of my Home Assistant recorder DB. I remove the moving average and instead transmit the power every 3s-60s. 3s is the minimum, 60s is the maximum and if the power changes +- 5W, then it transmits more frequently.

I recognize the above is a bit convoluted, but compared to the default YAML it seems to increase accuracy (compared to another sensor I compared with) and reduce traffic.

3. I don't really need the higher resolution data for power if I change the energy sensor to use the raw data so it reduces MQTT traffic, but make sure you understand the sensor dependencies.
4. The state_class:"total_increasing" tells Home Assistant that the integral always goes up and any reset to 0 should be interpreted as a negative energy usage, but a reset of the metering period. If I set this, then I could set restore:false which meant I never wrote values to flash and it reduced wear. The current recommendation is to set preferences.flash_write_interval:48h, but AFAIK that means you could cause issues with the reset to 0. The device_class:"energy" means Home Assistant will detect it as a sensor that can be directly added to the Energy dashboard.
5. I think my full template will better show why I'm doing what I'm doing. Note the difference between my Oven/Dryer appliance and the total energy. I care more about the total energy being really accurate than I care about the Oven/Dryer where I tolerate the 1s interval.

[ajacques]

I'm not using *abs anywhere in my filters, just *pos. My theory is that there is some error, but it's small enough not to be relevant. Even when my appliances are not running, they still consume some power. Additionally, from prior testing when I used *moving_avg a load of laundry consumed 204Wh whereas a non-averaged, faster sampling reported 126Wh. EDIT: Though I'm going to rerun this test because that was awhile ago.

I performed a test by trying a few different ways to monitor a single circuit that was turned off so no power flow. One sensor using just *pos and the other sensor with *moving_avg, *pos. After 13 hours, the *pos had 2Wh and the *moving_avg, *pos had 1Wh usage. Sure, there is some more error, but at 0.166Wh per hour error is low and and it looks to make the high power usages less accurate which is more important. Even the *moving_avg, *pos approach has an error of 0.077Wh per hour.

Given the noise is largely <1W, I also thought of just clipping off <1W the power sensor, but I'm also starting question everything because I've stared at these numbers so much.

EDIT: Next I need to rerun a full test of a single-leg and dual-leg to compare against the different integral sensors and see what variance I have.

[ajacques]

Update: I did another test comparing different ways of calculating the daily energy. tl;dr when the daily energy is computed on devices, there's minimal differences between the moving_avg and no moving_avg. Over the last 7 days, I saw <20Wh in variance between my current approach, *moving_avg with sum then integral, and *moving_avg with integral then sum, however this may coincide with less frequent usage of spiky appliances.

A larger difference comes in when using Home Assistant's Riemann sum helpers which is probably why I saw a big difference in the first place, but if you're using the total_daily_energy sensor with method:left,unit_of_measurement:"Wh",device_class:energy,state_class:total_increasing on device in the energy dashboard, then you get the best of all worlds.

The main benefit of using the copy approach is that I can independently control the update interval for power vs energy sensors and keep my HA recorder database small and efficient but continue to maintain high accuracy on energy. The dynamic 3s-60s update interval allows me to capture large changes, while throttling smaller changes. Given my usage patterns, I don't see any problems adding *moving_avg back into the chain.

[segdy]

Hi Adam, thank you again for the pretty thorough investigation.

Before going into details on the accuracy of integration, I have one basic question: are you using the API or MQTT?

I’ve been talking back and forth with @flaviut
on a major issue I mentioned briefly on this thread already: My Vue 2 seems to crash and restart at random times. Sometimes it takes up to 8hrs. To help me debugging, he even sent me another device which was working for him and surprisingly the issue occurred there too which likely rules out a hardware defect.

In my latest investigation, I disabled MQTT completely and only enable the HA API. The Vue2 is now running for over 51hrs!

If you’re wondering why I bother about this: When the Vue crashes, it does not save the state the of the daily_integration into NVM. As a result, the energy continues at a previous, much lower value. On HA, this is seen as decreased energy, messing up energy monitoring completely.

This comes at a big surprise to me, not least since MQTT is part of ESPHOME but I’m curious if people are even using it. Maybe it’s just me.

PS: I tried multiple versions of esphome on different system

[ajacques]

I use MQTT on all my ESPHome devices and don't see any issues and I have extensive monitoring to verify. If you're happy with HA API, then great, but if you want to fix MQTT, do you see this on any other ESP32s? My devices will restart if my MQTT server is offline for 15 minutes, such as when I'm doing OS upgrades.

Did you try the debug and add the Reset Reason sensor?

What MQTT broker are you using? Mosquitto will give a few different messages depending on how the device gets disconnected which may give a clue. Anything unique about how this server is running or the network connection? Another option is to watch the serial logs with the device turned on (but not plugged into mains) and see what it outputs.

On HA, this is seen as decreased energy, messing up energy monitoring completely.

Is it messed up on the Energy dashboard or just the charts? Do you mean negative watt-hours or do you mean 0 watt-hours for that time period? Negative watt-hours shouldn't happen if you set the state_class:increasing, though you will see the recorder chart go down. HA doesn't make this very intuitive.

[segdy]

I use MQTT on all my ESPHome devices and don't see any issues and I have extensive monitoring to verify.

Oh bummer. I thought this was it. I know it's absolutely hard to believe but this is the diff file between the config that has been crashing at least every 6-8hrs and the config that has been running for 80 hours:

--- flaviu-vue-test.yaml	2023-03-24 12:03:33.322408874 -0700
+++ flaviu-vue-test-2.yaml	2023-03-24 19:32:36.518892298 -0700
@@ -11,13 +11,16 @@
     type: esp-idf
     version: recommended
 
-mqtt:
-  broker: 10.227.79.10
-  id: vue2
-  topic_prefix: vue2
-  username: vue2
-  password: secret
-  discovery: True # Only if you use the HA API usually
+#mqtt:
+#  broker: 10.227.79.10
+#  id: vue2
+#  topic_prefix: vue2
+#  username: vue2
+#  password: secret
+#  discovery: True # Only if you use the HA API usually
+
+
+api:
 
 # enable ota updates
 ota: 

If you're happy with HA API, then great, but if you want to fix MQTT, do you see this on any other ESP32s?

Not that I know. Having said that, this is not something that I would have realized so far because a crash/reboot usually doesn't do much. In this case though it looses data which is why I see it.

My devices will restart if my MQTT server is offline for 15 minutes, such as when I'm doing OS upgrades.

Correct me if I'm wrong but any regular reboot will commit data (incl energy counters) to NVM and hence not loose data/mess up energy calculation. Those are fine.
Regular boots include everything that the watchdog controls, e.g. restart due to inactivity (15 min no API connection or no MQTT connection, no Wifi connection etc).

However, what I am seeing is that after the reboot (=uptime counter starts at 0 again) data is lost, i.e., it has not been committed to NVM and I strongly believe this is caused by a crash and not a watchdog timer or similar.

For example, I previously had both MQTT and api enabled and as written in the documentation, the device rebooted every 15min, visible by the uptime counter resetting to zero after 900. The energy waveforms were still correct though.

Did you try the debug and add the Reset Reason sensor?

The link is dead but I assume you mean https://esphome.io/components/debug.html ? I have not enabled either. This is a very good idea, I will try this, thank you!

What MQTT broker are you using?

Mosquitto on Debian.

Mosquitto will give a few different messages depending on how the device gets disconnected which may give a clue.

Also a good pointer, thank you!

Anything unique about how this server is running or the network connection?

Not to my knowledge. They are in different subnets, ESP is in a separate VLAN without internet access but nothing out of the ordinary I could think of

Another option is to watch the serial logs with the device turned on (but not plugged into mains) and see what it outputs.

That I have thought about but for the Vue2 it's kinda hard. I was thinking about risking the serial console though despite having it connected to the electrical panel / mains. If I could avoid it would be great though.

Is it messed up on the Energy dashboard or just the charts?

I am debugging this for weeks now and since then I am not even looking at the (broken) energy dashboard any more but just the raw data (=the charts).

Do you mean negative watt-hours or do you mean 0 watt-hours for that time period?

Neither. Literally DECREASING watt-hours (but not resetting to zero). Exactly as if the integration would have run for an hour or so but then the device reset without storing the latest value.

Here I am showing what I am observing:

You can see the device has restarted at 4:15am and 11:15 am; both without my action, no power outage etc. The 2nd reboot happened after nearly 7 hours!

In the "Total Daily Energy" plot you see that in both cases there is a big jump in in energy ... the Vue obviously lost its intermediate integration result and continued from an old value. The purple line shows the output of my Enphase Envoy (Solar) for comparison.

Negative watt-hours shouldn't happen if you set the state_class:increasing, though you will see the recorder chart go down. HA doesn't make this very intuitive.

I am still not completely clear about this ...
I understand that if the energy data goes to ZERO it corresponds to a fresh start. But just a drop as above messes things up I believe.
And even if not, I would still like to see the right energy consumption outside of the Energy dashboard.