Daily Energy meters seem to fluctuate instead of always going up...

[jcam]

A lot of the daily energy meters seem to look like this. not continuously increasing with a reset at midnight, but going up and then back down a bit then further up. As a result, the totals are quite a bit off from reality.
I am pretty sure this hasn't been the case all along, that a year-ish ago when I first deployed these, they had nice clean always increasing numbers. I'm not sure if this is a problem in my config, a bug in esphome, or what...

Any ideas what would be a good place to start to look into this?

I'm not sure which parts of my config would be useful to include vs just the whole thing; it isn't far off from the readme...

i2c:
  sda: 21
  scl: 22
  scan: false
  frequency: 200kHz  # recommended range is 50-200kHz
  id: i2c_a
[...]
sensor:
  - platform: emporia_vue
    i2c_id: i2c_a
[...]
    ct_clamps:
[...]
      - { phase_id: phase_b, input: "10", power: { id:  solar_b_power, filters: [ multiply: 1.195 ] } }
      - { phase_id: phase_a, input:  "9", power: { id:  solar_a_power, filters: [ multiply: 1.195 ] } }
    on_update:
      then:
        - component.update: total_solar_power
[...]
  - platform: template
    lambda: return max((id(solar_a_power).state + id(solar_b_power).state), 0.0f);
    update_interval: never
    id: total_solar_power
    unit_of_measurement: "W"
  - platform: total_daily_energy
    name: "Solar Daily Energy"
    power_id: total_solar_power
    accuracy_decimals: 0
    filters: *throttle_time

i tried adding this today to the total_daily_energy elements, but i'm not sure if it has made any difference:

    state_class: total_increasing
    device_class: energy
    method: left

[flaviut]

1. Is the time working properly in esphome? I've seen weird issues with the total_daily_energy sensors when folks have NTP blocked by firewall, or things like that.
2. Is the system rebooting frequently? Are there power cuts or maybe it's just rebooting on it's own?
3. What happens when you update your total_daily_energy sensors to have restore: false set? See the readme for examples.

[jcam]

1. time was set for sntp on this vue, and home assistant time on another, so today i switched it to home assistant time.
2. what would be a good way to check if it is rebooting? there don't appear to be power cuts
3. i can give restore:false a try tomorrow :)

[flaviut]

what would be a good way to check if it is rebooting? there don't appear to be power cuts

I guess keep the logs open? you can use the esphome logs command and redirect it to a file (depends on your OS). Or zoom in on the graph and see if there's missing data where the jumps are happening.

[zwbenedict]

I'm having the exact same issues. Random spikes of data on my total_daily_energy. This didn't start happening until I reflashed my emporia with the latest emporia-vue-local 2 days ago.

[jcam]

Which "method" are you using for the total_daily_energy calculation? apparently trapezoidal and right (where right is the default) don't work well when the same value may come through multiple times...

[jcam]

Well... since I changed multiple things I can't be sure which was the real solution. For one thing, the reset was apparently happening at 8pm my time, which is midnight UTC, now it reset at local midnight...
I did not set restore: false, and I do still have the 48h flash write frequency that has been there for the last couple of months.

Changes I've made:

Added to every total_daily_energy sensor:

    state_class: total_increasing
    device_class: energy
    method: left

Replaced SNTP time with homeassistant_time:
Old:

time:
  - platform: sntp
    id: my_time

New:

time:
  - platform: homeassistant
    id: homeassistant_time

The graph looks like this today vs the last 2 days:

[jcam]

Here is mine... It is somewhat confusing because it is monitoring the mains of two separate 200A services...

esphome:
  name: vue-garage

external_components:
  - source: github://emporia-vue-local/esphome@dev
    components: [ emporia_vue ]
    
esp32:
  board: esp32dev
  framework:
    type: esp-idf
    version: recommended

# Enable Home Assistant API
api:
  encryption:
    key: [...]
  services:
    - service: play_rtttl
      variables:
        song_str: string
      then:
        - rtttl.play:
            rtttl: !lambda 'return song_str;'

ota:
  password: [...]

# Enable logging
logger:
  baud_rate: 0   # don't send log messages over UART interface
  level: INFO

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

preferences:
  # the default of 1min is far too short--flash chip is rated
  # for approx 100k writes.
  flash_write_interval: "48h"

# Enable time component to reset energy at midnight
time:
  - platform: homeassistant
    id: homeassistant_time

mqtt:
  id: mqtt_client
  broker: 192.168.1.6
  username: !secret mqtt_username
  password: !secret mqtt_password
  discovery: false
  idf_send_async: true
  log_topic: null

output:
  - platform: ledc
    pin: GPIO12
    id: buzzer
  - platform: gpio
    pin: GPIO27
    id: buzzer_gnd

rtttl:
  output: buzzer
  on_finished_playback:
    - logger.log: 'Song ended!'

button:
  - platform: template
    name: "Two Beeps"
    on_press:
      - rtttl.play: "two short:d=4,o=5,b=100:16e6,16e6"

light:
  - platform: status_led
    name: "D3_LED"
    pin: 23
    restore_mode: ALWAYS_ON
    entity_category: config

i2c:
  sda: 21
  scl: 22
  scan: false
  frequency: 200kHz  # recommended range is 50-200kHz
  id: i2c_a

# these are called references in YAML. They allow you to reuse
# this configuration in each sensor, while only defining it once
.defaultfilters:
  - &throttle_avg
    # average all raw readings together over a 5 second span before publishing
    throttle_average: 5s
  - &throttle_time
    # only send the most recent measurement every 5 seconds
    throttle: 5s
  - &invert
    # invert and filter out any values below 0.
    lambda: 'return max(-x, 0.0f);'
  - &pos
    # filter out any values below 0.
    lambda: 'return max(x, 0.0f);'
  - &abs
    # take the absolute value of the value
    lambda: 'return abs(x);'

sensor:
  - platform: emporia_vue
    i2c_id: i2c_a
    phases:
      - id: phase_a  # Verify that this specific phase/leg is connected to correct input wire color on device listed below
        input: BLACK  # Vue device wire color
        calibration: 0.02281140024  # 0.022 is used as the default as starting point but may need adjusted to ensure accuracy
        # To calculate new calibration value use the formula <in-use calibration value> * <accurate voltage> / <reporting voltage>
        voltage:
          name: "Phase A Voltage"
          filters: [*throttle_avg, *pos]
        frequency:
          name: "Phase A Frequency"
          filters: [*throttle_avg, *pos]
      - id: phase_b  # Verify that this specific phase/leg is connected to correct input wire color on device listed below
        input: RED  # Vue device wire color
        calibration: 0.02167216913  # 0.022 is used as the default as starting point but may need adjusted to ensure accuracy
        # To calculate new calibration value use the formula <in-use calibration value> * <accurate voltage> / <reporting voltage>
        voltage:
          name: "Phase B Voltage"
          filters: [*throttle_avg, *pos]
        phase_angle:
          name: "Phase B Phase Angle"
          filters: [*throttle_avg, *pos]
    ct_clamps:
      - phase_id: phase_a
        input: "15"  # Verify the CT going to this device input also matches the phase/leg
        power:
          id: phase_a_power
          filters: [multiply: 4]
      - phase_id: phase_b
        input: "16"  # Verify the CT going to this device input also matches the phase/leg
        power:
          id: phase_b_power
          filters: [multiply: 4]
      # Pay close attention to set the phase_id for each breaker by matching it to the phase/leg it connects to in the panel
      - { phase_id: phase_a, input: "A", power: { id: gar_phase_a_power, filters: [ *pos ] } }
      - { phase_id: phase_b, input: "B", power: { id: gar_phase_b_power, filters: [ *pos ] } }
      - { phase_id: phase_a, input: "14", power: { id:  cir1, filters: [ *pos ] } }
      - { phase_id: phase_a, input: "13", power: { id:  cir3, filters: [ *pos ] } }
      - { phase_id: phase_b, input: "12", power: { id:  cir4, filters: [ *pos ] } }
#      - { phase_id: phase_a, input: "11", power: { id:  cir5, filters: [ *pos ] } }
      - { phase_id: phase_b, input: "10", power: { id:  solar_b_power, filters: [ multiply: 1.195 ] } }
      - { phase_id: phase_a, input:  "9", power: { id:  solar_a_power, filters: [ multiply: 1.195 ] } }
      - { phase_id: phase_a, input:  "1", power: { id: cir16, filters: [ *pos ] } }
      - { phase_id: phase_b, input:  "2", power: { id: cir17, filters: [ *pos ] } }
      - { phase_id: phase_b, input:  "3", power: { id: cir19, filters: [ *pos ] } }
      - { phase_id: phase_a, input:  "4", power: { id: cir20, filters: [ *pos ] } }
    on_update:
      then:
        - component.update: total_garage_power
        - component.update: total_house_power
        - component.update: total_house_from_grid
        - component.update: total_house_to_grid
        - component.update: total_solar_power
        - component.update: total_solar_power_draw
        - component.update: balance_power
  - { platform: copy, name: "House Phase A Power", source_id: phase_a_power, filters: *throttle_avg }
  - { platform: copy, name: "House Phase B Power", source_id: phase_b_power, filters: *throttle_avg }
  - { platform: copy, name: "Total House Power", source_id: total_house_power, filters: *throttle_avg }
  - { platform: copy, name: "House Power From Grid", source_id: total_house_from_grid, filters: *throttle_avg }
  - { platform: copy, name: "House Power To Grid", source_id: total_house_to_grid, filters: *throttle_avg }
  - { platform: copy, name: "Balance Power", source_id: balance_power, filters: *throttle_avg }
  - { platform: copy, name: "Garage Phase A Power", source_id: gar_phase_a_power, filters: *throttle_avg }
  - { platform: copy, name: "Garage Phase B Power", source_id: gar_phase_b_power, filters: *throttle_avg }
  - { platform: copy, name: "Total Garage Power", source_id: total_garage_power, filters: *throttle_avg }
  - { platform: copy, name: "Solar A", source_id: solar_a_power, filters: *throttle_avg }
  - { platform: copy, name: "Solar B", source_id: solar_b_power, filters: *throttle_avg }
  - { platform: copy, name: "Total Solar Power", source_id: total_solar_power, filters: *throttle_avg }
  - { platform: copy, name: "Solar Night Consumption Power", source_id: total_solar_power_draw, filters: *throttle_avg }
  - { platform: copy, name:  "Gar 2F Mini", source_id:  cir1, filters: *throttle_avg }
  - { platform: copy, name:  "Gar 2F Outlets", source_id:  cir3, filters: *throttle_avg }
  - { platform: copy, name:  "Gar 2F Lights", source_id:  cir4, filters: *throttle_avg }
#  - { platform: copy, name:  "Driveway Heat", source_id:  cir5, filters: *throttle_avg }
  - { platform: copy, name: "FR South Lights", source_id: cir16, filters: *throttle_avg }
  - { platform: copy, name: "Gar Otlts, 2F Closet", source_id: cir17, filters: *throttle_avg }
  - { platform: copy, name: "FR North Lights", source_id: cir19, filters: *throttle_avg }
  - { platform: copy, name: "FR Outlets", source_id: cir20, filters: *throttle_avg }
  - platform: template
    lambda: return id(phase_a_power).state + id(phase_b_power).state;
    update_interval: never
    id: total_house_power
    device_class: power
    state_class: measurement
    unit_of_measurement: "W"
  - platform: template
    lambda: return max(id(total_house_power).state, 0.0f);
    update_interval: never
    id: total_house_from_grid
    device_class: power
    state_class: measurement
    unit_of_measurement: "W"
  - platform: template
    lambda: return max(-id(total_house_power).state, 0.0f);
    update_interval: never
    id: total_house_to_grid
    device_class: power
    state_class: measurement
    unit_of_measurement: "W"
  - platform: total_daily_energy
    name: "House Energy From Grid"
    power_id: total_house_from_grid
    accuracy_decimals: 0
    state_class: total_increasing
    device_class: energy
    method: left
    filters: *throttle_time
  - platform: total_daily_energy
    name: "House Energy To Grid"
    power_id: total_house_to_grid
    accuracy_decimals: 0
    state_class: total_increasing
    device_class: energy
    method: left
    filters: *throttle_time
  - platform: template
    lambda: return id(gar_phase_a_power).state + id(gar_phase_b_power).state;
    update_interval: never
    id: total_garage_power
    unit_of_measurement: "W"
  - platform: total_daily_energy
    name: "Garage Daily Energy"
    power_id: total_garage_power
    accuracy_decimals: 0
    state_class: total_increasing
    device_class: energy
    method: left
    filters: *throttle_time
  - platform: template
    lambda: return max((id(solar_a_power).state + id(solar_b_power).state), 0.0f);
    update_interval: never
    id: total_solar_power
    unit_of_measurement: "W"
  - platform: template
    lambda: return max(-(id(solar_a_power).state + id(solar_b_power).state), 0.0f);
    update_interval: never
    id: total_solar_power_draw
    unit_of_measurement: "W"
  - platform: total_daily_energy
    name: "Solar Daily Energy"
    power_id: total_solar_power
    accuracy_decimals: 0
    state_class: total_increasing
    device_class: energy
    method: left
    filters: *throttle_time
  - platform: total_daily_energy
    name: "Solar Night Consumption"
    power_id: total_solar_power_draw
    accuracy_decimals: 0
    state_class: total_increasing
    device_class: energy
    method: left
    filters: *throttle_time
  - platform: template
    lambda: !lambda |-
      return max(0.0f, id(total_garage_power).state -
        id( cir1).state -
        id( cir3).state -
        id( cir4).state -
        id( cir16).state -
        id( cir17).state -
        id( cir19).state -
        id( cir20).state -
        id(total_solar_power).state);
    update_interval: never
    id: balance_power
    device_class: power
    state_class: measurement
    unit_of_measurement: "W"
  - platform: total_daily_energy
    name: "Balance Daily Energy"
    power_id: balance_power
    accuracy_decimals: 0
    state_class: total_increasing
    device_class: energy
    method: left
    filters: *throttle_time
  - { power_id:  cir1, platform: total_daily_energy, accuracy_decimals: 0, name:     "Gar 2F Mini Daily Energy", state_class: total_increasing, device_class: energy, method: left, filters: *throttle_time }
  - { power_id:  cir3, platform: total_daily_energy, accuracy_decimals: 0, name:  "Gar 2F Outlets Daily Energy", state_class: total_increasing, device_class: energy, method: left, filters: *throttle_time }
  - { power_id:  cir4, platform: total_daily_energy, accuracy_decimals: 0, name:   "Gar 2F Lights Daily Energy", state_class: total_increasing, device_class: energy, method: left, filters: *throttle_time }
  - { power_id: cir16, platform: total_daily_energy, accuracy_decimals: 0, name: "FR South Lights Daily Energy", state_class: total_increasing, device_class: energy, method: left, filters: *throttle_time }
  - { power_id: cir17, platform: total_daily_energy, accuracy_decimals: 0, name: "Gar Otlts, 2F Closet Daily Energy", state_class: total_increasing, device_class: energy, method: left, filters: *throttle_time }
  - { power_id: cir19, platform: total_daily_energy, accuracy_decimals: 0, name: "FR North Lights Daily Energy", state_class: total_increasing, device_class: energy, method: left, filters: *throttle_time }
  - { power_id: cir20, platform: total_daily_energy, accuracy_decimals: 0, name:      "FR Outlets Daily Energy", state_class: total_increasing, device_class: energy, method: left, filters: *throttle_time }

[flaviut]

Would you mind updating the readme to have method: left set where appropriate?

[jcam]

yeah i can put in a PR this weekend. i'm curious if this fixes things for @zwbenedict as well or if that's a different issue

[zwbenedict]

As of this morning, I am still getting wild spikes after adding restore: false to the config. I am going to add the method: left this morning and I'll report back in a few hours.

[jcam]

hmm. your errors do look a bit different from mine... that the overall graph is relatively stable makes me wonder if this is a frequency of reporting issue, rather than a value reporting issue, i.e. if influxdb gets 2 posts from the same meter with the same value at the same time is it adding them instead of ignoring one of them in the average? like if there are normally 2 values per minute and it takes an average by adding up those 2 and dividing by 2, is it doing the wrong thing when it gets 3 instead of 2? What does it look like when you do a graph of Wh instead of Wh.mean?