Why things don't add up in the app - the real reason (and fix)

[gekkehenkie11]

Over the years there's been a lot of discussion on the Emporia forum about why things don't add up in the app. It first was theorized by some users that the Emporia VUE probably would measure apparent power instead of real power. Which seemed to make sense: according to my VUE2, my Air conditioner would use about 800 Watt running but 120 Watt standy. I tested with some other devices that all came to about 6-8 Watt standby and a power factor near 0 in standby mode. So I thought the Emporia indeed measured apparent power for a long time.

Then I started looking into making an open source firmware for the Emporia Atmel chip: #173 and to my surprise I learned that the Emporia actually was measuring real power. In the world of samples, real power is simply V * I and then averaging that, what they indeed did! Then I called in the help of my brother who has a major in electrical engineering. I dumped buffers of voltage and current samples and supplied them to him. He first theorized, based on these samples, that it was a timing problem within the Emporia. But then when I supplied more samples, it seemed there were some problems that didn't make sense and couldn't be explained that way. Then he asked me to describe EXACTLY what I've been measuring and that finally led to the answer :) So here we go:

The problem arises for 3 phase users, however ALSO for some specific 2 phase users in North America. Our own hero @flaviut actually is one of those, going by this post:

emporia-vue-local/emporia-vue2-reversing#1 (comment)

Because this is the exact problem! Whenever somebody's electrical system has 120 degrees shift between the voltages and if they then supply 2 phases to a breaker, for example to connect a 220-240V air conditioner things go wrong...

Here on Aruba we have 3 phases: 127V between neutral and phase and 220V (and 120 degrees) between the phases. So the thing is, that THIS is what happens to the resulting voltage when you connect a breaker to two phases:

The resulting Voltage has a phase shift of 30 degrees and a multiplier of V3 (=1.73). So in this case you can not simply do V * I per sample and then simply use a 2.0 multiplier!! The correct way to calculate it in this scenario is:

(V1 - V2) * I

So we should take the difference between the two voltages and multiply THAT with the current!!

With this modification my Emporia VUE2 all of a sudden becomes super accurate :) I get close to 6W now in standby and close to 710 Watt when the air conditioner is on which is really close to what it indeed is (before, with multiplier 2.0 I got close to 800-830 which is too high).

EDIT There's actually a super simple alternative to this. Just use 2 breakers and add the power values up! With Emporia stock firmware this would never work since it would select the wrong phases, but I just tried it and if you lock the right phases this indeed works too! So you would just need an extra CT if you use 2 phases per breaker. Make sure that you don't use filters like 'abs' or 'pos' though, since one leg actually might become negative by design! For example my air conditioner in standby yields +30W on one leg and -22W on the other, so 8W in standby which is correct.

That's it!! I can't thank enough my brother for dedicating his time to help figure this out. Without him I'm sure I still would be banging my head against the wall about all this, haha :) Thanks bro :)

[g3gg0]

Great progress.
I wonder if the protocol you implemented is the same as the original one.
Didnt Emporia send the power calculation result against every phase?
thats what i try to autodetect in the esp application, which phase is the most likely one.
(very crude auto detection)

Does your approach also work for three phase systems?

[gekkehenkie11]

I wonder if the protocol you implemented is the same as the original one.

In the end the timing actually seems fine. So I just used stock firmware for the SAMD09, however, I modified it to take (V1-V2) * I instead of V * I. And it now finally produced the correct result for my devices that I connected between 2 phases.

Didnt Emporia send the power calculation result against every phase?

correct.

Does your approach also work for three phase systems?

Well I don't have any 'approach' just yet ;) The 3 steps above are the way to do it. It's going to be some work though. I guess we first need to expand the YAML file with the option for users with 120 degrees V shift who connect (some of their) breakers between 2 phases. And then we need to implement that extra 'configuration' packet from ESP -> SAMD09, and then have the SAMD09 process that and use (V1-V2) for those situations. And for the situations where the user just connected the breaker between L+N, so the 'normal' situation, we need to do just what they now do already: V * I

[gekkehenkie11]

BTW so if the above is implemented, then it also becomes super easy to calculate power factor. Remember that the SAMD09 already sends the power values (for each phase) and the current RMS to the ESP. Once we have the correct real power, the apparent power is simply the RMS values of V * I , like the SAMD09 already sends to the ESP. So then we can simply divide real power by apparent power and there's your power factor.

Or in other words, for single phase, or 2 phase users (with 180 degrees shift of course) they can already do that. No need to mod the SAMD09 in that case. All that needs to be done in that case is have @flaviut's firmware multiply (V RMS) * (I RMS), like it already gets from the SAMD09 (!), and then divide the real power (which it also already gets from the SAMD09) by apparent power.

Actually, 3 phase users can also already do it now. But only as long as they connect breakers between L+N of course. But that's the 'normal' scenario anyway. Only some devices that need 220V-240V will be connected between 2 phases anyway. I think in my home it are actually only the air conditioners, and maybe my oven that need 220V. BTW, haha, that just made me think of this post: #2 (reply in thread) which now finally makes sense too :) His problem was not the power factor, but his connection between 2 phases with 120 degrees shift.

[gekkehenkie11]

Actually, thinking about it, the easiest way would be to just make the ESP packet bigger. And then instead of sending for each CT (what it does already now) ->

V1 * I
V2 * I
V3 * I

We simply add for the 16 smaller CT's:

(V1-V2) * I
(V2-V3) * I
(V1-V3) * I

So instead of sending 3 power value's per CT, we send 6. That way there's no need to send any 'configuration message' to the SAMD09. We can then simply determine in the ESP firmware which of the 6 above to use. That's of course the reason they're sending those 3 value's right now too. So we can just expand on that. The only question is if the SAMD09 is fast enough to do this, I already ran into speed problems, not sure if there's enough time left to add 48 times a subtraction and multiplication. Would have to find out about that. If not, we could then either lower the sample rate or bit, or implement the configuration packet like I suggested above.

Either way, changing the communication between ESP and SAMD09 would either mean we'd need @flaviut's firmware to be able to handle that, so there should be some YAML setting to let it know which of the 2 types of 'sensor reading' packets to expect.

[gekkehenkie11]

I'm using myself this now, it's quick n dirty but works really well:

https://github.com/gekkehenkie11/Emporia-VUE-fix/tree/main

So I now have 8 ports configured for 2 phase connection and the others should be between phase and neutral. If you want to change this number you can simply change line 618 where it says

if (Muxnr < 4)

to for example

if (Muxnr < 2)

So then it would only use the first 4 ports for our special treatment. So if you only have 4 devices at home that need 2 phases then you could do it like that.

The port ordering is:

10, 2, 11, 3, 12, 4, 9, 1, 13, 5, 16, 8, 14, 6, 15, 7

So if you changed it into Muxnr < 2, it would then have ports 10, 2, 11 and 3 for 2 phases.

With some work it could become a lot nicer and easier (like described above) but for now I've invested already waaaay too much time into this and it works now very well. So for the moment I'll just leave it like this.

EDIT I've also just uploaded a precompiled version, ready to be flashed.

[gekkehenkie11]

LOL I actually have way more 220V devices in my house than I had thought :) All my split unit AC's, my solar, my oven, washing machine, coffee machine, tumble dryer, car charger and there are more that I haven't even identified are running between 2 phases. I now have one VUE completely flashed for 220V ports, so that also makes it easy to remember how it's all configured.

It's so damn awesome to FINALLY have correct value's. BTW I did find in all of my experiments that some of the CT's might have (slightly) higher or lower current measurements than others, so if after the above mod you still have some (slightly) different values than expected (like within 8%) you could try to change the CT, it fixed one last weird thing I saw.

[SamSavageSpeed]

Hello @gekkehenkie11,
Kudos to your brother & to you.
Would be great to have this somehow
integrated into ESPHome in Home Assistant.

[gekkehenkie11]

Hello @gekkehenkie11, Kudos to your brother & to you. Would be great to have this somehow integrated into ESPHome in Home Assistant.

Thanks! Yeah it would be AWESOME if we could just flash it from HA. It should be possible, in fact @odewdney wrote a micro python tool to flash the chip via the ESP: https://github.com/odewdney/emporiavue

[SamSavageSpeed]

Hello @gekkehenkie11,

I have 2 devices in 2 different buildings which
are connected with 2 phases without neutral.

I just found "Phase B Phase Angle" for both :
Building one : 120° and Building two : 240°

About your workaround : (V1-V2)*I
Is it the same calculation for 240° ?

Thanks in advance.

[SamSavageSpeed]

Hello @gekkehenkie11,

Actually, on my 2nd Home Vue, I already did more or less what you describe to
monitor both legs of 4 breakers with 8 clamps but I was not sure if it was fine :

  # Pay close attention to set the phase_id for each breaker by matching it to the phase/leg it connects to in the panel
  - { phase_id: phase_b, input:  "1", power: { id:  cir1, filters: [ *pos, multiply: 1 ] } }
  - { phase_id: phase_a, input:  "2", power: { id:  cir2, filters: [ *pos, multiply: 1 ] } }
  - { phase_id: phase_b, input:  "3", power: { id:  cir3, filters: [ *pos, multiply: 1 ] } }
  - { phase_id: phase_a, input:  "4", power: { id:  cir4, filters: [ *pos, multiply: 1 ] } }
  - { phase_id: phase_b, input:  "5", power: { id:  cir5, filters: [ *pos, multiply: 1 ] } }
  - { phase_id: phase_a, input:  "6", power: { id:  cir6, filters: [ *pos, multiply: 1 ] } }
  - { phase_id: phase_b, input:  "7", power: { id:  cir7, filters: [ *pos, multiply: 1 ] } }
  - { phase_id: phase_a, input:  "8", power: { id:  cir8, filters: [ *pos, multiply: 1 ] } }
on_update:
  then:
    - component.update: solar_power
    - component.update: cabinet_power
    - component.update: garage_prive_power
    - component.update: garage_location_power

  # The copy sensors filter and send the power state to HA        
  - { platform: copy, name: "Solar Power", source_id: solar_power, filters: *throttle_avg }
  - { platform: copy, name: "Cabinet Power", source_id: cabinet_power, filters: *throttle_avg }
  - { platform: copy, name: "Garage Privé Power", source_id: garage_prive_power, filters: *throttle_avg }
  - { platform: copy, name: "Garage Location Power", source_id: garage_location_power, filters: *throttle_avg }

• platform: template
  name: "Solar Power"
  lambda: return id(cir1).state + id(cir2).state;
  update_interval: never # will be updated after all power sensors update via on_update trigger
  id: solar_power
  device_class: power
  state_class: measurement
  unit_of_measurement: "W"

• platform: total_daily_energy
  name: "Solar Daily Energy"
  power_id: solar_power
  accuracy_decimals: 0

• platform: template
  name: "Cabinet Power"
  lambda: return id(cir3).state + id(cir4).state;
  update_interval: never # will be updated after all power sensors update via on_update trigger
  id: cabinet_power
  device_class: power
  state_class: measurement
  unit_of_measurement: "W"

• platform: total_daily_energy
  name: "Cabinet Daily Energy"
  power_id: cabinet_power
  accuracy_decimals: 0

• platform: template
  name: "Garage Privé Power"
  lambda: return id(cir5).state + id(cir6).state;
  update_interval: never # will be updated after all power sensors update via on_update trigger
  id: garage_prive_power
  device_class: power
  state_class: measurement
  unit_of_measurement: "W"

• platform: total_daily_energy
  name: "Garage Privé Daily Energy"
  power_id: garage_prive_power
  accuracy_decimals: 0

• platform: template
  name: "Garage Location Power"
  lambda: return id(cir7).state + id(cir8).state;
  update_interval: never # will be updated after all power sensors update via on_update trigger
  id: garage_location_power
  device_class: power
  state_class: measurement
  unit_of_measurement: "W"

• platform: total_daily_energy
  name: "Garage Location Daily Energy"
  power_id: garage_location_power
  accuracy_decimals: 0

That's good to know that it should be correct for this one :-) Here are last month's data to illustrate :

NB : More than 4 circuits here because some of them are monitored by other measurement devices.

But, for my other Vue, I use the 16 clamps each on a single leg of breaker
so I should either monitor only 8 of my 16 actuals circuits or merge them.

• I already merged circuits because I have more than 16 breakers.
  => I will have to analyze what I want to monitor closely or not
  and either merge more circuits or discard some circuits... :-/

[SamSavageSpeed]

Do you think Flaviut could/would integrate your findings in his project ?
To keep using a single clamp on a single leg to monitor more circuits :-)

[gekkehenkie11]

Well with a single CT you can't get correct value's if you have 120 degrees shift between the voltages, at least not without my firmware mod. I think the best way forward it so make it easy to flash the mod. @odewdney already wrote the script, it just needs integration into HA, someone would need to look into that. I don't have time for that though, but if that's available then it would become very easy to flash the mod and you could put a single CT on a 2 pole breaker with 2 phases.

And yes, your example YAML seems fine, you can add up the poles, just make sure the phases are correct.

[SamSavageSpeed]

Thanks for your answers/work/time. I will think about how to handle my first Vue with 16 clamps.
On one hand, I'm happy that you've confirmed that REAL power is actually mesured by Emporia
but, on the other hand, I'm very disappointed that they didn't adressed the shifting situation :-/

[gekkehenkie11]

Sure! BTW I found one mistake in your YAML, you should not use "POS" filter when adding up 2 phases. Then you'll make the same mistake as Emporia does in their firmware. Don't use pos for those CT's, make sure your have the right phases and that your CT's are correctly oriented. It's very well possible that one CT might become negative, especially when an air conditioner is in standby for example.

[SamSavageSpeed]

Thanks for your feedback. So I should remove *pos in the filters here below ?

  - { phase_id: phase_b, input:  "1", power: { id:  cir1, filters: [ *pos, multiply: 1 ] } }
  - { phase_id: phase_a, input:  "2", power: { id:  cir2, filters: [ *pos, multiply: 1 ] } }
  - { phase_id: phase_b, input:  "3", power: { id:  cir3, filters: [ *pos, multiply: 1 ] } }
  - { phase_id: phase_a, input:  "4", power: { id:  cir4, filters: [ *pos, multiply: 1 ] } }
  - { phase_id: phase_b, input:  "5", power: { id:  cir5, filters: [ *pos, multiply: 1 ] } }
  - { phase_id: phase_a, input:  "6", power: { id:  cir6, filters: [ *pos, multiply: 1 ] } }
  - { phase_id: phase_b, input:  "7", power: { id:  cir7, filters: [ *pos, multiply: 1 ] } }
  - { phase_id: phase_a, input:  "8", power: { id:  cir8, filters: [ *pos, multiply: 1 ] } }

Maybe even remove completly filters since multiply by 1 doesn't do anything ?

  - { phase_id: phase_b, input:  "1", power: { id:  cir1 } }
  - { phase_id: phase_a, input:  "2", power: { id:  cir2 } }
  - { phase_id: phase_b, input:  "3", power: { id:  cir3 } }
  - { phase_id: phase_a, input:  "4", power: { id:  cir4 } }
  - { phase_id: phase_b, input:  "5", power: { id:  cir5 } }
  - { phase_id: phase_a, input:  "6", power: { id:  cir6 } }
  - { phase_id: phase_b, input:  "7", power: { id:  cir7 } }
  - { phase_id: phase_a, input:  "8", power: { id:  cir8 } }

[gekkehenkie11]

Correct. And then make sure it's all configured correctly, meaning that you need to be sure that the phases are correct and that the values make sense. For example I have an AC between 2 phases, I get for example +30 on the one phase and -22 on the other in standby, resulting 8 Watt, which I know is correct. And when I turn the AC on, both get positive which is of course correct too. As a rule of thumb, it seems to be true that if you connect between 2 phases, one of the CT's need to be pointing in the other direction as the other one. So one arrow K->L should point toward the breaker and the other towards the load.

BTW in general I would advise not to use filters at all, unless you're 100% sure that what you're doing is correct. I find it way easier to spot mistakes when there are no filters. I do use some 'abs' myself for example, in those cases that I'm absolutely 100% sure that the CT is installed in the wrong direction, on a L+N breaker (!) and when it's difficult to take them out and switch direction, because there's no space for example. But again, when connecting between 2 phases you should never do that since a negative value on one of the 2 poles actually might be the correct outcome.

[SamSavageSpeed]

At the moment, the 8 clamps are put in the same direction on all legs of the 4 breakers.
Maybe I could keep the multiply filter afterall and just multiply by "-1" one of the leg ?
This way I could avoid changing orientation of the clamps ? Giving something like this:

• { phase_id: phase_b, input: "1", power: { id: cir1, filters: [ multiply: 1 ] } }
• { phase_id: phase_a, input: "2", power: { id: cir2, filters: [ multiply: -1 ] } }
• { phase_id: phase_b, input: "3", power: { id: cir3, filters: [ multiply: 1 ] } }
• { phase_id: phase_a, input: "4", power: { id: cir4, filters: [ multiply: -1 ] } }
• { phase_id: phase_b, input: "5", power: { id: cir5, filters: [ multiply: 1 ] } }
• { phase_id: phase_a, input: "6", power: { id: cir6, filters: [ multiply: -1 ] } }
• { phase_id: phase_b, input: "7", power: { id: cir7, filters: [ multiply: 1 ] } }
• { phase_id: phase_a, input: "8", power: { id: cir8, filters: [ multiply: -1 ] } }

Off course, on each of the 4 breakers, I should try some expected loads to verify.

[gekkehenkie11]

yeah multiplying with -1 is indeed an option.

[BlipRanger]

Following this thread. I just recently finished flashing my Gen 3 (NA with a handful of 220v breakers, only measuring a single leg) with this firmware and noticed that when compared to what was being reported by my power company, it was off by quite a bit.
So 2 quick questions:

• Does this same issue affect gen 3?
• Does the measure of power (as used here) differ much from consumption measured upstream by a power company?

[gekkehenkie11]

No don't short it, that WILL result in a fire indeed. It's indeed like @flaviut just explained.

Like explained, the VUE won't work in situations where you're getting multiple phases from your utility company and you have 120 degrees between 2 phases and feed those 2 phases to a device. So instead of feeding L+N, you feed L1+L2. Well, actually it WILL work, as long as you use esp home, put a CT on each leg, and lock the right phases to each leg and then sum the 2 legs (or use this mod as explained here and you can only monitor one leg, saving a CT). But all this won't work with stock firmware since the emporia is very likely to select the wrong phase(s) in this situation. Before understanding all this, I used to use Emporia stock firmware and my 220V air conditioner would for example show a 90 watt consumption in standby which is completely wrong. There is a long thread about this on the emporia forums. The first theory was that the emporia measured apparent power instead of real power. But that turned out to be wrong and the real issue is described in this thread.

Anyway I'd highly suggest not to experiment with all this stuff if you don't completely understand what you're doing.

[AndreKR]

So, to be clear, if I use ESPHome I don't need to flash the Atmel chip with the alternative firmware?

[flaviut]

If you are not an experienced embedded software engineer looking to tinker, do not try and flash the Atmel chip.

If you flash the Atmel, I'm not even sure it's possible to go back to the original…

See below

[gekkehenkie11]

@AndreKR You don't need to flash the Atmel chip. The advantage though is that by flashing it you can save some ports since you only need 1 CT per leg.
@flaviut Nah it's really not that much more difficult than flashing the ESP chip. The only thing is of course that you need some jlink connector. During the coding of my alternative firmware I messed up tons of times, resulting in a bricked device, but every time it was super easy to flash a new version and unbrick it. I dont think you can do much harm.

[flaviut]

I'm sorry about the FUD! I still don't think this is a beginner thing, but I was worried about being unable to backup the stock firmware image. Thanks for cleaning things up