Enphase field MTBF: M190: ~36 Years M215: ~316 Years M250: >357 Years

[pclifton]

And, so it begins. On the East Coast we have had a very cloudy December and January. Enphase updated all of my M190 firmware when I installed a new Envoy to 520-00008-r01-v01.08.01. Also note that four of my original 24 M190s have been replaced due to failures. It has been difficult given the weather to find a good sunny day for comparison, but my feeling was that the system efficiency went WAY DOWN. Note: EAST_TPM and CENTER_TPM have M190s (installed in late 2010), the WEST_TPM is all M250s (installed in 2015). While we had two sunny days where the M190s produced 180-190 watts back on January 19th, it looks like all twenty of the original M190s failed today, clipping power to 140 watts, here are the results for today.

The four M190s that have been replaced are showing a peak power of 181w, 182w, 187w, and 189w. It should be noted that the ones showing 187w and 189w are M190IGs. The worst of it is I now have twenty (20) M190s that peak out at 139-140 watts. This is completely unacceptable. During the middle part of the day my output is down 40w x 20 inverters = -800w

I don't look forward to trying to speak with someone at Enphase next week. Given what has been posted here I suspect that is a dead-end. I can try to get a "solar professional" from 40 miles away involved, but to be honest. I am considering another manufacturer. Looks like the micro-inverter experiment did not turn out too well.

Ken Clifton

 

[DaveEV]

I wonder what triggered the update of your system and also if they will eventually update you to the newer version..

I have no idea - the interesting thing is that the reported "Running Image" version did not change - it was the same before.
Possibly one thing that changed is whatever parm0/parm1 is which was logged as version 540-00089-r01-v01.08.53 in the logs. Unfortunately, the last time my inverters updated I don't have any record of that and the event log only holds 500 entries so it's impossible to see what happened beyond a couple days.

pclifton - that's really weird.

 

[AZDude]

.... The worst of it is I now have twenty (20) M190s that peak out at 139-140 watts. This is completely unacceptable.
I don't look forward to trying to speak with someone at Enphase next week. Given what has been posted here I suspect that is a dead-end. I can try to get a "solar professional" from 40 miles away involved, but to be honest. I am considering another manufacturer. Looks like the micro-inverter experiment did not turn out too well.

Ken Clifton

Yup, I had 4 inverters fail at the same time a couple years back with that exact symptom (maxed out at 140 watts) . As I recall there are 4 or 5 power transistors in the inverter and when one of them fails the output drops to 140 watts max. At the time Enphase told me they were happy to just pay me for the lost power instead of replacing the inverters, but they would replace them if I insisted. I insisted. They eventually sent out replacements after 'losing' the trouble ticket a couple times.

I figure in a couple years my entire array will end up getting swapped out with the M190-IGs which so far seem to be much more reliable.

I just wish that when they have to swap out a bunch of the M190s at once, that they would give you the option of going with the new cabling system instead of sending adapter cables out.

Oh well, I bought some of the new style cable off ebay. Once I swap over I'll see if anyone is interested in buying the adapter cables for adding new style inverters to their old style daisy chains.

 

[pclifton]

Yup, I had 4 inverters fail at the same time a couple years back with that exact symptom (maxed out at 140 watts) . As I recall there are 4 or 5 power transistors in the inverter and when one of them fails the output drops to 140 watts max. At the time Enphase told me they were happy to just pay me for the lost power instead of replacing the inverters, but they would replace them if I insisted. I insisted. They eventually sent out replacements after 'losing' the trouble ticket a couple times.

Thanks AZDude.
The only remedy that I will accept is the replacement of the twenty M190s. If that does not happen, then I will just publicize the heck out of this and move on. Just in case, I got a quote from Chilicon Power yesterday. I like their inverter design and local monitoring a lot. No dependence on a subscription to Enlighten or sending data out over the Web -- unless you want to.

Ken Clifton

 

[pclifton]

Today, I have heard from two other Enphase System owners with older M190s that received firmware updates between January 19th and January 24th. The firmware installed seems to be: 520-00008-r01-v01.08.00 or 520-00008-r01-v01.08.01 This firmware seems to initiate clipping to 140 watts on all older M190s.

So by sending out one firmware update it looks like Enphase has just hobbled 20 inverters for me and many others out there in the wild too. I am sure the motivation will be called "longevity" or "safety" related.

My prediction is that there will be a lot of unhappy people when this gets out.

Ken Clifton

 

[RegGuheert]

And, so it begins. On the East Coast we have had a very cloudy December and January.

It was cloudy here in December, but my January production has completely blown away all the records set in the previous five years. Specifically, I expect the system will have produced 1.3 MWh by the end of today, which is 16% higher than the previous record set in 2012.

This is true even though we received the largest single-storm snowfall in history last weekend. (Thanks to our Snow Rake!)

Enphase updated all of my M190 firmware when I installed a new Envoy to 520-00008-r01-v01.08.01. Also note that four of my original 24 M190s have been replaced due to failures. It has been difficult given the weather to find a good sunny day for comparison, but my feeling was that the system efficiency went WAY DOWN. Note: EAST_TPM and CENTER_TPM have M190s (installed in late 2010), the WEST_TPM is all M250s (installed in 2015). While we had two sunny days where the M190s produced 180-190 watts back on January 19th, it looks like all twenty of the original M190s failed today, clipping power to 140 watts, here are the results for today.

...snip picture...

The four M190s that have been replaced are showing a peak power of 181w, 182w, 187w, and 189w. It should be noted that the ones showing 187w and 189w are M190IGs. The worst of it is I now have twenty (20) M190s that peak out at 139-140 watts. This is completely unacceptable. During the middle part of the day my output is down 40w x 20 inverters = -800w

Yup, I had 4 inverters fail at the same time a couple years back with that exact symptom (maxed out at 140 watts) . As I recall there are 4 or 5 power transistors in the inverter and when one of them fails the output drops to 140 watts max.

This does sound like the same failure as pclifton is reporting. Typically I would expect multiple simultaneous failures to be due to damage from lightning. Since many areas of the East Coast experienced "thundersnow" during the blizzard last weekend, is there any chance there was a power surge on the grid nearby due to a lightning strike? If so, then you would not have seen the effects until the snow cleared from your panels. Anyway, just a thought.

I figure in a couple years my entire array will end up getting swapped out with the M190-IGs which so far seem to be much more reliable.

Agreed. I think they are an excellent solution for those with 60-cell PV modules (or who never experience cold weather). Most of my modules have 60 cells, but, thanks to some help by QueenBee, I now have twelve M190s attached to 72-cell PV modules, so I will try to keep old-style M190s in those positions for as long as possible. It will be interesting to see how long the best M190s will last.

I just wish that when they have to swap out a bunch of the M190s at once, that they would give you the option of going with the new cabling system instead of sending adapter cables out.

Me, too, but I don't expect them to do that since they often are only replacing individual units.

Oh well, I bought some of the new style cable off ebay. Once I swap over I'll see if anyone is interested in buying the adapter cables for adding new style inverters to their old style daisy chains.

I've also bought several used Engage cables for eventual replacement. I currently have Engage positions for 20 new-style inverters on my roof and a cable for another 16 positions not yet installed. Once installed, that will cover 36 of the 60-cell PV modules I have. I only need a few more to cover the entire array. Perhaps I will use a handful of the cables which come with the M190-IGs to make up the difference (but I doubt it).

 

[pclifton]

Typically I would expect multiple simultaneous failures to be due to damage from lightning. Since many areas of the East Coast experienced "thundersnow" during the blizzard last weekend, is there any chance there was a power surge on the grid nearby due to a lightning strike? If so, then you would not have seen the effects until the snow cleared from your panels. Anyway, just a thought.

No lightening or "thunder snow" here. Just to be very clear. The clipping began immediately after the new firmware was installed on January 24th. That firmware version was: 520-00008-r01-v01.08.01

As for any other theories about surges, please remember that I have a Delta-T protector on the Power Company Meter, a Square-D panel surge breaker on the Main Circuit Breaker panel, an Intermetnic Panel Protector at the subpanel at the first pole, and a Square-D protectors at each of the other poles. And, remember that nothing changed except on the older M190s. Each of the pole arrays with M190s has two replacement inverters that are still performing correctly.

I hope none of the rest of you get the firmware "update."

Ken Clifton

 

[RegGuheert]

No lightening or "thunder snow" here. Just to be very clear. The clipping began immediately after the new firmware was installed on January 24th. That firmware version was: 520-00008-r01-v01.08.01

Thanks for setting the record straight.

Given AZDude's experience, it sounds as if the firmware *somehow* makes some measurement of the impedance of the power devices in the inverter and throttles the power limit based on those measurements. Clearly something went *very* wrong with your update that triggered this throttling. I can only imagine that this situation is fully reversible via some remote action on Enphase' part. Let's hope so!

I hope none of the rest of you get the firmware "update."

Please note that Enphase updated the firmware in my M190s to that same version on December 16, 2015, before the record production I experienced during this month. (And, no, I do not think the firmware was the reason for the record, but rather the weather was the key. On the contrary, the new firmware likely cost me about 10 kWh of production during this month.)

I hope you can get your issue resolved soon.

 

[RegGuheert]

For everyone else following this thread, I want to post information pclifton sent me through PM:

Here is information about the inverters:

Older M190 with the new firmware clipping to 140 watts
800-00038-r08 Fri Apr 10, 2015 07:04 AM EDT 121011127350 OK 520-00008-r01-v01.08.01 - Sun Jan 24, 2016 04:40 PM EST fff-fffff-rff-vff.ff.ff 480-00002-r01-v00.0b.12 Sun Jan 31, 2016 01:56 PM EST

Newer M190 with the new firmware NOT clipping... Right now producing 183 watts
800-00065-r03 Sat Sep 12, 2015 10:01 AM EDT 121049463300 OK 520-00008-r01-v01.08.01 - Thu Dec 17, 2015 09:44 AM EST fff-fffff-rff-vff.ff.ff 480-00002-r01-v00.0b.12 Sun Jan 31, 2016 01:56

My inverters, which were activated only about ONE MONTH after pcliftons have the same HW Part No. of 800-00065-r03 as the ones in pclifton's system which are NOT experiencing the problem.

So it seems there is one of four possible explainations:
- Enphase is throttling the power limit on all Enphase M190 microinverters prior to HW Part No. 800-00065-r03
- Enphase is throttling the power limit on all Enphase M190 mircoinverters after a certain amount of time in the field (or activated before a particular date, like January 1, 2011. Perhaps they will occasionally update this date as the M190s age)
- The new firmware is somehow detecting if power device resistance raises above a certain level and triggers a power limiting mode if it does. Possibly the early hardware revision has power devices which trip this limit.
- The new firmware is designed to detect failure of the hardware that was not detected by previous versions.

I'm considering unplugging my Envoy from the internet in case Enphase intends to continue to cripple M190s as they age.

 

[RegGuheert]

It seems that AZDude is not the only early-build M190 owner who has experienced the 140W power limiting before the recent firmware update. I found a review from someone near me which said the following:

M190 inverters clipping at 140 W

2.00
11/29/2015
I had 18 panels with the M190 microinverters installed in January 2010 in our vacation home in Berkeley Springs, West Virginia. Beginning in October 2014 one of the inverters began clipping at 140W instead of the nominal 190 W. The second had the same problem beginning in April 2015. In September 2015, the third one failed in the same way and in October 2015 the fourth one started clipping at 140W. With this failure rate, I am surprised that there have not been more reports of the problem. To date I have not had any response from Enphase.

Gerald Share

 

[AZDude]

Today, I have heard from two other Enphase System owners with older M190s that received firmware updates between January 19th and January 24th. The firmware installed seems to be: 520-00008-r01-v01.08.00 or 520-00008-r01-v01.08.01 This firmware seems to initiate clipping to 140 watts on all older M190s.

So by sending out one firmware update it looks like Enphase has just hobbled 20 inverters for me and many others out there in the wild too. I am sure the motivation will be called "longevity" or "safety" related.

Ken Clifton

Well I just checked my array and as of Jan 25 all of my remaining original inverters are now maxing out at 140 watts. However, given that everyone else is apparently having the same problem, I don't expect to recieve replacements any time soon.

 

[AZDude]

So it seems there is one of four possible explainations:
- Enphase is throttling the power limit on all Enphase M190 microinverters prior to HW Part No. 800-00065-r03
- Enphase is throttling the power limit on all Enphase M190 mircoinverters after a certain amount of time in the field (or activated before a particular date, like January 1, 2011. Perhaps they will occasionally update this date as the M190s age)
- The new firmware is somehow detecting if power device resistance raises above a certain level and triggers a power limiting mode if it does. Possibly the early hardware revision has power devices which trip this limit.
- The new firmware is designed to detect failure of the hardware that was not detected by previous versions.
....

Given that all of the failures seem to have occured when they upgraded the firmware a week or so back, I don't think this is related to when they were installed, and given the wide variety of panels connected to these inverters, I doubt it has anything to do with impedence matching.

It could be related to the inverters part number. I suspect this is just a firmware error that only effects inverters produced prior to some design change.

If they can fix this by reprogramming them, that's fine. If not, they WILL replace mine. Personally I'd prefer the second option since it will mean that all of my inverters will be the new style.

 

[RegGuheert]

Given that all of the failures seem to have occured when they upgraded the firmware a week or so back, I don't think this is related to when they were installed,...

All inverters which failed at 140W, including yours, were activated in 2010. No inverters activate in 2011 or later have had the issue.

...and given the wide variety of panels connected to these inverters, I doubt it has anything to do with impedence matching.

The attached PV module has no relevance here. I'm talking about the resistance of the power switches, which is calculated by the voltage across the switch divided by the current, both of which are measured by the inverter. You would not have had the 140W failure of four of your inverters had they not been measuring switch impedance.

It could be related to the inverters part number.

I agree that is the most likely explanation. However, the fact that you and others have had the 140W failures of these older M190s due to a loss of a power devices means those other options cannot be excluded as real possibilities. For instance, if they changed the resistance threshold at which they trigger the 140W limit in the new firmware, that could result in exactly the result that pclifton is seeing if the older M190s had higher switch resistance. And it does seem reasonable that they would want to tighten up their telemetry on these devices to detect the onset of failure as early as possible.

I suspect this is just a firmware error that only effects inverters produced prior to some design change.

Me, too. But if that is the case, that implies that Enphase pushed this fix out to customers without first testing on ALL of the hardware versions of the M190s. I find that a bit hard to imagine given the number of inverters that must be impacted. Roughly HALF of the M190s I have in my database were activated in 2010 or earlier.

Drees is an interesting case, since it appears from his recent post that Enphase TRIED to upload SOMETHING to his inverters but perhaps nothing happened. I recall that he had M190s which had a serial number which began with "11" instead of the "12" like most of the rest of ours have. Drees also has experience no failures of any of his inverters. Could you please tell us the HW Part No for your M190s, drees?

If they can fix this by reprogramming them, that's fine. If not, they WILL replace mine. Personally I'd prefer the second option since it will mean that all of my inverters will be the new style.

It sounds like you got the new firmware and are also experiencing the problem. Is that correct? Also, can you please tell us the HW Part No of your original M190s?

 

[pclifton]

I just heard from the solar professional that I had to get involved to handle communications with Enphase. We have a case initiated. He was told by Enphase, that "yes" the update caused my clipping issue. He was further told that any inverter that is clipping to 140 watts after the update must be replaced.

I told the solar professional that I hope that means "Enphase replaces..." If I get a RMA I wil provide an update here.

I also heard from a friend in Minnesota about his Enphase M190s, similar vintage, similar clipping, same update. He was told by Enphase that they are working on a firmware resolution.

At this point it seems the only common item between Minnesota and North Carolina's information is that there is a firmware causation.

As an aside, I was told today that electrolytic capacitor failure can cause clipping similar to this in inverters. The figure of 20,000 hours life for electrolytic capacitors was mentioned. If I do the math correctly, my inverters are just over 5 years old. At 10 hours per day multiplied by 365 days by 5 years that comes up to 18,250 hours...

Respectfully,
Ken Clifton

 

[RegGuheert]

Thanks for the update.

He was told by Enphase that they are working on a firmware resolution.

This is the only thing that makes any sense to me. Imagine being Enphase CEO Paul Nahi standing in front of the BoD discussing the M190 warranty claims, one of your company's biggest financial hurdles: "We pushed out new firmware and broke 50% of the M190s remaining in the field so we decided to replace each and every one of them."

No, that wouldn't play well. Enphase has been working hard to reduce, or at the very least stretch out, their warranty claims through these firmware upgrades. Breaking massive quantities of inverters by pushing out bad firmware is financial suicide.

That firmware snafu will be fixed very soon, IMO.

As an aside, I was told today that electrolytic capacitor failure can cause clipping similar to this in inverters. The figure of 20,000 hours life for electrolytic capacitors was mentioned. If I do the math correctly, my inverters are just over 5 years old. At 10 hours per day multiplied by 365 days by 5 years that comes up to 18,250 hours...

We've discussed the capacitors in the M190s at length on this forum. I'm sure there have been a few random failures of capacitors, but I really don't think we are seeing a bunch of capacitors get to the end of their life. They used capacitors which were rated for either 4000 hours or 8000 hours at 105C and if you calculate their life in this application you come up with a 50-year life. Stress testing indicated likely a 30-year capacitor life, though Nichicon, the manufacturer, clearly states that any calculation over 15 years should be limited to 15 years. Hence the 15-year warranty on the M190s.

I personally think the failures are mostly related to failing solder joints or failing SOC attachment points.

 

[DaveEV]

FWIW:

I have M190 HW Part Num 800-00037-r08, running image "updated" to 520-00008-r01-v01.08.00 - Mon Jan 25, 2016 03:30 PM PST, no apparent clipping at 140W yet.

 

[AZDude]

Given that all of the failures seem to have occured when they upgraded the firmware a week or so back, I don't think this is related to when they were installed,...

All inverters which failed at 140W, including yours, were activated in 2010. No inverters activate in 2011 or later have had the issue.

None that we are aware of. If it is part number related, it could be that there are a few that sat on a shelf for a couple years before being installed. Or I could be wrong.

It occurs to me that there is another possibly reason for these failures. It's possible that the new firmware somehow damaged the inverters and they can't be fixed. Otherwise why didn't they just roll back to the previous version of the firmware?

One way or another, someone definitely screwed the pooch at Enphase.

The part number of my inverters is 880-72112-13
Hmm, maybe not. The above number is the part number that is actually printed on the inverter.
The number reported on Enlighten is: 800-00037-r08 (M190)

Now I'm wondering if the part number seen on Enlighten is changed by the firmware?

 

[RegGuheert]

I have M190 HW Part Num 800-00037-r08, running image "updated" to 520-00008-r01-v01.08.00 - Mon Jan 25, 2016 03:30 PM PST, no apparent clipping at 140W yet.

Thanks. So dress continues to be the exception to M190 failures. Perhaps there are benefits to only connecting 180W PV modules to the M190s.

It occurs to me that there is another possibly reason for these failures. It's possible that the new firmware somehow damaged the inverters and they can't be fixed.

Agreed that's a possibility. Let's hope that's not what happened!

Otherwise why didn't they just roll back to the previous version of the firmware?

One reason is that it can take DAYS for an Envoy to update the firmware on all the inverters in an array. They likely need to know that everything is done updating before sending another firmware revision.

Another reason would be that they should be sure they FULLY understand what went wrong and also that they FULLY understand the implications of the "fix" before they take a next step. As you motioned above, with power electronics firmware can and does destroy hardware. It is best to take steps only after careful deliberation.

One way or another, someone definitely screwed the pooch at Enphase.

Agreed. Overtime is being worked.

The part number of my inverters is 880-72112-13
Hmm, maybe not. The above number is the part number that is actually printed on the inverter.
The number reported on Enlighten is: 800-00037-r08 (M190)

Thanks. So, are your microinverters with that part number updated to the new firmware and are they now clipping at 140W?

Now I'm wondering if the part number seen on Enlighten is changed by the firmware?

I don't think they are. Hopefully that number cannot be changed by the firmware so that the hardware can be uniquely identified.

FWIW, the number on the label of my M190s is "880-72412 05".

 

[DaveEV]

One reason is that it can take DAYS for an Envoy to update the firmware on all the inverters in an array. They likely need to know that everything is done updating before sending another firmware revision.

Hmm, all 18 of mine updated within a couple hours according to the Envoy logs.

My earlier comment that I haven't seen any clipping was a bit premature - given that my inverters are only peaking around 110W right now. In fact, with 180W panels my inverters rarely spend any significant amount of time pushing more than 140W. I might get some edge of cloud effect peaks over 140W/ea by the end of the month, otherwise I have to wait until April to regularly start breaking 140W/inverter.

 

[RegGuheert]

One reason is that it can take DAYS for an Envoy to update the firmware on all the inverters in an array. They likely need to know that everything is done updating before sending another firmware revision.

Hmm, all 18 of mine updated within a couple hours according to the Envoy logs.

Note that the Envoy can only update the firmware in the microinverters during the daytime hours. Likewise, it cannot update microinverters which are covered with snow.

Here's what happened in January 2012 when I reported the first M190 failure in my system:
- Jan 11: New firmware transferred to Envoy by Enphase at 3:30PM. That day all microinverters received an image called 540-00036-r01-v01.02.03 (image_type="parm0"). They were all then commanded to reset before the sun went down.
- Jan 12: After the inverters awoke, the Envoy started to attempt to upload the image called 520-00008-r01-V01.03.01(image_type="pro load"). The Envoy attempted to update the microinverters to that image all day long, but all attempts failed. I'm not sure why.
- Jan 13: The Envoy again spent all day trying to update microinverters. When the sun went down, 18 of the 42 microinverters had successfully updated to the new firmware.
- Jan 14: The Envoy again spent all day trying to update microinverters. When the sun went down, 21 more microinverters had successfully been updated, bringing the total to 39.
- Jan 15: The Envoy completed the updating of the last three microinverters by 8:25AM.

In all, the firmware update which was initiated on January 11 took three full days and two partial days to complete.

Since that was close to the winter solstice, that was probably close to the worst case, barring a snowstorm.

The largest M190 system in my database contains 154 microinverters. I imagine it could take well over a week to fully update the firmware for all M190s in the array, assuming no snow.

It is certainly possible that they have improved the image update process since then, but I don't have any way to know.