philip
Posts: 122
Joined: Tue Nov 27, 2012 8:18 pm
Delivery Date: 25 Aug 2015
Leaf Number: 327341
Location: El Cajon, CA

Re: Enphase field MTBF: M190: ~44 Years M215: ~470 Years M250: >275 Years

Tue Nov 22, 2016 10:58 am

For what it's worth:

My M250 micros are running 520-00044-r01-v01.19.01

This is the image they were updated to the day they were first turned on: Sat Jun 20, 2015 05:08 PM PDT They were first powered on at 2:30pm that same day, so it took about 2.5 hours for all 20 to update.

My envoy is running R3.12.34 (2e8bfe) build date Wed Apr 29, 2015 02:28 PM PDT
2015 SV
Old 11/11/15: 2012SL, SOH76% 49.97AHr 35,644mi vin 26790
LEAF press kits

pclifton
Posts: 60
Joined: Thu Jan 26, 2012 4:51 pm
Delivery Date: 28 Jan 2012
Leaf Number: 17731
Location: Salisbury, NC
Contact: Website

Re: Enphase field MTBF: M190: ~44 Years M215: ~470 Years M250: >275 Years

Tue Nov 22, 2016 1:35 pm

RegGuheert wrote:I have a question for those of you who have experienced a failure of your Envoy: Did you notice any symptoms prior to the failure?
...
Any thoughts?


No problems or updates here. My two Envoys are both:
Current Software Version R3.12.34 (2e8bfe)
Software Build Date Wed Apr 29, 2015 02:28 PM PDT
Envoy Power Line Device 480-00008-r01-v02.0a.19

(No changes since 2015)

Thoughts: I wonder if a strong broadcast station has been updated or moved to your area? Another possibility might be that the power company is piloting broadband over power lines.

CFL bulbs are notorious for inducing noise on the power line. One bulb can disrupt a lot of communications. I had so much trouble with my communications over the years that I had to move both Envoys out next to the arrays in weatherproof and vented boxes. Venting is very important with them outside.

As far as failure indications. No, there was no advanced warning. My oldest Envoy just stopped working completely one day.

I wish you the best with getting it all resolved.

Ken Clifton
2 PHEVs Powered by 10.2 kW Solar
Topaz Blue 2013 Chevy Volt and Silver 2014 Chevy Volt
Traded My Ocean Blue 2012 LEAF SL on 2-2013
Wife Traded Her Cayenne Red 2012 LEAF SL on 11-2013
Web site: http://www.kenclifton.com

User avatar
RegGuheert
Posts: 6287
Joined: Mon Mar 19, 2012 4:12 am
Delivery Date: 16 Mar 2012
Leaf Number: 5926
Location: Northern VA

Re: Enphase field MTBF: M190: ~44 Years M215: ~470 Years M250: >275 Years

Tue Nov 22, 2016 4:05 pm

philip wrote:My M250 micros are running 520-00044-r01-v01.19.01
I have four new M250s which are running the following image:

520-00044-r01-v01.20.00 - Thu Jan 14, 2016 08:02 AM EST

I also have a bunch of M215IGs running that image.
philip wrote:This is the image they were updated to the day they were first turned on: Sat Jun 20, 2015 05:08 PM PDT They were first powered on at 2:30pm that same day, so it took about 2.5 hours for all 20 to update.
That's fast! Not one update succeeded today.
philip wrote:My envoy is running R3.12.34 (2e8bfe) build date Wed Apr 29, 2015 02:28 PM PDT
pclifton wrote:No problems or updates here. My two Envoys are both:
Current Software Version R3.12.34 (2e8bfe)
Software Build Date Wed Apr 29, 2015 02:28 PM PDT
Envoy Power Line Device 480-00008-r01-v02.0a.19
My new Envoy (currently in place) has the following information associated with it:

Model Number (on bottom of device) 880-0007 16 366
Type (from Enlighten) 800-00069-r05 (Envoy-R-NA)
Current Software Version R3.12.45 (f9b2fc)
Software Build Date Wed Aug 12, 2015 05:27 PM PDT
Envoy Power Line Device 480-00008-r01-v02.0a.19
pclifton wrote:Thoughts: I wonder if a strong broadcast station has been updated or moved to your area?
There is a new wireless internet provider a few miles away, but there is no way that is powerful enough to cause this issue.
pclifton wrote:TAnother possibility might be that the power company is piloting broadband over power lines.
I've wondered about this. My power company certainly reads the meter over the powerlines. It is certainly possible that they changed their PLC waveforms or put a repeater closer fairly recently.

But if that were the case, I probably wouldn't get five stars with the new Envoy when the house was all shut down.
pclifton wrote:CFL bulbs are notorious for inducing noise on the power line. One bulb can disrupt a lot of communications.
This is one thing that the representative from Enphase mentioned. It's likely something like this, but my testing was inconclusive since the old Envoy only ever reported ZERO bars. Also, eliminating loads one at a time caused microinverters to come online only gradually. There was no "silver bullet" to be had.
pclifton wrote:As far as failure indications. No, there was no advanced warning. My oldest Envoy just stopped working completely one day.
O.K. This is quite different. Both of my Envoys currently are working, but neither of them is working well. I have to say that my old Envoy is currently working better than the new one since when the house was "off" yesterday, it was talking to 52 inverters (with ZERO bars showing) while the new one could only locate 20 (with FIVE bars showing) under those same conditions. I can only think the two Envoys are experiencing different issues communicating.
pclifton wrote:I wish you the best with getting it all resolved.
Thanks, Ken, Philip and drees! I appreciate all your help with this issue!

Quick update from today:
- Signal level with the new Enovy is still indicating TWO bars (with the house running).
- No microinverters successfully received updated firmware today. All attempts failed.
- Communications with the (20 known) inverters was very poor. They were NEVER all seen at the same time and sometimes only TWO were working properly.
- The new Envoy spent the first half of the day trying to command the inverters to stop producing power (unless it was trying to update firmware). I think this must be the default setting for a new Envoy. I changed that setting on Enlighten, so hopefully it won't do that again tomorrow.
- Toward the end of the day I switched off three of the five breakers for the solar arrays. I only left on the old M250s on the roof and the old M190s in the field. During this period, I tried to perform updates and discover inverters. The updates failed, but four M190s in the field as well as one M250 on the roof were discovered. So this new Envoy now knows about 25 of the 54 inverters it should be able to see.
- I have now switched off the breaker for the field array in the hope that the Envoy will be able to update at least some of the old M250s in the morning when the sun comes up.

It's a bit frustrating that this all came on without any warning. Even with the rest of the house completely shut down, neither Envoy can communicate with all of the inverters. I'm just hopeful that the issue with the new Envoy has to do with the old firmware which is found in some of the inverters up there. If those can be successfully updated, perhaps things will return to normal. I just don't know...
RegGuheert
2011 Leaf SL Demo vehicle
10K mi. on 041413; 20K mi. (55.7Ah) on 080714; 30K mi. (52.0Ah) on 123015; 40K mi. (49.8Ah) on 020817; 50K mi. (47.2Ah) on 120717; 60K mi. (43.66Ah) on 091918.
Enphase Inverter Measured MTBF: M190, M215, M250, S280

User avatar
drees
Moderator
Posts: 6221
Joined: Fri Apr 23, 2010 3:51 pm
Location: San Diego

Re: Enphase field MTBF: M190: ~44 Years M215: ~470 Years M250: >275 Years

Tue Nov 22, 2016 11:31 pm

I suspect that your firmware updates and signal quality issues are related.

When I had signal quality issues with my system shortly after I installed my system, I resolved them by moving my Envoy as close as I could to the panel that feeds my arrays. In some cases, Enphase recommends installing the Envoy right on the combiner panel and installing noise filtering on the grid side to clean up the signal. Are you able to move your Envoy closer to the inverters?
'11 LEAF SL Powered By 3.24 kW Enphase Solar PV

User avatar
RegGuheert
Posts: 6287
Joined: Mon Mar 19, 2012 4:12 am
Delivery Date: 16 Mar 2012
Leaf Number: 5926
Location: Northern VA

Re: Enphase field MTBF: M190: ~44 Years M215: ~470 Years M250: >275 Years

Wed Nov 23, 2016 2:23 am

drees wrote:I suspect that your firmware updates and signal quality issues are related.
Agreed: the updates are failing because the Envoy is having problems communicating with the inverters.
drees wrote:When I had signal quality issues with my system shortly after I installed my system, I resolved them by moving my Envoy as close as I could to the panel that feeds my arrays. In some cases, Enphase recommends installing the Envoy right on the combiner panel and installing noise filtering on the grid side to clean up the signal. Are you able to move your Envoy closer to the inverters?
Yes, but only for the microinverters on my roof. There is a subpanel there which is electrically far from my main panel. Once upon a time I had considered the possibility of adding at 120V outlet below that panel and installing the Envoy there. But since it was working and that would require running an ethernet cable the length of the house it never happened. Now that I have brought the field array online, that solution no longer works. This is because the field array is connected directly into the main breaker panel. If I moved the Envoy closer to the subpanel for the roof array, I would be moving it farther from the field array.

But the outstanding question remains: What precipitated this problem? On Saturday, all was working well. On Sunday, communications were extremely bad. I currently have eight working hypotheses and since I have found no resolution, I have not been able to completely rule any of them out:

1) The PLC circuitry in the old Envoy slowly degraded over time and on Sunday when the temperature dropped significantly, it was no longer able to communicate with the arrays successfully. This doesn't explain the issues with the new Envoy, but perhaps there is a separate explanation for that Envoy's problems (which clearly ARE different).
2) The PLC circuitry in the old Envoy was damaged by some sort of surge. While there has not been any lightning recently, I DID have two recent instances of arc faults in my house. However, neither of those faults occurred on Saturday.
3) The PLC circuitry in one or more of the microinverters failed in some way which has caused noise on the power lines. I doubt this is the case since isolating the various portions of the array should have allowed full communications to be restored.
4) The power stage in one or more inverters is malfunctioning in some way which has caused noise on the power lines. As above, I doubt this is the case since isolating the various portions of the array should have allowed full communications to be restored.
5) Enphase has commanded the old Envoy to move to to newer communication protocols and/or waveforms on Saturday. I know that they recently updated the old Envoy's firmware. They told me that this occurred on November 4 of this year. The representative also claimed that no changes to the communications protocols have changed in three years. Still, I wonder if the Envoy may have tried to switch to a new protocol and/or waveforma on Sunday that it has never used before due to some combination of the new firmware, degraded signals and/or cold weather. Perhaps the 21 inverters with pre-2015 firmware are not able to communicate properly using those new protocols/waveforms. (Note that NONE of the inverters with the old firmware were in the system last time it was cold here.)
6) My power company has started communicating over their power lines with new waveforms and/or at higher power levels (one of Ken's proposals). This is a possibility, but if it were the case, I do not think my new Envoy would have been able to communicate with five bars if that were the issue.
7) Something in my house has malfunctioned and is now providing conducted EMI onto the power lines which is disrupting communications between the Envoy and the microinverters. Like the above, this is a possibility, but if it were the case, I do not think my communications issues would have persisted when the ENTIRE house was shut down. Neither Envoy was able to communicate with ALL of the inverters even when the house was completely quiet.
8) The old Envoy is malfunctioning AND the new Envoy is also malfunctioning (in other words, the new one never did not work right since I received it). This is certainly a possibility, but I am in denial about this being the case since it would be a major PITA to debug and resolve. I bought the new Envoy as "new" and all indications were that it really was in that condition. For instance there was still tape covering the ethernet port when I received the unit.

My favorite hypothesis at this point is #5. Even though the representative from Enphase tried to rule out this option, the fact that either an employee or some software from Enphase has triggered the inverter upgrades makes me think it might be a real problem. It also seems to be the only idea (except #8) which explains why the new Envoy cannot communicate even with five bars of signal quality indicated. But the following problem remains: If #5 IS the correct explanation, how do I get those inverters updated. drees' idea of moving the Enovy closer to the subpanel MIGHT help, but that would require quite a bit of effort and it is not a given that it can overcome such an issue.

For now, I will try to get the inverters to upgrade by switching most of them off while a small set updates. It is the easiest thing I can think to try. I should know if this approach is workable by about noon today. If that fails, I may switch the old Envoy back into place and see if it can successfully update the inverters. (At this point, I do not know if the old Envoy will even TRY to update the inverters since it never tried before.)

Thanks again to all for the assistance! Since I have no clear resolution at hand, I'm still open to suggestions.
RegGuheert
2011 Leaf SL Demo vehicle
10K mi. on 041413; 20K mi. (55.7Ah) on 080714; 30K mi. (52.0Ah) on 123015; 40K mi. (49.8Ah) on 020817; 50K mi. (47.2Ah) on 120717; 60K mi. (43.66Ah) on 091918.
Enphase Inverter Measured MTBF: M190, M215, M250, S280

User avatar
RegGuheert
Posts: 6287
Joined: Mon Mar 19, 2012 4:12 am
Delivery Date: 16 Mar 2012
Leaf Number: 5926
Location: Northern VA

Re: Enphase field MTBF: M190: ~44 Years M215: ~470 Years M250: >275 Years

Wed Nov 23, 2016 8:58 am

Perhaps a clue today: When the Sun came up, the new Envoy quickly recognized all of the new M250s on the one string I had switched on. So I switched on all of the strings and all 54 inverters were quickly found. Communication with the inverters was quite good around 8:00 AM with 52 of the 54 inverters communicating at that time. But as the Sun has gotten higher in the sky, the number of inverters which have stayed in contact with the Envoy has dropped to only 14 now at 11:00 AM. As such, I am putting forth a ninth hypothesis:

9) Resistance has built up in the wiring somewhere and that is causing noise on the powerlines which is disrupting PLC communications. To test this, I will turn of the Field array followed by the roof array to see if one or the other is the culprit. If it is the roof array, it may take quite a bit more work to determine what is causing the issue.
RegGuheert
2011 Leaf SL Demo vehicle
10K mi. on 041413; 20K mi. (55.7Ah) on 080714; 30K mi. (52.0Ah) on 123015; 40K mi. (49.8Ah) on 020817; 50K mi. (47.2Ah) on 120717; 60K mi. (43.66Ah) on 091918.
Enphase Inverter Measured MTBF: M190, M215, M250, S280

pclifton
Posts: 60
Joined: Thu Jan 26, 2012 4:51 pm
Delivery Date: 28 Jan 2012
Leaf Number: 17731
Location: Salisbury, NC
Contact: Website

Re: Enphase field MTBF: M190: ~44 Years M215: ~470 Years M250: >275 Years

Wed Nov 23, 2016 11:38 am

RegGuheert wrote:Perhaps a clue today: <snip/>
9) Resistance has built up in the wiring somewhere and that is causing noise on the powerlines which is disrupting PLC communications. To test this, I will turn of the Field array followed by the roof array to see if one or the other is the culprit. If it is the roof array, it may take quite a bit more work to determine what is causing the issue.


Question: How many inverters do you have per string?

When my first array was installed we had all 12 M190 inverters on one string. I evaluated it over a month, then due to voltage drop I rewired it to be two strings of six inverters (center fed). For all other poles I went with the center-fed arrangement from the start -- including the M250s, but I am literally at the end of the power line, and the poles are over 100 feet from the main panel.

Just a thought, but you may approach this by looking at how it is all connected. Of course the wiring is much better with the newer M215s and M250s -- much less chance of voltage drop / resistance I would guess.

Ken Clifton
2 PHEVs Powered by 10.2 kW Solar
Topaz Blue 2013 Chevy Volt and Silver 2014 Chevy Volt
Traded My Ocean Blue 2012 LEAF SL on 2-2013
Wife Traded Her Cayenne Red 2012 LEAF SL on 11-2013
Web site: http://www.kenclifton.com

User avatar
RegGuheert
Posts: 6287
Joined: Mon Mar 19, 2012 4:12 am
Delivery Date: 16 Mar 2012
Leaf Number: 5926
Location: Northern VA

Re: Enphase field MTBF: M190: ~44 Years M215: ~470 Years M250: >275 Years

Sat Nov 26, 2016 1:19 am

pclifton wrote:Question: How many inverters do you have per string?

When my first array was installed we had all 12 M190 inverters on one string. I evaluated it over a month, then due to voltage drop I rewired it to be two strings of six inverters (center fed). For all other poles I went with the center-fed arrangement from the start -- including the M250s, but I am literally at the end of the power line, and the poles are over 100 feet from the main panel.

Just a thought, but you may approach this by looking at how it is all connected. Of course the wiring is much better with the newer M215s and M250s -- much less chance of voltage drop / resistance I would guess.
My original design for the M190s had them set up as six strings of seven inverters on four breakers. Two pairs of strings were center-fed to minimize the voltage rise.

That went out the window when I switched to the Engage wiring on the roof. The reason is that I do not have (nor want) any junction boxes on my roof and I cannot feed two Engage cables through the 1" conduit which penetrates my roof. So now I have one string of seven and one string of 14 on the garage roof. That's not a real issue since the voltage drop for 14 M215s is only slightly worse than 7 M190s: both are limited to about 1 V max. I plan for a string of 12 and a string of 9 on the house roof, but right now I have an Engage string of 15 and an M190 sting of six still on the roof. I have what I need to make that change, but the difficulty of changing that all over to Engage is holding me back. Perhaps some day, but not now.

I do have a single string of 12 end-fed M190s in the field. I did that because when I set that up I only had one M190 pigtail. But now I have three more sitting in a box. I may switch that to two strings of six when I have a chance.

In any case, I do not think the wiring on the roof is involved in my current issues. The inverters have been connected the way they are all along and it has not impacted communications.

Problem update: No change.. Communications is still massively compromised. We had a couple of cloudy days and I was hopeful that some inverters would receive their updates, but it did not happen, even though I saw as many as 52 inverters communicating at one point.

But I did just think of something else to try: My SingBox Pro HD has not been working these past few days and I just put two-and-two together enough to realize that the power supply for that unit MIGHT just be the culprit. Those SlingBox power supplies only last about two years before they need replacement. I'm on the FOURTH one so far. However, those failures never caused a problem with the Enphase communications before. But this failure is different. Normally, the LED on the SlingBox goes off and the power supply gets cool. But this time the LED is still lit, the power supply is warm but the SingBox cannot communicate over the network. In any case, I have now unplugged that power supply. If that was the cause of the problems, I will know pretty soon after the Sun comes up this morning.

If this resolves the issue, I intend to leave the new Envoy in place until all of the microinverters updates are completed and then I will switch back to the old Envoy, putting the new one back into cold storage. I will say that it's nice to know I can switch them at will and either one can feed the information from the inverters back to Enphase without any changes to Enlighten. (At the time I called to get the new Envoy registered with my system, the support representative at Enphase told me I would have to involve them to switch over the inverters.)
RegGuheert
2011 Leaf SL Demo vehicle
10K mi. on 041413; 20K mi. (55.7Ah) on 080714; 30K mi. (52.0Ah) on 123015; 40K mi. (49.8Ah) on 020817; 50K mi. (47.2Ah) on 120717; 60K mi. (43.66Ah) on 091918.
Enphase Inverter Measured MTBF: M190, M215, M250, S280

User avatar
RegGuheert
Posts: 6287
Joined: Mon Mar 19, 2012 4:12 am
Delivery Date: 16 Mar 2012
Leaf Number: 5926
Location: Northern VA

Re: Enphase field MTBF: M190: ~44 Years M215: ~470 Years M250: >275 Years

Sat Nov 26, 2016 9:27 am

RegGuheert wrote:But I did just think of something else to try: My SingBox Pro HD has not been working these past few days and I just put two-and-two together enough to realize that the power supply for that unit MIGHT just be the culprit. Those SlingBox power supplies only last about two years before they need replacement. I'm on the FOURTH one so far. However, those failures never caused a problem with the Enphase communications before. But this failure is different. Normally, the LED on the SlingBox goes off and the power supply gets cool. But this time the LED is still lit, the power supply is warm but the SingBox cannot communicate over the network. In any case, I have now unplugged that power supply. If that was the cause of the problems, I will know pretty soon after the Sun comes up this morning.
That was it! Proper communications with the microinverters has been restored by simply unplugging the power supply for the SlingBox Pro HD. As of right now, all 12 of the M215IGs which had out-of-date firmware have been successfully updated and 8 of the 9 (used) M250s have also been successfully updated. I'm confident the last M250 will be updated today or tomorrow. Once that is complete, I will switch back to the old Envoy overnight so that no data is lost.

Interestingly, the improvement in signal quality was not detected by the Envoy: It showed TWO bars of signal quality both before and after I unplugged the offending power supply. That partially explains why I did not pick up on this when I turned off breakers one at a time on Monday.

As it turns out, the updates were to the same version of firmware that most of the other fourth-generation inverters currently have installed. The commands used looked like this:

For the M215IGs, first:
"Starting download of parm0, part number 540-00067-r01-v01.20.01
Completed download of parm0, part number 540-00067-r01-v01.20.01"
For the M250s, first:
"Completed download of parm1, part number 540-00060-r01-v01.20.01
Completed download of parm1, part number 540-00060-r01-v01.20.01"
For both types, then:
"Starting download of procload, part number 520-00044-r01-v01.20.00
Completed download of procload, part number 520-00044-r01-v01.20.00"

Assuming the last M250 successfully updates to this version, the system now contains the following inverters and Envoys Along with the version of firmware they have installed:

12: M190s (Operating): 520-00008-r01-v01.08.01
23: M190s (Cold spares): 520-00008-r01-v01.08.01
1: M190IG (Operating, Plastic case, 48Vmax): 520-00044-r01-v01.20.00
2: M190IG (Cold spares, Metal case, 60Vmax): Firmware unknown since these have never been installed
24: M215IGs (Operating, Plastic case): 520-00044-r01-v01.20.00
4: M215IGs (Operating, Metal case): 520-00044-r01-v01.22.00
13: M250s (Operating, plastic case): 520-00044-r01-v01.20.00
1: Envoy (Operating, Single USB): R3.7.28 (88072d)
1: Envoy-R-NA (Cold spare, Dual USB): R3.12.45 (f9b2fc)

Totals:
Microinverters:
Operating:
- 42-Fourth-generation (1-M190IG-48V, 28-M215IGs and 13-M250s)
- 12-Third-generation (12-M190s)
Cold Spares:
- 2-Fourth-generation (2-M190IG-60V)
- 23-Third-generation (23-M190s)
Envoys:
Operating:
- 1-Envoy (Single USB)
Cold spare:
- 1-Envoy-R-NA (Dual USB)

Thanks again for your help debugging this issue! Kudos to Tim at Enphase for calling this issue correctly within about one minute of the first and only support call.
RegGuheert
2011 Leaf SL Demo vehicle
10K mi. on 041413; 20K mi. (55.7Ah) on 080714; 30K mi. (52.0Ah) on 123015; 40K mi. (49.8Ah) on 020817; 50K mi. (47.2Ah) on 120717; 60K mi. (43.66Ah) on 091918.
Enphase Inverter Measured MTBF: M190, M215, M250, S280

User avatar
RegGuheert
Posts: 6287
Joined: Mon Mar 19, 2012 4:12 am
Delivery Date: 16 Mar 2012
Leaf Number: 5926
Location: Northern VA

Re: Enphase field MTBF: M190: ~44 Years M215: ~470 Years M250: >275 Years

Tue Nov 29, 2016 10:49 am

I can't win for losing! :(

I experienced a couple of incidents in he past month or so of arc faults occurring between the power cord and the power supply of a couple of our laptops. It was pretty scary and I felt fortunate that I was there to unplug them before they started a fire. This prompted me to install an AFCI outlet on this branch circuit to prevent a fire in case this happens again.

I purchased the unit last week and installed it this morning. The INSTANT I pressed the reset button on that unit to activate the circuit, communications with quite a few of my microinverters was lost. Grrr!

I suppose at this point I will purchase filters to put on any circuit where I intend to install AFCIs...
RegGuheert
2011 Leaf SL Demo vehicle
10K mi. on 041413; 20K mi. (55.7Ah) on 080714; 30K mi. (52.0Ah) on 123015; 40K mi. (49.8Ah) on 020817; 50K mi. (47.2Ah) on 120717; 60K mi. (43.66Ah) on 091918.
Enphase Inverter Measured MTBF: M190, M215, M250, S280

Return to “Solar”