E-MotorWerks JuiceBox - an open source 15kW EVSE

[valerun]

Nearly a month later, it's happening again. 2 days in a row again. I guess it's reboot time.
Valery, is this one of the "minor" bugfixes that you are reporting fixed below? I'll flash the firmware to be the newest, also so it gets power cycled.
But since it takes a month to reproduce the error, I won't know if it's fixed for a month when I go to use my car and realize it isn't charged.

Valery, let me know if you are already aware of this and/or what debugging steps you recommend.
I haven't rebooted it yet, so if there's a way to hook up debug output without resetting it, I can still do that.

Thanks Jay - since it seems to be related to rain, I might suspect GFI trip due to leakage. If water gets into any part of the energized system and creates a leakage to ground, this would be expected. Note that it does not have to be at the plug - could be somewhere else in the car, as well (although unlikely).

If it is indeed a GFI-related trip, there is not much we can do besides just disabling it altogether (not a great idea) or reducing the sensitivity of it (right now it's set to trip at ~10ma RMS). To reduce the sensitivity, you would need to make hardware change (add a parallel resistor to R5). You would also need to change sensitivity constant in firmware 'readV' function as the same sensor network is used to sense the AC voltage level.

Valery.

 

[Levi8than]

Apologies for being unclear. I don't think it's related to rain. I suspected rain the first time around because, well, it was raining the first time, in February.
But now here (San Jose, CA) we haven't had rain for a while, but we're expected to get some soon. So I don't think rain is at fault this time. I can leave it and not reset it again and see if it happens again tonight. I made it from 40% to 75% last night, but not all of the way to 80% where it's set. Yesterday it was 77%. Both time with the yellow fault light on the dash.

When it does fail, it seems to fail somewhere in 70's. I haven't had a case yet where I woke up to find no charge at all.

It's not rain. My current guess is that something goes wrong in the long running code, and it isn't recovering... is there a watchdog timer anywhere that could be triggering? Something that could toggle the relay mid-operation?
I'll leave the box (not reset it) to see if it keeps happening over the next few days, since I've got 2 days in a row again, just like last time. This will confirm my theory.

Nearly a month later, it's happening again. 2 days in a row again. I guess it's reboot time.
Valery, is this one of the "minor" bugfixes that you are reporting fixed below? I'll flash the firmware to be the newest, also so it gets power cycled.
But since it takes a month to reproduce the error, I won't know if it's fixed for a month when I go to use my car and realize it isn't charged.

Valery, let me know if you are already aware of this and/or what debugging steps you recommend.
I haven't rebooted it yet, so if there's a way to hook up debug output without resetting it, I can still do that.

Thanks Jay - since it seems to be related to rain, I might suspect GFI trip due to leakage. If water gets into any part of the energized system and creates a leakage to ground, this would be expected. Note that it does not have to be at the plug - could be somewhere else in the car, as well (although unlikely).

If it is indeed a GFI-related trip, there is not much we can do besides just disabling it altogether (not a great idea) or reducing the sensitivity of it (right now it's set to trip at ~10ma RMS). To reduce the sensitivity, you would need to make hardware change (add a parallel resistor to R5). You would also need to change sensitivity constant in firmware 'readV' function as the same sensor network is used to sense the AC voltage level.

Valery.

 

[davewill]

... I can leave it and not reset it again and see if it happens again tonight. I made it from 40% to 75% last night, but not all of the way to 80% where it's set. Yesterday it was 77%. Both time with the yellow fault light on the dash.

When it does fail, it seems to fail somewhere in 70's. I haven't had a case yet where I woke up to find no charge at all. ...

This sounds suspiciously like the problems a lot of people were having with the 2013s last year. A number of people got their OBCs replaced, but a subsequent software release was supposed to have fixed the issue. Is there any way you can test with a completely difference EVSE?

 

[Levi8than]

... I can leave it and not reset it again and see if it happens again tonight. I made it from 40% to 75% last night, but not all of the way to 80% where it's set. Yesterday it was 77%. Both time with the yellow fault light on the dash.

When it does fail, it seems to fail somewhere in 70's. I haven't had a case yet where I woke up to find no charge at all. ...

This sounds suspiciously like the problems a lot of people were having with the 2013s last year. A number of people got their OBCs replaced, but a subsequent software release was supposed to have fixed the issue. Is there any way you can test with a completely difference EVSE?

I have a 2013. Do you have a thread? But if this were the case... why would resetting the juicebox put the issue to bed for another month?
I have access to pay charge stations. But I've never done a full charge on one. Usually just enough to get me to where I need to go.

Let me tinker.

 

[locman]

davewill might be correct. It sounds like the PDM problem.
http://www.mynissanleaf.com/viewtopic.php?f=30&t=13194" onclick="window.open(this.href);return false;
There's a TSB on this. Bring it to the dealer and get the update before spending more time debugging.

I have a 2013. Do you have a thread? But if this were the case... why would resetting the juicebox put the issue to bed for another month?
I have access to pay charge stations. But I've never done a full charge on one. Usually just enough to get me to where I need to go.

Let me tinker.

 

[davewill]

I have a 2013. Do you have a thread? But if this were the case... why would resetting the juicebox put the issue to bed for another month? ...

Coincidence? When the problem first manifests, it's VERY intermittent, but gets progressively worse if not fixed.

Locman linked the thread.

 

[GlennD]

The relay used is nice but the ratings are either fantasy or Chinese amps are much smaller! Realistically half of 80A is doable without over heating. There should be no problem charging a RAV4 EV at 40A.

80A is a fire waiting to happen. The price is right on this relay but take its ratings with a boulder of salt! No one really needs 80A. The usual J1772 plugs are rated at 50A max and that is fine for most EV's.

 

[chris1howell]

The relay used is nice but the ratings are either fantasy or Chinese amps are much smaller! Realistically half of 80A is doable without over heating. There should be no problem charging a RAV4 EV at 40A.

80A is a fire waiting to happen. The price is right on this relay but take its ratings with a boulder of salt! No one really needs 80A. The usual J1772 plugs are rated at 50A max and that is fine for most EV's.

-Warning Sarcasm to follow-

Chinese Amps...???? They can't be Chinese Amps because Juicebox is "Genuinely Made in the USA product" and "All Juicebox Products are Proudly Made in the USA" I read it here on the internet...so it must be true.
http://emotorwerks.com/products/onl...-open-source-level-2-15kw-ev-charging-station

That can't be a
Chinese Relay
or J1772 cable
or power supply
or remote
or micro processor module
or premium enclosure
or RTC module
or display (actually not made in china but not made in the USA either)

whats left...? the Printed Circuit board is made in the USA according to the BOM?

Juicebox does not meet the definition of a US made product which must be "all or virtually all" made in the U.S. http://business.ftc.gov/documents/bus03-complying-made-usa-standard

 

[valerun]

The relay used is nice but the ratings are either fantasy or Chinese amps are much smaller! Realistically half of 80A is doable without over heating. There should be no problem charging a RAV4 EV at 40A.

80A is a fire waiting to happen. The price is right on this relay but take its ratings with a boulder of salt! No one really needs 80A. The usual J1772 plugs are rated at 50A max and that is fine for most EV's.

-Warning Sarcasm to follow-

Chinese Amps...???? They can't be Chinese Amps because Juicebox is "Genuinely Made in the USA product" and "All Juicebox Products are Proudly Made in the USA" I read it here on the internet...so it must be true.
http://emotorwerks.com/products/onl...-open-source-level-2-15kw-ev-charging-station

That can't be a
Chinese Relay
or J1772 cable
or power supply
or remote
or micro processor module
or premium enclosure
or RTC module
or display (actually not made in china but not made in the USA either)

whats left...? the Printed Circuit board is made in the USA according to the BOM?

Juicebox does not meet the definition of a US made product which must be "all or virtually all" made in the U.S. http://business.ftc.gov/documents/bus03-complying-made-usa-standard

how is this relevant to the discussion here, Chris? ;-)

 

[valerun]

The relay used is nice but the ratings are either fantasy or Chinese amps are much smaller! Realistically half of 80A is doable without over heating. There should be no problem charging a RAV4 EV at 40A.

80A is a fire waiting to happen. The price is right on this relay but take its ratings with a boulder of salt! No one really needs 80A. The usual J1772 plugs are rated at 50A max and that is fine for most EV's.

Hi Glenn - agree that one doesn't want to take the parts (any parts for that matter - Chinese or not) to its rated limits to allow for margin in operating conditions (e.g., Arizona sun shining on an alum box for 10 hours, etc.)

That said, we have tested these relays at 60A continuous for hours and recorded ~30-40C temp rise. If the ambient is below 50C, this is acceptable. Under AZ sun - maybe not. YMMV, as usual.

 

[valerun]

Apologies for being unclear. I don't think it's related to rain. I suspected rain the first time around because, well, it was raining the first time, in February.
But now here (San Jose, CA) we haven't had rain for a while, but we're expected to get some soon. So I don't think rain is at fault this time. I can leave it and not reset it again and see if it happens again tonight. I made it from 40% to 75% last night, but not all of the way to 80% where it's set. Yesterday it was 77%. Both time with the yellow fault light on the dash.

When it does fail, it seems to fail somewhere in 70's. I haven't had a case yet where I woke up to find no charge at all.

It's not rain. My current guess is that something goes wrong in the long running code, and it isn't recovering... is there a watchdog timer anywhere that could be triggering? Something that could toggle the relay mid-operation?
I'll leave the box (not reset it) to see if it keeps happening over the next few days, since I've got 2 days in a row again, just like last time. This will confirm my theory.

Thanks Joe. Got it. Any other correlation with environment variables you could recall?

Worst case, you can always bring your unit into our warehouse and we will swap for another one.

Valery

 

[GlennD]

The relay used is nice but the ratings are either fantasy or Chinese amps are much smaller! Realistically half of 80A is doable without over heating. There should be no problem charging a RAV4 EV at 40A.

80A is a fire waiting to happen. The price is right on this relay but take its ratings with a boulder of salt! No one really needs 80A. The usual J1772 plugs are rated at 50A max and that is fine for most EV's.

Hi Glenn - agree that one doesn't want to take the parts (any parts for that matter - Chinese or not) to its rated limits to allow for margin in operating conditions (e.g., Arizona sun shining on an alum box for 10 hours, etc.)

That said, we have tested these relays at 60A continuous for hours and recorded ~30-40C temp rise. If the ambient is below 50C, this is acceptable. Under AZ sun - maybe not. YMMV, as usual.

I guess we will have to differ about the relay overheating. Most people consider 85 degrees the absolute maximum temperature and try to keep temps well below that.

That said, I believe in reserves and it is good that you agree. In my own EVSE I have a 50A contactor for 30A.

 

[MarkInCNC]

Here is a case where a laser temp probe, such as this one from Harbor Freight, is easily worth it's cost in component savings, or maybe even fire avoidance!

I have found several battery hotspots in NiMH or Li-poly packs that were overheating and repaired them before any further damage was done. The JuiceBox build directions actually recommend having such a thermometer on hand to help with diagnosing any unusual hot spots when switching full power 240VAC.

 

[Levi8than]

Thanks Joe. Got it. Any other correlation with environment variables you could recall?
Worst case, you can always bring your unit into our warehouse and we will swap for another one.
Valery

No correlation. Did it again today, 3rd day in a row. I plugged it in around 50% last night, and she only charged to 65 with the same error light:

I just unplugged and plugged in again to reset the juicebox. Tonight, I'll plug in somewhere around 30-40% charge, and I'll have more information.
If the charge succeeds, then I'm putting my money on the Juicebox being at fault and would expect it to fail again in another month or so.
If the charge fails, then I'll print up the TSB suggested by davewill/locman and head off to my local dealership to see if they can repro it on their charger.

Valery: Thank you very much for being so quick to offer a part swap. If it turns out to be the Juicebox, I'll definitely take you up on that offer.

 

[Levi8than]

Summarizing. After a month of perfect performance, I saw 2 failures mid-charge w days in a row. Then I reset the JB by unplugging and plugging it back in.
About a month later, same failures mid charge 3 days in a row. Then I reset the JB by unplugging and plugging it back in.
Now I'm on day 4 of perfect charges again. The previous 3 days were to 80% with no issues, but this morning's charge was from 23% to 100% with no issues.

I think I've got enough data here to rule out coincidence, and also to rule out the TSB that's been suggested to me.
I also don't think there's an hardware issue with the JuiceBox itself. Because it works for a month, then begins to fail every night after that.
Rather, I want to walk through the code and simplify it, looking for a memory leak or unbounded pointer increment somewhere. How about with data storage on the flash? Any chance we're walking off the end of the earth?

Besides, I don't need any of that. I'm running the base model with no lcd and no wifi connectivity. So all I need is basic behavior. Everything else can be commented out. I think I'll try this soon and see if I come up with something.

Thanks Joe. Got it. Any other correlation with environment variables you could recall?
Worst case, you can always bring your unit into our warehouse and we will swap for another one.
Valery

No correlation. Did it again today, 3rd day in a row. I plugged it in around 50% last night, and she only charged to 65 with the same error light:

I just unplugged and plugged in again to reset the juicebox. Tonight, I'll plug in somewhere around 30-40% charge, and I'll have more information.
If the charge succeeds, then I'm putting my money on the Juicebox being at fault and would expect it to fail again in another month or so.
If the charge fails, then I'll print up the TSB suggested by davewill/locman and head off to my local dealership to see if they can repro it on their charger.

Valery: Thank you very much for being so quick to offer a part swap. If it turns out to be the Juicebox, I'll definitely take you up on that offer.

 

[valerun]

Summarizing. After a month of perfect performance, I saw 2 failures mid-charge w days in a row. Then I reset the JB by unplugging and plugging it back in.
About a month later, same failures mid charge 3 days in a row. Then I reset the JB by unplugging and plugging it back in.
Now I'm on day 4 of perfect charges again. The previous 3 days were to 80% with no issues, but this morning's charge was from 23% to 100% with no issues.

I think I've got enough data here to rule out coincidence, and also to rule out the TSB that's been suggested to me.
I also don't think there's an hardware issue with the JuiceBox itself. Because it works for a month, then begins to fail every night after that.
Rather, I want to walk through the code and simplify it, looking for a memory leak or unbounded pointer increment somewhere. How about with data storage on the flash? Any chance we're walking off the end of the earth?

Besides, I don't need any of that. I'm running the base model with no lcd and no wifi connectivity. So all I need is basic behavior. Everything else can be commented out. I think I'll try this soon and see if I come up with something.

Thanks Joe. Got it. Any other correlation with environment variables you could recall?
Worst case, you can always bring your unit into our warehouse and we will swap for another one.
Valery

No correlation. Did it again today, 3rd day in a row. I plugged it in around 50% last night, and she only charged to 65 with the same error light:

I just unplugged and plugged in again to reset the juicebox. Tonight, I'll plug in somewhere around 30-40% charge, and I'll have more information.
If the charge succeeds, then I'm putting my money on the Juicebox being at fault and would expect it to fail again in another month or so.
If the charge fails, then I'll print up the TSB suggested by davewill/locman and head off to my local dealership to see if they can repro it on their charger.

Valery: Thank you very much for being so quick to offer a part swap. If it turns out to be the Juicebox, I'll definitely take you up on that offer.

Thanks Joe - this is very helpful to know. Have you been able to catch it at the moment of failure? What happens then?

In the latest version of firmware (8.7.4 - to be posted tomorrow), we are implementing a watchdog timer (yes, we should have done it long time ago LOL) which will auto-reset JuiceBox if the MCU gets frozen. We did get 3-4 reports of JuiceBoxes freezing up and requiring unplug-replug reboot so this should help with that.

 

[valerun]

on an earlier post re 'Made in the USA' claim. We have done some calculations and can tell you guys that out of a $249 selling price of the JuiceBox Base Assembled unit, only ~$30 goes outside of the US. Yes, that includes ALL the parts made outside of the US (down to a capacitor). Our PCBs are made in the US, our PCB assembly is in San Jose, our unit assembly is in Redwood City and San Jose, our enclosure manufacturer is in Illinois, our enclosure machinist is in San Jose, etc.

In summary, 88% of your money goes to support not just US businesses but mostly local to SF Bay Area. You tell me what we should claim on our website...

Cheers,
Valery

 

[valerun]

Joe - just noticed your last question on memory utilization.

We have been quite diligent about controlling memory utilization (both RAM and Flash). We routinely have to refactor code for new releases when we add new features so I feel like we are reasonably on top of RAM budgeting. Also, in 30+ days of operation, all the functions would have been called multiple times and variables initialized and released many many times. We do not have any dynamic pointers in our code that would go astray. We do not have any malloc-like calls for dynamic memory allocation, either.

There has been a suggestion that our use of millis() functions for some timed activities could cause unexpected behavior after ~50 days as the 4-byte long unsigned int counter overflows. We then went through the code and confirmed that at worst that would result in incorrect power metering if the overflow happens during charge. But the basic unit operation should not be disturbed. To be on the safe side, we now have an 8-second watchdog timer (in 8.7.4 to be released tomorrow) that will reset the unit if it ever hangs.

I will re-iterate that so far by far the most common cause of intermittent charge interruptions (reported in ~2% of our units - just for reference) was attributed to false GFI trips. Some of those could be real (i.e. wet plug situation I described earlier). Some of those could be related to wire alignment (we have some instructions around that in our manual - more detailed now, including how to optimize your layouts etc - you might want to take a look). Some of those could be related to moisture entry in units (lids / glands not fully tightened or mounting holes not fully waterproofed, etc.)

To address some of these GFI-related issues, we will plan to (1) move to machined laminated bus structure for power wires - so there is no variation from unit to unit on wire layout, and (2) add a power resistor to control temperature & humidity inside the unit (managed by the main MCU). Any additional ideas welcome.

Valery

 

[Levi8than]

Thanks Joe - this is very helpful to know. Have you been able to catch it at the moment of failure? What happens then?

In the latest version of firmware (8.7.4 - to be posted tomorrow), we are implementing a watchdog timer (yes, we should have done it long time ago LOL) which will auto-reset JuiceBox if the MCU gets frozen. We did get 3-4 reports of JuiceBoxes freezing up and requiring unplug-replug reboot so this should help with that.

No, I've never caught it at the moment of failure. It's always been in the middle of the night, after a few hours of charging, but not at the end.
I'm not sure what happens.
If the pilot line were to stop, the car would just assume the cable were unplugged and wouldn't report an error.
My guess is that the pilot line is still reporting current, but the relay opens up. So the car reports an error because it's unable to charge.

Again this is just my guess. I haven't gotten a repro. And as it sits now I've got about a month to wait until I'll see one again.

 

[772]

In the latest version of firmware (8.7.4 - to be posted tomorrow), we are implementing a watchdog timer (yes, we should have done it long time ago LOL) which will auto-reset JuiceBox if the MCU gets frozen. We did get 3-4 reports of JuiceBoxes freezing up and requiring unplug-replug reboot so this should help with that.

This would we great... we've run into this issue about once a month so far. Not a huge problem, but it does occasionally mean the car doesn't charge up overnight. Looking forward to the firmware update, but I assume I would need some extra hardware to install it? I only have the base JB.