SOC-Meter CAN-Message and CAN-Do Log file formats

garygid · Oct 6, 2011

Logging with the SOC-Meter:

1. Typically is done via a good RS232-to-USB adapter
2. often connected to a laptop (or similar computer)
3. using a Win Operating System (for example XP, Vista, Win7)
4. running our free CAN-Do (or some compatible) program
5. which reads binary "Messages" sent by the SOC-Meter firmware
6. that runs on our SOC-Meter's AVR-CAN board.
7. The CAN-Do program adds time stamps,
8. and accumulates the Messages into memory.
9. There, the captured CAN-data can be:
a) examined and graphed
b) written to a CAN-Log Binary-Output file.
10. Later, these Log files can be
a) shared with others, or
b) read back into CAN-Do for further investigation.

Although the CAN-data is usually written to a Binary CAN-Log file
so that the data can be accessed later for further analysis,
it might just be used to display a real-time "Dashboard".

(For now, the following is from memory, so corrections might be needed.)

Format of CAN-Do's One-CAN Binary-Output Log file:

1. The one-CAN files contain data from just ONE of the CAN buses
2. are given the following extensions:
a) EV-CAN messages: ".evc"
b) AV-CAN messages: ".avc"
c) CAR-CAN messages: ".can"
d) QC-CAN messages: ".qcc"

These all have 12-byte binary records with the following format:

1. Two bytes of Time-Stamp (SS mm), seconds and milliseconds:
a. The high 6 bits of the first byte (SS) are the seconds, 0 to 59.
b. The low order 2 bits of the first byte (SS) , followed by the whole
second byte (mm), form a 10-bit value, a 0 to 999 count of milliseconds.

Note: In a few older Log files, these two Time-Stamp bytes might be zero.

2. Two bytes of Message ID (HMM) and Data Count (N): (as MM, then NH)
a. The high nibble (4 bits) of the second byte is the Data Count (N = 0 to 8).
Note: Other values of N (like 0xF) may occur for Special Messages.
b. The Low nibble of the second byte (H), followed by the whole first byte (MM)
form the 3-nibble Message ID, 0xHMM, usually between 0x100 (or 0x000) and 0x7FF.

Note: Our special pseudo-messages typically have IDs of 0xF00 to 0xFFF.
In new formats, Messages received with errors might have 0x800 added to
the Message 0xHMM value, to better separate the errors from the good data.

3. Eight bytes of Data:
There are always 8 bytes in the Log file, to make fixed-length "records".
If the data count (N above) is less than 8, the actual data bytes
have 0xFF bytes appended to form a full 8 bytes for the Log record.

CAN-Do refers to these bytes as D1 through D8, but others use the
"computer-talk" designations of D0 through D7. Yes, confusing.

This forcing of the unused bytes to 0xFF should be done where the
messages are collected, so that non-FF values can be used
by CAN-Do to help detect communication and buffering errors.

Format of CAN-Do's Multiple-CAN Binary-Output Log file:

The multiple-CAN Log files:
a) have the ".alc" (all-CAN) extension
b) have 13-byte binary records, a one byte prefix on the
standard one-CAN format described above.

1. One byte prefix for the CAN-Bus (Source) type:
(as I recall ... need to check)
The types are 0 to 4, with:
a) 1 = EV,
b) 2 = CAR,
c) 3 = AV,
d) 4 = QC,
e) 0 = All (messages or time stamps that apply to all types).
f) other values: Reserved
Note: The values 5, 6, 7, and 8 are now used, recalling roughly
as SC and XX for the Tesla S, and GPS (I need to check).

2. The 12 bytes of the One-Can format (see above).

SOC-Meter Message Format (from the AVR-CAN firmware):

Data from the SOC-Meter's Logging Port is an RS232
data stream (8, N, 1) at 115,200 baud, with no flow control.
NOTE: A GOOD RS232-to-USB adapter is needed to handle this data.
An adapter with a "real" FTDI chip is usually a candidate.
(See the "Good RS232-to-USB ..." thread in this sub-forum.)

These are usually 11-byte binary "records" with no specific delimiter, but
now a 13-byte version might be used if a Time-Stamp (SS MM) is included:

1. One Sync Byte to prefix the normal 12-byte message:
which is the EOR (Exclusive OR) of 0x53 with the two following bytes.
To decode, EOR the three sequential bytes, looking for 0x53 as the result.

2. Two bytes (the Message ID and Data Count (MM NH) as described above)

3. Eight bytes of Data (D1 to DN, with 0xFF padding, as described above)

Multi-CAN Message Format (serial incoming to CAN-Do):

Data from some multi-CAN Logging source is a stream of binary
bytes, usually a data stream through a Virtual Comm Port.

These are 14-byte binary "records" with no specific delimiter:

1. One Sync Byte (as described above):
T his is the EOR (Exclusive OR) of 0x53 with the two following bytes.
To decode, EOR three sequential bytes, looking for 0x53.

2. Two bytes (the Message ID and Data Count as described above)

3. Eight bytes of Data (0xFF padded, as described above)

4. Two Time-Stamp bytes, SS MM as described above.

5. One Source-byte, as described above.

Special Messages:

1. Date and Time Messages:
a) These have the 0xFFF message ID, with Data Length = 0xF,
so the "Two bytes of Message ID and Data Count" = 0xFF 0xFF
b) The 8 data bytes include the date and time.
Probably as an 8-byte seconds (and fraction?) past midnight the morning of Jan 1, 1970 ...
(but, at the moment, I do not remember the exact format.)

2. Event Message:
a) These have the 0xFFE message ID, with Data Length = 0xF,
so the "Two bytes of Message ID and Data Count" = 0xFE 0xFF
b) The 8 data bytes are an 8-character ASCII message,
that might have an "Continuation Message" preceeding it.

3. Continuation Message:
a) These have the 0xFFC message ID, with Data Length = 0xF,
so the "Two bytes of Message ID and Data Count" = 0xFC 0xFF
b) The 8 data bytes are an 8-character ASCII message,
that might have another "Continuation Message" preceeding it.

Note: There is a maximum of 2 continuation messages, sent
BEFORE the Event Message.

4. Special-Data Messages:
These usuallu have the 0xFFn message ID, with Data Length = 0xF,
so the "Two bytes of Message ID and Data Count" = 0xFn 0xFF
(where n = 0 through 9, with A, B, and D Reserved)
b) The 8 data bytes will be described later.
Although usually ASCII characters, there MIGHT be binary data.

TickTock · Oct 15, 2011

Thanks for sharing this Gary. I finally found some time to take a look. I was interested in plotting battery voltage versus gids to confirm the expected discharge curve. Anyway it does look just as expected. I am going to paste the perl code below if anyone wants to use it as a boiler plate for their own studies.

Code:

#!/usr/bin/perl -w
# ----------------------------------------  
# File       : battvsgids.pl  
# Author     : TickTock
# Date       : October 15, 2011  
# ----------------------------------------  
my $EVCfile = @_;
my ($fileName,$i,$firstTime,$currentTime,$buf,$time1,$time2,$msgid1,$msgid2,@data,$gids,$battery);

if ($#ARGV < 0) {                               # print out usage line
        die "Usage: battvsgids evcfile\n";
}

$fileName = shift(@ARGV);

$i=0;
$firstTime=0;
$currentTime=0;
open(EVCfile, $fileName) ||
            die "Can't open file: $fileName $!\n";
	binmode EVCfile;
	while (read(EVCfile, $buf, 12)){
	($time1,$time2,$msgid1,$msgid2,@data) = unpack("CCCCCCCCCCCC", $buf);
		$time=$currentTime+(($time1>>2)*1000 + (($time1 & 3)*256+$time2))/1000;
		$msglen=$msgid2>>4;
		$msgid=($msgid2&0xf)*256+$msgid1;
            last if (length($buf) < 12);
			#last if ($i++ > 250);
			if ($msgid==0xfff) { #Timestamp
				$time=$time-$currentTime;
				$currentTime=($data[3]<<24)+($data[2]<<16)+($data[1]<<8)+$data[0]-$time-$firstTime;
				if($firstTime==0) {
					$firstTime=$currentTime;
					$currentTime=0;
					}
				}
			elsif($msgid==0x5bc) { #gids
				$gids=($data[0]<<2)+($data[1]>>6);
				printf ("%4.3f\t%d\t%4.1f\n",$time,$gids,$battery);
				#printf ("%4.3f %1x %03x %02x %02x %02x %02x %02x %02x %02x %02x\n",$time,$msglen,$msgid,@data);
				}
			elsif($msgid==0x1db) { #battery voltage
				$battery=(($data[2]<<2)+($data[3]>>6))/2;
				#printf ("%4.3f %d %4.1f\n",$time,$gids,$battery);
				}
	}
close(EVCfile);

garygid · Oct 15, 2011

Looks Good.

I am planning to add a function in CAN-Do to output a ".csv" file,
the user selecting multiple "recipe" items for the fields.

I also need to allow the user to select "trigger" field(s). When data
for this field (these fields) is extracted (as you did with $gids above),
the program will write a new line to the ".csv" file.

However, your example reminds me that I should also include
the choice of a Log-record number field, and a date-time field.

Also, should I include a choice of a "space" seperator,
to be used instead of (or in addition to) the "comma" seperator?

Thanks, and ...
Glad you got the Log-Reading working.
Cheers, Gary

TickTock · Oct 16, 2011

Here is a composite plot of battery voltage versus gids from two logs using the perl above. 1) discharge: a log I took as I drove from 6miles on the GOM to dead and 2) charge: the subsequent log of a charge from LBW to 100% (missing the log of the charge from dead to LBW which I performed using a generator away from home). I also backed out the IR drop across the series resistance to smooth the curve (plotted parameter is battery_volts - battery_amps*0.07). ESR appears to be about 70mOhms (determined empirically - anything larger or smaller resulted in more modulation with battery_amps).

garygid · Oct 16, 2011

Pack Charging "resistance" of 70 mOhms:

Charging at 100A with this 0.070 Ohm PACK resistance (only about
0.0007 Ohm per cell-pair) would be only a 7 volt rise in Battery Pack
voltage as the charging current is being "forced" into the Battery pack
(about 700 watts, a substantial heating rate, but perhaps only for
a half hour, so 700 x 0.5 = 350 watt-hours).

With our normal L2 charging (less than 10 amps going in),
that would be only approximately a 0.7 volt drop (only 7 watts,
for maybe 5 hours = 35 watt-hours).

Good estimates to start with, I suspect.
Great use of your EV-CAN bus Logs.
Good work, TickTock.

Even if this 70 milli-ohm Pack-Charging value turns out to be a bit low,
or represents the battery only when operating in a "best/better"
temperature (or age) range, it is indicative of a very good
battery characteristic.

Note: The discharge "resistance" MIGHT be different.

TickTock · Oct 16, 2011

I converged on the 70mOhms using the discharge log. Not enough variation in the charging current to distinguish IR drop from battery voltage. I think this ESR is part of Nissan's magic formula enabling passing battery cooling. Since discharge is endothermic, if they can keep the IR lossses in the battery lower then the heat absorbed from the chemical reaction, they don't have to worry about cooling during operation. Puts the cooling onus on the charging operation which makes a lot of sense (noone cares if you throttle charging to keep temp down, but we certainly would notice if it was done during driving). I think this is also why regen is limited to 40kW.

TickTock · Oct 16, 2011

OK. One more program. I thought a graphical view of the messages coming across might by revealing so I modified the code to generate a EKG-like display showing the activity of each type of message over time. Below is a snapshot of a short log I took with the car unplugged and canbus off, pressing the timeron/off button, waiting a few seconds, and pressing it again. X axis is time.

Here is the code:

Code:

#!/usr/bin/perl -w
# ----------------------------------------  
# File       : canekg.pl  
# Author     : TickTock
# Date       : October 16, 2011  
# ----------------------------------------  
my $EVCfile = @_;
my ($fileName,$i,$j,$pulse,$firstTime,$currentTime,$buf,$time1,$time2,$msgid1,$msgid2,$msgid,$smsgid,$msglen,@data,@signals,@names,$signals);

if ($#ARGV < 0) {                               # print out usage line
        die "Usage: canekg evcfile\n";
}

$fileName = shift(@ARGV);

$signals="";
open(OUTfile, ">$fileName.out");
print OUTfile "\n";
$j=1;
$firstTime=0;
$currentTime=0;
$time=0;
open(EVCfile, $fileName) ||
	die "Can't open file: $fileName $!\n";
binmode EVCfile;
while (read(EVCfile, $buf, 12)){
	($time1,$time2,$msgid1,$msgid2,@data) = unpack("CCCCCCCCCCCC", $buf);
	$time=$currentTime+(($time1>>2)*1000 + (($time1 & 3)*256+$time2))/1000;
	$msglen=$msgid2>>4;
	$msgid=($msgid2&0xf)*256+$msgid1;
	$smsgid=sprintf("%02x",$msgid);
	last if (length($buf) < 12);
	if ($msgid==0xfff) { #Timestamp
		$time=$time-$currentTime;
		$currentTime=($data[3]<<24)+($data[2]<<16)+($data[1]<<8)+$data[0]-$time-$firstTime;
		if($firstTime==0) {
			$firstTime=$currentTime;
			$currentTime=0;
			}
		}
	else {
		if ($signals!~$smsgid) {
			$names{$smsgid} = $j++;
			$signals=$signals."\t".$smsgid;
			}
		$pulse=((($data[0]+$data[1]+$data[2]+$data[3]+$data[4]+$data[5]+$data[6]+$data[7])&0xff)-0x80)/256;
		printf OUTfile ("%4.3f" . "%s" . "%3.2f\n",$time,"\t"x$names{$smsgid},$pulse+10*$names{$smsgid});
		}
	}
close(EVCfile);
print OUTfile "time".$signals;
close(OUTfile);

Since the output is tab delimited text, it gets rather large for large logs.

TickTock · Nov 28, 2011

Gary,
I don't think I have the time fields figured out quite right. I am pretty sure the top six bits of the first byte count off seconds, but I cannot figure out any algorithm that works for the rest of the bits. Can you offer any clues?

garygid · Nov 29, 2011

The remaining 2 (low) bits (nn) of the first byte and the 8 bits (bbbbbbbb) of the 2nd byte are a 10-bit (nnbbbbbbbb) value: the number of milliseconds (0 to 999).

TickTock · Nov 29, 2011

garygid said:
The remaining 2 (low) bits (nn) of the first byte and the 8 bits (bbbbbbbb) of the 2nd byte are a 10-bit (nnbbbbbbbb) value: the number of milliseconds (0 to 999).

This is what I thought (and is reflected in the code I published earlier in this thread), but it doesn't seem to be right. When I decode it as such and plot my results versus the resulting time, the messages appear "bursty" (coming in clumps) as seen in this thread:

http://www.mynissanleaf.com/viewtopic.php?f=31&t=6702&start=29

My graphs look much more continuous if I just plot the messages sequentially, ignoring the timestamp:

http://www.mynissanleaf.com/viewtopic.php?f=31&t=6702&start=22

Is it possible they are getting queued up in the CAN controller on the AVRCAN and being read (and timestamped) in blocks?

garygid · Nov 29, 2011

What version of CAN-Do are you using to get and save your Log data?

You could look the time stamp data on the main page of CAN-Do.

Or, use a hex editor, or email a zipped copy of the log to me.

Maybe there is a bug in the way that CAN-Do gets the time-stamps from the OS, or in how it adds the 2-byte time stamps to the stored data?

TickTock · Nov 29, 2011

Can-Do v1.5.2

Can-Do reports the same time as I get (exhibiting the same bursty behavior)

I did use HexEdit a bit, too, to figure everything out. Those 10bits do seem to depict 0-999 as you indicate but that means either the messages are arriving in bursts, or the logger is timestamping them in bursts. Is the timestamp applied by the SOCguage or Can-Do? It could be my RS232 USB driver is queuing the messages up before delivering in blocks to Can-Do if Can-Do is doing the timestamping.

garygid · Nov 30, 2011

CAN-Do adds the two time-stamp bytes.
The AVR-CAN does not really know about "time".

Yes, if you have hardware buffering of message bytes into "blocks" in your UART, or driver, ... or some other OS or system buffering of the COMM data before CAN-Do "wakes up", that could be causing the clumping.

CAN-Do looks at the number of bytes in its "RS232 buffer", and tries to process (sync and time stamp) the message as soon as 11 bytes are available. Each message is (should be) treated individually.

If another buffer is waiting for perhaps 220 bytes of data before passing them on to the application, that would cause 20 messages to be "clumped".

How many messages are in your "clumps", and how close are the time stamps within the clump?

What is the time jump between youe clumps?

TickTock · Dec 3, 2011

OK. Here is a plot of the delta time between messages as indicated in the log file (blue) as well as the count of messages in each burst (red). It looks like about every 0.5 seconds I get a burst of about 360 messages

garygid · Dec 3, 2011

Does my log file of "EV-log of 35 mile drive..." at
http://www.wwwsite.com/puzzles/cando/" onclick="window.open(this.href);return false;
have the same "clumping" problem?

TickTock · Dec 3, 2011

I didn't see any clumping in your evc file. But... I tried a few of my own .evc files and I don't see it on them either. This appears to be unique to the car can so maybe this is how they arrive from the car. Do you can a car can log I can try? I tried your Carsplit.can in your 3-Can log but it appears to be a different format (all timestamps were 0.000 - in fact there didn't seem to be any real data).

garygid · Dec 3, 2011

The 3-CAN log was initially ONE log file (".alc") that contains all the messages, from whatever source. It is POSSIBLE that I did not yet have the time-stamping working when I made the ".alc" file.

If one loads an "alc" file, one should be able to save just the EV, CAR, or AV messages seperately, into ".evc", ".can", or ".avc" files respectively.

Try that and see if I have a bug in my "write-log" function.

-----
Since you get good (non-clumped) EV time-stamps, CAN-Do MIGHT have a bug when loging only the CAR-CAN messages.

garygid · Jan 25, 2012

The first post (essentially the same information, slightly rewritten) contains:

Note: The name change of this Thread.

1. The format of the binary CAN-Messages output from the SOC-Meter.
2. The format of the one-CAN binary Log files written by CAN-Do.
3. The format of the binary all-Can Log file output by CAN-Do.
4. The formats of various Pseudo-Messages.

garygid · Sep 24, 2013

I just added the description of the incoming Multi-CAN data
stream, usually from a logging data source, to the first post.

garygid · Jul 21, 2014

In some drivers that handle the virtual comm ports in the PC,
the clustering of bytes into USB "blocks" can be adjusted down
from perhaps a default of 256 bytes per block to a much lower
16 or so bytes. I believe that this "grouping" of data bytes, for
greater efficiency in the USB data stream, is the cause of the
apparent "clustering" of the CAN messages. Using the much
lower minimum bytes per block, and a low latency time (the
maximum time that the data bytes are buffered before a
USB block is sent) seems to almost eliminate this problem.

SOC-Meter CAN-Message and CAN-Do Log file formats

Help Support My Nissan Leaf Forum:

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Similar threads

Join the conversation!

Join today and get all the highlights of this community direct to your inbox. It's FREE!