What I want to do is this:

I have already sniffed/monitored the exchange between the app and the USB device.

I don't really have much idea about what's going on, but I'd like to decode (reverse engineer) it first to understand what's what.

I'd appreciate any pointers to help me with this.

Then, I'd like to be able to write my own app to initiate and handle communication with the device. So, I believe I need some code to locate the correct USB device and talk to it.

What do you use for this sort of thing?

<redundant_text>
Here is a little background about why I am asking this:

Since I have diabetes, I have kissed every frog of blood glucose test devices out there. All were write-only. By that I mean this: You do the test, but unless you manually write down the test result somewhere else, it is kept in the device's memory for a while.

My princess has finally come up --sort of. There is now a device, namely 'Contour USB' which not only stores plenty of readings, it also has a USB interface which you can attach to a computer a software reads it to produce historical performace charts etc.

Well, that is the does not work quite like that in practice.

And the reason is the software that comes with it. It's Java (.jar) file that --more often than not-- can not start up unless you try repetitively [I tried this on WinXP, Win7-X64 and OSX] .

Second, and worse problem is, it is a lousy piece of $%^&. Looks like someone put it together in 20 mins, it's absolutely useless. Cannot even export data or produce useful charts.
</redundant_text>

Adem wrote

What I want to do is this: I have already sniffed/monitored the exchange between the app and the USB device.

May I ask how you did that? I'm searching a usable but not very costly USB monitoring thingy for Windows(XP).

For a small controller built using an ATmega64 and the abandoned Philips PDIUSBD11 I did it on low level. The operating system gives me a raw device, /dev/ugen0 on FreeBSD, and I have translated a small subset of the USB headers. These are used to talk to the device sending user requests implemented by the maker of the small controller board.

I can show some code snippets if you like.

Second, and worse problem is, it is a lousy piece of $%^&. Looks like someone put it together in 20 mins, it's absolutely useless. Cannot even export data or produce useful charts.

I have a similar problem with another type of hardware, but the USB analyzers (hardware) or sniffing software are very comfortable, they do on the fly protocol decoding and the like, but those have their price. And this price is amazing. ;)

You can use libusb for this. It allows you to communicate with usb devices from userspace (i.e. no need to write a kernel driver). It works on linux, windows and osx.

You can use some kind of usb snooping software to reverse engineer all the transfers. I've used this approach with success. It's important for you to familiarise yourself with the usb protocol, though.

Note that there aren't any fpc bindings for libusb 1 (that I know of), but that should be easy to generate.

There is a list for libusb, but I'm willing to help. This probably isn't the right list, though.

Henry

I forgot because I don't use it:

http://wiki.lazarus.freepascal.org/Hardware_Access#USB

There is libusb, which might be simpler to use and is already translated to fpc. And it does have a windows counterpart called libusb-w32 or -win32. You can find that on sourceforge, too.

Please note that this is for the legacy libusb (0.1.x). Libusb1.0 supports linux, windows and mac osx from the same source base. See http://www.libusb.org/ for more details.

I think a lot of open source projects (modems, webcams and so on) rely on SniffUsb run on Windows. However there's also
http://biot.com/blog/usb-sniffing-on-linux although I've not tried it- for the obvious reason that most things that you want to sniff don't have Linux drivers.

I've written an app to control a Velleman board using raw USB calls on Linux using Lazarus- somebody had already decoded the comms and the exercise was fairly painless.

The problems are going to start when the (Windows) driver contains code that has to be downloaded to a microcontroller in the device.

I have a similar problem with another type of hardware, but the USB analyzers (hardware) or sniffing software are very comfortable, they do on the fly protocol decoding and the like, but those have their price. And this price is amazing. ;)

You could always do what I do when reverse-engineering serial comms: get the file from the capture device, convert it into text in a convenient format, and then massage it using a sequence of Perl filters.

Marc Santhoff wrote

May I ask how you did that? I'm searching a usable but not very costly USB monitoring thingy for Windows(XP).

I used USBTrace [ http://sysnucleus.com/ ]. It's not at all cheap [195 USD].

I am using Windows (will later port to *nix or --if I can-- to OSX when I release it as freeware)

USBTrace exports in XML format with content such as below [this a short <g> sample of 200,000 lines I got for one session alone]

I apologize for dumping on you this, but I thought you might like to see what the USBTrace output looks like too.

<Request Number = "5088" Dir = "[OUT]">
<LogType>URB_FUNCTION_BULK_OR_INTERRUPT_TRANSFER</LogType> <Param>
<Name>Length</Name>
<Value>0x48</Value>
</Param>
<Param>
<Name>USBD Status</Name>
<Value>USBD_STATUS_SUCCESS (0x0)</Value>
</Param>
<Param>
<Name>EndpointAddress</Name>
<Value>0x81</Value>
</Param>
<Param>
<Name>PipeHandle</Name>
<Value>0x86363A94</Value>
</Param>
<Param>
<Name>TransferFlags</Name>
<Value>0x3 ( USBD_TRANSFER_DIRECTION_IN USBD_SHORT_TRANSFER_OK )</Value> </Param>
<Param>
<Name>TransferBufferLength</Name>
<Value>0x40</Value>
</Param>
<Param>
<Name>TransferBuffer</Name>
<Value>0x86ED6AD8</Value>
</Param>
<Param>
<Name>TransferBufferMDL</Name>
<Value>0x0</Value>
</Param>
<Param>
<Name>UrbLink</Name>
<Value>0x0</Value>
</Param>
</Request>
<Request Number = "5089" Dir = "[IN]">
<LogType>URB_FUNCTION_BULK_OR_INTERRUPT_TRANSFER</LogType> <Param>
<Name>Length</Name>
<Value>0x48</Value>
</Param>
<Param>
<Name>USBD Status</Name>
<Value>USBD_STATUS_PENDING (0x40000000)</Value>
</Param>
<Param>
<Name>EndpointAddress</Name>
<Value>0x81</Value>
</Param>
<Param>
<Name>PipeHandle</Name>
<Value>0x86363A94</Value>
</Param>
<Param>
<Name>TransferFlags</Name>
<Value>0x3 ( USBD_TRANSFER_DIRECTION_IN USBD_SHORT_TRANSFER_OK )</Value> </Param>
<Param>
<Name>TransferBufferLength</Name>
<Value>0x0</Value>
</Param>
<Param>
<Name>TransferBuffer</Name>
<Value>0x86ED6AD8</Value>
</Param>
<Param>
<Name>TransferBufferMDL</Name>
<Value>0x858FD770</Value>
</Param>
<Param>
<Name>UrbLink</Name>
<Value>0x0</Value>
</Param>
</Request>

I believe all those capitalized labels are constants. What the others are I have no idea yet.

In my case, health considerations took precedence, I'm afraid.

Mark Morgan Lloyd wrote

I think a lot of open source projects (modems, webcams and so on) rely on SniffUsb run on Windows.

I am extremely new to this subject --I have done serial port stuuf, but that was more than a decade ago.

USB seems quite a bit of different beast.

http://sysnucleus.com/usbtrace_cd_vendor.html

USBTrace, I mentioned in another post, talks about lists about a dozen 'Device Class Decoders' at this URL [ halfway down the page, http://sysnucleus.com/usbtrace_download.html ] plus how to write a vendor specific 'Device Class Decoder' [
http://sysnucleus.com/usbtrace_cd_vendor.html ].

If I need to go through all this, before decyphering the actual information I am after, it seems I have bit more than I can chew ;)

The problems are going to start when the (Windows) driver contains code that has to be downloaded to a microcontroller in the device.

Hmmm.. This device (Contour USB) installs a Java application --claimed to be usable in Windows, Linux and OSX.

Would you say it is also installing a device driver too?

I think this device is some form of mass storage device; but I am not sure.

How can I tell whether it does install a device driver also --in Windows platofrm.

You could always do what I do when reverse-engineering serial comms: get the file from the capture device, convert it into text in a convenient format, and then massage it using a sequence of Perl filters.

USBTrace seems to be able to help in filtering and/or turning the data exchange into an organized blurb [it darn well should; for that price.]

I didn't know that, thanks.

I'll test it, thanks.

http://biot.com/blog/usb-sniffing-on-linux although I've not tried it- for the obvious reason that most things that you want to sniff don't have Linux drivers.

The problem is the software for the device running on windows and behaving badly. So regardless which os the target program I'd write would be running on, sniffing has to be done on windows.

I think in this case it is not possible. The device has masses of control knobs and the software can switch them one by one - I'm not sure if it is a good idea at all to try, but the original program is really ... ahem ... funny.

That's in the price range of similar software, look there:

http://www.usblyzer.com/purchase.htm

At least they offer a 30 days fully functional demo. Another kind of pet are hardware analyzers, starting at around 800$:

http://www.beyondlogic.org/usb/protocolanalysers.htm

I'm still not willing to pay one third of the price the device to look at has costed itself. I'm pretty close to exchanging that thing itself, saving all the work making a protocol spec and a program ...

Then libusb should be best for you.

Very interesting, thanks. :)

<Request Number = "5088" Dir = "[OUT]"> <LogType>URB_FUNCTION_BULK_OR_INTERRUPT_TRANSFER</LogType> <Param>

[...]

Yes, look at the headers of your OS and libusb or their pascal translation. And maybe some USB starters document, explaining basic communication protocol.

Chapter 8 there is an example:

http://www.beyondlogic.org/usbnutshell/usb1.htm

The above just means that it's doing some bulk/interrupt transfers on endpoint 1. It's strange that it gives the endpoint address as 0x81, since that implies an "in" direction, and one of the transfers say it's an "out". What you are really interested in are the control transfers the software issues to set up and control the device. There will probably be bulk/interrupt reads after some control transfers, I'd guess.

Henry

Henry Vermaak wrote

You can use some kind of usb snooping software to reverse engineer all the transfers. I've used this approach with success. It's important for you to familiarise yourself with the usb protocol, though.

The thing one has to be careful of here is when the same pid:vid can refer to several different implementations. The Windows software knows how to pick up a private submodel number, somebody reverse-engineering the transfer doesn't.

Note that there aren't any fpc bindings for libusb 1 (that I know of), but that should be easy to generate.

http://wiki.lazarus.freepascal.org/Hardware_Access#libusb

I don't really know what you mean. Do you mean that the software can do a control transfer to query the model number, then have different communications methods depending on that? Either way you'll always be able to see all the messages on the bus, so I don't see how this is a problem.

> Note that there aren't any fpc bindings for libusb 1 (that I know of), > but that should be easy to generate. http://wiki.lazarus.freepascal.org/Hardware_Access#libusb

No, these bindings are for the old (0.1) libusb.

Henry

I've used SnoopyPro. It's free and worked well.
http://sourceforge.net/projects/usbsnoop/

Henry

Adem wrote

Would you say it is also installing a device driver too?

You're the one with the hardware and software. I'm saying nothing.

I think this device is some form of mass storage device; but I am not sure.

I'd suggest that the thing you should be doing /first/ is plugging it into a (Linux) system and getting as much information as possible about what it describes itself as. If it is simply a mass storage device then it is probably creating files which the Java app is transferring and trying to process. The fact that it takes several attempts to start up could be bad hardware design- I'm far from being an expert in this area but there are various requirements about how a device boots on connection, including changing pullup termination.

How can I tell whether it does install a device driver also --in Windows platofrm.

Don't know, the only Windows I use is NT as an application server and that has very limited USB capabilities. A good starting point would be looking for any software that dissects the software installer or the uninstall files it creates, a brute-force search for files with interesting names or incriminating messages including copyright notices, and- obviously- a thorough Google for the vid:pid
(vendor_identifier:product_identifier) and any other related hardware or software.

USBTrace seems to be able to help in filtering and/or turning the data exchange into an organized blurb [it darn well should; for that price.]

Since it's XML there's obviously lots of processing tools around. But the first thing is to find the underlying device class etc. so that you can see the wood for the trees: if you spent months working through your 200K trace only to find that it was entirely routine mass-storage management you'd feel pretty sick.

Beware of devices that need firmware from a driver. Beware of devices that start off looking like a mass storage or CD device and need a special command sequence ("Zero-CD" etc.) to make them operational.

Java should be a natural choice for this sort of thing. Unfortunately Sun dawdled for years before they even introduced serial support, meaning that Java made no significant penetration into e.g. lab equipment support. I've come across it being used for mini-PLC programming and that's about all.

I'm not sure about this but I suspect that the same class or archive that supports serial ports also supports USB. Anything at all that you can deduce about the Java classes and any custom libraries is likely to be a useful resource. And don't forget to look inside files for developer copyright messages and URLs, and allow that some of that might be Unicode so non-trivial to search.

So to summarise: work out what class of USB device you've got, inspect the Java, and then (only) if necessary use standard tools on the XML. I believe that all of these are well-understood by practitioners in appropriate fields (but not necessarily by me :-)

[Notes copied from elsewhere]

Here's something particularly interesting from the Universal Software Radio Peripheral (USRP) documentation. Note that the FX2 is an 8051-derivative acting as the USB slave. My notes in [brackets].

-----8<-----
The USRP itself contains no ROM-based firmware, merely a few bytes that specify the vendor ID (VID), product ID (PID) and revision [I presume in a serially-accessed NV device, otherwise the defaults for the microcontroller]. When the USRP is plugged in to the USB for the first time, the host-side library sees an un-configured USRP. It can tell it's un-configured by reading the VID, PID and revision. The first thing the library code does is download the 8051 code that defines the behavior of the USB peripheral controller. When this code boots, the USRP simulates a USB disconnect and reconnect.

When it reconnects, the host sees a different device: the VID, PID and revision are different. The firmware now running [in the 8051] defines the USB endpoints, interfaces and command handlers. One of the commands the USB controller now understands is load the FPGA. The library code, after seeing the USRP reconnect as the new device, goes to the next stage of the boot process and downloads the FPGA configuration bit stream.

Once the firmware has been downloaded to the FX2, it sets an internal register and reboots itself, this time presenting custom product and vendor ID?s, which the host detects as the FX2 disconnecting followed by the connection of a custom USB device. ----->8-----

So this third vid:pid pair could then be used to load kernel modules to implement a Bluetooth or GPS stack, not to mention more mundane applications like an AM or FM radio.

I suspect that several of these steps are common to a large number of USB devices with internal firmware. If that is that case then the firmware doesn't so much process data being transferred but mediates between commands on the USB bus and setup being sent to the application-specific chip with data being transferred by hardware when available.

[End copied notes]

That's obviously going to complicate things if you're only sniffing a single device (small group of endpoints) or a single class. The sniffing software (and any decoders) are not going to be able to say "device x:y is now killing itself and will be resurrected as z:t" unless somebody's already reverse-engineered the loader- not impossible but not very likely either.

Mark Morgan Lloyd wrote

So to summarise: work out what class of USB device you've got, inspect the Java, and then (only) if necessary use standard tools on the XML. I believe that all of these are well-understood by practitioners in appropriate fields (but not necessarily by me :-)

Right. So a few minutes Googling takes me to
http://ubuntuforums.org/showthread.php?p=8439613 which implies that the cable has an FTDI converter 1a79:6001 [checks] which Linux 2.6.32 should support (older kernels might need a patch). The handheld device is serial hence courtesy of __j__:

-----8<-----
For what it's worth, getting results from the Contour using the serial connector is pretty straightforward:

1. When you plug the meter in and press the "M" button, the device will send an ENQ (ASCII code 5) to signal that's it's listening.
2. You reply with an ACK (ASCII code 6), saying "send me the next record."
3. The device will send you a record. To signal you received it and want the next one, send another ACK.
4. Repeat until you've exhausted all the records.

The first record you get will be a header of sorts. The next 2 are misc information. Everything after that are glucose measurement records (or control solution).
----->8-----

If in doubt put a 'scope on the interface to the handheld device, you might need a level converter before you can attach a standard comms analyzer.

So if that's correct then the app uses standard serial comms, there might be a driver but it's purely to support the FTDI interface. You could probably write one that looked very similar in Lazarus and then extend it for database interface or whatever tickles your fancy. Then tell Bayer you've done it and see if they are friendly.

BUT I would very strongly suggest not trying to change any of the device's settings under software control since you might screw up its calibration, unless you can get interfacing details.

Strictly, somebody has been naughty here since if I remember correctly Java's license specifically excludes its use in medical equipment because of safety/liability issues. Under the circumstances I think that you must understand that if you use Lazarus for this sort of thing any problems are your own.

I'm familiar with this scenario, we use it in our own devices, except that we keep our vid and pid the same. The above method abuses the concept of having a vid and pid (it's the same product, why different pid?). In windows, the cypress driver can load a script that writes the firmware to the device when it is plugged in, then the device re-enumerates. In linux, I manually interpret the hex file and upload the data to the device.

You'd very quickly notice this happening. You can also see from the driver inf file whether there are any scripts involved. You can also learn all the vid and pid combinations from the inf file, so you'd know what to look for.

Henry

This is the ftdi_sio driver, which has been in the kernel for more than 10 years.

Henry Vermaak wrote

This is the ftdi_sio driver, which has been in the kernel for more than 10 years.

Sure, the driver has been in the kernel for more than ten years but there are continual additions to the list vid:pid identifiers that cause the driver module to be loaded. If you look at the patch at the bottom of the URL I cited:

+ * Bayer Ascensia Contour blood glucose meter USB-converter cable.
+ * http://winglucofacts.com/cables/
+ */
+#define BAYER_VID 0x1A79
+#define BAYER_CONTOUR_CABLE_PID 0x6001
+
+/*

And /that/ went into 2.6.30- hardly ten years old.

Sure it's abusing the standard- just like taking more current than you've asked for abuses it, or handling termination incorrectly. Doesn't stop it happening.

You still don't need a patch. You can test it via the module params or even an entry in /sys... (new_id). Sorry for being pedantic, I digress.

Marc, Mark, Henry and everyone else,

Thank you guys, these are plenty to start with.

I'll need some time to digest them all and then try to decide on an attack angle --I'll also get back to you, if I may, on the way.

So, once more, I'd like to thank you all.

right but one should be able to note the vid:pid (did i get that right?) attached to a particular USB port and note that it changes within a specific time period to a secondary and then within another certain time frame to a tertiary vid:pid... as these will occur within a (presumably) very short time period (guessing less than 2 or 3 seconds), it would appear to be "not a human plugging, unplugging and switching devices" because a human won't be able to do that in that short a time frame... plus there that if a human /did/ try to do that, it would likely (?) result in the sequence starting all over and running thru the three steps...

one might also have a table that notes the vid:pid of each stage... determining those is something else, altogether ;)

i would assume (yeah, doncha hate that?) that when there are no more records in the device that it would send a NAK to the ACK instead of a record?? the NAK meaning there are no more records...

ie: -> == from device and <- == to device

-> ENQ
<- ACK
-> DATA (header)
<- ACK
-> DATA (misc data 1)
<- ACK
-> DATA (misc data 2)
<- ACK
-> DATA (record 1)
<- ACK
:loop
-> DATA (record 1+x)
<- ACK
(increment x)
(goto loop until NAK)
-> NAK (no more records)
(then whatever shutdown code may be necessary)

the above depicting the transfer of only two records of glucose data in the device...

Mark Morgan Lloyd wrote

Sure it's abusing the standard- just like taking more current than you've asked for abuses it, or handling termination incorrectly. Doesn't stop it happening.

true... and those folk doing that should know better (for one thing) and it also enables others to take a batch of wet noodles and beat them over their collective heads for not following the published and accepted standards...

when coders stop working arounds other's bugs, then all code advances, grows, and evolves... i've seen this (forcibly) done with great effect of progress in numerous situations ;)

It may be tricky to note the change, but you're only really interested in seeing with what vid/pid it ends up with. Device manager shows you this. As I've noted, you can look in the inf file, since that should have all the vid/pid combinations in it already.

Henry

What could be tricky is when you need to run the device in linux and you need to upload the firmware manually, since you may have to do it transfer by transfer. I've never had to do this, luckily.

Anyway, this is a bit off topic now, since Adem's device just shows up as a serial port (we actually use one of these ftdi chips in one of our products).

Henry

waldo kitty wrote

i would assume (yeah, doncha hate that?) that when there are no more records in the device that it would send a NAK to the ACK instead of a record?? the NAK meaning there are no more records...

Or something like that. Note that the posting doesn't specify whether the data is ASCII or binary- this takes me /way/ back to peering at TD830 protocol decodes, not to mention the HPs currently on my desk...

right... i was unaware of that when i wrote the above... that they do that is a GoodThing<tm> :)

Henry Vermaak wrote

Anyway, this is a bit off topic now, since Adem's device just shows up as a serial port (we actually use one of these ftdi chips in one of our products).

sure... i was just tossing out ideas from the sidelines ;) :)

It depends on if the device is full speed or high speed. High speed device line analyzer can be very expensive.
As for me, i bought a Beagle USB 480 for $1200, and that was just about 14 months ago. I'd would recommend that
you get a software usb sniffer, sniffusb
(http://www.pcausa.com/Utilities/UsbSnoop/default.htm); as for myself, i like usbmon, a linux kernel module that allows you to spy usb traffic. Install a windows VM in linux, use the program, and look at the usbmon log. that's a good way to go. By the way, you also need to know what usb class and endpoint pipe type the usb device implements. If you're not familiar, please read the usb 2.0 specifications before you start on any of this.

Toan