Technical

Using CFFI to Debug C from Python

Recently I’ve been working on projects that require tools that interoperate between operating systems and languages. In particular I’ve been running on OSX in Python and Objective C while communicating with a particular hardware module in an embedded device to perform firmware upgrades.  

The firmware upgrade workflow has three parts, preparing the firmware file to be sent to the target device, sending the file, and receiving the file on the target  writing it to flash. Each stage of this process utilizes a different tool running in a different environment. I use Python to pre-process firmware images so that they are ready to be sent to the target device, Objective C for speaking to Apple’s hardware APIs and actually sending the file, and C on the target device.

To ensure that the firmware isn’t corrupted on its way to the device, a Cyclic Redundancy Check is computed at different points in the workflow and compared. A precise definition of a CRC is a topic for another blog post, but for our purposes it is sufficient to say that a CRC is a number that can be computed based on some input. Different inputs generate deterministically different output numbers, so these can be compared in order to verify that any given set of inputs is the same. For the firmware update process the inputs are the firmware files, and we can use a CRC to check if even a single bit of the firmware file has changed.

There are a variety of parameters that can be tuned when calculating a CRC but to generate the same value exactly the same parameters must be used every time. For example the hardware module in the target device has 6 options, each of which has multiple choices available.  So many choices make CRCs adaptable for many uses, but they also make it more difficult to debug what’s going wrong when outputs don’t match for a given set of identical inputs.

Ok, with that out of the way lets return to firmware updates. To verify that the firmware file made it through all three parts of the workflow, the same CRC must be conducted in three places. In firmware there is a hardware CRC module to use, in Objective C I found a CRC implementation in pure C, and in Python there were modules that provided such functions. Getting each of these three implementations of the same CRC function to agree was an onerous task, to phrase it lightly. After getting the Objective C code to match the hardware module, I was left with the Python code in disagreement. After hours of debugging I discovered the useful trick that inspired this blog post.

I was aware that you could wrap C in such a way that it could interoperate with Python, but I had never done it myself. I reasoned that I could avoid all my CRC headaches if I used exactly the same library in Python that I had already debugged for Objective C. Frustratingly most intermediaries require writing glue code in a specialized language or figuring out an arcane blend of headers and compiler flags. After beginning to write glue code for Cython, one of the intermediaries, I discovered cffi for Python. An FFI, or “Foreign Function Interface”, is a means by which one language can invoke functions of another. Python’s cffi allows the programmer to compile C directly into a shared library and to call it from Python. However its method of doing this is what’s special. Cffi allows the programmer to more or less copy-paste C code into cffi methods in Python, then ask cffi to compile it. That’s it! Cffi appears to dump it into a .c file, mix in its own magic, compile it on the spot, and make it immediately available for import.

Here is an example of compiling a function from C into Python, and calling it. First run the code to build the library:

from cffi import FFI
ffi = FFI()
ffi.cdef("int foo(uint8_t *buffer_in, uint8_t *buffer_out, int len);")
ffi.set_source("_example",
         """ 
         #include 
         int foo(uint8_t *buffer_in, uint8_t *buffer_out, int len)     {         
                  for(int i = 0; i < len; i++) {
                           buffer_out[i] = buffer_in[i] ^ 0xFF;
                  }     
                  return 0;
         } """)
ffi.compile()

Then import it and invoke away!

from _example import ffi, lib
b_in = bytes(‘a’ * 100)
b_out = bytes(‘0’ * len(b_in))
result = lib.foo(b_in, b_out, len(b_in)) 
print b_out

And with that, I was able to directly pull my C CRC library into Python without writing any glue code and deliver the tools I needed to the client. So next time you, o’ despondent programmer, find yourself searching for a quick way of interoperating between C and Python, consider using cffi to bridge the gap.