Python – How do I handle member stuffing in C structures when reading cffi.buffer using numpy.frombuffer?

I have to read the array of C structures returned from the dll and convert it to a numpy array. The code uses Python’s cffi module.

The code works so far, but I don’t know how to handle member stuffing in the structure of the np.frombuffer hint:

ValueError: buffer size must be a multiple of element size

Here is my code :

from cffi import FFI
import numpy as np

s = '''
    typedef struct
    {
        int a;
        int b;
        float c;
        double d;
    } mystruct;
    '''

ffi = FFI()
ffi.cdef(s)

res = []

#create array and fill with dummy data
for k in range(2):

m = ffi.new("mystruct *")

m.a = k
    m.b = k + 1
    m.c = k + 2.0
    m.d = k + 3.0

res.append(m[0])

m_arr = ffi.new("mystruct[]", res)

print(m_arr)

# dtype for structured array in Numpy
dt = [('a', 'i4'),
      ('b', 'i4'),
      ('c', 'f4'),
      ('d', 'f8')]

# member size, 20 bytes
print('size, manually', 4 + 4 + 4 + 8)

# total size of struct, 24 bytes
print('sizeof', ffi.sizeof(m_arr[0]))

#reason is member padding in structs

buf = ffi.buffer(m_arr)
print(buf)

x = np.frombuffer(buf, dtype=dt)
print(x)

Any ideas to deal with this cleanly?

Edit:

If I add an extra number to the dtype where padding should occur, it seems to work :

dt = [('a', 'i4'),
      ('b', 'i4'),
      ('c', 'f4'),
      ('pad', 'f4'),
      ('d', 'f8')]

Why does the filling happen there? (Win7, 64-bit, Python 3.4-64-bit).

But this is not the best approach. The real code is much more complex and dynamic, so it should be possible to handle this somehow, right?

Probably the most convenient method is at numpy dtype constructor Use the keyword align=True.This will be populated automatically.

dt = [('a', 'i4'),
      ('b', 'i4'),
      ('c', 'f4'),
      ('d', 'f8')]

dt_obj = np.dtype(dt, align=True)
x = np.frombuffer(buf, dtype=dt_obj)

(See also Numpy doc) about structured arrays.)