C – How to measure the delay of receiving packets precisely or approximately

I’m trying to measure the latency when a packet enters the Rx buffer and is copied to application memory. I’m measuring it with this code :

struct timespec start, end;

clock_gettime(CLOCK_REALTIME, &start);
recvfrom(sock, msg, msg_len, 0, &client, &client_addrlen);
clock_gettime(CLOCK_REALTIME, &end);

I know this doesn’t measure latency precisely. However, I can calculate the average delay by receiving many packets, measuring each packet, and calculating them. Is there any way to measure latency more precisely? (For example, latency = (time to complete recvfrom() - (time from which the NIC received the packet)).

For devices and device drivers, I’m using Mellanox connectx-3 and mlx4_en.

I was able to get almost exact numbers via recvmsg().

References

• The most helpful link ( https://stackoverflow.com/a/47329376/5215330 ).
• Network timestamp ( https://www.kernel.org/doc/Documentation/networking/timestamping.txt )
• Linux Precision Time Protocol ( http://linuxptp.sourceforge.net/ ).

Code

I’m reproducing the code in the first link. This code is not ready to run, but just a snippet of working code.

static struct timespec handle_time(struct msghdr *msg) {
    struct cmsghdr *cmsg = CMSG_FIRSTHDR(msg);
    struct scm_timestamping *ts = (struct scm_timestamping *)CMSG_DATA(cmsg);
    return ts->ts[0];
}

...
char ctrl[64];
char *msg = malloc(64);

int val = SOF_TIMESTAMPING_RX_HARDWARE | SOF_TIMESTAMPING_RX_SOFTWARE
        | SOF_TIMESTAMPING_SOFTWARE | SOF_TIMESTAMPING_RAW_HARDWARE;

setsockopt(sock_fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val));

 user buffer
struct iovec iov = {
    .iov_base = msg,
    .iov_len = msg_len,
};

 ancillary message header 
struct msghdr m = {
    .msg_name = &client_addr,           // struct sockaddr_in
    .msg_namelen = client_addrlen,      // socklen_t
    .msg_iov = &iov,
    .msg_iovlen = 1,
    .msg_control = &ctrl,
    .msg_controllen = sizeof(ctrl),
};

while (1) {
    memset(msg, 0, msg_len);
    num_received = recvmsg(sock_fd, &m, 0);
    start = handle_time(&m);
    clock_gettime(CLOCK_REALTIME, &end);

if (verbose) {
        double elapsed_time = time_diff(start, end) / 1000;
        total_elapsed += elapsed_time;
        count++;
        printf("%f us %f us\n", elapsed_time, total_elapsed / count);
    }

if (sendto(sock_fd, msg, msg_len, 0, (struct sockaddr *) &client_addr, client_addrlen) < 0) {
        perror("\nMessage Send Failed\n");
        fprintf(stderr, "Value of errno: %d\n", errno);
    }
}

The key point is to use setsockopt() and recvmsg(). The key mechanism is that when you set an option for a socket FD, the kernel sets a timestamp based on the timestamp flag. After you set them, if you receive a message with struct msghdr, the kernel will audit the timestamp in SW or HW. When you view the data, you will be able to get 3 timestamps. This information can be interpreted as follows:

The structure can return up to three timestamps. This is a legacy
feature. At least one field is non-zero at any time. Most timestamps
are passed in ts[0]. Hardware timestamps are passed in ts[2]. ts[1] used to hold hardware timestamps converted to system time. Instead, expose the hardware clock device on the NIC directly as a HW PTP clock source, to allow time conversion in userspace and optionally synchronize system time with a userspace PTP stack such as linuxptp. For the PTP clock API, see Documentation/driver-api/ptp.rst.

For more information, see 2.1 in Documentation/networking/timestamping.txt.

If you want to see the hardware timestamp, then you need to have a specific hardware (reference this comment) and change its characteristics with ioctl(). However, there is a handy tool called Linuxptp that does the job.