Linux – What memory leaks can occur outside of the view of the heap analyzer for GHC

I have a program exhibiting memory leaking behavior. It gradually occupies all system memory until it fills up all swap space, and then the operating system kills it. This happens every few days.

I’ve done an extensive dissection of the heap in a number of ways (-hy, -hm, -hc) and tried limiting the heap size (-M128M) adjusting the generation number (-G1), but whatever I do, the heap size always stays the same and low (in kB, not MB or GB). However, when I observe the program in htop, its resident memory is steadily climbing.

This indicates to me that the memory leak is coming from somewhere outside the GHC heap. My program uses dependencies, specifically Haskell’s yaml library that wraps the C library libyaml, and the leak may lie in the number of external pointers it has to assign to objects via libyaml.

I have three questions:

1. Where else in a Haskell program can a memory leak occur besides the GHC heap?
2. What tools can I use to track these?
3. What changes do I need to make to my source code to avoid these types of leaks, as they seem different from the more common space leaks in Haskell?

This does sound like the outer pointer is not done correctly. There are several possible reasons for this:

The
1. underlying C library is not freeing memory correctly.
2. The Haskell library is not set up for finalization correctly.
3. The ForeignPtr object is not freed.

I think it’s actually quite possible that option 3. If RTS consistently finds enough memory in the first generation GC, it won’t bother running the main collection. Fortunately, this is the easiest to diagnose. Just have your program run System.Memory.performGC frequently. If you fix it, then you have found the error and can adjust how often you want to do it.

Another possible problem is that you might have external pointers in long-lived thunk or other closures. Make sure you don’t.

When using a wrapped C library, a particularly strong possibility is that the wrapper function will return a ByteString with the underlying array allocated by the C code. Therefore, any ByteString you return from yaml may be off-heap.