Bug 2119

I confirm this intermittent error. I could reproduce it in another Ubuntu 14.04 machine. I did not managed to reproduce it in a Debian machine. I will investigate further.

The example dummy-network of the fd-net-device required the SimulatorImplementationType to be set to ns3::RealtimeSimulatorImpl. After adding this change I could no longer reproduce the memory leak. This fix was committed with changeset 4be177372dc4. 

I'll leave the bug report open until the automated valgrind tests on the Ubuntu 14.04 64-bit server confirm the problem is no longer present.

By changing the SimulatorImplementationType are we just papering over a real issue, presumably in DefaultSimulatorImpl?  Or the threading required for FdNetDevice?

In this case, the FdNetDevice requires the RealtimeSimulatorImpl because there are external elements to the simulator that work in real time (such as the socketpair pipes used in the example to communicate the devices). Using the DefaultSimulatorImpl caused the example not to run correctly (e.g. ARP replies and request not synchronized, blocking the ICMP exchange). The non-freed memory might have been a consequence of the simulation not executing properly.

Thanks for the explanation.

Alina, it seems now that 'dummy-network' and 'realtime-dummy-network' are essentially identical programs; should 'dummy-network' be removed?  

If not removed, we need to provide dummy-network only when bld.env["ENABLE_REAL_TIME"] is set, because since you committed the patch, it is building but crashing at runtime on OS X with:
assert failed. cond="uid != 0", msg="Assert in TypeId::LookupByName: ns3::RealtimeSimulatorImpl not found", file=../src/core/model/type-id.cc, line=567
libc++abi.dylib: terminating

You are right Tom. I will remove the dummy-network example in this case.

I checked the other test for the fd-net-device (the fd2fd-onoff example), which also uses the DefaultSimulatorImpl and a socketpair, and it is also producing intermittent memory leaks in my Ubuntu 14.04. It makes sense not to use the DefaultSimulatorImpl when using external elements which might have real-time constraints (e.g. socketpairs), but since despite the memory leaks the fd2fd-onoff example seems to complete correctly, there might be something else going on. I would like to take more time to look into this issue before deciding to remove the dummy-network test. For the time being I will revert my last changeset, so the tests in OS X don't fail, until I have more information.

Created attachment 2104 [details]
[FdNetDevice] Patch to free buffers with incoming traffic that were not processed at the end of the simulation

Created attachment 2105 [details]
[DefaultSimulatorImpl] Patch to free events remaining in the m_eventsWithContext list at the end of the simulation

After a closer inspection, the memory leak problem seems to come from the simulation finishing before all packets are processed (This does not happen when using the RealtimeSimulatorImpl). 

Problem description: The FdNetDeviceFdReader, in the FdNetDevice, reads incoming traffic into buffers (using a dedicated thread), and these buffers are passed as arguments to the ForwardUp callback for processing. For each buffer, a new ForwardUp Event is created and scheduled on the main simulation thread (using the scheduleWithContext method). When using the DefaultSimulatorImpl, it might happen that the simulation ends before all Events are executed, and this causes the memory allocated for the unprocessed Events, and for the corresponding buffers, to remain un-freed at the end of the simulation. There are two separated causes for the memory leak when using the DefaultSimulatorImpl:

1. Buffers allocated with incoming traffic not being freed when Events remain non-executed.

2. Non-executed Events not being freed at the end of the simulation when they are scheduled with ScheduleWithContext.

The two patches attached solve these issues. The first patch (attachment 2104 [details]) replaces the m_pendingReadCount counter, used in the FdNedDevice, by a queue that holds pointers the unprocessed buffers, so that any remaining buffers can be freed in the FdNetDevice destructor. The second patch (attachment 2105 [details]) fixes the DefaultSimulatorImpl so that the DoDispose method destroys any remaining events in the m_eventsWithContext list.

Though these patches should fix the intermittent memory leaks for the dummy-network test, it still remains the issue of whether it makes sense to use the DefaultSimulatorImpl with the FdNetDevice when a socket pair is used. This combination can produce wrong simulation results (e.g. 100% packet loss in the dummy-network example when run with "./test.py --example=dummy-network --grind --retain --verbose" although this does not happen when run with "./waf --run=dummy-network"). If you are ok with these patches, I will proceed and apply them. 

Regarding the dummy-network test (and also the fd2fd-onoff test), I am not sure whether to remove them or to leave them.

I'm ok with the patch to default-simulator-impl.cc.

An alternative would be to just call ProcessEventsWithContext() at the beginning of DoDispose, which will move all the pending events-with-context to the regular event list.

DefaultSimulatorImpl::DoDispose (void)
{
  NS_LOG_FUNCTION (this);
  ProcessEventsWithContext ();
  while (!m_events->IsEmpty ())
    {
      Scheduler::Event next = m_events->RemoveNext ();
  ...

Patches committed (with Peter's suggested modification) in changeset 941beab1b849