Bug 2131

I already said in mail, but it's worth commenting here too.

TCP (like any L4 protocol) could not know in advance the sender address - the most could have more than one.
The routing protocol is called (at least) twice: once to find out what is the sender address, the other to find the route - in case of the original sender, the output interface.

We *could* do it in one call, but this is *not* to be done - for the following reasons.

Many applications have a "connection" concept. TCP is a connection-oriented protocol, but also on UDP you can "connect" a socket, meaning that the application expects that there's something invariant in the data flow: the sender / destination IP numbers and ports (plus the protocol) 5-tuple.

For TCP this is obvious, but what does it means ?
That the sender address must be chosen only once - then it must be kept for the whole  connection. In other terms it is the sender address to be memorized, NOT the route. The route can change, or be lost. The sender address must be maintained.
In other terms, the sender address should be fixed when you send the SYN.

For UDP, we could do a similar thing, even if the protocol doesn't really state this.

About TcpL4Protocol::Send in particular, the sender address is passed as an argument and it must not be undefined. The problem is NOT there.

The local address is *already* set once: in TcpSocketBase::SetupEndpoint ().
As a consequence, for TCP the problem is not a problem. For UDP... maybe.

Given my current understanding, the problem is invalid.

Is the problem here that the header passed into the route request does not have the source address set?

I agree with Tommaso that asking for a route for every packet at this point in the code is not necessarily wrong, since it suppresses the call to find a route at the next layer down (IP); i.e. if we were not to obtain this route here, IPv4L3Protocol would later do the same per-packet lookup.

(In reply to Tom Henderson from comment #2)
> Is the problem here that the header passed into the route request does not
> have the source address set?
> 
> I agree with Tommaso that asking for a route for every packet at this point
> in the code is not necessarily wrong, since it suppresses the call to find a
> route at the next layer down (IP); i.e. if we were not to obtain this route
> here, IPv4L3Protocol would later do the same per-packet lookup.

I'm not an expert of IP layer (I'm barely sufficient in TCP after 2 years of work..) so maybe me (and Morteza) are confused.

The question here is if the call

route = ipv4->GetRoutingProtocol ()->RouteOutput (packet, header, oif, errno_);

should be called for each outgoing packet (#1) or the result of the first call can be cached (#2).

In the case #1, I need to know if Morteza (added in CC) needs absolutely this patch for ECMP or it is only a "cosmetic" (in an optimization sense) patch.

For case #2, I'm not sure about this phrase from Tommaso:
> The route can change, or be lost. The sender address must be maintained.

I think the phrase is right for all hosts inside the path, but not for the endpoint. I mean, there are two cases.

a) I'm who send the SYN. My address is fixed (and so the output interface, which is the parameter Ptr<NetDevice> oif passed by TcpSocketBase). Why I need to worry about a possible route change inside my host? I'm passing down all the packet through the same interface. Otherwise my connection will broke (because ip src changed).

b) I'm who send the SYN-ACK. My address is fixed, and so on... having as a result that, inside me, the route can not change (as in case a).

Giving this, I think caching the Ptr<Ipv{4,6}> route in the tcp/udp endpoints can be done safely.

(In reply to natale.patriciello from comment #3)
> (In reply to Tom Henderson from comment #2)
> > Is the problem here that the header passed into the route request does not
> > have the source address set?
> > 
> > I agree with Tommaso that asking for a route for every packet at this point
> > in the code is not necessarily wrong, since it suppresses the call to find a
> > route at the next layer down (IP); i.e. if we were not to obtain this route
> > here, IPv4L3Protocol would later do the same per-packet lookup.
> 
> I'm not an expert of IP layer (I'm barely sufficient in TCP after 2 years of
> work..) so maybe me (and Morteza) are confused.
> 
> The question here is if the call
> 
> route = ipv4->GetRoutingProtocol ()->RouteOutput (packet, header, oif,
> errno_);
> 
> should be called for each outgoing packet (#1) or the result of the first
> call can be cached (#2).
> 
> In the case #1, I need to know if Morteza (added in CC) needs absolutely
> this patch for ECMP or it is only a "cosmetic" (in an optimization sense)
> patch.

ECMP shouldn't rely on the TCP implementation to work. I haven't checked how ECMP works and/or why it should know the intended route, but I bet there's a workaround. Perhaps one can infer the route from the src/dst pair (hashed?) and the next hop address.

> For case #2, I'm not sure about this phrase from Tommaso:
> > The route can change, or be lost. The sender address must be maintained.
> 
> I think the phrase is right for all hosts inside the path, but not for the
> endpoint. I mean, there are two cases.
> 
> a) I'm who send the SYN. My address is fixed (and so the output interface,
> which is the parameter Ptr<NetDevice> oif passed by TcpSocketBase). Why I
> need to worry about a possible route change inside my host? I'm passing down
> all the packet through the same interface. Otherwise my connection will
> broke (because ip src changed).
> 
> b) I'm who send the SYN-ACK. My address is fixed, and so on... having as a
> result that, inside me, the route can not change (as in case a).
> 
> Giving this, I think caching the Ptr<Ipv{4,6}> route in the tcp/udp
> endpoints can be done safely.

No, it can't. A node could have a "virtual" IP address that is masqueraded by IP routing itself. TCP is and should not be interested into the outgoing interface, provided that it works.
Let me give you an example, perhaps it will be clearer.
Think to IP over IP. TCP is aware of the outer IP address, but the routing is made by the inner IP. Moreover, the inner IP address could change (tunneling) and even change interface. All the MIP cases fall in this category.

Don't be fooled by the fact that one can bind a socket to a particular output interface. That is purely because it's handy to be able to force a particular output interface. Moreover, for IPv6 is mandatory (link-local are "valid" on all the interfaces).

Anyway, it could be safe to store a route in TCP right now, but it's not wise. It's not something that would be done in real implementations and it's not something future-enhancements-proof.

Anyway, I triple checked the code we're discussing about. Maybe we're looking for the tree instead of the forest.

Code:

      Ptr<NetDevice> oif (0); //specify non-zero if bound to a source address
      if (ipv4->GetRoutingProtocol () != 0)
        {
          route = ipv4->GetRoutingProtocol ()->RouteOutput (packet, header, oif, errno_);
        }
      else
        {
          NS_LOG_ERROR ("No IPV4 Routing Protocol");
          route = 0;
        }


First, the above code is BUGGED (serendipity for the win).
Second, what the above code does ?

The bug:
"Ptr<NetDevice> oif (0);" should be erased.
It is masquerading the outermost "oif" definition and it is overwriting the of that was passed to the function. BAD.

What the code does is to 1) *verify* that a route exist (i.e., the packet can be sent), considering the output interface preferences (if any) and the src/dst pairs, and 2) *avoid* that the above check is done in IP.

If we want to move the above call in IP and change the Ipv[4,6]::Send signature to have an of instead of a route, we could also talk about it. However, we would just move the call from one layer to another. Moreover, we would have to cope with the IP over IP cases again, where we'd have to prevent (or live with) multiple RouteOutput calls.
Moving the call wouldn't simplify / fix the fact that RouteOutput is called for every packet.

Anyway, I stand my point: RouteOutput *must* be called for every packet. If there's a connection loss, it will be watched by the routing protocol, the route will be removed (or updated) and L4 protocols will know.
If we cache the route will have to invalidate it as soon as it's gone fishin'. Again, not a wise thing. 

Cheers,

T.

(In reply to Tommaso Pecorella from comment #1)
> I already said in mail, but it's worth commenting here too.
> 
> TCP (like any L4 protocol) could not know in advance the sender address -
> the most could have more than one.
> The routing protocol is called (at least) twice: once to find out what is
> the sender address, the other to find the route - in case of the original
> sender, the output interface.
> 
> We *could* do it in one call, but this is *not* to be done - for the
> following reasons.
> 
> Many applications have a "connection" concept. TCP is a connection-oriented
> protocol, but also on UDP you can "connect" a socket, meaning that the
> application expects that there's something invariant in the data flow: the
> sender / destination IP numbers and ports (plus the protocol) 5-tuple.
> 
> For TCP this is obvious, but what does it means ?
> That the sender address must be chosen only once - then it must be kept for
> the whole  connection. In other terms it is the sender address to be
> memorized, NOT the route. The route can change, or be lost. The sender
> address must be maintained.
> In other terms, the sender address should be fixed when you send the SYN.

I am interested to know a scenario where a route is invalidated but the source address (corresponding to that route) is not?
> 
> For UDP, we could do a similar thing, even if the protocol doesn't really
> state this.
> 
> About TcpL4Protocol::Send in particular, the sender address is passed as an
> argument and it must not be undefined. The problem is NOT there.
> 
> The local address is *already* set once: in TcpSocketBase::SetupEndpoint ().
> As a consequence, for TCP the problem is not a problem. For UDP... maybe.
> 
> Given my current understanding, the problem is invalid.

It is very hard to invalid the problem from your explanation.

(In reply to Tom Henderson from comment #2)
> Is the problem here that the header passed into the route request does not
> have the source address set?
 
I guess this is interesting since adding source IP address would change the operation of RouteOutput(). I think this is a good mechanism to validate a route.

> I agree with Tommaso that asking for a route for every packet at this point
> in the code is not necessarily wrong, 
> since it suppresses the call to find
> route at the next layer down (IP); i.e. if we were not to obtain this route
> here, IPv4L3Protocol would later do the same per-packet lookup.

The question is who should find a route for a packet? L4 or L3? 
I think we are violating the layering concept here!

(In reply to Tommaso Pecorella from comment #4)
> (In reply to natale.patriciello from comment #3)
> > (In reply to Tom Henderson from comment #2)
> > > Is the problem here that the header passed into the route request does not
> > > have the source address set?
> > > 
> > > I agree with Tommaso that asking for a route for every packet at this point
> > > in the code is not necessarily wrong, since it suppresses the call to find a
> > > route at the next layer down (IP); i.e. if we were not to obtain this route
> > > here, IPv4L3Protocol would later do the same per-packet lookup.
> > 
> > I'm not an expert of IP layer (I'm barely sufficient in TCP after 2 years of
> > work..) so maybe me (and Morteza) are confused.
> > 
> > The question here is if the call
> > 
> > route = ipv4->GetRoutingProtocol ()->RouteOutput (packet, header, oif,
> > errno_);
> > 
> > should be called for each outgoing packet (#1) or the result of the first
> > call can be cached (#2).
> > 
> > In the case #1, I need to know if Morteza (added in CC) needs absolutely
> > this patch for ECMP or it is only a "cosmetic" (in an optimization sense)
> > patch.
> 
> ECMP shouldn't rely on the TCP implementation to work. I haven't checked how
> ECMP works and/or why it should know the intended route, but I bet there's a
> workaround. Perhaps one can infer the route from the src/dst pair (hashed?)
> and the next hop address.
> 
> > For case #2, I'm not sure about this phrase from Tommaso:
> > > The route can change, or be lost. The sender address must be maintained.
> > 
> > I think the phrase is right for all hosts inside the path, but not for the
> > endpoint. I mean, there are two cases.
> > 
> > a) I'm who send the SYN. My address is fixed (and so the output interface,
> > which is the parameter Ptr<NetDevice> oif passed by TcpSocketBase). Why I
> > need to worry about a possible route change inside my host? I'm passing down
> > all the packet through the same interface. Otherwise my connection will
> > broke (because ip src changed).
> > 
> > b) I'm who send the SYN-ACK. My address is fixed, and so on... having as a
> > result that, inside me, the route can not change (as in case a).
> > 
> > Giving this, I think caching the Ptr<Ipv{4,6}> route in the tcp/udp
> > endpoints can be done safely.
> 
> No, it can't. A node could have a "virtual" IP address that is masqueraded
> by IP routing itself. TCP is and should not be interested into the outgoing
> interface, provided that it works.
> Let me give you an example, perhaps it will be clearer.
> Think to IP over IP. TCP is aware of the outer IP address, but the routing
> is made by the inner IP. Moreover, the inner IP address could change
> (tunneling) and even change interface. All the MIP cases fall in this
> category.
> 
> Don't be fooled by the fact that one can bind a socket to a particular
> output interface. That is purely because it's handy to be able to force a
> particular output interface. Moreover, for IPv6 is mandatory (link-local are
> "valid" on all the interfaces).
> 
> Anyway, it could be safe to store a route in TCP right now, but it's not
> wise. It's not something that would be done in real implementations and it's
> not something future-enhancements-proof.
> 
> Anyway, I triple checked the code we're discussing about. Maybe we're
> looking for the tree instead of the forest.
> 
> Code:
> 
>       Ptr<NetDevice> oif (0); //specify non-zero if bound to a source address
>       if (ipv4->GetRoutingProtocol () != 0)
>         {
>           route = ipv4->GetRoutingProtocol ()->RouteOutput (packet, header,
> oif, errno_);
>         }
>       else
>         {
>           NS_LOG_ERROR ("No IPV4 Routing Protocol");
>           route = 0;
>         }
> 
> 
> First, the above code is BUGGED (serendipity for the win).
> Second, what the above code does ?
> 
> The bug:
> "Ptr<NetDevice> oif (0);" should be erased.
> It is masquerading the outermost "oif" definition and it is overwriting the
> of that was passed to the function. BAD.
> 
> What the code does is to 1) *verify* that a route exist (i.e., the packet
> can be sent), considering the output interface preferences (if any) and the
> src/dst pairs, and 2) *avoid* that the above check is done in IP.
> 
> If we want to move the above call in IP and change the Ipv[4,6]::Send
> signature to have an of instead of a route, we could also talk about it.
> However, we would just move the call from one layer to another. Moreover, we
> would have to cope with the IP over IP cases again, where we'd have to
> prevent (or live with) multiple RouteOutput calls.
> Moving the call wouldn't simplify / fix the fact that RouteOutput is called
> for every packet.
> 
> Anyway, I stand my point: RouteOutput *must* be called for every packet.
> If
> there's a connection loss, it will be watched by the routing protocol, the
> route will be removed (or updated) and L4 protocols will know.
> If we cache the route will have to invalidate it as soon as it's gone
> fishin'. Again, not a wise thing. 



> 
> Cheers,
> 
> T.

(In reply to Tommaso Pecorella from comment #4)
> (In reply to natale.patriciello from comment #3)
> > (In reply to Tom Henderson from comment #2)
> > > Is the problem here that the header passed into the route request does not
> > > have the source address set?
> > > 
> > > I agree with Tommaso that asking for a route for every packet at this point
> > > in the code is not necessarily wrong, since it suppresses the call to find a
> > > route at the next layer down (IP); i.e. if we were not to obtain this route
> > > here, IPv4L3Protocol would later do the same per-packet lookup.
> > 
> > I'm not an expert of IP layer (I'm barely sufficient in TCP after 2 years of
> > work..) so maybe me (and Morteza) are confused.
> > 
> > The question here is if the call
> > 
> > route = ipv4->GetRoutingProtocol ()->RouteOutput (packet, header, oif,
> > errno_);
> > 
> > should be called for each outgoing packet (#1) or the result of the first
> > call can be cached (#2).
> > 
> > In the case #1, I need to know if Morteza (added in CC) needs absolutely
> > this patch for ECMP or it is only a "cosmetic" (in an optimization sense)
> > patch.
> 
> ECMP shouldn't rely on the TCP implementation to work. I haven't checked how
> ECMP works and/or why it should know the intended route, but I bet there's a
> workaround. Perhaps one can infer the route from the src/dst pair (hashed?)
> and the next hop address.
> 
> > For case #2, I'm not sure about this phrase from Tommaso:
> > > The route can change, or be lost. The sender address must be maintained.
> > 
> > I think the phrase is right for all hosts inside the path, but not for the
> > endpoint. I mean, there are two cases.
> > 
> > a) I'm who send the SYN. My address is fixed (and so the output interface,
> > which is the parameter Ptr<NetDevice> oif passed by TcpSocketBase). Why I
> > need to worry about a possible route change inside my host? I'm passing down
> > all the packet through the same interface. Otherwise my connection will
> > broke (because ip src changed).
> > 
> > b) I'm who send the SYN-ACK. My address is fixed, and so on... having as a
> > result that, inside me, the route can not change (as in case a).
> > 
> > Giving this, I think caching the Ptr<Ipv{4,6}> route in the tcp/udp
> > endpoints can be done safely.
> 
> No, it can't. A node could have a "virtual" IP address that is masqueraded
> by IP routing itself. TCP is and should not be interested into the outgoing
> interface, provided that it works.
> Let me give you an example, perhaps it will be clearer.
> Think to IP over IP. TCP is aware of the outer IP address, but the routing
> is made by the inner IP. Moreover, the inner IP address could change
> (tunneling) and even change interface. All the MIP cases fall in this
> category.
> 
> Don't be fooled by the fact that one can bind a socket to a particular
> output interface. That is purely because it's handy to be able to force a
> particular output interface. Moreover, for IPv6 is mandatory (link-local are
> "valid" on all the interfaces).
> 
> Anyway, it could be safe to store a route in TCP right now, but it's not
> wise. It's not something that would be done in real implementations and it's
> not something future-enhancements-proof.

I think TCP hold a route in real implementation. Are you sure about this?
> 
> Anyway, I triple checked the code we're discussing about. Maybe we're
> looking for the tree instead of the forest.
> 
> Code:
> 
>       Ptr<NetDevice> oif (0); //specify non-zero if bound to a source address
>       if (ipv4->GetRoutingProtocol () != 0)
>         {
>           route = ipv4->GetRoutingProtocol ()->RouteOutput (packet, header,
> oif, errno_);
>         }
>       else
>         {
>           NS_LOG_ERROR ("No IPV4 Routing Protocol");
>           route = 0;
>         }
> 
> 
> First, the above code is BUGGED (serendipity for the win).
> Second, what the above code does ?
> 
> The bug:
> "Ptr<NetDevice> oif (0);" should be erased.
> It is masquerading the outermost "oif" definition and it is overwriting the
> of that was passed to the function. BAD.
> 
> What the code does is to 1) *verify* that a route exist (i.e., the packet
> can be sent), considering the output interface preferences (if any) and the
> src/dst pairs, and 2) *avoid* that the above check is done in IP.
> 
> If we want to move the above call in IP and change the Ipv[4,6]::Send
> signature to have an of instead of a route, we could also talk about it.
> However, we would just move the call from one layer to another. Moreover, we
> would have to cope with the IP over IP cases again, where we'd have to
> prevent (or live with) multiple RouteOutput calls.
> Moving the call wouldn't simplify / fix the fact that RouteOutput is called
> for every packet.
> 
> Anyway, I stand my point: RouteOutput *must* be called for every packet. If
> there's a connection loss, it will be watched by the routing protocol, the
> route will be removed (or updated) and L4 protocols will know.
> If we cache the route will have to invalidate it as soon as it's gone
> fishin'. Again, not a wise thing. 
> 
> Cheers,
> 
> T.

Just a "global" reply. However, first things first.

What is a route.
Excluding Source Routing (which is deprecated in IPv6), a "route" is represented by the next hop IP address and its metric to reach a destination.
The next hop can become invalid if it disappears or if there is a better next hop.
The next hop unreachability can be due to a link failure or a node failure. The node could or could not be able to detect promptly this failure.

The above explanation is valid for routers and for source nodes, with our without a specific routing protocol.

> I am interested to know a scenario where a route is invalidated but the source
> address (corresponding to that route) is not?

IPv6 - Route Redirect ICMP message. The SYN would use a gateway and the SYN-ACK another one. The "route" is modified. 
Mobile IP - switching from a HoA to a CoA (or between two CoAs)
IPv4 or IPv6 - gateway failover mechanisms.

> The question is who should find a route for a packet? L4 or L3? 
> I think we are violating the layering concept here!

This is a bit harder to explain. Any protocol can ask what is a route to a destination, but routing is a L3 thing.
Again, L4 (TCP, UDP and so on) have to probe the routing table to discover the source address, but that means little or nothing.
Moreover, IETF decided that strict layer separation wasn't really necessary (a not-so-wise decision, but it wasn't so evident at that time). As a consequence, there are a lot of these "quirks" in TCP/IP.
Anyway, routing is done at L3. Any layer can access the routing table (the IP address is used as L4 and L7 address too). However, L3 has the final decision. Said so, an implementation is free to optimize things.

> I think TCP hold a route in real implementation. Are you sure about this?

99% sure. I.e., I save a 1% to take into account my non-omniscence.

Now, back to this bug - I don't like to go into discussions about if angels are male or female.

TCP ask for the route for every packet. Yes, because IP doesn't.
The alternatives are:
1) to let IP do it. I'm against for the reasons I said.
2) to cache the route (the gateway). In this case we'd need a way to purge an outdated route and to trigger an appropriate action by TCP.

I could consider the second option if someone could explain why there is this need and what are the benefits of this.
I suspect that ECMP needs this information (the route), but - again - my question is: isn't there any other way to achieve the same without modifying how TCP works ?
If it's mandatory to have the same route for all the TCP packets in a flow, the routing can safely implement a hash table and provide a cached reply for all the calls. There's no need to cache the route in TCP.

Hi Tommaso, thanks for your answers.

> TCP ask for the route for every packet. Yes, because IP doesn't.
> The alternatives are:
> 1) to let IP do it. I'm against for the reasons I said.
> 2) to cache the route (the gateway). In this case we'd need a way to purge
> an outdated route and to trigger an appropriate action by TCP.
> 
> I could consider the second option if someone could explain why there is
> this need and what are the benefits of this.
> I suspect that ECMP needs this information (the route), but - again - my
> question is: isn't there any other way to achieve the same without modifying
> how TCP works ?
> If it's mandatory to have the same route for all the TCP packets in a flow,
> the routing can safely implement a hash table and provide a cached reply for
> all the calls. There's no need to cache the route in TCP.

First thing to note here is that this is not a *bug* (I never think of this as a *bug*). Although, with my current understanding, I think, we are talking about a design decision that has been made for ns-3 and its possible limitations (since there are always tradeoffs in any design). 

Second thing to note here is that ECMP does not require a route, it actually *should* provide a route - it is an extra ability of the IP layer.

With current deployment of TCP (more specifically its interaction with IP layer), ECMP faces some limitations, which are discussed as follows:

1) HASH-based ECMP. It does not work properly when a host is multi-homed. The reason is TcpL4Protocol queries routing per-packet basis with an incomplete information (i.e. source IP address is not passed to routeOutput). Thus, hash of five-tuple may offer the same route for different TCP flows destine to a single destination. However, if we argue that we do need to consider multi-homed scenarios, then hash-based ECMP works perfectly with current deployment since when RouteInput() is called (between routers), all elements of five-tuple can be used by routing to make its forwarding decision.

2) Round-robin based ECMP. It does not work at all when a host is multi-homed since RouteOutput is called per-packet basis (i.e. it is not possible to assign routes (correspond to different network interfaces) to TCP sockets in circular manner).

I am not sure yet if we can find a workaround for this limitation other than thinking about changing the current deployment (I actually prefer not to change current deployment as its implications are not completely clear to me).

My current deployment of ECMP modes is available publicly, in case anyone is interested to use it (http://users.sussex.ac.uk/~mk382/).

I am very grateful for all of our discussions on this topic so far since they significantly advanced my understanding.

Regards,
Morteza

(In reply to Morteza Kheirkhah from comment #9)
> Hi Tommaso, thanks for your answers.
> [...]
> First thing to note here is that this is not a *bug* (I never think of this
> as a *bug*). Although, with my current understanding, I think, we are
> talking about a design decision that has been made for ns-3 and its possible
> limitations (since there are always tradeoffs in any design).

I totally agree. Sorry if I overlooked the scope. Still, I consider the actual design a better alternative than a cache. Of course, since we're talking about design, we can disagree.

> Second thing to note here is that ECMP does not require a route, it actually
> *should* provide a route - it is an extra ability of the IP layer.
> [...]
> 1) HASH-based ECMP. It does not work properly when a host is multi-homed.
> The reason is TcpL4Protocol queries routing per-packet basis with an
> incomplete information (i.e. source IP address is not passed to
> routeOutput). 

THIS is a friggin' bug. The header must provide all the "known" data, otherwise the routing protocol is entitled to choose the source address it wants.
BUG, BUG, BUG.
If we fix this, Hash should work, right ?

> 2) Round-robin based ECMP. It does not work at all when a host is
> multi-homed since RouteOutput is called per-packet basis (i.e. it is not
> possible to assign routes (correspond to different network interfaces) to
> TCP sockets in circular manner).

I don't know exactly how Round Robin ECMP should work. However, it's best to move the discussion elsewhere.

> I am not sure yet if we can find a workaround for this limitation other than
> thinking about changing the current deployment (I actually prefer not to
> change current deployment as its implications are not completely clear to
> me).
> 
> My current deployment of ECMP modes is available publicly, in case anyone is
> interested to use it (http://users.sussex.ac.uk/~mk382/).

I think that a better understanding will lead to a more realistic and implementable design. Once you're happy with the design choices, you can also change the deployment.

> I am very grateful for all of our discussions on this topic so far since
> they significantly advanced my understanding.

I'm happy to discuss as well, I learnt some new stuff too to better explain my points :)

Cheers,

T.

As a reference, the two bugs we found (serendipity?) are Bug #2135 and #2137.

(In reply to Tommaso Pecorella from comment #10)
> (In reply to Morteza Kheirkhah from comment #9)
> > Hi Tommaso, thanks for your answers.
> > [...]
> > First thing to note here is that this is not a *bug* (I never think of this
> > as a *bug*). Although, with my current understanding, I think, we are
> > talking about a design decision that has been made for ns-3 and its possible
> > limitations (since there are always tradeoffs in any design).
> 
> I totally agree. Sorry if I overlooked the scope. Still, I consider the
> actual design a better alternative than a cache. Of course, since we're
> talking about design, we can disagree.
> 
> > Second thing to note here is that ECMP does not require a route, it actually
> > *should* provide a route - it is an extra ability of the IP layer.
> > [...]
> > 1) HASH-based ECMP. It does not work properly when a host is multi-homed.
> > The reason is TcpL4Protocol queries routing per-packet basis with an
> > incomplete information (i.e. source IP address is not passed to
> > routeOutput). 
> 
> THIS is a friggin' bug. The header must provide all the "known" data,
> otherwise the routing protocol is entitled to choose the source address it
> wants.
> BUG, BUG, BUG.
> If we fix this, Hash should work, right ?

Hi Tommaso,

Yes, I think it solves everything. I reported this in bug 1929 one year go, but you closed my bug report, by indicating that this is not a bug? :D Please find its link as follows: https://www.nsnam.org/bugzilla/﷒0﷓

Regards,
Morteza