Subject: Re: [lisp] A Simple BGP-based Mobile Routing System for the Aeronautical Telecommunications Network

From: Dino Farinacci <[email-masked]>
Date: Mon, 5 Jun 2017 15:08:34 -0700

> LISP’ers,

Fred, thanks for the email. Sorry for the delay on my response.

> Since the closing of the Routing Research Group (RRG), I have been focused
> on furthering the development of my own technologies that began there,
> and really haven't paid much attention to LISP - even to the point that I never
> even took the time to read the LISP specs. That has changed over the past
> couple of days, and now I think I have something to discuss that should
> interest this group.

Well a lot of machinery has been added for the various new and modern use-cases coming up these days.

> I am involved in a working group that is designing an IPv6-based Internetwork
> for the worldwide civil aviation air traffic management system. The network
> will be known as the Aeronautical Telecommunications Network with Internet
> Protocol Services (ATN/IPS). It will involve air traffic controllers communicating
> with aircraft that use aviation data links with low bandwidth and high delay
> properties (with some links having bandwidth as low as 32Kbps!). Each aircraft
> will receive a /56 IPv6 prefix taken from a shorter ATN/IPS service prefix
> (e.g., 2001:db8::/32).

Okay.

> Aircraft may undergo mobility events throughout their various phases of flights,
> such as handing off between cell towers, switching from a terrestrial data link
> to a satellite link, etc. So, the network needs to be designed to handle most
> mobility events at the intra-site level and only propagate mobility events to
> the network core when there is a major movement between sites. To that
> end, I have published a document titled:

Note, have a look at draft-farinacci-lisp-predictive-rlocs-02. It has been called for adoption for a working group document. With this mechanism, we minimize packet loss an wouldn’t have to have a control-plane topology carry data packets as you are suggesting below.

> "A Simple BGP-based Mobile Routing System for the Aeronautical
> Telecommunications Network":
> https://datatracker.ietf.org/doc/draft-templin-atn-bgp/
> 
> This document describes a simple BGP-based hierarchy in an overlay network
> using tunnels. But after my reading over the past couple of days, I have come
> to realize that it is an example of a LISP+ALT topology.

Well if you extend the LISP-ALT from the map-resolvers/map-servers to the xTR, then you have that push mechanism available. And then you know where an ETR is for a given EID. 

> However, for this specific use case I see a possible departure from the
> recommendations of RFC6836. Namely, the packets produced by aviation data

Because the topology was provisioned and capacity planned for moving small Map-Request messages that had different delivery requirements then packet data.

> links need to be treated as precious commodities that are to be delivered
> with the highest degree of reliability possible, which means that they must
> not be dropped due to an on-demand mapping table lookup. This means
> that the packets should be routed over the LISP+ALT topology unless or
> until a mapping is discovered. RFC6836 recommends against this in the
> general sense, but I believe what we have here is an exception.

I don’t think you need to. I think the reason you suggest that is because a LISP-ALT topology knows “where topologically” an EID is. But if you have that, you could just encapsulate to the last-hop BGP xTR. So the LISP-ALT topology wouldn’t be needed for packet data.

> The packets produced by aircraft need to be treated as precious commodities

That is what everyone says. Just not your industry.  ;-)

> by the network since there is no way to tell if the aircraft will be able to
> retransmit if the packet is dropped - each packet should be considered
> as potentially the aircraft's very last transmission before falling silent.

Is there a multicast requirement?

> So, what I think we should consider is whether there is a class of packets
> that should *always* be routed over the LISP+ALT topology, i.e., instead
> of serving as data probes. This could also give rise to a hybrid topology
> where safety-of-flight-critical packets are unconditionally forwarded over
> the ALT topology while lower priority packets are forwarded from ITR to
> ETR in the true LISP sense.

Give us your opinion about the suggestions above.

> Please have a look at my document and consider this use case I am
> describing. Please post comments to the list.

I did. We used this idea in many places to have a “push-based” overlay using the BGP protocol. I probably have implemented 3 different variants of it. So it is a well-known tool.

Thanks,
Dino

> 
> Thanks- Fred
> [email-masked]
> _______________________________________________
> lisp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lisp

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 8 Jun 2017 15:51:48 +0000

Hi Dino,

> -----Original Message-----
> From: Dino Farinacci [mailto:[email-masked]]
> Sent: Monday, June 05, 2017 3:09 PM
> To: Templin, Fred L <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [lisp] A Simple BGP-based Mobile Routing System for the Aeronautical Telecommunications Network
> 
> > LISP’ers,
> 
> Note, have a look at draft-farinacci-lisp-predictive-rlocs-02. It has been called for adoption for a working group document. With this
> mechanism, we minimize packet loss an wouldn’t have to have a control-plane topology carry data packets as you are suggesting
> below.

Thanks, I read the document. It looks like the approach focuses on trying to hit
a moving target, and an airplane might be a good example use case. That is good,
but not exactly the case I was concerned about. I am concerned about an airplane
that sends a single, isolated packet that gets routed through some ground
network domain to an ITR that has no mapping cache entry for the destination.
The ITR then has to issue a map-request and wait for a map-reply to come back,
and in the interim the isolated packet could be lost. I would rather see the
packet make steady forward progress toward the destination even if it might
need to take a slightly longer path.

> Well if you extend the LISP-ALT from the map-resolvers/map-servers to the xTR, then you have that push mechanism available.And
> then you know where an ETR is for a given EID.

I think where my document might differ from classical LISP-ALT (if there is such
a thing) is that the xTRs are also BGP routers but they have only a partial set of
EID-to-RLOC mappings. And, for any EID for which the xTR does not have a
mapping, it forwards packets to  a "default" BGP router that does have the
full set of mappings.

> requirements then packet data.

That sounds like a network planning consideration. The system could be sized
to accommodate data packets as well as control. And, I am not talking about
large packets at high data rates - I am expecting that only a few initial data
packets are forwarded over the longer path before route optimization
discovers a direct ITR->ETR path, and/or safety-of-flight critical small data
packets that need to be delivered with the highest possible reliability.

> I don’t think you need to. I think the reason you suggest that is because a LISP-ALT topology knows “where topologically” an EID is. But
> if you have that, you could just encapsulate to the last-hop BGP xTR. So the LISP-ALT topology wouldn’t be needed for packet data.

Based on what you are saying, I think there may be a difference between
"classical" LISP-ALT and what my document is proposing. In my design, ITRs
that are also BGP routers have only partial mapping information initially and
so don't know how to reach the last-hop BGP xTR until a route optimization
message is received. The (partial topology / full topology) design reduces
the number of BGP updates and reduces the BGP convergence time.

> Is there a multicast requirement?

Yes, good point. The system will need to support multicast. I don't address
this in the current document version, but it will need to be addressed.

> I did. We used this idea in many places to have a “push-based” overlay using the BGP protocol. I probably have implemented 3
> different variants of it. So it is a well-known tool.

The other thing to mention about BGP is this. Packets sent over the same paths
that carry BGP peerings maintained by TCP keepalives have the assurance that
the path has been tested before any data packets try to use the path. In contrast,
in my understanding of LISP when an ITR receives a map-reply from an ETR, it
knows that the ETR is up but does not yet know if the ITR->ETR path is working
because that path has not been tested yet. So, sending packets without testing
the ITR->ETR path first is a leap of faith in a certain sense and could result in a
black hole if that path is somehow blocked. In the approach I am suggesting,
ITR->ETR paths are always tested first before any data packets are sent.

Thanks - Fred

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 9 Jun 2017 08:44:09 -0700

> but not exactly the case I was concerned about. I am concerned about an airplane

You were concerned about the “first packet drop while looking up the mapping” issue. In a mobility case, one could argue, you’d have this for every move. The reason I pointed you to the document is to show that you would not have it for every move.

> need to take a slightly longer path.

We don’t find this in practice exactly for the mechanisms you state below. Most xTRs are behind a NAT or firewall of some flavor, so the xTRs ususally install 0.0.0.0/0 and 0::/0 in their map-caches when they get a list of RTRs from the mapping system when they register. The RLOCs for the default map-cache entries is the list of RTRs. That is part of the nat-traversal design in draft-ermagen-lisp-nat-traversal.

> EID-to-RLOC mappings. And, for any EID for which the xTR does not have a

That is what I suggested above. We have always said, for ANY mapping database system, you could extend the mapping system functionality (i.e. map-resolver and map-server) into the xTRs so they would participate in LISP-ALT or LISP-DDT.

> mapping, it forwards packets to  a "default" BGP router that does have the

That is what an RTR does for nat-traversal, signal-free multicast, LISP-RE, and LISP interworking.

> to accommodate data packets as well as control. And, I am not talking about

Right. LISP-ALT was a logical tunnel topology where data packets could take sub-optimal routes, where smaller, less frequently sent Map-Requests could endure some sub-optimality.

> packets are forwarded over the longer path before route optimization

Right. But many application writers say they want consistent latency from the network rather than designed in inconsistent latency.

> the number of BGP updates and reduces the BGP convergence time.

Yes, that is called a Map-Reply  ;-)

You are proposing the exact same machinery that already exists, just using different nouns and verbs. ;-)

> this in the current document version, but it will need to be addressed.

Is it required to support multicast in the underlay? Or is it sufficient to support a multicast overlay with a unicast underlay?

> the path has been tested before any data packets try to use the path. In contrast,

We already have that with LISP proper. When an ITR populates its map-cache, it sends RLOC-probes to each RLOC to test reachability. It can also test reachability *while* it is sending data with echo-noncing in the data-plane. It can use either or both. And when it uses both, echo-noncing can suppress RLOC-probes to reduce message load.

> knows that the ETR is up but does not yet know if the ITR->ETR path is working

It knows the ETR’s RLOC and there are many mechanisms of varying quality to determine if the RLOC is reachable from the ITR’s topological perspective.

> ITR->ETR paths are always tested first before any data packets are sent.

If you couple NAT-traversal with RLOC-probing, an ITR always has a map-cache entry for EVERY destination and tests the RTR reachability with Info-Requests and RLOC-probes.

So you do not have single packet loss and you have tested the data path.

Dino

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 14 Jun 2017 23:33:11 +0000

Hi Dino,

> Sent: Friday, June 09, 2017 8:44 AM
> You were concerned about the “first packet drop while looking up the mapping” issue. In a mobility case, one could argue, you’d have
> this for every move. The reason I pointed you to the document is to show that you would not have it for every move.

I re-read the document. My concern is for a Roaming-EID sending to a
fixed-EID when the ITR used by the Roaming-EID does not yet have a
mapping for the fixed-EID. That is the reverse direction from what is
discussed in *-predictive-rlocs AFAICT.

> We don’t find this in practice exactly for the mechanisms you state below. Most xTRs are behind a NAT or firewall of some flavor, so
> the xTRs ususally install 0.0.0.0/0 and 0::/0 in their map-caches when they get a list of RTRs from the mapping system when they
> register. The RLOCs for the default map-cache entries is the list of RTRs. That is part of the nat-traversal design in draft-ermagen-lisp-
> nat-traversal.

There is no NAT or firewall, and no IPv4 in the system I am considering.
But, even if the ITR has a default route to an RTR, the RTR would still
have to consult the mapping service if it does not already have a
mapping, right?

> That is what I suggested above. We have always said, for ANY mapping database system, you could extend the mapping system
> functionality (i.e. map-resolver and map-server) into the xTRs so they would participate in LISP-ALT or LISP-DDT.
> 

Please let me know how you think the simple stub/core routing arrangement
in my document maps to the RTR concept. In my document, the stub ASBRs
correspond to ITRs that have only partial topology information and the core
ASBRs are default routers that have full topology information. In AERO, the
stub ASBRs are AERO Servers and the core ASBRs are AERO Relays.

> Right. LISP-ALT was a logical tunnel topology where data packets could take sub-optimal routes, where smaller, less frequently sent
> Map-Requests could endure some sub-optimality.

Consider the possibility that some classes of data packets (e.g., safety-of-flight
critical short text messages) always take the sub-optimal but stable route
while other classes of data packets (e.g., non safety-critical motion video)
take the route-optimized path which may be subject to disruption. Same
destination; multiple paths.

> Right. But many application writers say they want consistent latency from the network rather than designed in inconsistent latency.

Rerouting from a slower path to a faster path can happen anywhere in the
network and at any time. 

> You are proposing the exact same machinery that already exists, just using different nouns and verbs. ;-)

I am proposing a set of default routers with full topology information that can
forward data packets over suboptimal routes when an xTR does not have
a mapping. The xTR can then go fetch a mapping while continuing to allow
data packets to flow through the default router so that route optimization
happens in parallel to data packet progression instead of dropping or
delaying initial packets. That is different than what I understand about LISP.

> Is it required to support multicast in the underlay? Or is it sufficient to support a multicast overlay with a unicast underlay?

Either/or should work fine.
> We already have that with LISP proper. When an ITR populates its map-cache, it sends RLOC-probes to each RLOC to test reachability.
> It can also test reachability *while* it is sending data with echo-noncing in the data-plane. It can use either or both. And when it uses
> both, echo-noncing can suppress RLOC-probes to reduce message load.

That is not quite the same as what I am talking about. In the design I am
describing, no data packet is ever delayed, and no data packet is ever sent
over a path that has not already been tested in the ITR->ETR direction.

> If you couple NAT-traversal with RLOC-probing, an ITR always has a map-cache entry for EVERY destination and tests the RTR
> reachability with Info-Requests and RLOC-probes.
> 

I think understand what you are saying, but it is not quite the same.

BTW, I arrived at the design I am describing based on my development of the
AERO proposal. That design is based on the notion of a Non-Broadcast, Multiple
Access (NBMA) virtual interface model for forwarding encapsulated packets to
neighbors without explicit tunnels. That model had its origins in RFC2529, which
was then adapted for RFC4214/RFC5214. I then took the concept into the Routing
Research group, where RFC6179 emerged as one of the IRTF docs. And, that
model has now been extended to the design we are talking about here.

This is not to say that I think it pre-dates LISP or anything like that, but it does go
back quite a long time and has been further developed along an independent
path that did not interact with the LISP development path. That we are seeing
similarities is not surprising - but there are still important differences.  

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 15 Jun 2017 16:22:40 +0000

Dino,

> Rerouting from a slower path to a faster path can happen anywhere in the

Slower/faster is not the correct distinction - both paths would be carried over
high-bandwidth backbone links (e.g. 1Gbps).

The correct distinction is longer (delay) path vs. shorter (delay) path. The longer
delay path would be maintained by BGP keepalives while the shorter delay path
would not.

Thanks - Fred

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 15 Jun 2017 19:48:43 -0700

> discussed in *-predictive-rlocs AFAICT.

Victor Moreno made a suggestion that has not been added yet to the document where the predictive-RLOCs in the RLE path can have the map-cache primed with the EID of the stationary node. So when packets come to an RLOC that the roaming-EID has landed on, when it returns packets to the stationary EID, the RLOC doesn’t have to send a Map-Request.

> There is no NAT or firewall, and no IPv4 in the system I am considering.

Okay, that simplifies the use-case.

> have to consult the mapping service if it does not already have a

Yes, but it is acting in aggregate for all ITRs that default to it. So it reduces the chances of sending a Map-Request.

> in my document maps to the RTR concept. In my document, the stub ASBRs

The stubs are ITRs, the core are RTRs. It’s exactly they same idea but you use different terms.

> stub ASBRs are AERO Servers and the core ASBRs are AERO Relays.

If the RTRs have all the EID->RLOC information since it was pushed via BGP, then there are no packet drops (for the EIDs that matched the pushed data).

> destination; multiple paths.

You can do this with RTR service-chaining.

> a mapping. The xTR can then go fetch a mapping while continuing to allow

And I am proposing all xTRs have a default route to a set of RTRs that have full routing information.

> delaying initial packets. That is different than what I understand about LISP.

You can default to a coarse EID-prefix which is the longer route and if a Map-Request is sent later and a more specific-EID is returned, then that can correspond to the shorter route.

> over a path that has not already been tested in the ITR->ETR direction.

Sending packets over suboptimal paths IS delaying data. I’m just saying.  ;-)

> neighbors without explicit tunnels. That model had its origins in RFC2529, which

I don’t know what the definition of “explicit tunnels” means. You should define it. But LISP has dynamically encapsulated packets. There is no “tunnel setup” in the protocol.

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 15 Jun 2017 19:49:32 -0700

Right, I got that.

Dino

> On Jun 15, 2017, at 9:22 AM, Templin, Fred L <[email-masked]> wrote:
> 
> Dino,
>

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 20 Jun 2017 17:17:39 +0000

HI Dino,

> Sent: Thursday, June 15, 2017 7:49 PM
> Victor Moreno made a suggestion that has not been added yet to the document where the predictive-RLOCs in the RLE path can have
> the map-cache primed with the EID of the stationary node. So when packets come to an RLOC that the roaming-EID has landed on,
> when it returns packets to the stationary EID, the RLOC doesn’t have to send a Map-Request.

Right, I heard others make that suggestion too (at the last ICAO meeting
in May). But, if the number of potential correspondent nodes (fixed of
mobile) is large then pre-fetching EID-RLOC mappings may not be feasible.

> The stubs are ITRs, the core are RTRs. It’s exactly they same idea but you use different terms.

Autonomous System Border Router (ASBR) is a BGP term that I think has
been around for a long time.

> If the RTRs have all the EID->RLOC information since it was pushed via BGP, then there are no packet drops (for the EIDs that matched

You mean that if all communications went from ITR->RTR->ETR then
there would be no packet loss? That would be a dogleg path where
you would really rather do ITR->ETR.

What I am suggesting is to start out with a dogleg path, then discover
a direct path but continue to use the dogleg path until the direct path
has been tested. After that, either go direct from ITR->ETR or continue
to send some classes of packets over the dogleg path and send other
classes over the direct path, i.e., multipath.

> And I am proposing all xTRs have a default route to a set of RTRs that have full routing information.

That is what I said, except that I am using the BGP terminology ASBR
and the AERO terminology Client/Server/Relay.

> You can default to a coarse EID-prefix which is the longer route and if a Map-Request is sent later and a more specific-EID is returned,
> then that can correspond to the shorter route.
> 

There may be a difference in delay between the direct path and the dogleg
path, yes. But, the dogleg path follows the same path as the BGP peerings
between ASBRs which are maintained by TCP keepalives. So, the dogleg
path is high-assurance while the direct path can fail either on first-use or
at some later time - both result in packet loss.

> I don’t know what the definition of “explicit tunnels” means. You should define it. But LISP has dynamically encapsulated packets.

The term "explicit tunnels" comes directly from the title of RFC2529. It is
not used again in the document, but the document says that "configured
tunnels" are not needed where I take "configured" and "explicit" to mean
the same thing.
  
> There is no “tunnel setup” in the protocol.

Right there is no point-to-point tunnel setup in the traditional sense, but
the tunnel is NBMA with a single ingress and with multiple egress points
that are dynamically discovered. Just like neighbors on an NBMA link.

Thanks - Fred

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 20 Jun 2017 11:57:45 -0700

> you would really rather do ITR->ETR.

You put the RTR in the core so its on path for most flows.

> classes over the direct path, i.e., multipath.

That can be done with LISP where the RTR sends an SMR (Solicit-Map-Request or a Map-Notify) to the ITR to get the specific mapping so it can go direct.

Dino

---

From: "Templin, Fred L" <[email-masked]>
Date: Tue, 20 Jun 2017 19:57:54 +0000

Hi Dino,

> Sent: Tuesday, June 20, 2017 11:58 AM

This is the same thing as a Redirect in the terminology of IP links. The
automatic tunneling used by LISP and AERO can be accurately modeled
by  the NBMA link model.

The difference is that AERO openly acknowledges the NBMA link
nature whereas LISP avoids it (there is no reference to "link" or
"interface" wrt tunneling in the LISP specs that I could find).

Thanks - Fred