1. Why this post?

Earlier in the series we built an L3VPN: VPRN (Nokia) and MPLS L3VPN (Huawei). The transport job was done by LDP. In the Segment Routing post we removed LDP from the core and let OSPF distribute the labels.

Now we put the two together: the same L3VPN, but no LDP. An SR tunnel will carry the customer packet.

This is a concept post. There are no commands, only the logic. In the next two posts we build the same thing on Nokia and on Huawei. Once the concept sits, those posts are only a translation into commands.

At a glance

An L3VPN does two separate jobs, and the two are independent. One is “which customer does this packet belong to?” (the VPN label, MP-BGP). The other is “how do I get to the far PE?” (the transport tunnel). When SR arrives, only the second job changes.

QuestionWho answers it?Does it change with SR?
Which customer, which VRF does the packet enter?The VPN label, distributed between PEs by MP-BGPNo
How do PEs learn the customer routes?MP-BGP (VPNv4) carries them. The RD makes the prefix unique, the RT picks which VRF imports itNo
Which path and which label to reach the far PE?The transport tunnel: an LDP LSP or an SR tunnelYes
Does the P router see the customer?No, it only looks at the top labelNo

There is also a trap, which I explain in detail in the middle of the post: which tunnels a VPN may use is a setting. Even when SR tunnels are ready in the core, if the VPN’s tunnel resolution setting doesn’t allow SR, the route isn’t used. BGP is up, OSPF is up, and there is no ping.

2. The two jobs of an L3VPN: a postal analogy

Imagine you send a parcel. The parcel carries two kinds of information:

  • A city sorting code: at the cargo hubs the parcel follows this code to the right city. The transit hubs only read this code, they know nothing about the contents or the recipient.
  • An apartment number: once the parcel reaches the city, the local courier looks at the apartment number and drops it at the right door. The transit hubs that read the city code never use this number.

An L3VPN works the same way. Two labels are pushed onto the packet:

PostL3VPNWho puts it there?
City sorting codeTransport labelThe tunnel mechanism: LDP or SR
Apartment numberVPN label (service label)MP-BGP, the value the ingress PE learns from the far PE

The transport label sits at the very top and the VPN label underneath. P routers only look at the top one. The bottom label only means something to the egress PE: “this packet belongs to VRF 100”.

A small subtlety: what the VPN label points to depends on the PE’s label allocation mode. Nokia’s default is one label per VRF, which is why this post says “VRF 100’s label”. Some platforms and settings allocate a label per prefix or per CE/next-hop. Then the label points to a narrower forwarding action. Either way it tells the egress PE “in which context to forward the packet”.

One point to note: who put the transport label there doesn’t concern the VPN. It can be LDP, SR or RSVP-TE. The VPN only says “give me a tunnel to reach the far PE”, and how the tunnel was built is not its business.

3. Where does SR fit into this picture?

Exactly at the source of the transport label. Compare the two cases:

Core with LDPCore with SR
Who produces the transport label?Every router, agreeing with its neighbor through LDPLabel = the next router’s SRGB base + the index of the target PE. OSPF tells everyone the index
Label towards PE2 (example)Changes hop by hop: 17, 42Every router in this lab has SRGB 20000, so it’s 20004 all the way (20000 + index 4)
VPN labelMP-BGP, the ingress PE learns itThe same
VRF, RD, RT, PE-CEThe same

The “20004 all the way” in the table is because this lab’s SRGBs are identical. The Prefix-SID index means the same thing across the network, but the numeric value of the MPLS label depends on each router’s SRGB. If the SRGBs differ, routers change the value when they swap the label. I showed this with a three-vendor lab in section 10 of the SR post.

So the control plane of the L3VPN (MP-BGP, RD, RT) doesn’t change at all. VRF and RT settings, PE-CE routing, all the same. The only change is this: which tunnel PE1 uses to reach the next-hop (4.4.4.4) it learned from BGP.

4. The journey of a packet

CE1 and CE2 are two sites of the same customer. A packet goes from CE1 to CE2’s LAN (192.168.2.1). Follow it step by step below. With the switch on the left you can pick two scenarios: first watch the working case, then the case where no tunnel is selected.

A VPN packet in an SR tunnel: two labels
VPN ANokiaNokiaNokiaVPN AIP

Four things to take from the working scenario:

  1. Two lookups happen. PE1 first searches the customer’s VRF for 192.168.2.0/24 (a BGP VPN route: next-hop 4.4.4.4, VPN label 100007). Then it makes a second lookup to reach 4.4.4.4: which tunnel exists for this next-hop? The answer: the SR tunnel, label 20004. The two labels come from here.
  2. P learned nothing. P only looked at the top label 20004 and knew it was a SID belonging to PE2. The VPN label or the customer’s IP meant nothing to P.
  3. PE2 decides what happens at the last hop. Depending on the Prefix-SID’s PHP behavior, the second-to-last router either pops the transport label or carries it all the way to the egress PE. Nokia SR OS always advertises its own SIDs with “don’t pop” (the NP-flag). So in this Nokia lab P doesn’t pop; it hands 20004 to PE2. PE2 first terminates its own SID label, then looks at the VPN label. On Cisco and Huawei the default is usually PHP: there P pops the transport label and only the VPN label reaches PE2. The outcome is the same either way; only the label stack on the last link differs.
  4. The VPN label is constant end to end. PE2 advertised 100007 over BGP and PE1 uses it as is. The transport label can change hop by hop (with PHP it is even popped at the last hop), but the VPN label travels under the packet all the way to PE2.

5. “The route is there but there’s no ping”: the most common trap

The second scenario is the real lesson of this post. The situation:

  • OSPF is up, SR tunnels show in the tunnel table.
  • The MP-BGP VPNv4 session is Established.
  • The VRFs are right, RD and RT match.
  • But CE1 can’t ping CE2 and PE1’s VRF table has no remote LAN.

No error message, no neighbor went down. Why?

The BGP route reached PE1, but for a BGP route to be usable its next-hop has to resolve. It’s like getting an address (“go to 4.4.4.4 and find VRF 100 there”) without knowing how to get there. The VPN route points at PE2’s loopback as the next-hop. PE1 has to resolve that next-hop to a suitable transport tunnel.

Which tunnels count as suitable depends on the VPN’s tunnel resolution setting, and that varies by vendor:

  • On Huawei and Cisco the default picks the LDP LSP. That’s why it works in an LDP core without writing anything. Moving to SR doesn’t change that default by itself.
  • On Nokia a VPRN has no default tunnel resolution: even in an LDP core you have to write auto-bind-tunnel (we wrote it for LDP too in the VPRN post). If resolution any was configured earlier, the service can move to a suitable SR tunnel by itself once LDP is gone. If resolution filter allows only ldp, it won’t.

In this lab the tunnel resolution setting is left out on purpose. That’s why the VPN route can’t become active.

The symptom is always the same: BGP up, tunnel present, no remote route in the VRF. If you look in the wrong places it can take hours (“is something wrong with BGP?”, “is the RT wrong?”). The right question is: which tunnel does the next-hop of the VPN route resolve to?

How does each vendor select the tunnel?

How you tell itNote
Nokia SR OSauto-bind-tunnel inside the VPRN: resolution any or resolution filter + resolution-filter sr-ospfAccording to Nokia’s documentation, SR tunnels are used in VPRN and EVPN through auto-bind-tunnel. The default resolution value does no tunnel resolution, so with no setting the VPN route can’t be used. With LDP present, any prefers LDP (Pref 9 < 10)
Huawei VRPA tunnel policy: tunnel select-seq sr-lsp inside tunnel-policy, attached to the VPN instance with tnl-policyThe default policy picks LSPs. Huawei’s own SR-MPLS BE L3VPN example uses tunnel-prefer segment-routing under the global segment-routing instead of a policy. Both exist, depending on platform and release
Cisco IOS-XEDefault: if there is an LDP label, LDP is preferred. To prefer SR: set-attributes and sr-label-preferred under segment-routing mplsThere is no lab for Cisco in this post. The information comes from the note in the SR post

6. What happens when migrating from LDP to SR?

In a real network you don’t say “turn LDP off tonight”. The migration happens in stages:

  1. Build SR next to LDP. On Nokia two tunnels show up for the same destination: LDP and SR-OSPF. In the tunnel table the lower Pref wins (in the SR post’s example LDP is 9, SR-OSPF is 10). So just because you built SR, the VPN doesn’t move to it.
  2. Switch the service to SR. On Nokia with resolution filter + sr-ospf in the VPRN, on Huawei with sr-lsp in the tunnel policy. Now the service uses SR and LDP is still on.
  3. Test. Ping, traffic, the label stack. In the lab, if something breaks, going back to LDP is one line. A real operator network has hundreds of VPRNs, different forwarding policies and service dependencies. There, rolling back is a job that needs planning just like the migration.
  4. Turn LDP off. This is the last step. Before you do it, verify that every service resolves to SR tunnels.

The critical point: before LDP is removed, you have to verify that the VPN services can resolve their next-hops over SR tunnels. Otherwise some VPN routes won’t resolve and you lose reachability. For example, on Nokia, if the tunnel resolution allows only ldp, the service goes down. With resolution any it can move to a suitable SR tunnel by itself. Step 2 is a separate step to remove that uncertainty: you choose which tunnel the service uses.

7. FAQ

Does the VPN label change with SR? No. MP-BGP provides the VPN label and PE2 picks it according to its label allocation mode (per VRF by default on Nokia). It was the same in the LDP core.

Does the P router have to know SR? Yes. P must also run SR in OSPF and know the SIDs, because it reads the top transport label. That’s why in the last task of the lab you turn SR off on P and see that the VPN goes down while BGP stays up.

What if SR isn’t on in the PEs? The PEs must also speak SR in the IGP: so the ingress PE can push the label, it needs to know its neighbor’s SRGB and the target’s SID.

Does the customer have to know SR or MPLS? No. The CE speaks plain IP. Labels only travel from PE to PE.

Is there a separate label class for VPNs in SR? No. The VPN label is the classic service label distributed by BGP. SR doesn’t change it.

Can the label stack be more than 2 labels? Yes. With SR-TE the ingress PE can push a list of labels (for waypoints, for example). Then there are several transport labels above the VPN label. In the simple case of this post there is a single transport label. SR-TE is the subject of a separate post.

8. Try it yourself

The same scenario is ready on the Simulation page (Turkish UI) and in the Labs section (Turkish):

In both, the core runs SR (no LDP) and the L3VPN is ready. But the VPN’s tunnel is not selected: first you find out why the ping doesn’t go through, then you fix it and bring the VPN up. In the end you turn SR off on P and see what happens to the VPN when the tunnel disappears.

For the step-by-step commands, see the next posts: Nokia and Huawei.

9. Summary

  • An L3VPN has two jobs: which customer (VPN label, MP-BGP) and how to reach the far PE (transport tunnel). The two are independent.
  • With SR only the source of the transport tunnel changes: OSPF distributes the label instead of LDP. VRF, RD, RT and MP-BGP stay the same.
  • PE1 makes two lookups: first the customer route in the VRF (VPN label), then the tunnel for the next-hop (transport label). Two labels are pushed: transport on top, VPN underneath.
  • P only looks at the top label. Whether the label is popped at the last hop depends on how PE2 advertises its SID: Nokia doesn’t ask for PHP (P carries the label to PE2), on Cisco and Huawei the default is usually PHP. PE2 finds the right VRF with the VPN label.
  • The trap: BGP up, SR tunnel present, but if the VPN’s tunnel resolution setting doesn’t allow SR there is no remote route in the VRF. On Nokia auto-bind-tunnel, on Huawei the tunnel policy decides it.
  • When migrating, first verify that the services resolve to SR tunnels, then turn LDP off.

For the whole series, see MPLS from Scratch (Turkish).

Sources