SDWAN: Application-Aware Routing

Overview

This feature ensures traffic to use the best path depending on the business requirements such as mission-critical and delay-sensitive applications. It has the ability to dynamically steer the traffic to different circuits depending on the circuit's condition which are packet loss, latency, and jitter. These are the SLA parameters that have to be set as it influences the traffic flow from the service side (which is from the local side of the vEdge) to what circuit certain traffic will prefer. It considers these factors in path selection and is possible with the use of BFD probes which are sent every second by default and performance is recorded by the vEdge.

The Application-Aware routing policy affects only traffic that is flowing from the service side to the tunnel side (WAN) of the vEdge router. This is a type of centralized data policy which is configured in the vManage and once the policy is activated it pushes it to the vSmart(s) and pushes again the policy to the vEdge routers that are configured to learn this policy.

I've created a lab that shows how this feature works.

In the scenario below, the controllers are cloud-based, and there are 3 sites: Site 100 and Site 200, and Site 10 (controllers). Sites 100 and 200 have a pair of vEdges and connected to a Layer3 switch. Two loopbacks have been configured on the Layer3 switch of both sites to test the traffic flows.

The objective is for the traffic from Site 100 to Site 200 to prefer a circuit based from the following requirements.

• FTP traffic to 172.16.21.0/24 - Use MPLS circuit
• HTTP/HTTPS traffic to 172.16.21.0/24 - Use Internet circuit
• SQL traffic to 172.16.21.0/24- Use MPLS circuit
• All other types of traffic to 172.16.21.0/24 and 172.16.22.0/24 should continue to use the default behavior which is full-mesh connectivity.

Topology

Summary Steps

1. Define the Groups of Interest.
2. Configure the App-Aware Policy.
3. Apply to Site(s).
4. Pre-verification of traffic flows.
5. Activate the App-Aware Policy.
6. Post-verification of traffic flows.

Define the Groups of Interest

Navigate to Configuration > Policies > Centralized Policy > Add Policy.

Define the Groups of Interest. Click Application and New Application List to define which applications will be matched in the App-Aware Policy. Select the Applications to be matched and set an Application List Name. Click Add.

Create a Site List for Site 100 and click Add. (A range of sites can also be defined and/or separated by commas e.g. 100-200, or 100,101,102.)

Create the Service VPN and click Add. (A range of sites can also be defined and/or separated by commas.)

Create the Data Prefix List and click Add. 172.16.21.0/24 will be set later as the destination prefix on the App-Aware Policy configuration. As you can see in the left-hand corner, there is also an option that says "Prefix". The difference between the two is that "Data Prefix List" is used for Traffic Data Policies (Data Plane). Configuring App-Aware Policy is on the Data Plane. While "Prefix" is used for manipulating routes in the Control Plane which we will not perform in this scenario.

Define the SLA. Enter a SLA Class List Name, and then the Packet Loss, Latency and Jitter. Click Add.

Now that we have done the Groups of Interest, next is to configure the App-Aware Policy. Click Next.

This will take you to the Configure Topology and VPN Membership page. Routes will not be manipulated in this scenario so we will skip this part. Click Next.

Configure the App-Aware Policy

In this page "Configure Traffic Rules" is where we will configure the App-Aware Policy. Under Application Aware Routing, click Add Policy and Create New.

Click Sequence Type and Sequence Rule.

The Applications, Data Prefix List, and SLA Class that were defined earlier will be matched in this policy. And for simplicity the same SLA Class is used. Also set the Policy Name and Description as required. Take note that the Application/Application Family List cannot be combined under a single sequence, so 3 different sequences have been created.

Application/Application Family List: FTP

Destination Data Prefix List: DATAPL-172-16-21-0
SLA Class: SLA

Preferred Color: mpls

Application/Application Family List: HTTP-HTTPS

Destination Data Prefix List: DATAPL-172-16-21-0
SLA Class: SLA

Preferred Color: biz-internet

Application/Application Family List: SQL

Destination Data Prefix List: DATAPL-172-16-21-0
SLA Class: SLA

Preferred Color: mpls

Leave the Default Action to its default setting.

Here's what it looks like after configuration. And click Save Application Aware Routing Policy.

The Application Aware Routing Policy has been created. Click Next.

Apply to Site(s)

This is where you will define what Sites will be using the policy. Enter a Policy Name and Policy Description, and click Application-Aware Routing. Since we have not touched the other policies there is no need to define anything under the Topology, Traffic Data, and Cflowd. 

Click New Site List and VPN List. Select SITE100 and VPN1, and click Add. Imagine if you have hundreds or thousands of sites that will use this same policy, you can define Site List objects and select it under this option. This can also be applied to multiple Service VPNs. Multiple objects of Site List and VPN List can be added.

You may click Preview to see what the configuration looks like which will be pushed by vManage to the vSmart(s). vSmart(s) will then push this policy to the vEdge(s) that have been defined. Click Save Policy to proceed to the next page.

This takes you to the main page of Centralized Policy.

But before we activate the policy, let's first verify the traffic flows. Navigate to Monitor > Network. And select vEdge1/vEdge2 > Troubleshooting > Simulate Flows. Both are on the same site - Site 100.

Pre-verification of traffic flows

Verify traffic to Site 200 for prefix 172.16.21.0/24. For simplicity let’s just use vEdge1, but it has the same output for vEdge2. As shown in the screenshots below, all traffic going to Site 200 (for both prefixes 172.16.21.0/24 and 172.16.22.0/24) uses both MPLS and Internet circuits. Later we will see the difference after we activate the App-Aware Policy.

 Select VPN1 and the VPN1 source interface and enter the Destination IP. Leave the Application blank and click Simulate.

Normal traffic

For other types of traffic, enter or select the Application in the drop-down menu.

FTP

HTTP/HTTPS

SQL

Traffic to 172.16.22.0/24

Activate the App-Aware Policy

Return to Configuration > Policies and activate the policy. This will be pushed by vManage to the vSmart(s) and then from vSmart(s) to the vEdge(s).

Post-verification of traffic flows

Navigate back again to Monitor > Network > vEdge/vEdge2 > Troubleshooting > Simulate Flows.

Normal traffic is the same as expected. Next, we will see the traffic flow for the different applications.

FTP traffic to 172.16.21.0/24 prefers the MPLS circuit.

If compared to FTP traffic to the other prefix 172.16.22.0/24, it uses both circuits.

HTTP/HTTPS traffic to 172.16.21.0/24 prefers the Internet circuit.

SQL traffic to 172.16.21.0/24 prefers the MPLS circuit.

LAB: How to resolve common OSPFv2/OSPFv3 peering issues (PART 2)

This is the continuation of my previous post. Click here if you haven't seen it yet.

You may download the UNL file from this link.
RS-TSHOOT-OSPF (END STATE)
Note: Do not extract the ZIP file. Import the ZIP file in your EVE-NG lab.

Links:
R1 - R2: Incorrect network statement
R1 - R3: Router ID duplicate
R1 - R4: OSPFv3 - Incorrect OSPFv3 process ID
R1 - R5: Hello and Dead interval mismatch
R1 - R6: OSPF Network Type
R1 - R7: OSPFv3 - Missing Stub flag
R1 - R8: Passive Interface
R1 - R9: OSPFv3 - MTU mismatch
R1 - R10: OSPFv3 - Incorrect password in Key Chain
R1 - R11: Authentication mode mismatch
R1 - R12: OSPFv3 - Distribute list
R1 - R13: OSPFv3 - IPv6

Below are the details in resolving the OSPF issues. Hope this helps!


R1 - R2: Incorrect network statement
R1 can peer with R2, but cannot ping R2's Loopback.

R1#sh ip ospf neighbor

Neighbor ID     Pri   State           Dead Time   Address         Interface
2.2.2.2           1   FULL/DR         00:00:38    10.1.12.2       GigabitEthernet0/0

R1#sh ip route 2.2.2.2      
% Network not in table

R1#ping 2.2.2.2 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
.....
Success rate is 0 percent (0/5)

 On R2, Loopback0 has not been enabled with OSPF.

R2#sh ip ospf interface brief
Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Gi0/0        10    2               10.1.12.2/30       1     DR    1/1

R2 is not advertising the correct IP. R2's Lo0 is 2.2.2.2.

R2#sh run | sec router
router ospf 10
 router-id 2.2.2.2
 network 2.2.2.3 0.0.0.0 area 2
 network 10.1.12.0 0.0.0.3 area 2

Corrected the network statement

R2(config)#router ospf 10
R2(config-router)#no  network 2.2.2.3 0.0.0.0 area 2
R2(config-router)#network 2.2.2.2 0.0.0.0 area 2

R2's Lo0 is now enabled with OSPF.

R2#sh ip ospf interface brief
Interface    PID   Area            IP Address/Mask    Cost  State Nbrs F/C
Lo0          10    2               2.2.2.2/32         1     LOOP  0/0
Gi0/0        10    2               10.1.12.2/30       1     DR    1/1

R1 and R2 Loopbacks are now reachable.

R1#ping 2.2.2.2 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 3/5/9 ms

 R2#ping 1.1.1.1 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 2.2.2.2
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 5/8/13 ms
R2#

R1 - R3: Router ID duplicate
Logs on R1 and R3 indicate a duplicate router ID.

R1#
*Apr 19 04:56:33.222: %OSPF-4-DUP_RTRID_NBR: OSPF detected duplicate router-id 1.1.1.1 from 10.1.13.2 on interface GigabitEthernet0/1

R3#
*Apr 19 04:48:31.277: %OSPF-4-DUP_RTRID_NBR: OSPF detected duplicate router-id 1.1.1.1 from 10.1.13.1 on interface GigabitEthernet0/0

The router ID configured on R3 is 1.1.1.1. It should be 3.3.3.3.

R3#sh run | sec router
router ospf 10
 router-id 1.1.1.1
 network 3.3.3.3 0.0.0.0 area 3
 network 10.1.13.0 0.0.0.3 area 3

Fixed the router ID and the loopbacks are now reachable.

R3(config)#router ospf 10
R3(config-router)#router-id 3.3.3.3
R3(config-router)#
*Apr 19 04:59:05.533: %OSPF-5-ADJCHG: Process 10, Nbr 1.1.1.1 on GigabitEthernet0/0 from LOADING to FULL, Loading Done

R1#ping 3.3.3.3 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 5/6/12 ms

R3#ping 1.1.1.1 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 3.3.3.3
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 6/7/9 ms

R1 - R4: OSPFv3 - Incorrect OSPFv3 process ID
R4's Lo0 is enabled with the incorrect OSPFv3 process ID. It should be process ID 100 instead of 1000.

R4#sh ospfv3 int brief
Interface    PID   Area            AF         Cost  State Nbrs F/C
Gi0/0        100   4               ipv4       1     DR    1/1
Lo0          1000  4               ipv4       1     LOOP  0/0

R4#sh run | sec router
router ospfv3 1000
 !
 address-family ipv4 unicast
 exit-address-family
router ospfv3 100
 router-id 4.4.4.4
 !
 address-family ipv4 unicast
 exit-address-family
 !
 address-family ipv6 unicast
 exit-address-family

Removed OSPFv3 process ID 1000 and reconfigured R4's Lo0 to enable OSPFv3.

R4(config)#no router ospfv3 1000
R4(config)#int lo0
R4(config-if)#ipv6 enable
R4(config-if)#ospfv3 100 ipv4 area 4
R4(config-if)#do sh run | sec router
router ospfv3 100
 router-id 4.4.4.4
 !
 address-family ipv4 unicast
 exit-address-family
 !
 address-family ipv6 unicast
 exit-address-family

The loopbacks are now reachable.

R1#ping 4.4.4.4 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/5/8 ms

R4#ping 1.1.1.
*Apr 19 05:15:55.848: %SYS-5-CONFIG_I: Configured from console by console1
R4#ping 1.1.1.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 4.4.4.4
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 6/7/12 ms

R1 - R5: Hello and Dead interval mismatch
Hello and Dead Interval on R5 does not match with R1.

R1#sh ip ospf int g0/3
GigabitEthernet0/3 is up, line protocol is up
  Internet Address 10.1.15.1/30, Area 5, Attached via Interface Enable
  Process ID 10, Router ID 1.1.1.1, Network Type BROADCAST, Cost: 1
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           1         no          no            Base
  Enabled by interface config, including secondary ip addresses
  Transmit Delay is 1 sec, State DR, Priority 1
  Designated Router (ID) 1.1.1.1, Interface address 10.1.15.1
  No backup designated router on this network
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5

R5#sh ip ospf int | begin Gig
GigabitEthernet0/0 is up, line protocol is up
  Internet Address 10.1.15.2/30, Area 5, Attached via Network Statement
  Process ID 10, Router ID 5.5.5.5, Network Type BROADCAST, Cost: 1
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           1         no          no            Base
  Transmit Delay is 1 sec, State DR, Priority 1
  Designated Router (ID) 5.5.5.5, Interface address 10.1.15.2
  No backup designated router on this network
  Timer intervals configured, Hello 9, Dead 41, Wait 41, Retransmit 5
    oob-resync timeout 41
    Hello due in 00:00:01
  Supports Link-local Signaling (LLS)
  Cisco NSF helper support enabled
  IETF NSF helper support enabled
  Index 1/2/2, flood queue length 0
  Next 0x0(0)/0x0(0)/0x0(0)
  Last flood scan length is 0, maximum is 0
  Last flood scan time is 0 msec, maximum is 0 msec
  Neighbor Count is 0, Adjacent neighbor count is 0
  Suppress hello for 0 neighbor(s)

R5#sh run int g0/0
Building configuration...

Current configuration : 168 bytes
!
interface GigabitEthernet0/0
 ip address 10.1.15.2 255.255.255.252
 ip ospf dead-interval 41
 ip ospf hello-interval 9
 duplex auto
 speed auto
 media-type rj45
end

Remove the OSPF hello and dead interval commands in R5 so that it uses the default timers to match with R1. The OSPF network type of both links is Broadcast.

R5(config)#int g0/0
R5(config-if)#no ip ospf dead-interval 41
R5(config-if)#no ip ospf hello-interval 9
R5(config-if)#do sh ip ospf int g0/0
GigabitEthernet0/0 is up, line protocol is up
  Internet Address 10.1.15.2/30, Area 5, Attached via Network Statement
  Process ID 10, Router ID 5.5.5.5, Network Type BROADCAST, Cost: 1
  Topology-MTID    Cost    Disabled    Shutdown      Topology Name
        0           1         no          no            Base
  Transmit Delay is 1 sec, State DR, Priority 1
  Designated Router (ID) 5.5.5.5, Interface address 10.1.15.2
  No backup designated router on this network
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5

The loopbacks are now reachable.

R1#ping 5.5.5.5 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 5.5.5.5, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/5/10 ms

R5#ping 1.1.1.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 5.5.5.5
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 6/6/8 ms

R1 - R6: OSPF Network Type
R1 and R6 OSPF network types are not compatible.

R1#sh run int g0/4
Building configuration...

Current configuration : 172 bytes
!
interface GigabitEthernet0/4
 ip address 10.1.16.1 255.255.255.252
 ip ospf network point-to-multipoint
 ip ospf 10 area 6
 duplex auto
 speed auto
 media-type rj45

R6#sh run int g0/0
Building configuration...

Current configuration : 148 bytes
!
interface GigabitEthernet0/0
 ip address 10.1.16.2 255.255.255.252
 ip ospf network point-to-point
 duplex auto
 speed auto
 media-type rj45

Changed R1's OSPF network type to point-to-point so that it matches R6. Loopbacks are now reachable.

R1(config)#int g0/4
R1(config-if)#ip ospf network point-to-point
*Apr 19 05:33:08.942: %OSPF-5-ADJCHG: Process 10, Nbr 6.6.6.6 on GigabitEthernet0/4 from LOADING to FULL, Loading Done

R1#ping 6.6.6.6 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 5/5/7 ms

R6#ping 1.1.1.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 6.6.6.6
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/9/14 ms

R1 - R7: OSPFv3 - Missing Stub flag
Missing stub configuration on R7.

R1#sh run | sec router
router ospfv3 100
 router-id 1.1.1.1
 !
 address-family ipv4 unicast
  area 7 stub
  area 10 authentication key-chain KEYS
 exit-address-family
 !
 address-family ipv6 unicast
 exit-address-family

R7#sh run | sec router
router ospfv3 100
 router-id 7.7.7.7
 !
 address-family ipv4 unicast
 exit-address-family
 !
 address-family ipv6 unicast
 exit-address-family

Applied stub configuration in R7. Loopbacks are now reachable.

R7(config-router-af)#router ospfv3 100
R7(config-router)#address-family ipv4 unicast
R7(config-router-af)#area 7 stub
*Apr 19 05:41:30.312: %OSPFv3-5-ADJCHG: Process 100, IPv4, Nbr 1.1.1.1 on GigabitEthernet0/0 from LOADING to FULL, Loading Done

R7(config-if)#do sh ospfv3 neighbor

          OSPFv3 100 address-family ipv4 (router-id 7.7.7.7)

Neighbor ID     Pri   State           Dead Time   Interface ID    Interface
1.1.1.1           1   FULL/DR         00:00:38    7               GigabitEthernet0/0

R1#ping 7.7.7.7 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 7.7.7.7, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/4/8 ms

R7#ping 1.1.1.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 7.7.7.7
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 7/9/15 ms

R1 - R8: Passive Interface
Passive-interface default is configured in R8. So the link between R1 and R8, and R8's Lo0 is not active in OSPF. R1's interface(Gi0/6) towards R7 is also in passive interface.

R8#sh ip protocols | begin Passive
  Passive Interface(s):
    GigabitEthernet0/0
    GigabitEthernet0/1
    GigabitEthernet0/2
    GigabitEthernet0/3
    Loopback0
    RG-AR-IF-INPUT1
  Routing Information Sources:
    Gateway         Distance      Last Update
  Distance: (default is 110)

R8#sh run | sec router
router ospf 10
 router-id 8.8.8.8
 passive-interface default
 network 8.8.8.8 0.0.0.0 area 8
 network 10.1.18.0 0.0.0.3 area 8

Enabled OSPF in R8's Gi0/0 and Lo0, and R1's Gi0/6. Loopbacks are now reachable.

R1(config)#router ospf 10
R1(config-router)#no passive-interface g0/6

R8(config)#router ospf 10
R8(config-router)#no pass
R8(config-router)#no passive-interface g0/0
R8(config-router)#no passive-interface lo0
*Apr 19 05:49:01.233: %OSPF-5-ADJCHG: Process 10, Nbr 1.1.1.1 on GigabitEthernet0/0 from LOADING to FULL, Loading Done

R1#ping 8.8.8.8 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/3/7 ms

R8#ping 1.1.1.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 8.8.8.8
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 7/8/11 ms

R1 - R9: OSPFv3 - MTU mismatch
The MTU of R1's interface (Gi0/7) towards R9 is 1499 as opposed to R9 that is 1500(default).

R1#sh run int g0/7
Building configuration...

Current configuration : 166 bytes
!
interface GigabitEthernet0/7
 ip address 10.1.19.1 255.255.255.252
 ip mtu 1499
 duplex auto
 speed auto
 media-type rj45
 ipv6 enable
 ospfv3 100 ipv4 area 9
end

R1#sh ospfv3 neighbor

          OSPFv3 100 address-family ipv4 (router-id 1.1.1.1)

Neighbor ID     Pri   State           Dead Time   Interface ID    Interface
4.4.4.4           1   FULL/DR         00:00:37    2               GigabitEthernet0/2
7.7.7.7           1   FULL/BDR        00:00:34    2               GigabitEthernet0/5
9.9.9.9           1   EXSTART/DR      00:00:36    2               GigabitEthernet0/7
R1#
R1#
R1#
R1#ping 9.9.9.9 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 9.9.9.9, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
.
*Apr 19 05:52:43.049: %OSPFv3-5-ADJCHG: Process 100, IPv4, Nbr 9.9.9.9 on GigabitEthernet0/7 from EXSTART to DOWN, Neighbor Down: Too many retransmissions....
Success rate is 0 percent (0/5)

Removed MTU command in R1 to match R9's MTU. Loopbacks are now reachable.

R1(config)#int g0/7
R1(config-if)#no  ip mtu 1499
*Apr 19 05:55:03.975: %OSPFv3-5-ADJCHG: Process 100, IPv4, Nbr 9.9.9.9 on GigabitEthernet0/7 from LOADING to FULL, Loading Done
R1#ping 9.9.9.9
*Apr 19 05:55:09.221: %SYS-5-CONFIG_I: Configured from console by consoles
R1#ping 9.9.9.9 so lo0
R1#ping 9.9.9.9 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 9.9.9.9, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/5/10 ms

R9#ping 1.1.1.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 9.9.9.9
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/8/10 ms

R1 - R10: OSPFv3 - Incorrect password in Key Chain
The password in the key chain does not match. R10's password was misconfigured as there's a whitespace after the password.

R1#sh run | sec router
router ospfv3 100
 router-id 1.1.1.1
 !
 address-family ipv4 unicast
  area 7 stub
  area 10 authentication key-chain KEYS

R10#sh run | sec router
router ospfv3 100
 router-id 10.10.10.10
 !
 address-family ipv4 unicast
  area 10 authentication key-chain KEYS
 exit-address-family

R1#show key chain
Key-chain KEYS:
    key 1 -- text "cisco"
        accept lifetime (always valid) - (always valid) [valid now]
        send lifetime (always valid) - (always valid) [valid now]

R10#show key chain
Key-chain KEYS:
    key 1 -- text "cisco "   (there's a whitespace)
        accept lifetime (always valid) - (always valid) [valid now]
        send lifetime (always valid) - (always valid) [valid now]

Reconfigured with the correct password in R10. Loopbacks are now reachable.

R10(config)#key chain KEYS
R10(config-keychain)# key 1
R10(config-keychain-key)#  key-string cisco

R1#ping 10.10.10.10 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.10, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 3/6/10 ms

R10#ping 1.1.1.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 10.10.10.10
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/9/11 ms

R1 - R11: Authentication mode mismatch
The authentication mode in R1's Gi0/9 interface towards R11 is not configured with the same mode in R11.

R1#sh run int g0/9
Building configuration...

Current configuration : 200 bytes
!
interface GigabitEthernet0/9
 ip address 10.1.21.1 255.255.255.252
 ip ospf authentication
 ip ospf message-digest-key 1 md5 cisco
 ip ospf 10 area 11
 duplex auto
 speed auto
 media-type rj45

R11#sh run int g0/0
Building configuration...

Current configuration : 210 bytes
!
interface GigabitEthernet0/0
 ip address 10.1.21.2 255.255.255.252
 ip ospf authentication message-digest
 ip ospf message-digest-key 1 md5 cisco
 duplex auto
 speed auto
 media-type rj45
 no cdp enable

Updated R1's Gi0/9 authentication mode

R1(config)#int g0/9
R1(config-if)#ip ospf authentication message-digest
*Apr 19 06:13:12.785: %OSPF-5-ADJCHG: Process 10, Nbr 11.11.11.11 on GigabitEthernet0/9 from LOADING to FULL, Loading Done

R1#ping 11.11.11.11 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 11.11.11.11, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 3/5/9 ms

R11#ping 1.1.1.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 11.11.11.11
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 7/9/14 ms

R1 - R12: OSPFv3 - Distribute list
The distribute-list configured in R12 is causing connectivity issues between R1 and R12 loopbacks.

R12#sh run | sec router
router ospfv3 100
 router-id 12.12.12.12
 !
 address-family ipv4 unicast
  distribute-list 1 in
!
R12#sh access-lists  1
Standard IP access list 1
    10 deny   1.1.1.1 (1 match)
    20 permit any (8 matches)

Removed the distribute-list. Loopbacks are now reachable.

R12(config)#router ospfv3 100
R12(config-router)#address-family ipv4 unicast
R12(config-router-af)#no distribute-list 1 in
R12(config-router-af)#do sh ospfv3 neighbor

          OSPFv3 100 address-family ipv4 (router-id 12.12.12.12)

Neighbor ID     Pri   State           Dead Time   Interface ID    Interface
1.1.1.1           1   FULL/BDR        00:00:31    12              GigabitEthernet0/0

R1#ping 12.12.12.12 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 12.12.12.12, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/4/11 ms

R12#ping 1.1.1.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 12.12.12.12
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/9/10 ms

R1 - R13: OSPFv3 - IPv6
OSPFv3 for IPv4 and IPv6 is not enabled in R1's Gi0/11. Network statement for OSPFv2 is not needed in R1. Incorrect OSPFv3 process ID for IPv6 in R13's Lo0. The process ID used should be 100.

R1#sh run int g0/11
Building configuration...

Current configuration : 117 bytes
!
interface GigabitEthernet0/11
 ip address 10.1.23.1 255.255.255.252
 duplex auto
 speed auto
 media-type rj45

router ospf 10
 router-id 1.1.1.1
 network 1.1.1.1 0.0.0.0 area 0
 network 10.1.13.0 0.0.0.3 area 3
 network 10.1.18.0 0.0.0.3 area 8
 network 10.1.23.0 0.0.0.3 area 13   (not needed as OSPFv3 is used for peering)

R13# sh run int lo0
Building configuration...

Current configuration : 149 bytes
!
interface Loopback0
 ip address 13.13.13.13 255.255.255.255
 ipv6 address 2001:DB8:13::13/64
 ospfv3 100 ipv4 area 13
 ospfv3 10 ipv6 area 13

Enabled OSPFv3 for both IPv4 and IPv6 in R1's Gi0/11 interface, and removed the unnecessary network statement in R1 under OSPFv2. Also fixed the OSPFv3 process ID for IPv6 in R13.

R1(config)#router ospf 10
R1(config-router)#no  network 10.1.23.0 0.0.0.3 area 13
R1(config)#int g0/11
R1(config-if)#ipv6 enable
R1(config-if)#ospfv3 100 ipv4 area 13
R1(config-if)#ospfv3 100 ipv6 area 13
*Apr 19 06:24:32.915: %OSPFv3-5-ADJCHG: Process 100, IPv4, Nbr 13.13.13.13 on GigabitEthernet0/11 from LOADING to FULL, Loading Done
*Apr 19 06:26:43.066: %OSPFv3-5-ADJCHG: Process 100, IPv6, Nbr 13.13.13.13 on GigabitEthernet0/11 from LOADING to FULL, Loading Done

R13(config)#int lo0
R13(config-if)#no ospfv3 10 ipv6 area 13
R13(config-if)#ospfv3 100 ipv6 area 13

Loopbacks are now reachable.

R1#ping 13.13.13.13 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 13.13.13.13, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/3/5 ms

R1#ping 2001:DB8:13::13   
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:DB8:13::13, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/3 ms

R1#sh ipv6 route ospf
IPv6 Routing Table - default - 4 entries
Codes: C - Connected, L - Local, S - Static, U - Per-user Static route
       B - BGP, HA - Home Agent, MR - Mobile Router, R - RIP
       H - NHRP, I1 - ISIS L1, I2 - ISIS L2, IA - ISIS interarea
       IS - ISIS summary, D - EIGRP, EX - EIGRP external, NM - NEMO
       ND - ND Default, NDp - ND Prefix, DCE - Destination, NDr - Redirect
       RL - RPL, O - OSPF Intra, OI - OSPF Inter, OE1 - OSPF ext 1
       OE2 - OSPF ext 2, ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2
       la - LISP alt, lr - LISP site-registrations, ld - LISP dyn-eid
       a - Application
O   2001:DB8:13::13/128 [110/1]
     via FE80::5200:FF:FE0D:0, GigabitEthernet0/11

R13#ping 1.1.1.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 13.13.13.13
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 7/9/14 ms

R13#ping 2001:DB8:1::1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:DB8:1::1, timeout is 2 seconds:
Packet sent with a source address of 2001:DB8:13::13
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 6/7/11 ms

FINAL VALIDATION:
R1 is able to learn all the Loopbacks from its peers via OSPFv2/OSPFv3.

R1#sh ip route ospf | b Gate
Gateway of last resort is not set

      2.0.0.0/32 is subnetted, 1 subnets
O        2.2.2.2 [110/2] via 10.1.12.2, 00:13:12, GigabitEthernet0/0
      3.0.0.0/32 is subnetted, 1 subnets
O        3.3.3.3 [110/2] via 10.1.13.2, 00:13:12, GigabitEthernet0/1
      5.0.0.0/32 is subnetted, 1 subnets
O        5.5.5.5 [110/2] via 10.1.15.2, 00:13:12, GigabitEthernet0/3
      6.0.0.0/32 is subnetted, 1 subnets
O        6.6.6.6 [110/2] via 10.1.16.2, 00:13:12, GigabitEthernet0/4
      8.0.0.0/32 is subnetted, 1 subnets
O        8.8.8.8 [110/2] via 10.1.18.2, 00:13:12, GigabitEthernet0/6
      11.0.0.0/32 is subnetted, 1 subnets
O        11.11.11.11 [110/2] via 10.1.21.2, 00:13:12, GigabitEthernet0/9
R1#
R1#sh ip route ospfv3 | b Gate
Gateway of last resort is not set

      4.0.0.0/32 is subnetted, 1 subnets
O        4.4.4.4 [110/1] via 10.1.14.2, 00:10:01, GigabitEthernet0/2
      7.0.0.0/32 is subnetted, 1 subnets
O        7.7.7.7 [110/1] via 10.1.17.2, 00:10:01, GigabitEthernet0/5
      9.0.0.0/32 is subnetted, 1 subnets
O        9.9.9.9 [110/1] via 10.1.19.2, 00:10:01, GigabitEthernet0/7
      10.0.0.0/8 is variably subnetted, 25 subnets, 2 masks
O        10.10.10.10/32 [110/1] via 10.1.20.2, 00:10:01, GigabitEthernet0/8
      12.0.0.0/32 is subnetted, 1 subnets
O        12.12.12.12 [110/1] via 10.1.22.2, 00:10:01, GigabitEthernet0/10
      13.0.0.0/32 is subnetted, 1 subnets
O        13.13.13.13 [110/1] via 10.1.23.2, 00:10:01, GigabitEthernet0/11

R1(tcl)#foreach var {
+>(tcl)#1.1.1.1
+>(tcl)#2.2.2.2
+>(tcl)#3.3.3.3
+>(tcl)#4.4.4.4
+>(tcl)#5.5.5.5
+>(tcl)#6.6.6.6
+>(tcl)#7.7.7.7
+>(tcl)#8.8.8.8
+>(tcl)#9.9.9.9
+>(tcl)#10.10.10.10
+>(tcl)#11.11.11.11
+>(tcl)#12.12.12.12
+>(tcl)#13.13.13.13
+>(tcl)#2001:DB8:13::13
+>(tcl)#} { ping $var source lo0}
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/9 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 3/5/9 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/5/8 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/5/6 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 5.5.5.5, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/4/9 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.6.6.6, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 3/5/6 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 7.7.7.7, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/4/7 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 5/6/9 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 9.9.9.9, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/4/9 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.10, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 6/7/10 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 11.11.11.11, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 3/3/4 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 12.12.12.12, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/5/9 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 13.13.13.13, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 3/4/8 ms
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:DB8:13::13, timeout is 2 seconds:
Packet sent with a source address of 2001:DB8:1::1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/3 ms
R1(tcl)#

REFERENCE:
Below are some helpful commands:
ping x.x.x.x source Lo0
show ip ospf neighbor
show ip ospf interface brief
show ip route
show ip route ospfv3
show ipv6 interface brief
show ipv6 route
show ipv6 route ospf
show ipv6 ospf neighbor
show ospfv3 neighbor
show ospfv3 int brief
show ospfv3 int G0/0
show ospfv3 database

You may use this TCL script to verify connectivity from R1 to all the peer Loopbacks. Run the commands from the privilege prompt.

tclsh
foreach var {
1.1.1.1
2.2.2.2
3.3.3.3
4.4.4.4
5.5.5.5
6.6.6.6
7.7.7.7
8.8.8.8
9.9.9.9
10.10.10.10
11.11.11.11
12.12.12.12
13.13.13.13
2001:DB8:13::13
} { ping $var source lo0} 


If you haven't set up your own lab yet in GCP(Google Cloud Platform), refer to my previous post using the links below.
RS Lab Setup

HOW TO: Set-up EVE-NG via Google Cloud Platform (GCP) - PART 1