VMware NSX : DLR Routes LIFs Update on ESXi hosts

Le but de cet article est de montrer comment les Logical Interfaces (LIFs) au moment de leur configuration depuis la Logical Control VM (DLR VM) sont poussées via NSX Controllers au sein des ESXI hosts (data plane) de la transport Zone.

Topologie

4x Logicals Swithes déjà créer: APP, WEB, DB et Transit

La DLR Control VM est déjà déployée, mais non configurée (pas de LIFs configurée)

Le but est de créer les 3x Gateways (.254) pour les VXLAN APP, Web et DB et de voir comment celles-ci sont poussées sur les ESXi hosts

Le scénario ne sera joué que sur le Logical Switch Web

Vu des Logicals Switchs
Notez le VNI ID du Logical Switch Web = 5000

$C:\Users\zecevici\Desktop\NSX\blog_dlr\2017-06-26_10h59_45.jpg$

Vu de la DLR VM
Aucune Interface ou Logical Interfaces (LIFs) n’est configurée.
Par conséquent, aucune gateway (passerelle) n’est disponible au niveau des ESXi (data plane).

$C:\Users\zecevici\Desktop\NSX\blog_dlr\2017-06-26_10h49_18.jpg$

Test Ping depuis la VM web01
Étant donné que la gateway (10.150.2.254) n’est pas encore configurée, le ping ne fonctionne pas.

$C:\Users\zecevici\Desktop\NSX\blog_dlr\2017-06-26_10h55_58.jpg$

NSX Controller Logical router Information
Au niveau du NSX Controller, les commandes suivantes permettent d’avoir des informations sur la DLR VM.

nsx-controller # show control-cluster logical-routers instance all
LR-Id      LR-Name                                            Universal Service-Controller Egress-Locale                        In-Sync    Sync-Category
0x1388     default+edge-5                                     false     10.3.3.101         local                                No         NORMAL

nsx-controller #

LR-ID = 0x1388 = ID de la DLR VM
LR-Name = default+edge-5 = Nom de la DLR VM
Service-Controller = 10.3.3.101 = le NSX Controller en charge de la DLR VM

Aucune connexion vers la DLR VM n’est présente, car aucune LIFs n’est configurée pour le moment.

nsx-controller # show control-cluster logical-routers connections 0x1388

nsx-controller #

Netcpa.xml to Verbose
Le service netcpad est responsable de la communication TCPoSSL avec les NSX Controllers.
De ce fait, /usr/lib/vmware/netcpa/etc/netcpa.xml est le fichier d’information du service netcpad.
De base, ce service a un fichier de log /var/log/netcpa.log qui log les informations en mode “info”. Ici, nous allons demander que le service nous envoie plus de logs en passant le level en verbose.

   <outputToFiles>false</outputToFiles>
   <!-- log level is one of the 2 users configurable setting -->
   <!-- configurable levels : panic, error, warning, info, verbose, trivia -->
   <!-- log levels are listed in increasing amount of logging done -->
   <!-- use trivia to get most detailed logging for providing logs to VMware Support -->
   <maxFileSize>524288</maxFileSize>
   <maxFileNum>8</maxFileNum>
   <level>info</level>
   <outputToSyslog>true</outputToSyslog>
   <syslog>
I /usr/lib/vmware/netcpa/etc/netcpa.xml [Modified] 31/159 19%

   <outputToFiles>false</outputToFiles>
   <!-- log level is one of the 2 user configurable setting -->
   <!-- configurable levels : panic, error, warning, info, verbose, trivia -->
   <!-- log levels are listed in increasing amount of logging done -->
   <!-- use trivia to get most detailed logging for providing logs to VMware Support -->
   <maxFileSize>524288</maxFileSize>
   <maxFileNum>8</maxFileNum>
   <level>verbose</level>
   <outputToSyslog>true</outputToSyslog>
   <syslog>
      <facility>local4</facility>
      <ident>netcpa</ident>
   </syslog>
I /usr/lib/vmware/netcpa/etc/netcpa.xml [Modified] 31/159 19%

net-vdr command
Cette commande est disponible uniquement sur les ESXi.
Elle permet d’avoir pas mal d’informations concernant les instances DLR.
Je vous laisse regarder les options.

Options Format:
--------------
--instance          Instance Commands can also use -I
--route             Route Commands can also use -R
--conection         Connection Commands can also use -C
--lif               Lif Commands can also use -L
--nbr               Neighbor Commands can also use -N
--stats             Stats Commands can also use -S
--syncState         CP sync state commands
--cplane            Control Plane Commands
--bridge            Bridge Commands can also use -b
--mac-address-table Bridge MAC address table commands
--tunables          VDR instance tunable parameters
--preunload         Prepare the module for unload. can also use -P
--brief             Summarized output of the command
-D                  Designated Instance or VXLAN Multicast IP <x.x.x.x>
-i                  IP Address <x.x.x.x>
-M                  Ip Mask <x.x.x.x>
-m                  Mac address <aa:bb:cc:dd:ee:ff>
-n                  Name <String>
-s                  Virtual Switch Name <String>
-p                  VDR port Id <String/Decimal>
-v                  VLAN or VXLAN id or any value <Decimal>
-z                  sedimented LIF (default is distributed)
-f                  mac entry flags
-r                  Reset option
--verbose|-V        More verbose logging
[root@pod-vesxi01:~]

VDR Instance Information
Si vous regarder les options ci-dessus, la commande suivante permet de lister toutes les DLR instances connues.
Ici, aucune VDR instance n’est listée. Ceci est normal, car aucune LIFs n’est configurée au niveau de la DLR VM, aucune LIFs n’est pousée au niveau des ESXi hosts.

[root@pod-vesxi01:~] net-vdr -I -l

VDR Instance Information :
---------------------------

[root@pod-vesxi01:~]

Configuration de la Logical Interface Web (Web Gateway)

Name = to-web
Type = Internal
Connected to = Logical Switch Web
IP Address = 10.150.2.254/24

$C:\Users\zecevici\Desktop\NSX\blog_dlr\2017-06-26_10h56_34.jpg$

Netcpa.log
Important – tail -f /var/log/netcpa.log doit être lancé avant de cliquer sur OK lors de la configuration de la Web LIF. Au sein de ce log, on peut voir les échanges entre le Control Plane (NSX Controller) et le Data Plane (ESXi host) concernant la création de la LIF.

Number of VDRs = 1
VDR Name = default+edge-5 (DLR VM Name)
VDR ID = 5000 (Web VXLAN)
Controller = 10.3.3.101 port 1234
sockname = 10.3.3.11 (ESXi host)
IP = 10.3.3.11 (ESXi host)
Num of added entries = 1
LIF UPDATE
ROUTE UPDATE

[root@pod-vesxi01:~] tail -f /var/log/netcpa.log
2017-06-26T12:18:41.439Z verbose netcpa[FFC2DB70] Set config view vdr instance vdr name = default+edge-5, vdr id = 5000, auth token = 3c1ac646-6174-48a3-8917-      d2ada2f8f4b1, universal = false, localEgress = false
2017-06-26T12:18:41.439Z info netcpa[FFC2DB70] Added vdr instance vdr name = default+edge-5, vdr id = 5000, auth token = 3c1ac646-6174-48a3-8917-d2ada2f8f4b1,       universal = false, localEgress = false
2017-06-26T12:18:41.439Z verbose netcpa[FFAE8B70] VsfwdClient: got a message type 9, len 224
2017-06-26T12:18:41.439Z verbose netcpa[FFAE8B70] VsfwdClient: message to vsfwd type 9, len 218
2017-06-26T12:18:41.439Z verbose netcpa[FFAE8B70] VsfwdClient: sent message to vsfwd
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70] type CFG_TYPE_VDR_INSTANCE
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70] Max 85 vdrInstances can be sent per VmkLink msg
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70]        VDR INSTANCE LIST: len = 80
-->     TYPE:ADD
-->     Number of VDRs: 1
-->     VDRs:   VDR NAME : default+edge-5, VDR ID :5000 isUniversal :0 localEgressRequired :0
-->
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70]         VDR INSTANCE LIST: len = 80
-->     TYPE:ADD
-->     Number of VDRs: 1
-->     VDRs:   VDR NAME : default+edge-5, VDR ID :5000 isUniversal :0 localEgressRequired :0
-->
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70] Vdrb: send a message to the dataplane
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70] Sent 1 vdrInstances to kernel
2017-06-26T12:18:41.440Z info netcpa[FFC2DB70] Vdrb: sharding elem ID is created for 5000
2017-06-26T12:18:41.440Z info netcpa[FFC2DB70] App VDRB : 0 register connection to existing controller to 10.3.3.101 port 1234
2017-06-26T12:18:41.440Z info netcpa[FFC2DB70] Vdrb: core app ready on 10.3.3.101:0
2017-06-26T12:18:41.440Z info netcpa[FFC2DB70] Vdrb:VDRB : 0 add userID 5000 to session 10.3.3.101
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70] sockname 10.3.3.11, linkChangeEntry[0].cpIp 184746762
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70]         LINK CHANGE: len = 17
-->     OP:UP
-->     Number of VDRs: 1
-->     VDRs:   VDR ID :5000    IP: 10.3.3.11
-->
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70] Vdrb: send a message to the dataplane
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70] sent link up to kernel vdrId 5000
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70] sockname 10.3.3.11, linkChangeEntry[0].cpIp 184746762
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70] sending 17 bytes to vse
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70] sent link up to edge vdrId 5000, fd 19
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70] VdrID 5000 - VSE Instance report msg from cache sent to controller.
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70] recvUpdateMsg = 71
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70] localeId in the RPRT msg is 00000000-0000-0000-0000-000000000000
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70] HandleVdrbExternalMsg: ctrlMsgType 7, internal msg type 3, vdrId 5000
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70] Sent edge link up to kernel
2017-06-26T12:18:41.440Z verbose netcpa[FFC2DB70] type
2017-06-26T12:18:41.440Z info netcpa[FFC2DB70] No flap edge CP link for vdr id 5000
2017-06-26T12:18:41.440Z verbose netcpa[FFCCFB70] Received from kernel vdrb internal message type 4
2017-06-26T12:18:41.440Z verbose netcpa[FFCCFB70] recvUpdateMsg = 72
2017-06-26T12:18:41.440Z verbose netcpa[FFCCFB70] localeId in the RPRT msg is 42369634-2d0a-1987-29c3-8c960cce8eaf
2017-06-26T12:18:41.440Z verbose netcpa[FFCCFB70] HandleVdrbExternalMsg: ctrlMsgType 7, internal msg type 3, vdrId 5000
2017-06-26T12:18:41.440Z verbose netcpa[FFCCFB70]        INSTANCE REPORT: len = 61
-->     VDRID: 5000, Change: JOIN
-->     VSE LocaleId : = 42369634-2d0a-1987-29c3-8c960cce8eaf
-->
2017-06-26T12:18:41.441Z verbose netcpa[FFED7B70] Send a message (type 7 len 68) to controller 10.3.3.101:0:1234, from source port 50799
2017-06-26T12:18:41.441Z verbose netcpa[FFED7B70] Send a message (type 9 len 20) to controller 10.3.3.101:0:1234, from source port 50799
2017-06-26T12:18:41.443Z verbose netcpa[FFED7B70] Get a message from controller type 7 len 84 protocolID 1025,  from source port 50799
2017-06-26T12:18:41.443Z verbose netcpa[FFED7B70] Received from controller vdrb message type 7
2017-06-26T12:18:41.443Z verbose netcpa[FFED7B70]        INSTANCE REPORT: len = 77
-->     VDRID: 5000, Change: JOIN
-->     VSE LocaleId : = 42369634-2d0a-1987-29c3-8c960cce8eaf
-->     Gen Number UPDATE: len = 16
-->     VdrID:5000
-->     GenNum :20033
-->     NumUpdateTlv :1
-->
-->
2017-06-26T12:18:41.551Z verbose netcpa[FFF18B70] Vdrb: send a message to the dataplane
2017-06-26T12:18:42.465Z verbose netcpa[FFE14B70] Get a message from controller type 7 len 123 protocolID 1025,  from source port 50799
2017-06-26T12:18:42.465Z verbose netcpa[FFE14B70] Received from controller vdrb message type 7
2017-06-26T12:18:42.465Z verbose netcpa[FFE14B70]          LIF UPDATE: len = 116
-->     VdrID:5000
-->     Num of removed entries: 0
-->     Num of added entries: 1
-->     Gen Number UPDATE: len = 16
-->     VdrID:5000
-->     GenNum :20034
-->     NumUpdateTlv :1
-->
-->
2017-06-26T12:18:42.466Z verbose netcpa[FFE14B70] Vdrb: send a message to the dataplane
2017-06-26T12:18:44.443Z verbose netcpa[FFD51B70] Start recv on fd 19
2017-06-26T12:18:44.444Z verbose netcpa[FFD51B70] recv : received 28 bytes, status: Address family not supported by protocol
2017-06-26T12:18:44.444Z verbose netcpa[FFD51B70] Handle recv buffer from fd 19
2017-06-26T12:18:44.444Z verbose netcpa[FFD51B70] recvUpdateMsg = 73
2017-06-26T12:18:44.444Z verbose netcpa[FFD51B70] HandleVdrbExternalMsg: ctrlMsgType 7, internal msg type 5, vdrId 5000
2017-06-26T12:18:44.444Z verbose netcpa[FFD51B70]          ROUTE UPDATE: len = 28
-->     VdrID:5000
-->     Num of removed entries: 0
-->     Num of added entries: 0
-->     ROUTE UPDATE: len = 28
-->     VdrID:5000
-->     Num of removed entries: 0
-->     Num of added entries: 0
-->
-->
2017-06-26T12:18:44.447Z verbose netcpa[FFF18B70] Vdrb: send a message to the dataplane
2017-06-26T12:18:44.551Z verbose netcpa[FFE55B70] Get a message from controller type 7 len 163 protocolID 1025,  from source port 50799
2017-06-26T12:18:44.551Z verbose netcpa[FFE55B70] Received from controller vdrb message type 7
2017-06-26T12:18:44.551Z verbose netcpa[FFE55B70]        LIF UPDATE: len = 156
-->     VdrID:5000
-->     Num of removed entries: 0
-->     Num of added entries: 1
-->     ROUTE UPDATE: len = 156
-->     VdrID:5000
-->     Num of removed entries: 0
-->     Num of added entries: 0
-->     LIF UPDATE: len = 156
-->     VdrID:5000
-->     Num of removed entries: 0
-->     Num of added entries: 0
-->     Gen Number UPDATE: len = 16
-->     VdrID:5000
-->     GenNum :20035
-->     NumUpdateTlv :3
-->
-->
2017-06-26T12:18:44.551Z verbose netcpa[FFE55B70] Vdrb: send a message to the dataplane
2017-06-26T12:18:50.030Z verbose netcpa[FFF18B70] Get a message from controller type 2 len 10 protocolID 0,  from source port 50799
2017-06-26T12:18:50.030Z verbose netcpa[FFF18B70] Core: Recvd KeepAlive from 10.3.3.101:0
2017-06-26T12:18:52.486Z verbose netcpa[FFED7B70] Core: send keepAlive message to 10.3.3.101:0
2017-06-26T12:18:52.486Z verbose netcpa[FFED7B70] Send a message (type 2 len 10) to controller 10.3.3.101:0:1234, from source port 50799
2017-06-26T12:19:00.030Z verbose netcpa[FFED7B70] Get a message from controller type 2 len 10 protocolID 0,  from source port 50799
2017-06-26T12:19:00.030Z verbose netcpa[FFED7B70] Core: Recvd KeepAlive from 10.3.3.101:0
2017-06-26T12:19:01.966Z verbose netcpa[FFD51B70] Start recv on fd 19
2017-06-26T12:19:01.966Z verbose netcpa[FFD51B70] recv : received 24 bytes, status: Address family not supported by protocol
2017-06-26T12:19:01.966Z verbose netcpa[FFD51B70] Handle recv buffer from fd 19
2017-06-26T12:19:01.966Z verbose netcpa[FFD51B70] recvUpdateMsg = 74
2017-06-26T12:19:01.966Z verbose netcpa[FFD51B70] HandleVdrbExternalMsg: ctrlMsgType 7, internal msg type 5, vdrId 5000
2017-06-26T12:19:01.966Z verbose netcpa[FFD51B70]         ROUTE UPDATE: len = 24
-->     VdrID:5000
-->     Num of removed entries: 1
-->     Num of added entries: 0
-->
2017-06-26T12:19:01.966Z verbose netcpa[FFE55B70] Send a message (type 7 len 31) to controller 10.3.3.101:0:1234, from source port 50799
2017-06-26T12:19:01.967Z verbose netcpa[FFE55B70] Get a message from controller type 7 len 47 protocolID 1025,  from source port 50799
2017-06-26T12:19:01.967Z verbose netcpa[FFE55B70] Received from controller vdrb message type 7
2017-06-26T12:19:01.967Z verbose netcpa[FFE55B70]         ROUTE UPDATE: len = 40
-->     VdrID:5000
-->     Num of removed entries: 1
-->     Num of added entries: 0
-->     Gen Number UPDATE: len = 16
-->     VdrID:5000
-->     GenNum :20036
-->     NumUpdateTlv :1
-->

VDR Instance Information
La même commande que tout à l’heure nous donne cette fois-ci plus d’informations.

Vdr Name = default+edge-5 = Le Nom de la DLR VM qu’on a vu au niveau du NSX Controller
Vdr ID = 0x1388 = ID de la DLR VM qu’on a vu au niveau du NSX Controller
Number of Lifs = 1 = Web LIF
Number of Routes = 1 = Web gateway
Controller IP = 10.3.3.101 = NSX Controller en charge de la DLR VM

[root@pod-vesxi01:~] net-vdr -I -l

VDR Instance Information :
---------------------------

Vdr Name:                   default+edge-5
Vdr Id:                     0x00001388
Number of Lifs:             1
Number of Routes:           1
State:                      Enabled
Controller IP:              10.3.3.101
Control Plane IP:           10.3.3.11
Control Plane Active:       Yes
Num unique nexthops:        0
Generation Number:          0
Edge Active:                Yes

VDR Route Table information
La commande ci-dessous nous donne des informations sur la table de routage
On voit que la Route 10.150.2.0 (WEB subnet) est disponible au niveau du kernel de l’ESXI Host

Destination = 10.150.2.0
GenMask = 255.255.255.0
Gateway = 0.0.0.0 (Local, Distributed in Kernel)
Interface= 13880000000a (Web LIF)

[root@pod-vesxi01:~] net-vdr -R -l default+edge-5

VDR default+edge-5 Route Table
Legend: [U: Up], [G: Gateway], [C: Connected], [I: Interface]
Legend: [H: Host], [F: Soft Flush] [!: Reject] [E: ECMP]

Destination      GenMask          Gateway          Flags    Ref Origin   UpTime     Interface
-----------      -------          -------          -----    --- ------   ------     ---------
10.150.2.0       255.255.255.0    0.0.0.0          UCI      1   MANUAL   478        13880000000a

VDR Lif Information
La commande suivante nous donne plus d’informations concernant les LIFs de notre DLR VM.
Ayant configuré qu’une seule LIF (Web LIF), une seule entrée est disponible

Name = 13880000000a = Nom de la LIF Web
Mode = Routing, Distributed, Internal = Interface disitribué sur tous les ESXi hosts
ID = Vxlan:5000 = VXLAN Web
IP(Mask) = 10.150.2.254 (255.255.255.0) = IP de la LIF Web

[root@pod-vesxi01:~] net-vdr -L -l  default+edge-5

VDR default+edge-5 LIF Information :

Name:                13880000000a
Mode:                Routing, Distributed, Internal
Id:                  Vxlan:5000
Ip(Mask):            10.150.2.254(255.255.255.0)
Connected Dvs:       DVS
VXLAN Control Plane: Enabled
VXLAN Multicast IP:  0.0.0.1
State:               Enabled
Flags:               0x2288
DHCP Relay:          Not enabled

NSX Controller connections to DLR VM
Cette fois-ci, on voit que 3 ESXi ont établi une connexion avec la DLR VM (0x1388).
En effet, la Web LIF a été poussée sur tous les ESXI hosts (3x hosts) du transport Zone.

nsx-controller # show control-cluster logical-routers connections 0x1388
Id     IP               Version Locale-Id                            Sync-State
3      10.3.3.12        6.2     00000000-0000-0000-0000-000000000000 OK
4      10.3.3.13        6.2     00000000-0000-0000-0000-000000000000 OK
6      10.3.3.11        6.2     00000000-0000-0000-0000-000000000000 OK

NSX Controller Interfaces Information
Les 2x commandes ci-dessous ressemblent à celles exécuter au niveau des ESXI host (net-vdr)
Elles donnent des informations sur les interfaces de la DLR VM.

Interface = 13880000000a = Interface LIF Web
ID = 5000 = VXLAN 5000
IP = 10.150.2.254/24 = IP de la Web LIF

nsx-controller # show control-cluster logical-routers interface-summary 0x1388
Interface                        Type   Id                       IP[]
13880000000a                     vxlan  5000(0x1388)             10.150.2.254/24
nsx-controller #
nsx-controller # show control-cluster logical-routers interface 0x1388 13880000000a

Interface-Name:   13880000000a
Logical-Router-Id:0x1388
Id:               0x1388
Type:             vxlan
IP:               10.150.2.254/24
DVS-UUID:         514e3650-09af-0ea6-a954-51cadecffcc0
Mac:              02:50:56:56:44:52
Mtu:              1500
Multicast-IP:     0.0.0.1
Designated-IP:
Flags:            0x280
Bridge-Id:
Bridge-Name:
DHCP-relay-server:

Test Ping depuis la VM web01

$C:\Users\zecevici\Desktop\NSX\blog_dlr\2017-06-26_11h03_26.jpg$

VMware NSX : DLR LIFs Update on ESXi hosts

VMware NSX : DLR Routes LIFs Update on ESXi hosts

Leave a Reply Cancel reply

VMware NSX : DLR Routes LIFs Update on ESXi hosts

Share this :

Leave a Reply Cancel reply