Stack-wise Virtual with Catalyst 9K Switches!

Lately I’ve been configuring and deploying a lot of Cisco Catalyst 9K series switches. The design calls for Stack-wise virtual on 9606 and 9500 series switches. So, I thought I’d take some time to share with you what exactly is Stack-wise Virtual, how to configure it, and some of the caveats I’ve found in the process. Join me!

What is Stack-wise Virtual?

Let’s start at the the beginning and answer the question – what is Stack-wise Virtual? Stacking has been around forever in the Catalyst lineup. You can physically, or virtually, stack switches so their control plane and MAC tables combine. While logically they are many switches, from a control plane standpoint they function as a single switch. So, a whole stack of switches can now be controlled as a single switch.

Stack-wise virtual is how we connect switches that lack the physical stack ports found on the 9300 and older 3K series, and some 2900 series switches. If you ran the tride and true 6K or 4500 series platform you may recall the Virtual Switch System, or VSS, which did the same thing. This is not to be confused with the Nexus VPC – the major difference here is that with Nexus VPC the switches still operate independently of each other from a control and management plane perspective. You’re logically managing two separate switches. With Stackwise-Virtual, and the older VSS, you have two physical switches but you were logically managing a single a switch, just as you would with physically stacked switches.

Use Case – the Why?

Why use Stackwise? Well with Stackwise we can address a redundant pair of switches as a single logical switch, rather than manage two separate switches. We can load balance aggregated links across the two different physical switches. We can also take advantage of Control and Management Plane redudancy features like ISSU and Non-Stop Forwarding.

If we didn’t use Stackwise then we would have to give some additional attention to Spanning Tree, and perhaps add in a First Hop Redundancy Protocol like VRRP or HSRP. With Stackwise the rest of the network will see these two redundant switches as a single logical switch hosting the L2/L3 boundary, and no need to implement FHRP configuration so we can IP address space/utilization.

Requirements

There are some software requirements for Stackwise-Virtual depending on the hardware platform you’re going to be running.

• 16.8.1 or newer provides support for the Catalyst 3850XS and all 9500 series switches
• 16.9.1 or newer on the Catalyst 9400 series
• 16.12.1 or newer on the Catalyst 9606 chassis switches – this is dual sup support only – so only one sup card per chassis. If you need/want quad supervisor card support you’ll need IOS-XE 17.2.1 or newer.

I think it’s important to add here that the quad-sup support is not built-in until 17.2.1. If you dual sup cards per chassis and running a version of IOS previos to 17.2.1 then one sup card in each chassis will just not be used. The card won’t even be a hot standby. So if you lose a sup card in a chassis then Stackwise will failover to the other chassis until the failed sup card is replaced. 17.2.1 will make full use of all four supervisor cards.

Cabling

You’ll also need to make the connections between the switch pairs. There’s two connections for the Switch Virtual Link, and two connections for the Dual Active Detection.

The Switch Virtual link is used send control plane and data plane traffic between the two switches. So this is a high speed, high bandwidth connection. Pictured below the Switch Virtual Links are the Active Optical Cables with 40G QSFPs.

The Dual Active Detection links are used to detect a switch failure so the stand by switch can assume control, if needed. Lower bandwidth, but high speed links are used. Pictured below these links are using 10G SR SFP+ and LC-LC connections.

Licensing

Thankfully there’s no specific license required for Stackwise-Virtual. This is a feature available in the base licensing.

UPDATE: There is now a requirement for Network Advantage licensing o the CAT 9K platform.

Configuration

Let’s configure Stack-wise Virtual! Here I’ve got 2 Cat 9500’s. On switch one I’m going to run the following commands:

On Switch 1:

switch#config t
switch(config)#
switch(config)#stackwise-virtual
Please reload the switch for Stackwise Virtual configuration to take effect
Upon reboot, the config will be part of running config but not part of start up config.
Switch(config-stackwise-virtual)# domain 10
Switch(config-stackwise-virtual)#exit
Switch(config)#interface Hu1/0/51
Switch(config-if)# description STACKWISE-VIRTUAL
Switch(config-if)# stackwise-virtual link 1
WARNING: SVL configuration will be ignored  on lower (1G) speed.
WARNING: All the extraneous configurations will be removed for HundredGigE1/0/51 on reboot
INFO: Upon reboot, the config will be part of running config but not part of start up config.
Switch(config-if)#interface Hu1/0/52
Switch(config-if)# description STACKWISE-VIRTUAL
Switch(config-if)# stackwise-virtual link 1
WARNING: SVL configuration will be ignored  on lower (1G) speed.
WARNING: All the extraneous configurations will be removed for HundredGigE1/0/52 on reboot
INFO: Upon reboot, the config will be part of running config but not part of start up config.
Switch(config-if)#
Switch(config-if)#interface TwentyFiveGigE1/0/47
Switch(config-if)# description STACKWISE-VIRTUAL
Switch(config-if)# stackwise-virtual dual-active-detection
WARNING: All the extraneous configurations will be removed for TwentyFiveGigE1/0/47 on reboot.
INFO: Upon reboot, the config will be part of running config but not part of start up config.
Switch(config-if)#interface TwentyFiveGigE1/0/48
Switch(config-if)# description STACKWISE-VIRTUAL
Switch(config-if)# stackwise-virtual dual-active-detection
WARNING: All the extraneous configurations will be removed for TwentyFiveGigE1/0/48 on reboot.
INFO: Upon reboot, the config will be part of running config but not part of start up config.
witch(config-if)#^Z
Switch#copy run start
Destination filename [startup-config]?
Building configuration...
[OK]
Switch#reload
Proceed with reload? [confirm]

So let’s break this down a little. The Domain tells the switches what domain they are in. When two switches share the same domain they know they are paired together. In this example configuration the domain is 10. Any domain number from 1 to 255 is valid.

The interface descriptions are not required, I do it just for posterity sake. I’m a deployment engineer so after it’s deployed another engineering team will use it. I like to label everything I do so the engineers that follow me know what’s what. But you can see I configure the faster speed ports for the virtual link, and the lower speed links for the Dual Active Detection.

Let’s hop over to Switch 2!

On Switch 2:

The commands on switch two are going to be exactly the same, except for the first set of commands. We need to tell Switch 2 that it’s, well, Switch 2. We do that by:

Switch#switch ren
Switch#switch renumber 2
WARNING: Changing the switch number may result in a configuration change for that switch.  The interface configuration associated with the old switch number will remain as a provisioned configuration. New Switch Number will be effective after next reboot. Do you want to continue?[y/n]? [yes]:
Switch#

So, after the reload this will now be Switch 2. Before we reload we need to configure our Stackwise ports just like we did on Switch 1. For Switch 2 I will skip the descriptions because those will not stick through the reload.

Switch#
Switch#config t
Enter configuration commands, one per line.  End with CNTL/Z.

Switch(config)#stackwise-virtual
Please reload the switch for Stackwise Virtual configuration to take effect
Upon reboot, the config will be part of running config but not part of start up config.
Switch(config-stackwise-virtual)#domain 10
Switch(config-stackwise-virtual)#interface Hu1/0/51

Switch(config-if)# stackwise-virtual link 1
WARNING: SVL configuration will be ignored  on lower (1G) speed.
WARNING: All the extraneous configurations will be removed for HundredGigE1/0/51 on reboot
INFO: Upon reboot, the config will be part of running config but not part of start up config.
Switch(config-if)#interface Hu1/0/52

Switch(config-if)# stackwise-virtual link 1
WARNING: SVL configuration will be ignored  on lower (1G) speed.
WARNING: All the extraneous configurations will be removed for HundredGigE1/0/52 on reboot
INFO: Upon reboot, the config will be part of running config but not part of start up config.
Switch(config-if)#interface TwentyFiveGigE1/0/47

Switch(config-if)# stackwise-virtual dual-active-detection
WARNING: All the extraneous configurations will be removed for TwentyFiveGigE1/0/47 on reboot.
INFO: Upon reboot, the config will be part of running config but not part of start up config.
Switch(config-if)#interface TwentyFiveGigE1/0/48

Switch(config-if)# stackwise-virtual dual-active-detection
WARNING: All the extraneous configurations will be removed for TwentyFiveGigE1/0/48 on reboot.
INFO: Upon reboot, the config will be part of running config but not part of start up config.
Switch(config-if)#exi
Switch(config)#exi
Switch#copy run start
Destination filename [startup-config]?
Building configuration...
[OK]
Switch#reload
Proceed with reload? [confirm]

Upon reboot switch one is going to delay boot-up and wait for Switch 2 to be detected and join the Stack.

After both switches are booted you can use show switch to verify the stack. You may see HA Sync in Progress, like this:

witch#sho switch
Switch/Stack Mac Address : b0c5.3ce8.0001 - Local Mac Address
Mac persistency wait time: Indefinite
                                             H/W   Current
Switch#   Role    Mac Address     Priority Version  State
-------------------------------------------------------------------------------------
*1       Active   b0c5.3ce8.3480     1      V02     Ready
 2       Standby  b0c5.3ce8.4a60     1      V02     HA sync in progress



Switch#

When you see that you won’t actually be able to modify the configuration of the switches until that HA Sync is complete. It will take a few minutes, even on a bare bones Stack-wise Virtual set of switches.

To Verify your Stackwise configuration use show stackwise-virtual:

Switch#sho stackwise-virtual
Stackwise Virtual Configuration:
--------------------------------
Stackwise Virtual : Enabled
Domain Number : 10

Switch  Stackwise Virtual Link  Ports
------  ----------------------  ------
1       1                       HundredGigE1/0/51
                                HundredGigE1/0/52
2       1                       HundredGigE2/0/51
                                HundredGigE2/0/52

Switch#

You’ll see the ports configured and in use on each switch.

You’ll also see interface numbers change. Switch 1 ports will be 1/0/1 and switch 2 will be 2/0/1. Let’s break that down for moment. The first number is the physical switch, so if it’s 1 we’re addressing Switch 1 interfaces, if it’s 2 then we are addressing Switch 2 interfaces. The next number refers to a module, like an insertable accessory card. The Catalyst 9300’s have these, a Network Interface Module that’s installed on the very end of the switch. This isn’t available on the 9500 platform so it’ll likely always be 0. The last number is the interface number itself. So, 2/0/35 is port 35 on Switch 2.

And that’s it! You’ve now configured Stackwise Virtual. Read on for some additional notes that will save you some time.

Chassis Platforms

This same exact procedure can be used on the Catalyst 9606 platoform. However, when rebooting you’ll notice something a little bit different on the Interface number scheme. The interface numbers will be something like 1/5/0/24 – 1 is the Chassis, 5 is the line card – on the 9606 you can have 4 line cards in slots 1, 2, 5, and 6, with slots 3 and 4 being reserved for Supervisor cards – the 0 refers to modules and currently there are none available for the 9606 platform, but that may change, and the last number is the interface number. So, in the previous example, you have Chassis 1, the line card is slot 5, no network modules, and interface 24.

Caveats

Some important caveats to be aware of:

• Port Channels now range from 1 – 192 on the Cat 9K Platform – this is not specific to Stackwise Virtual. On older Catalyst Platforms you could have 301, or 405, or 550. You’ll need to update your configuration to make sure your new port channels are within that valid range.
• Additionally Port Channel numbers 127 and 128 are reserved for Stackwise virtual. So, if you are currently using those numbers you’ll need to modify your configuration to use alternate Port Channel interfaces.
• VLAN 4094 – This is reserved for Stackwise. So, if you’re using that VLAN currently you’ll need to plan to migrate that off to another VLAN.

Additionally, software upgrades can be both easier and a little bit trickier at the same time. With Stackwise Virtual you can make use of ISSU, or In State Software Upgrade. With ISSU you can have one supervisor performing the upgrade and another supervisor actively forward traffic. When one sup is done it’ll come up on the new version, assume command and take over as the “active” switch. The Active will then become the Standby and then perform the upgrade. Any connections that are not dual-homed to both switches will go down.

You can read more on ISSU supported software upgrades here, and be sure to check the notes on the version of software you looking to upgrade to for specific ISSU support or caveats.

That’s it!

That’s Stackwise Virtual! Hopefully, if you’re considering implementing Stackwise Virtual in your network this helped you out! If you have some more questions that I didn’t cover in my article you can checkout the full details on Stackwise Virtual from Cisco, here. As always shout out in the comments, or yell at me on Twitter. Thanks for stopping by!