Connecting FEX (5k to 2k)

December 1, 2013·

I was pretty excited to get my hands on some of the Cisco Nexus product line which focuses on switching in the data center. The gear I have is an evaluation unit so I made some notes and one thing I was impressed is how easy it is to connect a Nexus 5K to a Nexus 2k also known as a fabric extender. I have kept it simple as the gear I have is a single Nexus 5596 along with a single Nexus 224TP-E who needs redundancy right? So in this post I’ll explain how to connect a 5k Nexus to a 2k extender. Let’s get started!

So I needed a couple of SFPs as the only way to connect the 5k to the 2k is a FET-10G SFP along with that was some multimode fiber (OM3) with LC connectors. Looking at the simple diagram below our end goal is to be able to configure the 2k from the 5k. That is the point of the 2k extenders less switches to manage in the data center.

Once we have made the physical connections lets power both switches on, we won’t need to worry about the 2k switch at all as along we have a physical connection to a parent switch the 5k in our case, but that also could be 7k switch. Just an FYI, I’ve already ran through the simple CLI wizard to setup a username and password but other than that let’s get configuring by entering the familiar command config t

User Access Verification
demo5k login: admin
Password:
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2013, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and

http://www.opensource.org/licenses/lgpl-2.1.php

demo5k# config t

Now in configuration mode we have to turn on a feature in the NX-OS to enable FEX, we do this by issuing the “feature FEX” command. The Nexus runs on efficiency and only loads what it needs or configured. This is familiar approach that Linux uses which loads what is installed and nothing else.In the output below we can see all the features that the NX-OS supports followed by the command to turn on FEX.

demo5k(config)# feature ?
  bgp             Enable/Disable Border Gateway Protocol (BGP)
  cts             Enable/Disable CTS
  dhcp            Enable/Disable DHCP Snooping
  dot1x           Enable/Disable dot1x
  eigrp           Enable/Disable Enhanced Interior Gateway Routing Protocol
                  (EIGRP)
  fcoe            Enable/Disable FCoE/FC feature
  fcoe-npv        Enable/Disable FCoE NPV feature
  fex             Enable/Disable FEX
  flexlink        Enable/Disable Flexlink
  hsrp            Enable/Disable Hot Standby Router Protocol (HSRP)
  http-server     Enable/Disable http-server
  interface-vlan  Enable/Disable interface vlan
  lacp            Enable/Disable LACP
  msdp            Enable/Disable Multicast Source Discovery Protocol (MSDP)
  ntp             Enable/Disable NTP
  ospf            Enable/Disable Open Shortest Path First Protocol (OSPF)
  ospfv3          Enable/Disable Open Shortest Path First Version 3 Protocol
                  (OSPFv3)
  pim             Enable/Disable Protocol Independent Multicast (PIM)
  port-security   Enable/Disable port-security
  private-vlan    Enable/Disable private-vlan
  privilege       Enable/Disable IOS type privilege level support
  ptp             Enable/Disable PTP
  rip             Enable/Disable Routing Information Protocol (RIP)
  scp-server      Enable/Disable SCP server
  sftp-server     Enable/Disable SFTP server
  ssh             Enable/Disable ssh
  tacacs+         Enable/Disable tacacs+
  telnet          Enable/Disable telnet
  udld            Enable/Disable UDLD
  vpc             Enable/Disable VPC (Virtual Port Channel)
  vrrp            Enable/Disable Virtual Router Redundancy Protocol (VRRP)
  vtp             Enable/Disable Vlan Trunking Protocol (VTP)

demo5k(config)# feature fex
demo5k(config)#

Now that our FEX feature is active let’s configure the port that has the 2k connected and in this example I have it as ethernet port 48 on our 5k. The commands are petty simple, I first want to go into interface followed by changing the mode of interface and finally giving the interface a FEX associate ID.

NOTE: Since this design is a single stack we need to issue the FEX associate command on the physical interface. If you want redundancy you would usually put the FEX associate command with channel-groups and vPCs.

Another helpful FYI is if you have channel-groups you don’t need to put the FEX associate command on the physical interface.

demo5k(config)# interface ethernet1/48
demo5k(config-if)# switchport mode fex-fabric
demo5k(config-if)# fex associate 100
demo5k(config-if)# exit
demo5k(config)#

Give it a couple of minutes and you should see the 2k come online :)

2013 Dec 01 07:28:37 demo5k %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 100 is online
2013 Dec 01 07:28:37 demo5k %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-100 On-line
2013 Dec 01 07:28:39 demo5k %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 100 is online

Let’s verify that our FEX is all good by issuing show fex 100 which shows the FEX and the 5k version. (Notice they match versions) You can also find the serial number, model number, the number links and the status of fabric when using this command. (The 5K)

demo5k# show fex 100
FEX: 100 Description: FEX0100   state: Online
  FEX version: 6.0(2)N1(2) [Switch version: 6.0(2)N1(2)]
  Extender Serial: SSI153804CQ
  Extender Model: N2K-C2248TP-E-1GE,  Part No: 73-13671-01
  Pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/48
  FCoE Admin: false
  FCoE Oper: true
  FCoE FEX AA Configured: false
  Fabric interface state:
    Eth1/48 - Interface Up. State: Active

demo5k# show fex
  FEX         FEX           FEX                       FEX
Number    Description      State            Model            Serial
------------------------------------------------------------------------
100        FEX0100                Online   N2K-C2248TP-E-1GE   SSI153804CQ

You can also go into detail on the FEX by issuing the “show fex detail” which shows almost the same information has “show fex” but it includes the FEX ports, the status of those FEX ports and what port the fabric is connected. (Also called the 5k) as well as helpful logs for the 2k at the end of command.

demo5k# show fex detail
FEX: 100 Description: FEX0100   state: Online
  FEX version: 6.0(2)N1(2) [Switch version: 6.0(2)N1(2)]
  FEX Interim version: 6.0(2)N1(2)
  Switch Interim version: 6.0(2)N1(2)
  Extender Serial: SSI153804CQ
  Extender Model: N2K-C2248TP-E-1GE,  Part No: 73-13671-01
  Card Id: 149, Mac Addr: 70:81:05:00:99:42, Num Macs: 64
  Module Sw Gen: 12594  [Switch Sw Gen: 21]
  post level: complete
 pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/48
  FCoE Admin: false
  FCoE Oper: true
  FCoE FEX AA Configured: false
  Fabric interface state:
    Eth1/48 - Interface Up. State: Active
  Fex Port        State  Fabric Port
       Eth100/1/1  Down     Eth1/48
       Eth100/1/2  Down     Eth1/48
       Eth100/1/3  Down     Eth1/48
       Eth100/1/4  Down     Eth1/48
       Eth100/1/5  Down     Eth1/48
       Eth100/1/6  Down     Eth1/48
       Eth100/1/7  Down     Eth1/48
       Eth100/1/8  Down     Eth1/48
       Eth100/1/9  Down     Eth1/48
      Eth100/1/10  Down     Eth1/48
      Eth100/1/11  Down     Eth1/48
      Eth100/1/12  Down     Eth1/48
      Eth100/1/13  Down     Eth1/48
      Eth100/1/14  Down     Eth1/48
      Eth100/1/15  Down     Eth1/48
      Eth100/1/16  Down     Eth1/48
      Eth100/1/17  Down     Eth1/48
      Eth100/1/18  Down     Eth1/48
      Eth100/1/19  Down     Eth1/48
      Eth100/1/20  Down     Eth1/48
      Eth100/1/21  Down     Eth1/48
      Eth100/1/22  Down     Eth1/48
      Eth100/1/23  Down     Eth1/48
      Eth100/1/24  Down     Eth1/48
      Eth100/1/25  Down     Eth1/48
      Eth100/1/26  Down     Eth1/48
      Eth100/1/27  Down     Eth1/48
      Eth100/1/28  Down     Eth1/48
      Eth100/1/29  Down     Eth1/48
      Eth100/1/30  Down     Eth1/48
      Eth100/1/31  Down     Eth1/48
      Eth100/1/32  Down     Eth1/48
      Eth100/1/33  Down     Eth1/48
      Eth100/1/34  Down     Eth1/48
      Eth100/1/35  Down     Eth1/48
      Eth100/1/36  Down     Eth1/48
      Eth100/1/37  Down     Eth1/48
      Eth100/1/38  Down     Eth1/48
      Eth100/1/39  Down     Eth1/48
      Eth100/1/40  Down     Eth1/48
      Eth100/1/41  Down     Eth1/48
      Eth100/1/42  Down     Eth1/48
      Eth100/1/43  Down     Eth1/48
      Eth100/1/44  Down     Eth1/48
      Eth100/1/45  Down     Eth1/48
      Eth100/1/46  Down     Eth1/48
      Eth100/1/47  Down     Eth1/48
      Eth100/1/48  Down     Eth1/48
Logs:
12/01/2013 07:28:31.344277: Module register received
12/01/2013 07:28:31.347237: Registration response sent
12/01/2013 07:28:31.696577: create module inserted event.
12/01/2013 07:28:31.697697: Module Online Sequence
12/01/2013 07:28:37.285556: Module Online

Believe it or not you now have a working 5k with a connected FEX, you could plug devices in and be ready to roll on a single stack 2k and 5k. One thing I’ve been impressed with the NX-OS it’s that familiar feeling of IOS with some improved differences, Cisco has good documentation on the common and different features when comparing NX-OS and IOS which is below. Like always I hope this post is helpful and I have a couple more Nexus posts in the works but other than that enjoy the remaining of 2013! Cisco Nexus 7000 NX-OS/IOS Comparison Tech Notes

Exploring the Nexus 1000 V REST API (blogs.cisco.com)
FC and FCoE versus iSCSI - “Network-centric” versus “End-Node-centric” provisioning (brasstacksblog.typepad.com)

GNS3 1.0 Campaign Creating a SPAN Port on a 2960

Connecting FEX (5k to 2k)

Related articles