Cisco Nexus Review, Testing & Verification of the embedded NX-OS Packet Capture Tools
Introduction
Cisco NX-OS offers a robust set of features to guarantee the uninterrupted operation of the Nexus series switches at the Data Centers. Taking advantage of the Cisco NX-OS architecture which is running on Linux OS, makes it easy to embed productive tools used by network administrators who are familiar with Linux-based environments. Two of the tools Cisco offers with the NX-OS are the Ethanalyzer and tcpdump. The Network administrators no longer have difficulty gaining visibility of the nature of the traffic flowing through their network.
This article is part of the vPC Nexus-Indeni series. It starts with a review of the embedded to NX-OS Linux based packet captures tools. It follows a brief introduction to the Wireshark application and describes the Wireshark-like functions provided by Cisco NX-OS. Then, we discuss how to use the Wireshark protocol analyzer for real-time analysis of traffic. Finally, it is demonstrated in the lab environment how can easily we can exploit the credentials of a user who logged in via telnet to a Nexus switch by taking advantage only of the NX-OS packet capture tools and the Wireshark. Lastly, this article highlights how Indeni’s analysis can eliminate network security risk and resolve the problem.
Cisco NX-OS Ethanalyzer & tcpdump packet captures tools
The Ethanalyzer is a tool mainly used to troubleshoot control plane and traffic destined to switch CPU.
The Cisco NX-OS Ethanalyzer has the next main features and limitations:
- Capture packets sent or received by the Supervisor.
- Set the number and length of packets to be captured.
- Display packets with summary or detailed protocol information.
- Open and save packet data captured.
- Filter packets captured or displayed on many criteria.
- It cannot provide any type of alerting or action when the network experiences problems. However, the Ethanalyzer might help to determine the cause of the problem.
- Normally, it cannot capture data plane traffic that is forwarded in hardware. However, there is a trick to be achieved with extra configuration. In particular, if we add the “log” keyword to the Access Control Entries (ACEs) to send copies of the matching traffic to the supervisor. This mechanism allows the supervisor to receive the traffic of interest and allows Ethanalyzer to capture it. This needs to be implemented with cautions since may have impact to the CPU utilization of the Nexus switch.
Tcpdump is an NX-OS CLI application that can be used to capture and prints out a description of the contents of packets on a network interface. It supports several command options and features such as the -w flag, which causes it to save the packet data to a file for later analysis, and the -r flag which causes it to read from a saved packet file rather than to read packets from a network interface. In all cases, only packets that match expression will be processed by tcpdump. Finally, the tcpdump tool is not supported by all the Nexus series switches.
Nexus and Indeni Lab setup
This section describes the Cisco Nexus switches, Indeni platform and Software versions deployed at the lab environment to test the NX-OS ethanalyzer and tcpdump packet analyzer tools. Then, the captures are collected and analyzed in the Wireshark to exploit the credentials and track the activity of the user.
The following network diagram illustrates the Network topology as well as the private IP address allocation of the remote user, TFTP server and the Indeni platform.
The software-version information of the Network devices and Indeni equipped to the lab are found in the following table.
The following chart provides an easy mnemonic in order to elaborate the numbering convention of the deployed NX-OS versions.
Here’s a brief description of the used Cisco Nexus switches:
Nexus 5548UP: A powerful and widely deployed Nexus switch commonly used as “End of Row” at a Data Center with thirty-two fixed Unified ports 1/10 Gbps fixed SFP+ on the base chassis along with one expansion slot. In addition to the fixed interfaces, the Nexus 5548UP has one expansion module.
Nexus 9504: It provides high performance 1, 10, 25, 40, 50 and 100GE GE in a compact 7-rack-unit modular chassis. This versatile modular switch is designed for high-density, end-of-row, and high-performance aggregation-layer deployments in both traditional and Cisco Application Centric Infrastructure (ACI)–enabled data centers
Nexus Traffic Capture
A user named “nexus-admin” is created to the Nexus switch. The privilege level of the “nexus-admin” user is configured to network-admin (root level). Next, we investigate how we can get the credentials (username/password) and track the activity of the “nexus-admin” user by taking advantage of the embedded packet capture analyzer tool of the nexus.
The new user is configured to the NX-OS and all the users are displayed to the output.
N5k-UP(config)# username nexus-admin password indeni123!@# role network-admin N5k-UP# sh user-account user:admin this user account has no expiry date roles:network-admin user:cisco this user account has no expiry date roles:priv-15 san-admin user:nexus-admin this user account has no expiry date roles:network-admin
First, we login through SSH to the Nexus 5k in order to set a traffic capture by taking advantage of the Ethanalyzer NX-OS command.
Now, the ethanalyzer is activated to monitor all the traffic toward the mgmt port and for port 23 (telnet). The capture file is saved to the bootflash of the Nexus.
N5k-UP# ethanalyzer local interface mgmt capture-filter "tcp port 23" limit-captured-frames 100 write bootflash:///telnet-capture2
The ‘capture-filter’ option is used to select which packets to display or save to bootflash during the capture. The capture-filter option has the same syntax, and thus the flexibility, of the Linux tcpdump utility. The limit-captured-frames command option is set to 100 and refers to the maximum value for the number of frames to be captured (the default is equal to 10 frames). To sum up, the Ethanalyzer by running this command shows the 100 packets which match the filter “tcp port 23” capturing only the telnet traffic toward the management interface.
The next output illustrates that there are two active ssh sessions to the Nexus switch. The first ssh session is initiated by the Indeni which analyses the Nexus switch, and the Network Administrator has established the second ssh session.
N5k-UP# sh users NAME LINE TIME IDLE PID COMMENT admin pts/0 Aug 29 11:47 . 7384 (10.10.8.216) session=ssh nexus-admin pts/1 Aug 29 11:47 . 7443 (10.10.8.243) session=ssh
A new telnet session is initiated by the user. Here, we used the unsecured telnet instead of the ssh protocol. The new telnet session is illustrated to the next figure and is highlighted with a star by the command output.
N5k-UP# sh users NAME LINE TIME IDLE PID COMMENT admin pts/0 Aug 29 11:47 . 7384 (10.10.8.216) session=ssh nexus-admin pts/1 Aug 29 11:47 . 7443 (10.10.8.243) session=ssh nexus-admin pts/2 Aug 29 11:47 . 7502 (10.10.8.243) *
As soon as the telnet session is established the NX-OS capture starts to collect frames and save to the predefined bootflash. Meanwhile a brief sequence of the packets is illustrated to the NX-OS terminal.
Capturing on mgmt0 2018-08-29 11:47:48.031669 10.10.8.243 -> 10.10.8.111 TCP 58231 > telnet [SYN] Seq=0 Win=8192 Len=0 MSS =1260 WS=8 2018-08-29 11:47:48.032051 10.10.8.111 -> 10.10.8.243 TCP telnet > 58231 [SYN, ACK] Seq=0 Ack=1 Win=173 60 Len=0 MSS=1260 2018-08-29 11:47:48.054152 10.10.8.243 -> 10.10.8.111 TCP 58231 > telnet [ACK] Seq=1 Ack=1 Win=17640 Le n=0 2018-08-29 11:47:48.055360 10.10.8.243 -> 10.10.8.111 TCP 58231 > telnet [FIN, ACK] Seq=1 Ack=1 Win=176
The capture reaches the maximum defined value of the 100 frames and stops. The new file has been created and can be found in the bootflash.
N5k-UP# pwd bootflash: N5k-UP# dir | i capture2 10595 Aug 29 11:48:20 2018 telnet-capture2
Trying to read the capture file by running the “show file” NX-OS command proves that the format of the file is not readable.
N5k-UP# show file bootflash:///telnet-capture2
@Aÿÿä[µ{BB`Oª áü`KE4<
�
o�wäKë/ wXä[3}<<ü`KÆ`Oª E,ñ@Rf
o
�ãwú¯ÚäKë0`CЧPìä[Ó<<`Oª áü`KE(<@L
�
o�wäKë0ú¯ÚPDè½-ä[@Ø<<`Oª áü`KE(<@K
However, NX-OS support embedded reader to view the captured frames directly from the terminal.
N5k-UP# ethanalyzer local read bootflash:///telnet-capture2 2018-08-29 11:47:48.031669 10.10.8.243 -> 10.10.8.111 TCP 58231 > telnet [SYN] Seq=0 Win=8192 Len=0 MSS =1260 WS=8 2018-08-29 11:47:48.032051 10.10.8.111 -> 10.10.8.243 TCP telnet > 58231 [SYN, ACK] Seq=0 Ack=1 Win=173 60 Len=0 MSS=1260 2018-08-29 11:47:48.054152 10.10.8.243 -> 10.10.8.111 TCP 58231 > telnet [ACK] Seq=1 Ack=1 Win=17640 Le n=0 2018-08-29 11:47:48.055360 10.10.8.243 -> 10.10.8.111 TCP 58231 > telnet [FIN, ACK] Seq=1 Ack=1 Win=176 40 Len=0
The details of each collected frame can be provided to the NX-OS output by adding the detail option to the last command.
N5k-UP# ethanalyzer local read bootflash:///telnet-capture2 detail Frame 1 (66 bytes on wire, 66 bytes captured) Arrival Time: Aug 29, 2018 11:47:48.031669000 [Time delta from previous captured frame: 0.000000000 seconds] [Time delta from previous displayed frame: 0.000000000 seconds] [Time since reference or first frame: 0.000000000 seconds] Frame Number: 1 Frame Length: 66 bytes Capture Length: 66 bytes [Frame is marked: False] [Protocols in frame: eth:ip:tcp] Ethernet II, Src: 88:1d:fc:60:4b:c6 (88:1d:fc:60:4b:c6), Dst: 8c:60:4f:aa:c2:e1 (8c:60:4f:aa:c2:e1) Destination: 8c:60:4f:aa:c2:e1 (8c:60:4f:aa:c2:e1) Address: 8c:60:4f:aa:c2:e1 (8c:60:4f:aa:c2:e1) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
As mentioned in the previous section, the tcpdump linux NX-OS tool is another option for packet capturing and is not offered by all the Nexus series switches. So, a Nexus 9k series switches which supports this tool is used to test it. Seen below, the linux tool can be executed from the bash-shell of the Nexus 9k switch.
N9k(config)# feature bash-shell N9k(config)# exi N9k# run bash bash-4.2$ tcpdump --help tcpdump version 4.1.1 libpcap version 1.2.1 Usage: tcpdump [-aAbdDefIKlLnNOpqRStuUvxX] [ -B size ] [ -c count ] [ -C file_size ] [ -E algo:secret ] [ -F file ] [ -G seconds ] [ -i interface ] [ -M secret ] [ -r file ] [ -s snaplen ] [ -T type ] [ -w file ] [ -W filecount ] [ -y datalinktype ] [ -z command ] [ -Z user ] [ expression ]
By following the command instructions and options we run as root (sudo) a packet capture this time by taking advantage of the tcpdump tool.
bash-4.2$ sudo tcpdump -w file.pcap -i Eth1-1 tcpdump: WARNING: Eth1-1: no IPv4 address assigned tcpdump: listening on Eth1-1, link-type EN10MB (Ethernet), capture size 65535 bytes �^C4 packets captured 4 packets received by filter 0 packets dropped by kernel bash-4.2$
Traffic Capture Analysis & credentials exploit
Next, we use the Wireshark open source tool, for the analysis of the collected packets from the Nexus switch. In short, the Wireshark, formerly known as Ethereal, is the world’s foremost open source network protocol analyzer. The Wireshark offers powerful and multiple options for the analysis of the captured packet data file. Wireshark’s packet capturing is performed using the pcap library, which is also the format used by the NX-OS captures.
Then we open a TFTP server to collect the capture file from the Nexus, and run the next commands to upload the capture file to the TFTP server.
N5k-UP# copy bootflash: tftp: Enter source filename: telnet-capture2 Enter vrf (If no input, current vrf 'default' is considered): management Enter hostname for the tftp server: 10.10.8.243 Trying to connect to tftp server...... Connection to Server Established. TFTP put operation was successful Copy complete, now saving to disk (please wait)... N5k-UP#
The file has been successfully uploaded to the TFTP server as is illustrated to the next capture.
Then we change the extension of the collected file to .pcap and we open it by using the Wireshark application. All the frames are displayed. Then, we select the “Analyze” Menu and the “TCP stream” option as is illustrated below.
The output reveals the credentials of the user. It is so simple!
In addition, all the NX-OS commands and outputs executed by this user have been captured and provided from the capture analysis. Below is the execution of the “show ver” NX-OS command.
The packet analysis of the capture file from the Nexus revealed the credentials of the user, the executed commands and the relevant outputs of each command by executing only few steps in the Wireshark.
Cisco Nexus Telnet Service and Indeni
Indeni monitors and analyzes metrics related to the telnet service to eliminate the embedded to the protocol security risks as highlighted to the previous section.
The live config tab of the Indeni provides useful information about the users and telnet service status of the Nexus switch as is displayed below.
Indeni raises an alert when identifies that the Telnet service is enabled to the Nexus switch. Remediation steps are provided how to resolve this issue.
As soon as we connect to this switch to troubleshoot, we then validate that the telnet service is enabled by running the next NX-OS command:
N5k-UP# show telnet server telnet service enabled
Finally, we follow the remediation steps provided by Indeni and we disable the telnet feature to the Nexus switch.
N5k-UP(config)# no feature telnet N5k-UP(config)# sh telnet server telnet service not enabled
Indeni live-config is updated and the alarm is cleared. The nexus switch is no longer affected by the embedded vulnerabilities of the telnet protocol.
Summary
This article describes the packet-analysis tools embedded in the Cisco NX-OS. The capabilities and limitations of the NX-OS packet capture tools are described before being tested in the Lab. The steps to configure the ethanalyzer and tcpdump are described and implemented at the Lab environment. Testing and verification commands proves the fully operation of the tools. In addition, we demonstrate how we can easily exploit the credentials of a user who logged in via telnet to a nexus switch and track his activity by using the Wireshark packet analysis tool. Finally, we how the Indeni analyzes the Nexus switch to get info about this vulnerability and how we can protect the Nexus switch by simply following the Indeni Remediation steps.
Learn more by joining the Indeni Community.