CCIE Homelab Tips, Tricks, & Thoughts CSR1000V memory optimization

With the completion of my Master’s degree I now have more free time to start preparing for my deep dive into my CCIE studies. Recently I’ve been working on getting my lab environment together. What I’ve decided was to do a mixture of both physical hardware and virtual instances which includes a few “pods” of routers and switches along with a ESXi Server running both Cisco VIRL and a large number of CSR100V instances. There are both pros and cons to using either hardware or virtual instances, something I will probably cover at a later time, including a rundown of my lab environment once its completed.

Today I want to go over something neat I discovered when researching how to deploy CSR1000Vs for my lab, disabling large pages in ESXi. Disabling Large Pages in ESXi is essentially deduplication for memory. This allows for a very large memory savings when running multiple instances of similar VMs (such as 20 CSR1000V instances!). For me on 10 CSR1000V instances I saw RAM use go from 33GB down to 12GB, which allows for even more instances to be installed on the same amount of hardware.

Enabling this feature takes a matter of minutes to implement, and gives a great memory use savings.

*Please note*: I have not done extensive research on the interworking of what enabling this feature does. I have not personally run into any performance issues running this on my CCIELab ESXi host which runs mostly CSR instances with a few other VMs. If you are going to deploy this on a host that is used for other things please take baselines and use caution when deploying. I would also recommend not deploying this feature in production without first consulting a VMware expert.

Now to the fun part!

First lets take a note of what memory usage is currently at:

To have the change take affect, the VMs have to be powered on after the change is made, so next is to power down all of your VMs.

After the VMs are powered down follow the following steps:

1. Navigate to Configuration
2. Under Software: Select Advanced Settings

3. Select “Mem”
4. Locate “Mem.AlocGuestLargePage” and set its value to 0

5. Select Ok
6. Power on VMs

It will take a few minutes (roughly 5 – 10) for the CSRs to fully boot up and for ESXi to deduplicate memory. I recommend turning up as many instances you can and coming back in 10 minutes to enjoy all of the additional memory you just acquired.

And of course, a screenshot of my 10 CSR100Vs running with less memory utilization ūüôā


Sending commands to multiple Terminal Sessions at one time

A few weeks ago I had to hunt down PCs but had no idea where they physically were or what switchports they were connected to. To determine the location of a PC the following steps were taken:

  1. Ping the PC IP address РThis refreshes the ARP and MAC-Address tables with the current MAC address and ports
  2. Determine the MAC address from the ARP table on a layer three device < show ip arp | include IP_ADDRESS >
  3. Trace the MAC address to it’s port < show mac address-table address MAC_ADDRESS >

While in a normal environment, the most switches that have to be touched is usually two (the default gateway, and the switch the device is connected to). However, for this instance 9 switches were daisy chained together in a ring like fashion… Throw in the fact roughly 35 devices needed to be traced I got tired of clicking through all of the terminal sessions and tracing the MAC address I got annoyed pretty quickly… Luckily I remembered something that stuck out to me from one of the wonderful CBT Nuggets of the VIRL series by Anthony Sequeira. He had mentioned the Command (Chat) Window in SecureCRT that could be used to send the same commands to multiple sessions at the same time. After poking around for a few minutes I figured out how to enable the chat window and have it send commands to every active session. Once setup tracking down those PCs was a lot less time consuming!

Setting up Command Chat is pretty straight forward.

First enable Command (Chat) Window under View:

After the Chat Window has been enabled SecureCRT will have a blank box at the bottom of the screen as seen below:

The chat window alone will not send commands to all of the active sessions, that feature must b enabled by right clicking in the chat window and selecting “Send Commands to All Sessions”:

After that anything typed into the chat window will be sent to all active sessions!

Here is the command window in action:



As with anything always be mindful of what you are doing! Having sessions open that you may forget about and then push commands to can be very very bad!!

A few examples of when this can come in handy are:

  • Tracing MAC addresses
  • Tracing routes across the network
  • Creating¬†the same¬†VLANS across multiple switches
  • Pushing standard configurations for initial configurations in VIRL
  • Many more!

Two months fly by quick!

Wow its been two months already!!

Lots of things have been going on that have kept me away from doing much of anything outside besides work and school work. Luckily I am officially done with my Master’s degree, which means I will have plenty of time to start creating more posts like I had originally intended. With the completion of my Master’s degree I will be able to start preparing for the CCIE written.

That’s it for today, just wanted to say that things aren’t dead!

To homelab or not to homelab‚Ķ.? That is a question.. (Part 2) – Diagrams

In my previous post in this series I went over the gradual growth of my lab from just being on my desk to having its very own 42u network rack. Since then, my lab has grown and changed. Previously I was using the virtual switch right on my esxi hosts (which were 5.1), along with having separate local storage and managing them individually.

Today my virtual environment is managed by vCenter, has centralized storage, runs ESXi 5.5, and utilizes Cisco Nexus 1000v for distributed switching. I plan to do another post eventually on my centralized storage setup running FreeNAS so I won’t be talking directly about it too much. Instead, through building out my virtualized environment I realized that I don’t actually have any diagrams for my home network / lab.

The first diagram I started to work on is for my “server” environment. This includes my storage servers as well as my ESXi hosts and the guests running on the guests. It took a bit of time to figure out a good way to represent how the VMS are connected to the network but I think I found a good way of doing it.


Going through the diagram:

  • ESXi-1 (C1100)
    • 72 GB ECCRAM
    • iSCSI LUN
    • Dual connected IP and iSCSI connectivity
    • 25 Configured VMs
      • 11 Active
  • ESXi-2 (C1100)
    • 72 GB ECC RAM
    • iSCSI LUN
    • Dual connected IP and iSCSI connectivity
    • 29 Configured VMs
      • 11 Active
  • unRAID (WhiteBox)
    • 8GB RAM
    • Single DataStore (1 Disk Parity) [ 22TB]
    • Dual connected IP connectivity
  • FreeNAS (C2100)
    • 24GB ECC RAM
    • Single DataStore (1 Disk Parity) [ 22TB]
    • Dual connected IP connectivity
    • Quad connected iSCSI connectivity
    • Dual PSU
  • Access Switch
    • WS-C2960S-48LPS-L
  • Other
    • APC BR1500G (1500VA/865W)
    • APC BR24BPG (additional four 9Ah batteries for UPS)


Next steps will be to add the rest of the equipment, and my CCIE Lab. Can’t forget about that ūüôā

Upgrading ESXi 5.X, the easy way

I am currently in the process of collecting the several pieces I need for my next lab upgrade (shared storage for my ESXi hosts) and as part of my preparation I needed to upgrade my hosts to the latest version of ESXi. I spent about 45 minutes reading through all of the documentation on upgrading and then attempting to find the upgrade tool mentioned in VMware documentation. Through my search I discovered a very simple way to upgrade 5.x

  1. Download the update zip from VMware.
  2. Upload the update image to a Datastore on the host with the following steps:
    1. Browse to Configuration > Hardware > Storage
    2. Right click on the Datastore where you want to store the image and click on “Browse Datastore…”:ESUP1
    3. Upload the update file:ESUP2
  3. Once uploaded, turn down all VMs and put the host into maintenance mode.
  4. SSH to the ESXi host (may require SSH to be turned on) and enter the following command:
    esxcli software vib install -d /UPDATE_LOCATION/
  5. After the upgrade finishes you will see a similar screen:esxi upgrade
  6. Reboot
  7. Once your host finishes booting you are all set and ready to turn up your VMs (don’t forget to exit maintenance mode)!

I was able to use this method to upgrade my 5.1 host and my 5.5U1 host to 5.5U2 in a manner of minutes (the 5.1 host took a little longer to come back after the reboot, just give it time).

As always, test before implanting anything in production, make proper backups, verify upgrade paths, etc.

Happy upgrading!

Lets just go ahead and use DTP & VLAN 1‚Ķ Part 1: Attacking DTP – getting those server files

In my previous post, I discussed the vulnerabilities introduced from using the defaults of DTP and VLAN1 along with ways to mitigate the vulnerabilities. In this post a basic example of attacking DTP will be reviewed.

To make things easier to follow the following diagram will be used throughout the series:VH

Before the attack, for demonstration purposes, we verify that the attacker’s switchport (fa 0/14) is not a trunk:


We will also verify the inability of the attacker to access the target server (due to the router’s ACL):a1

Now that we have verified the inability for the attacker to gain access to the server we can begin the attack.

The first phase of the attack will be to trick the switch into thinking its connected to another switch and negotiate a trunk link. This will be completed through spoofing DTP packets, which can be seen below:at1

We can see the switch port flap as it resets, and then its verified that the attacker’s port is now a trunk:Untitled3

Now that the switch believes its connected to another switch via a trunk connection its now possible to create virtual interfaces for any VLAN allowed across the trunk (all by default):at2

As can be imagined, with the attacker able to successfully trick the connected switch into thinking s/he is also a switch almost anything is possible. For today, we will simply pull a file off of the server which is supposed to be protected by an ACL on the router:


With DTP enabled on a port it takes a matter of seconds to trick a switch into thinking its connected to another switch, this is why its very important to configure any port that is not a trunk port in use as an access port with other appropriate security configurations.

Lets just go ahead and use DTP & VLAN 1… Part 0: What using DTP & VLAN 1 means

By default, DTP auto negation is enabled on Cisco switches on all layer 2 ports and they are placed in VLAN 1. These two defaults allow for an easy way to just deploy a switch, or attach another switch to gain more port density, without needing any configuration knowledge. While this is very helpful, the use of VLAN 1 and leaving DTP auto negation on has been widely accepted as standard use for data ports and in turn has left the ability for someone with physical access to gain access to other VLANs and the devices in them.

In part 0 of this series we are going to go over the theory of why the use of DTP and VLAN 1 could be used to allow for an attacker to execute a VLAN hopping attack.


Dynamic Trunking Protocol (DTP) is a Cisco proprietary protocol used to allow for trunks to automatically form between switches without requiring any configuration knowledge when they are plugged into each other. DTP sends updates every 60 seconds over links with DTP enabled and includes the switch’s DTP type, interface status, and VTP domain.

Attacking DTP:

While there really isn’t much to say about DTP (even at a technical level) having DTP on allows for a security hole that can easily be exploited to¬†allow full access to any VLANs the switch has access to. There are two different ways to take advantage of DTP both of which are pretty straight forward.

The first method involves simply plugging in another Cisco switch, that the attacker has control over, with DTP enabled. Once the two switches form a trunk, the attacker can then configure the switch they control and put a device in any VLAN that’s allowed across the trunk (which is all by default). SS

Figure 1 – Adding a third switch

The second method involves forging DTP frames to trick the victim switch into forming a trunk with the attackers computer. Once the trunk is formed, the attacker can forge other frames and packets that will allow them deeper access into the network similar to having their own switch but without the hassle of having a switch on-site for the attack. ST

Figure 2 РSwitch Spoofing

Mitigating DTP attacks:

Just as DTP and the attacks associated with the protocol are simple, so are the steps for mitigating the attacks.

1. Set any port not used as a trunk explicitly as an access port
SW-1 (config)# interface fa 1/0/1
SW-1 (config-if)# description Access Port
SW-1 (config-if)# switchport mode access

2. On ports that are trunks, disable DTP negotiation
SW-1 (config)# interface gig 1/0/1
SW-1 (config-if)# description Trunk to SW-2
SW-1 (config-if)# switchport trunk encapsulation dot1q
SW-1 (config-if)# switchport mode trunk
SW-1 (config-if)# switchport nonegotiate

SW-2 (config)# interface gig 1/0/1
SW-2 (config-if)# description Trunk to SW-1
SW-2 (config-if)# switchport trunk encapsulation dot1q
SW-2 (config-if)# switchport mode trunk
SW-2 (config-if)# switchport nonegotiate

Unlike Cisco proprietary ISL, standards based 802.1Q does not encapsulate the original frame. Instead it adds a 32-bit field between the source MAC address and the EtherType/length fields. Another difference between ISL and 802.1Q is the concept of a native VLAN. While ISL doesn’t have a native VLAN in 802.1Q a native VLAN will not be tagged when frames for that VLAN are flowing over a trunk link by default.

Attacking the native VLAN:

The use of the native (untagged) VLAN allows for an attack called VLAN hopping. This attack allows for an attacker to craft frames that are “double tagged” which¬†have two 802.1Q tags in the frame. The attack itself would be performed in the following manner:


Figure 3 РDouble Tagging

1. The attacker crafts a frame with two 802.1Q tags, the first tag will be for the native VLAN 1 (in this case the default VLAN 1) and the second tag will be for the target VLAN (in this case VLAN 10).

2. SW-1 receives the frame on a port set as the native VLAN and strips the first 802.1Q header. Next, it will forward the frame out any trunk links it has that are set for native VLAN 1. In this case the frame will be forwarded to SW-2 over its trunk link.

3. SW-2 receives the frame on its trunk and see that it is destined for a device in VLAN 20. Assuming the target’s MAC address is already in the MAC table, the frame has its second 802.1Q header removed (because its going to be forwarded over an access port which doesn’t allow 802.1Q headers) and forwarded to the target.

As can be seen above, the attack does require two switches to be trunked together in order to allow for it to be successful.

Protecting the native VLAN:

There are three easy ways to protect against VLAN hopping attacks, only one is required but the use of all three is highly recommended.

1. Do not use the native VLAN on any access port
SW-1 (config)# interface fa 1/0/1
SW-1 (config-if)# description Access Port
SW-1 (config-if)# switchport access vlan 20

2. Set the native VLAN to an unused VLAN (must be configured on both switches of a trunk)
SW-1 (config)# interface gig 1/0/1
SW-1 (config-if)# description Trunk to SW-2
SW-1 (config-if)# switchport trunk native vlan 999

SW-2 (config)# interface gig 1/0/1
SW-2 (config-if)# description Trunk to SW-1
SW-2 (config-if)# switchport trunk native vlan 999

3. Tag native VLAN traffic (must be configured on both switches of a trunk)
SW-1 (config)# interface gig 1/0/1
SW-1 (config-if)# description Trunk to SW-2
SW-1 (config-if)# switchport trunk native vlan tag

SW-2 (config)# interface gig 1/0/1
SW-2 (config-if)# description Trunk to SW-1
SW-2 (config-if)# switchport trunk native vlan tag


While defaults are wonderful to have and make certain tasks easier, it must be taken into consideration what those defaults actually enable what type of holes could be opened in a security plan when used. My next post will go into greater detail of using these vulnerabilities as a jumping off point on gaining access to other systems on a network.