Remote Lab Access and Control

A requirement I’ve quickly come to realize with building my lab is remote access into lab my equipment. This requirement is two fold, I don’t feel like always sitting in my basement to build topologies and I’m not always home when I will be studying. This need naturally led me to acquiring a terminal server, which was very helpful in fulfilling my first need of not having to always hang out in the basement when studying. I didn’t like the idea of always leaving my lab equipment on wasn’t exciting to me as I don’t like wasting electricity, so I found a Remote Power Control (RPC) unit also known as a switched PDU.

I enjoyed setting everything up so I figured I’d share the configuration steps I took to get two devices communicating with each other and functioning. The two devices I used were a Opengear IM7200 terminal server and a Avocent (Cyclades) PM10. The setup is pretty straightforward with minimal steps.

First you need to make sure the RPC unit is cable properly, for the PM10 a serial console connection is made of a UTP straight through cable from one of the serial ports on the Opengear terminal server to the “In” port on the PM10. You can daisy chain multiple PM10s together by going from the “out” port to the “in” port on the next PM10, but I recommend setting up each PM10 as an individual serial port on the terminal server. This gives more flexible control and you won’t lose multiple RPCs if you have a failure “up stream” in the daisy chain. After the cabling is taken care of its time to move on to the fun part, configuration!

The first configuration component is to configure the serial port on the IM7200 to the PM10. To do so navigate to the Serial Port configuration section: ynwxbpg

The next step is to configure the serial port connected to the PM10 by editing the port on the IM7200:um6g9qy

The following settings are specific to the PM10 connection and need to be configured on the serial port on the IM7200 for connectivity:rbf9hxzoihsmtcSettings include:

  • Label – Port name you would like
  • Baud Rate – 9600
  • Data Bits – 8
  • Parity – None
  • Stop bits – 1
  • Flow control – None
  • Port Pinout – Cisco Straight (X1)
  • Terminal Type – ansi

In addition to the required serial settings, the serial port must be set to a device type of “RPC” so that the terminal server knows how to handle the port:r8igm0v

Next navigate to the RPC configuration under Serial & Networks:mj3mucf

Next click on ” Add RPC”:ijx5dch

Next setup the RPC configuration on the IM7200 withe the following settings:dly8d0pSettings include:

  • Connected via – Serial Port previously configured
  • RPC Type – Cyclades PM10
  • Name – Whatever you would like to name it
  • Outlets (optional) – set it to 10 or leave it as default for auto-probing
  • Username / Password – Set to admin/password for PM10
  • Log Status – Enabled (Checked)
  • Log Rate – Setting you would like

The next step is to configure serial ports connected to console ports on the devices controlled by the RPC with the Power Menu enabled:twmrllo

The last step is to setup a Managed Device for each device to be controlled by the RPC, to do so navigate to “Managed Devices” under Serial & Networks:kawabbv

Click “Add Device”:fr3x9gy

Finally configure the device with a name, assigned console port, and assigned RPC port:f8ohwtg

After configuration the devices can be managed under devices:tlbf8dt

Or right from the console sessions via the terminal server:yla5om8

Happy labbing!

Cisco ISE REST API & Python

I’ve been faced with a fun little challenge on how to make sure our ISE deployment has every NAD (Network Access Device) configured appropriately to allow for successful EAP communications. Originally I was planning on utilizing a CSV and the bulk import tool to regularly import new devices into ISE as they were built. This allows for a number (small or large) of devices to be imported into ISE without taking too much time. This has worked well in the past but creates an reliance on making sure the CSV is proper and that someone (me) still has to manually login and import the file. With that I decided to look into other possibilities to remove the “me” from the process flow. At first I was looking into ways to automatically populate the CSV and then script out away to login to ISE and force the bulk import. While that option would work, it seemed to be too complicated to really deploy and rely on. I finally decided to give another whack at using the REST API ( I had previously tried years prior with ACS but did not have much luck).

There are a two things that need to be done on ISE prior to being able to utilize the REST API. The below screens and settings are based on ISE 2.2 but are similar between all recent releases of ISE:

  1. Create an account that will be utilized for the REST calls.  To do this navigate to: Administration > Admin Access > Administrators > Admin Users and click on “Add”:ialbmj4
    Currently there are two different access types you can assign: Read/Write or ReadOnly. For the code about to be run, we need Read/Write.
  2. Enable ERS (External RESTful Services) to allow REST calls. To do this navigate to: Administration > System > Settings > ERS Settings then select “Enable ERS for Read/Write” under the Primary Administration Node:fsdtf17
    This setting must be enabled after each upgrade as its set to disabled during the upgrade. If you plan to utilize the REST API, I recommend adding to your upgrade documentation / process that the REST API is enabled at the end of the upgrade.

After a user account is created and ERS is enabled the REST API can be utilized via HTTPS on port 9060. API documentation can be found at: https://ISE-PAN-IP:9060/ers/sdk

Now that the API is exposed its now for some fun! But first some cautions / warnings..

  1. This is by no means a tutorial about REST API or Python.
  2. You really should have a good understanding about REST API before enabling it. I’m still skeptical about the security around access when it comes to REST.
  3. You should never use a production system to develop code that makes changes to it.
  4. Use the code shown at your own risk!

And a few notes about the code…

  1. The below code is not complete, and needs tweaking to be functional. Its intention is simply to show a proof of concept for automating device creation.
  2. The code calls ‘nad.xml’ which is a separate XML file (can be found on my github repository). I will not be going over the file in this tutorial, but can be manipulated for actual use.
  3. The final output is not pretty and may not be complete depending on the number of devices being imported.
  4. The code below is a picture due to me not knowing how to easily paste code that looks nice on wordpress. A copy of the code can by found on my github repository.
  5. The IP address or FQDN of your ISE PAN needs to be updated prior to running the code.
  6. A proper authorization key needs to be added prior to running the code. This will be from the account you created earlier.
  7. It would be a good idea to create a variable for the ISE PAN information to use for multiple URLs.
  8. It would be a good idea to create a variable for the authorization information to use for multiple calls.

Now its really time for the fun part!

The below code is intended to do two things: Bulk create network devices in ISE and to verify the status of the bulk job:

kdtfnke

Code Breakdown:

  • Lines 2 – 8 are simply to deal with importing the XML file. You could just include it in the script and assign it to the payload variable (referenced in line 18) but that doesn’t make this usable in a production environment.
  • Line 11 is the URL used for the first API call. Don’t forget to update with your ISE PAN information.
  • Line 15 is where you should update your authorization information.
  • Line 18 has an extra variable in the request which is “verify” set to “False”. This lets you ignore certificate warnings. In my lab I didn’t bother deploying trusted certs so I needed this.
  • Line 18 is the actual API call being pushed. If you do not care about the status you could simply end here or just print the response.
  • Line 21 grabs just the location header from the response from the API call. The location header is a URL containing the BULK ID that is parsed from the entire URL to use for the second portion of the script.
  • Line 28 is the URL used for the second API call + the BULK ID. Don’t forget to update with your ISE PAN information.
  • Line 32 is where you should update your authorization information.
  • Line 36 has an extra variable in the request which is “verify” set to “False”. This lets you ignore certificate warnings. In my lab I didn’t bother deploying trusted certs so I needed this.

Lets see the code in action!

First we will look to see whats configured in ISE for network devices:9aczrfjNow lets run the script:hghklfn

As can be seen the XML containing 10 network devices was still in progress when the status check was run. If this was production code there are multiple options to avoid this such as a timer could be implemented, the user could be asked when to run the check, or constant checking until it completes.

Now lets see what we have in ISE:qghpxe9Ten brand new devices!

The API in Cisco ISE has many different functions that can allow for the creation, modification, or deletion of several different objects outside of network devices. This is just one example of the power that is available for automating functions within ISE that have been around for a while.

 

IKEv2 with RSA Signatures

Currently my studies have taken me on an adventure into the wonderful world of Cisco Security. I am studying for the 300-209 (SIMOS) certification exam which is VPN technologies including DMVPN, FlexVPN, and a few other flavors of VPN.I find it interesting that so many try very hard to avoid having to implement security because its hard, or because that’s not how they did it at their old job. In this post I am going to go over the simple process of adding RSA Signatures into a DMVPN phase 3 deployment.

 

Below is the high level topology I’ve been  using for my DMVPN labs:

DMVPNTopologyAs can been seen, its a single hub deployment with two spokes (one with two routers and the other with a single router). I am utilizing two separate tunnel types to allow for the addition of another hub (Datacenter) later on down the road. For this post, addressing doesn’t matter too much as I am only going over the parts that are required for RSA Signatures to work.

Lets start with the basics of what’s already in place:

  • NTP (Time) – All routers are already configured with NTP to have synchronized time. This is important for certificates as they have Valid From and Valid To fields that are enforced.
  • DMVPN Phase 3 with Pre-Shared Keys (PSKs)
    • This includes all parts from IKEv2 Policies, all the way to tunnel protection being configured. Maybe another time I’ll go over my lab but there’s a good amount of posts about DMVPN already so not sure it’s really worth while.
  • Routing
    • Both on the FVRF (Front Door VRF) and Internal VRF
  • PKI Infrastructure
    • I setup a single Signing CA for my lab environment with auto enrollment to allow for easy certificate signing on my routers.

Now that what’s in place has been gone over, lets see if things actually working.

dmvpn1

As can be seen, from DMVPNR2 (which is the DMVPN hub for the second tunnel), both remote locations are currently using PSKs and sessions are in a healthy state.

And now the fun part,  actually authenticating the CA, getting a signed RSA signature, and utilizing it with the current DMVPN configuration! As note, any configurations done on a device will be in italics, and any variable that can be changed will be in red like the following: hostname Branch2-R1

1.Generate a RSA keypair for your router:
crypto key generate rsa modulus 2048 label Branch2-R1.dmpvn.com

2.Configure the CA trustpoint:
crypto pki trustpoint TrustedCA
    enrollment url http://10.0.0.6
    rsakeypair Branch2-R1.dmpvn.com
    fqdn Branch2-R1.dmpvn.com
    subject-name CN=Branch2-R1,O=dmpvn.com
    revocation-check crl none

3. Authenticate CA
crypto pki authenticate TrustedCACA Authenticate.jpg

4.Enroll with the CA
crypto pki enroll TrustedCA

CA Enroll

The following fields will need to be filled out:

  • Password – This is needed to revoke the certificate
  • The option to include the Serial number in the subject name (Yes or No)
  • The option to include IP address in the subject name (Yes or No)
    • If yes – it must be configured
  • Accept the request to a certificate from CA (Yes or No)

The certificate request process runs in the background and syslogs are shown when the certificate is received from the CA. The command show crypto pki certificate verbose TrustedCA can be utilized to view the certificate fingerprints for the configured CA.

5. Create a certificate map
crypto pki certificate map CMAP 10
    issuer-name co CA
For my lab environment, I simply created a certificate map that looks for the word “CA” in the issure-name field of the certificate. This can be a very specific variable to allow for strict control over what exactly is allowed for authentication.

6. Create / Modify IKEv2 profile for RSA Signature based authentication
crypto ikev2 profile FVRF-IKEv2
    identity local dn
    match certificate CMAP
    authentication local rsa-sig
    authentication remote rsa-sig
    pki trustpoint TrustedCA

It’s important to make sure you add the authentication local and remote commands for rsa-sig, without them PSKs will still be used! In addition, if the profile is being reused and had the configuration for PSKs, the command for PSKs is NOT overwritten when the rsa-sig command is issued and need to be removed with the “no authentication remote pre-share” if it is no longer desired to utilize PSKs for IKEv2 SAs.

 

And that’s it! Any new SA for DMVPN IPSEC tunnels will be created utilizing RSA. Now, the way Cisco IOS maintains security SAs, they will be active with their negotiated security settings until the SA is renegotiated (if supported) or torn down. This means the old sessions (if RSA was added to an already configured deployment) will have been setup with PSKs. The easiest way to “refresh” these SA’s is to flap the tunnel. Extreme caution should be used when pushing changes to production environments, and as always test in your lab first!

Hiding (filtering) a specific user from reporting in Cisco ISE

I ran into an interesting problem preparing for an 802.1x deployment – the authentications report in Cisco ISE was full of all the network devices checking to make sure ISE was still available (health checks). As seen below the load balancer’s keep alive fill the logs pretty much on their own, imagine trying to troubleshoot a login issue!1 YUCK!

Something else I found interesting that my Google Foo (or knowledge of ACS and how to filter out a certain user) was no match for trying to find a solution for my issue. Because of this, I decided a quick how-to on this would be helpful (I can’t be the only person who will want to filter out such an annoying problem).

First Navigate to Administration > System > Logging:2

Once in the System Settings for Logging, navigate to “Collection Filters”:3

At this point, the rest is pretty straight forward. But for completeness I am going to finish the whole process, so click “Add”:4

After that just fill in the type of attribute you want to filter (Username, Policy Set Name, NAS IP Address, Device IP Address, or MAC Address), the Value for the selected attribute, and the Filter Type (Filter All, Filter Passed, FIlter Failed, or Bypass Suppression [with time limit]). Finally, click “Submit”!

5

For me, it made the most sense to filter the username used for the monitors, and to only filter on passes for that username. This allows me to use the least amount of filters, and if a health monitor fails for any reason will show up in the reporting still.

Final result (don’t mind the old logs, I was too impatient to wait for them to clear):

6

Happy troubleshooting!

Controlling Traffic to a Virtual Server on F5

There are multiple ways to control what traffic is allowed or not allowed through a BIG-IP or for specific Virtual Servers (VS). The following method uses F5’s AFM (Application Firewall Manager) to create security policies which are then applied to a specific VS. For this method example, traffic from three specific hosts will be allowed to a specific VS, while all other traffic is blocked. The below diagram illustrates the environment used:
diagram

In this example the VS (192.168.35.100) is setup for TCP port 80 (HTTP) traffic only. This within itself controls some of the allowed traffic flow (assuming there are not other VS configured to accept all traffic) by only allowing port 80 traffic destined to the VS to be processed. An address list is created with the addresses that are allowed to the VS and then a Network Firewall Policy Rule is created and applied to the specified VS with an accept action. To finish this method, a block rule is added to the Network Firewall Rule Policy rule-set to drop all other traffic.

The first step is to create a Address List by going to Security > Network Firewall > Address Lists and either clicking the plus sign under Address Lists:
AddressLists

or selecting Create on the next screen:
CreateAddressList

Enter the following Information (Only Name and Addresses are required):
CreatingAddressList Click finish when completed. A side note, the first time I did this I spent a few moments figuring out you must push enter after typing the address in the address field.

The next step is to create the Rule Policies by going to Security > Network Firewall > Active Rules and clicking the plus sign under Active Rules:
ActiveRules

or selecting Add on the next screen:
AddRule

Enter the following information:

  • Context: Virtual Server – VS_Name
  • Policy: New – Policy_Name
  • Rule Properties:
    • Name: Rule_Name
    • Description: Rule_Description
    • Source:
      • Address/Region: Specify
      • Type: Address List
      • List: Address_List (From last step)
    • Action: Accept

NewRule Select Repeat to move onto the next rule.

For the next rule include the following information:

  • Rule Properties:
    • Name: Rule_Name
    • Description: Rule_Description
    • Source:
      • Address/Region: Any
    • Action: Drop

NewRule2 Click Finished when all set. When adding the block rule, or any other rule for that matter, make sure to add to the current policy. Otherwise, the current policy will be overwritten along with any rules in those policies. In this example creating the blocking rule improperly would overwrite the policy with the allow rule and block ALL traffic to the VS.

Once completed, it is now time to test. Attempting to browse to http://192.168.35.100 from an “allowed IP address” should allow access to the site while browsing from an IP address that isn’t on the list should be blocked.

HSRP – No longer for the weak of heart

Often times, a remote location requires high availability. This high availability will include two separate routers, each with their own WAN circuit connecting back to the Data center.

In order for workstations at the remote site to be able to utilize the “active” router a FHRP (First Hop Redundancy Protocol) must be used. There are three major FHRPs: HSRP (Hot Standby Router Protocol), VRRP (Virtual Router Redundancy Protocol), and GLBP (Gateway Load Balancing Protocol). VRRP is a standards based FHRP while GLBP and HSRP are Cisco proprietary FHRPs.

In Cisco only organizations, HSRP has a higher tendency to be used as an FHRP and will be covered today. With HSRP, a virtual MAC address is used to respond to ARP requests from workstations trying to resolve the MAC address of their default gateway. Within HSRP, the Active router is determined based on priority (default priority is 100). The highest priority router becomes the active router for the HSRP group. The HSRP active router handles frames destined to the virtual MAC address and forwards traffic normally. HSRP1

Figure-1 HSRP: Normal Operations

In the event of the HSRP active router failing, the standby router will take over and start handling traffic for the virtual MAC address. This allows for workstations to continue sending packets back to the Data center without requiring manual intervention of the workstations.

HSRP2
Figure-2 HSRP: Active Router Switchover

Lets now review a basic HSRP configuration. Within a basic configuration only two settings are required:

  • The interface must be configured with its own unique IP address
  • The standby group needs to be assigned it’s virtual IP address

In addition to the above two configurations, it is also recommended to set the priority on the desired active router to something higher than the default (which is 100). Preempt should also be configured as well on the routers, this allows for the router with a higher score to immediately take over as active.

Left router configuration:
code-1

Right router configuration:
code-2

As seen above, both routers have their own unique IP address on their interfaces while they share the same virtual IP address. In addition, the right router does not have a priority configured meaning it will use the default priority of 100.

Within a basic HSRP configuration, the only way a HSRP switchover occurs is when the heartbeat between the Active router and the Standby router is broken. This means that if a fail-over occurs upstream (BGP peering is lost, the WAN circuit fails, etc.) of the HSRP active router, the active router will stay active and end up causing a traffic black hole.

HSRP3
Figure-3 HSRP: Traffic Black Hole

HSRP supports the ability to track interfaces, IP SLA Objects, and IP SLA Object groups. In a typical HSRP configuration with tracking, interface tracking is used to track the upstream interface connecting to the WAN.

Interface tracking can be easily accomplished with the following command:
standby 0 track FastEthernet 0/1 25

The command itself is pretty explanatory, first you specify the standby group you are applying the configuration to. Next you specify what you are doing with the standby group (“track”). Then you specify what you are tracking, in this case it will be FA0/1. Finally, you give a value to how many points you want to decrement if the tracked object goes into a “down” state. In this scenario, where the active router has a priority of 110 and the standby router of 100, if the interface on the active router goes down it would loose 25 points from its priority making it 85. This score, which is lower than the default 100, would cause the standby router to take over as the active router to forward traffic.

The complete interface configuration with interface tracking looks like:
code-3

Lets now take a step back to look at the bigger picture to see why interface tracking only works for a small subset of WAN failures.

HSRP4
Figure-4 HSRP: The Bigger Picture

A typical HSRP configuration with basic interface tracking has the desired active router configured with a higher priority than the default priority left on the standby router. The active router than has interface tracking setup to track the interface connected to the WAN. If the tracked interface fails, the priority score is decremented to a score below the default priority (100) set on the standby router, allowing the standby router to take over as active.

HSRP5
Figure-5 HSRP: Interface Tracking

During an upstream failure, basic HSRP tracking will not see the failure further up the traffic flow (the interface will stay up). This causes a similar traffic black hole situation seen earlier where the active router stays active but has no path back to the Datacenter.

HSRP6
Figure-6 HSRP: WAN Traffic Black Hole

With a more advanced tracking configuration it is possible to track further upstream either into the WAN circuit providers network or into the Data center itself. In order to track upstream, IP SLA (or IP SLA Object Group) is required to keep track of flow through the traffic path.

*********Read this part*********

There are a few considerations that must be made when implanting a more advanced tracking configuration:

  • Does asynchronous routing negatively impact applications / services?
  • Is there more than one Data center?
  • Is there a routing protocol between the two routers?

If you answered yes to any of the above questions, the following configuration will cause routing problems in your network unless properly tweaked to handle return traffic from the WAN router(s) at the Data center(s), and to prevent routing loops between the two routers at the remote location.

*********I hope you read it*********

To combat an upstream WAN failure, IP SLA tracking will be implemented. Ideally you will already have an IP SLA responder somewhere in your Data center, but if there is none, any device can be used just remember that you want it to be very fault tolerant.

The configuration of the IP SLA for HSRP tracking has three parts; the IP SLA itself, the track object, and special attributes required for it to all work together.
code-4

The HSRP configuration itself will remain the same as before except instead of tracking the interface, the track object will be trackedcode-5

Lets break the commands down and review what each one does.

Within the ip sla 11 configuration there are two parts needed. The first section of the first part is icmp-echo states that the IP SLA test will be an echo and its destination. The second section is source-ip and is used to force the router to send its icmp from the interface that owns the IP address, in our case we want the WAN interface to be used (this prevents issues seen when the router is aware of the routes the other HSRP router has). The second part of the ip sla 11 command is the frequency at which the test will run, this is in seconds.

With the track configuration we create a track object (object 10 for this example), assign it to a ip sla (the ipa sla previously created), and reachability is used to inform the track object that the reachability of the IP SLA will be used for its state. Within the track configuration its possible to set a up and / or down delay. This delay is used to prevent a flapping state, and is tracked in seconds. A 15 second delay is a pretty safe delay to make sure the circuit is really down or really up, keep in mind that the up delay means that even though the IP SLA is down, traffic will still be forwarded into the probably black hole.

Finally we have two additional required configuration options. The first command, ip sla schedule 11 life forever start-time now starts the IP SLA which is used by the tracked object. It must be started otherwise the track object will be marked down. The command causes the IP SLA to start immediately and run forever. The second command, ip sla enable reaction-alerts allows for the IP SLA to be tracked properly through the system.

One change on the interface configuration will be required also, this is standby 0 track 10 decrement 50. This tells the router for the HSRP group to track object 10 and if it goes down to decrement the score by 50.

For completeness, the configuration of the other router is below:
code-6

 

There is no interface tracking on the standby router due to the fact that it is your standby device and decrementing points for HSRP would provide no positive impact. With that said, its possible to utilize the tracking configuration examples to build an EEM script for custom alerting (but that’s for another time).

 

As always, every network is different and careful consideration should be used before implementing any technology. It is always best to test!

 

Finally we will see the HSRP configuration in action. Below is the output from the Active HSRP router:
example-1

First we verify that it is in fact the active HSRP router. This is done by using the command show standby. As can be seen “Active router is local” which means it is in fact the active router. During this example, the interface on R01 was shutdown going into the WAN cloud this simulates the WAN circuit at the Data center failing but still leaving the remote locations as up and with BPG neighborship. IP SLA 21 was created to immediately announce the connection being down. this shows that IPA SLA 10 waits 15 seconds before triggering. Once triggered HSRP state moves into the speak state (the other router immediately takes over as active) and then into the standby state. We can then verify with show standby who the active router is, which in this case is no longer the local router.

Bonus (router taking back active role):
example-2