Complexity & Change Management - A Lesson for the Week

Complexity in IT is common place. No matter how hard we try to reduce complexity some solutions are complex. Sometime over the life-cycle of a system, changes cause the once simple solution to be complex.

Change Management - "Change management is an approach to shifting/transitioning individuals, teams, and organizations from a current state to a desired future " - from wikipedia. Every organization has different practices ranging from the formal to the informal.

My organization has a fairly informal change management process, outside of certain changes going through a form in the help desk system, changes are on the less documented side. Now let me be clear with 7 members of the IT department, most of us are all aware of the changes that are happening. Yes there is room for improvement into the process.

So to the lesson this week, needing to elevate the Windows Domain and Forest to 2008 level instead of the mixed mode we have been in. Being in mixed mode is now limiting us in GPO tasks and other important projects moving forward.  Server admin's task was to demote 6 DC's across two domains this week.

This started off by consulting the Department's - Jedi Holocrons

Image from moddb.com

That starts by yelling over the cube walls "Hey Dave! If we are going to do this what are we going to break? " I rattle off the things that I know that will be effected plus a couple more that are maybes.

Demotion of DC's start, and are moving a quick clip. Server Admin works with our developer to make sure some custom user provisioning process get moved.

Bump number one - Several web apps point at the demoted dc's for authentication. Fix for one was change in web.config and iisreset. Other was a bit more complex as in change authentication.config then push that into the app.

Bump number two - Custom user provisioning code is hard coded to specific DC's. Stood over the developer's shoulder to verify changes were correct, then install all the dependencies for it to run on the server that we moved the code to.

So hopefully the last DC will get demoted this weekend without any trouble and we can move forward.

Luckily only one set of our users were affected by these bumps, and yes that was faculty/staff opposed to students.

So what seems to be a simple process turns into a bigger one with the lack of complete documentation and change management processes. I am not an advocate of ITIL or a strict rigid change management process, but having a process will help.

In this situation we could have avoided the bumps if some more documentation was kept and read through. But by understanding the overall picture and knowing what we needed to get through we were able to work through the bumps.

PacketFence 3.5

So back in August of last year, I wrote about the previous version of PacketFence that I ran for many years, version 1.6.7 [See Here]

In that post I stated that I would write more about the newer version, well months have past, going to change plans about that post due to the fact that version 3.6.1 is out.

So I will highlight some of the big differences between the versions.

• No more ARP spoofing - Multiple options, including use of snmp traps and port-security, to 802.1x mac security.
• Effectiveness of trapping - Once a client has been identified they are switched to a vlan, instead of having the router address spoofed. It works fast and with less overhead.
• Scale of server - Due to the fact that traffic is not all trunked into the server the server can handle more tasks and is more responsive to the admin requests.
• 3.5 has included many of the components under the control of the PacketFence processes- freeradius being a big one.

Problems that still are being addressed: Xbox 360's dhcp fingerprint is not detected correctly, so that they do not auto-register. Some of the reports still need work as they take a long time to run (some of these have been fixed in 3.6.x)

Features that I still need to try or want to implement:

• Guest Access - PacketFence has a guest portal, this would be ideal to the separate SSID/system currently used on my campus.
• Game system registration form - In 3.6.x there is a self-service form to register Xbox's and other systems that my not auto-register. This would help the manual process done now. 

So this is a quick follow-up to the post from August on my PacketFence install.

Xbox Live - Cisco Wireless

So I had this problem creep up ever since the students showed back up on campus for the spring semester.

From the trouble ticket: "My Xbox won't connect to Live"

What? What do you mean it won't connect. I saw xboxes connected to the wireless network all over campus. Then the flood of tickets stated coming in. Most of the students know by now that they need to register there xbox on the network via a helpdesk ticket. This is due to the problems with DHCP fingerprint with our registration system, Packetfence. If you want to know more about that read some of my other posts or encourage me to finish some about the updated version.

Anyway back to the problem at hand, Xboxes not connecting to Live. I have tickets out the wazoo about them, grumble grumble, search up and down. MS points to reboot your router and reconnect. Yeah that will happen.

So last friday, I got an xbox 360 from a student to test with. Connects to Wifi, but that is as far as it will go. Connected to the wired network, tests clean to live, except the NAT, which is a fact of life. Back to the wifi, tests show no response to icmp. Hmmm. Without dancing through some hoops, I do not have the proper tools to capture the wireless traffic. Ok lets do the next best that I can do, track xlate and conns through the firewall and then compare them.

Shortened and simplified
Wired side:

• Client builds xlate
• UDP to Live address
• TCP connection to several other Live addresses

Wireless side:

• Client builds xlate
• UDP to Live address (Same address as wired side)
• Teardown connections - Nothing more

So this behavior has me thinking, am I fragmenting packets? But why? Where?

I finally came across the following article after exhaustive searching: http://revolutionwifi.blogspot.com/2010/07/fragmentation-in-controller_02.html 

Andrew refers to LWAPP Fragmentation in  L3 LWAPP transport - Check doing that. Ok so what I am using. I read down farther to the prevention, and I check my controllers/ap's. Adjust TCP MSS not checked on any of my AP's.

Enabled that on all the ap's on the controllers and then checked back with students. 

Responses back, "It's working, I can connect to Live"

Why did this come up? 

After last semester I upgraded the code from 6.0.xxx to 7.0.235.3 on my WLC's because of the MS Windows 8 issue. So either this was un-done in the upgrade or was not a factor in the previous code. Not sure, but since I have a fix, I am not going back to find out. Time to move forward with the many other projects.

Hopefully this helps someone in the future. Since connecting an xbox to a enterprise wireless network is not really covered in MS help.

Also I would like to thank Andrew von Nagy @revolutionwifi for his blog article, Josh O’Brien @joshobrien77 for pushing me to post and to keep looking. 

Note: This may need some edit but just trying to actually push it out the door before it ends up on the cutting room floor. 

PacketFence - 1.6.7

So I run a NAC solution called PacketFence. http://packetfence.org This is an Open-Source project. At the time I started using it, I had mainly used windows boxes, some VMS (scary), but very little Linux. The project was brought to my attention via a student employee, who was much more proficient with Linux than I. He setup a working copy on a desktop machine as a proof of concept. Seemed to work and was customisable and it was the right price, FREE.

The method used in PacketFence 1.6... was ARP Spoofing. Yes, I did just say arp spoofing. For the configuration/process:

1. Trunk Vlan to server
2. Give the PF server ip address on vlan
3. Tell PF that it was trapping on that network
4. Every 60 seconds it ran it's "Arp Gun"
5. Inject mac address of PF server in as router of systems that were not registered
6. Client would go to PF server as gateway 
7. PF server display captive portal 
8. User Registered with captive portal 
9. PF would "release user" Giving them back the correct address of the Production Router

This method worked fairly well. Some problems with it:

1.  Trapping/Registration didn't happen right away, could take hours.
2.  Clients that had the mac of the router on their subnet could place static entry in their arp table and bypass registration/trapping. 
3. A/V- Security suites would detect the mac change of the router and throw warnings. 
4. Overhead of listening on several vlans

After running this for many years, I decided that some changes were in order for the design of the network, and trunking these vlans into the PF server would not be feasible. Also with the effectiveness of trapping lessening it was time for an upgrade. 

Next Post will be on the upgrade process and current version. Stay tuned. 

NAC - Sounds Scary - First Thoughts

So for the last 12 years that I have run the network at my University we have tried various methods to manage student devices on the network.

• Paper Forms (Dumb ass Idea)
• Port-Security (Static, goes in with above)
• VMPS (If I remember correctly, you know the Cisco vlan switch-a-roo server) Kinda worked,. Custom Front-end written by a former student.
• PacketFence 1.6 - (Ran that version for like 4 years)
• PacketFence 3.5 - (Started at 3.2 testing, Prod today is at that version)

NAC can be hard and painful to implement. As you see from above I've used a couple of versions of PacketFence. I am happy with the product. I've been very pleased at the work the lead developers have done since giving the project a backing a few years ago.

I plan to run through the deployment that I have just about completed soon. So stay tuned. Just not to close, it is August and students are starting to return to campus, which mean if I wasn't busy before, I will be now.

Lessons - Newbies

I was thinking the other day, what lessons need to be learned by everyone who wants to have a career in networking.
This topic comes up in a lot of different places, I've seen several articles over at packetpushers.net related to getting started and changing gears. I think about this a lot as I have students who have networking in their major, who work for the IT department, have some weird questions. We have a Cisco Academy, which the lab is on the other side of the wall from my cubicle.

So lets start with the list, in no particular order:

• Operation of Spanning Tree Protocol - I get yeah it prevents loops by.... What does it do when it is off and you put the loop in the network? What happens to a router which is setting up a vpn tunnel and looking for traffic from the lan to build that tunnel and you put a loop in the network? (Short answer: NO Tunnel) My last post was about STP: http://mrfogg97.blogspot.com/2012/04/spanning-tree-stp-love-it-hate-it.html
• 802.1Q operation between switch vendors, not everybody configures like Cisco.
• You will forget to save the config at some point, especially if you have multiple vendors gear - see point above.
• Shutting down the wrong interface - ie the one that connects you to the equipment, learned that one very early, had to walk across campus to fix my error.
• RTFM - Read Read Read, then ask questions.
• Multicast operations - because it is in use in a lot of places that you may not realize. Avaya One-X Quick Edition phones use it to save voice mails  between all the units on the network.
• Your gonna fail, learn from it
• Know the difference between a hub and a switch - use the terms correctly
• Any business, bigger than a 50 people probably has more than one subnet / vlan. That means that you can not just plug any cable into any switch port, as a rule.
• Structured Cabling is there for a reason, keep it neat. (I'm still working on this at times, past sins)
• If you encounter one of these:  It takes two cables to complete the patch. Switch to mid-span , mid-span to patch panel. See RTFM - the ports are labeled.

Avaya Mid-Span

So lets put this one in the bag... We are all still learning, keep an open mind. These are some of the things I see or have done. 

Disclaimer: This may be disjointed post, but I needed to get it out. Comments are welcome. 

Spanning Tree - STP - Love it & Hate it

So every network engineer I talk to seems to have the same relationship with Spanning Tree. The love it when it works and hate it when it works. Spanning tree is used to prevent loops in the network. Link to Read more, as I will not explain the whole thing here.

When I posted the following picture on twitter, it generated some chatter.

What??? That could be a problem. Especially since those are both loops. Now if these are both cabled to two ports on the same switch and STP is on, one of the ports will go into blocking mode. If STP is off we have a loop and then we have lost all control. Which should be a lesson for these younglings in which they have to learn. This lesson is best taught in multi-vendor networks as there are different default settings for STP between vendors.

This picture was taken in the Computer Science Networking Lab. Two things about the lab from my prospective:

1. I give them a single feed to the rest of the network, one vlan and a small dhcp scope in which to make things easy on them (There is more address space for them to use outside the scope, use at their own risk) 
2.  Any problems I have in which the rest of my network is threatened, shutdown their feed.

Now back to the photo, is STP in play, is there a loop? 

No loop! except in the cable. Each jack in this case represents a different rack, so the wall plates are "stations" to work at. This keeps the younglings from stacking onto of each other doing labs.

This is the way they chose to bring production network to the other side of the room to the lab rack. This was done by our Cyber-Defense team, last year's team can be found here.  

No worries on this one but it is right inside the door and makes me double-take every time I see it.

Thanks to all that responded to the post on twitter.