Where did the time go? - December 2013 Edition

The other day in reading a few blog posts I notice that some of my favorites were a little neglected. I was about to give them a little jab over twitter, then I thought when was your last post? Umm, Umm, Yeah got myself. August 25th. It's December, where did the time go?

I realize that I am not the master of content, churning out daily posts or even weekly posts. But I think it was time for something.

I could go down a list of thing of how we got to this point but let me filter out the more technical highlights.

Things that have kept me busy since the last post:

• Fall Semester started - now is almost over, couple more days.
• Limited packetfence work other that keep it running. FreeRadius issue - patched with restart script
• Cloud storage vendor change
• Twinstrata implementation - Lots of data move from one provider to another
• Office 365 Implementation - Still in process
• Initial setup was contracted, production migrations up to us
• Migrations scheduled into early next year (2014)
• Personal device Printing - Still in process
• IOS/AirPrint - Papercut
• Bonjour over Cisco controller based wireless
• Family - busy with activities for the kids

With Christmas break coming up, unlike some who get a change freeze, I get a change window, including some during the middle of the day.

Looking forward past the current items: 

• Testing of new gear
• Airtight AP - received that in late November, have to dig into the interface to see all the nerd knobs
• Brocade switches
• Received 2 - 6610 and 2 - 3450 switches to evaluate uses as campus switches
• The 6610 has the potential to be used as a core replacement, with the stacking and L3 features. 
• Getting back to OpenDaylight. 
• Writing a few more posts. The list above has more than enough topics to write on, just have to do it.

If anyone picked up on that little tidbit a few lines back, yes my core switch is slated for EOS in a couple years, so I think it is time to evaluate what is out in the market. 

PacketFence 4.0.x Game System Registration

I came across a little snafu in the Gaming System registration in Packetfence 4.0.X. In PacketFence you can have a page that allows users to register systems that don't have web browsers on them. You can access it by https://<packetfence ip>/gaming-registration .

Great since Xbox's are notorious for not getting registered, due to collision in the OS fingerprinting. This is a well documented problem. In my previous install they would fingerprint to OEM Wireless Router, this time it seems RIM BlackBerry. Either way they miss my auto-registration violation rule, which I may just get rid of.

So trouble started when several students tried to register their Xbox's using the registration page.

 They would authentication to the first page then enter the MAC address:

 Then it would throw this error:

So what gives looks valid. Turns out there is a file: <PF install dir>/lib/pf/web/gaming.pm Which has a list of the first part of MAC addresses allowed to be entered into this page. Added the first 3 octets of the address to the file in the correct spot. Save the file and you are back in business.

I recommend that you verify the manufacturer  of the mac. I use this site: http://www.coffer.com/mac_find/ this allows you to verify that it is a microsoft mac. Hope this helps if you come across this problem.

Packetfence 4.0.5 - Notes

Packetfence 4.0.5 was released on 8-12-2013. Slight bit of craziness, since there were a couple problems with it. But Inverse issued patches within hours. Now the downloaded version is stable and works fairly well.

I however ran into a couple bumps with that upgrade/patch.

First one was I ran into this bug: http://www.packetfence.org/bugs/view.php?id=1676
Which basically the radius server was throwing SOAP errors and would not return the correct vlan, and then after a few hours just stop running. The patch listed here: https://github.com/inverse-inc/packetfence/commit/4861189ba7faf680eef257d5b1c157d7260fe0de  Was missing/didn't apply to the update. So I found the line in the source and edited it to reflect the change. This resolved the issue and the radius server has been running since.

The other was the retrieval for role for gaming device registration. This was a quick two lines added to module described here: https://github.com/inverse-inc/packetfence/commit/36bacc02289afb01a1abd38420585c7f792a4511

At this point I have to freeze the code, no more updates as this is completely in production. And the students are back on campus. The only changes I think we be made are page edits to fix the wording or add more information to the pages.

Update to this post:

After experiencing some problems, I found that my installation is still at 4.0.1 - which is why these patches were not there. Only a certain part of PF was upgraded. Not anything to make it matter.
 {Link to new post when done}

7Signal - Thoughts [Pre-WFD5]

[I have received a Webex presentation and in-person demo of the 7Signal product. This is also a stream of thought post.]

7Signal is a wireless performance and optimization company. Their products do not provide wifi, they monitor and test your wireless network, so that you can optimize it and get the best potential out of your investment.

There solution is listed below- [straight from the data sheet]

Solution components

Sapphire consists of three elements that measure, record, report, alarm, analyze, troubleshoot and verify 

WLANs:

1. Sapphire Eye: Unobtrusive ceiling-mounted scanners that measure large wireless coverage areas.

2. Sapphire Sonar Server: Sonar test servers are located in close proximity to application servers. 

Sonar Server is the endpoint for user experience measurements performed by the Eye units. Sonar 

reports results back to Eye units and then forwards those reports to the Carat Management Server.

3. Sapphire Carat Management Server: A centrally located Carat server stores, manages and 

analyzes the collected data from the Eyes. It provides reports and alarms and includes analyzer 

software. 

So there is the Sapphire Eye which you mount on the ceiling in your environment in the area in which you want to monitor and test. The Eye has 7 directional antennas which are utilized in all of the test. To say that it is "unobtrusive" I will beg to differ. It looks like a upside down flower pot. I know that I have some areas in which if I hung it from those ceilings, someone would hit their head. With that said, I understand the reason why it it so big, the antennas are not your cheap usb adapter 1.5 db antennas. They are big! There is a compass feature in the unit so that you can always know the direction of the antennas no matter how you mount it. 

The software is in two pieces, server software which runs/controls the test that the Eye performs, collects data and such. Second piece is the management console which you can access the reports and data that has been collected and processed. Graphs and charts and all sorts of data is displayed.
Since I didn't drive the console and have only a limited feel for it. Watch the videos and read the data sheets for a better look.

My explanation of the product is: "Hang a wireless engineer from the ceiling with all his tools 24/7 and get data. Giving you a real time site survey and analysis"

Pricing:
As explained, you can go with a CAPEX model, OPEX, or hybrid of the two. These models allow some flexibility depending on your organization.

My feelings without naming price, is it is on the expensive side. (I'm pretty cheap so take that how you will... Actually just contact a partner and find out for yourself.) If you take the offering and break it down into what a professional wireless engineer would bring to the job, then if think they are on par.

The OPEX model has a lot of intrigue. The list price is like buying an mid-level AP per month per Eye.

Include or not depending on options is a start-up analysis/recommendations from their engineers.

I was ask if I thought is was worth it from a colleague. I believe it has value and great potential for me, not sure that I can justify the price for my organization. The costs used in their calculator seem not to be in my ballpark as ROI and such. (If I could plug my own values in, might change a bit.) Now there is flexibility to move the EYEs around into spots that are having trouble and then work through that area.   

Currently I am thinking more about the product and the value. I will be watching the WFD5 stream to see what they present. I hope to update this post in the coming weeks as I will have flushed out some ideas further.

Packetfence Guest Email Domains Check - Updated

So in my quest for latest version of Packetfence 4.0.x I am enabling the guest access. This is done so that guest account creation for wireless will be reduced to only the special cases. The user contact info is captured so I know who is on the network from a rough perspective.

Inside of Packetfence the guest module check the email against the local domain of the packetfence server so you can block your normal users from getting guest accounts. Since they expire in a short fashion and a limited network access it really does the trick.

I ran into a little problem, multiple email domains which need to be blocked. After a post to the listserv and a response that indicated to look in the guest.pm module here is what I did:

my $email_type = pf::Authentication::Source::EmailSource->meta->get_attribute('type')->default;
    my $source = &pf::authentication::getAuthenticationSourceByType($email_type);
    if ($source) {
        unless (isenabled($source->{allow_localdomain})) {
            # You should not register as a guest if you are part of the local network
            my $localdomain = $Config{'general'}{'domain'};

# Added explicated second domain for check below 

my $localdomain1 = ‘second domain name’;

            if ($cgi->param('email') =~ /[@.]$localdomain$/i) {
                return ($FALSE, $GUEST::ERROR_EMAIL_UNAUTHORIZED_AS_GUEST, [ $localdomain ]);
}

if ($cgi->param('email') =~ /[@.]$localdomain1$/i) {
                return ($FALSE, $GUEST::ERROR_EMAIL_UNAUTHORIZED_AS_GUEST, [ $localdomain1 ]);
            }
        }
    }

I added a second check for another explicitly defined domain $localdomain1 yes I could change that but it was quick and dirty. Added is the code in orange. 

Tested and this does exactly what I wanted it to. Just thought I would share.

Updated: 7-25-2013

On the advice of Inverse, I took the above code blob with the surrounding sub and transferred it to the custom.pm and changed the section to reference the pf::web::guest so that this new sub would take precedence.

Thanks again to the Inverse Team.

Packetfence 4.0.1 - First load

As the summer window of working on projects before the day students return is rapidly shrinking, I needed to start on my Packetfence upgrade. If you read my blog before you may know I am a pretty big user of it.

As I wrote back in May it is a fresh install, no direct upgrade path. Getting started, I loaded up a vm with CentOS 6.4 and started down the PacketFence Admin Guide to install. The install guide is fairly straightforward to follow. There is a section on install which on RHEL/CentOS system you have to add additional repos to get things going.

Great been here, not sure I remember the last one from the last install. Ok no big deal, right? I hit a small roadblock with this one. There is only 1 copy of that repo, and it was down. Argh!

Read some more into the doc:

 Debian and Ubuntu
All the PacketFence dependencies are available through the official repositories.

Ok scrap the CentOS 6 install, load up Debian and lets rock. Everything went smooth after that.

PacketFence install within 15 minutes of Debian system being up.

Initial Impressions:

Web Configurator: Awesome to help get things setup.
New Admin Portal: Awesome, leaps and bounds better than 3.5x, the amount you can change inside the interface is great.  I was editing the AUP_Text and save and refreshing the captive portal with the changes live. (Couldn't do that before.)

System performance: Not in production yet so can't say.

The interface is snappy. The search is great with the ease that you can add on more rules to search for to narrow down the person/node that you are looking for.

Guest management is just what I was looking for.

This is NAC that Excites. Hats off to the Inverse.ca team for this work.

Certification - Where do I start from?

So I have been working in IT since 1996, did mostly PC work until 2000 when I started doing this network thing... We had all this Cisco gear, and I wanted to go get my Cisco certs. But it was clear that they were not going to help in my current employment (Financially). Put it on the back burner, kids came along, further back.... Less and less Cisco gear, some staleness in learning on my part, also additional responsibilities. Finally decided to get my Masters degree (price was right, working in EDU does have some perks).

Now after starting this blog, using twitter, connecting with others in the industry, I have decided that maybe it is time to move ahead with a certification. My goal is not too accumulate certs for the sake of doing so. It really is about learning and growing.

Daily tasks cover lots of topics not just networking. Server Admin, Telecom, Networking, PC, Database, Security, etc. The "All other duties as assigned" part of the job description fits quite well. This happens when working in a department of IT department of 7, err maybe 8 now.

On my goals in my review this year I put down obtain one certification. The $64,000 question is which one?
That is what I am asking, I would appreciate some feedback as to options?  Cisco, HP, Security related. Just looking for a place to start with, and the value of it.

So please leave a comment, send me a idea on twitter.

Thanks

PacketFence 4.0

Well, the Inverse team released a new version of PacketFence. Current release is 4.0. This is a jump up from 3.6.1. http://www.packetfence.org/news/2013/article/packetfence-40-released.html


Highlights from the news release:

New Features

• Brand new Perl-based Web administrative interface using the Catalyst framework
• New violation actions to set the node's role and deregister it
• Support for scanning dot1x connections for auto-registration by EAP-Type
• Support for auto registering dot1x node based of the EAP-Type
• New searchable MAC Addresses module to query all existing OUI prefixes
• New advanced search capabilities for nodes and users
• New memory object caching subsystem for configuration files
• Ubuntu packages

Enhancements

• Authentication sources can now be managed directly from the GUI
• Roles (previously called categories) are now computed dynamically using authentication sources
• Portal profiles and portal pages are now managed from the GUI
• Fingerprints and User Agents modules are now searchable
• Translated all remediation pages to French
• Updated Brazilian Portuguese and Spanish translations

Lots of good features listed. I am looking forward to the new admin ui. That is something in the current version which needed help. A heavily loaded system moved slow trying to admin it. 

New search is something which was needed. If you didn't get it quite right the first time you could be sitting a while when waiting for the results to load.

Overall the list of changes looks like a big step forward in usability for the system.

A big gotcha listed in the release notes is the recommendation to start with a fresh install. With the major rewrite it would be difficult to just upgrade in place. See UPGRADE document.

I will be starting my migration in the next month or so. I plan on posting a review once I get my system updated.

OpenDaylight SDN on Windows

So following the tutorial from @networkstatic found here: http://networkstatic.net/opendaylight-openflow-tutorial/ I was able to easily modify the steps and install/build OpenDaylight on Windows.

I used Windows Server 2008 R2 as my base system.
Components added:

• Git for Windows : http://code.google.com/p/msysgit/downloads/list
• Apache Maven: http://maven.apache.org/download.cgi
• Java JDK/JRE: http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

Install Git for Windows, I selected the option during install to use the windows command shell for git.

Extract Apache Maven from the zip file. Add the path of the extracted files to the system path so you can call maven from anywhere. Makes life a little easier when you get to the directory inside of the clone source.

Install the JDK/JRE and add an environmental variable: JAVA_HOME and the path to the JDK. Things don't go without it.

Once you have the prerequisites done you can pull down the source from git.opendaylight.org

From a command prompt:

git clone http://git.opendaylight.org/gerrit/p/controller.git
Next change directories into: controller\opendaylight\distribution\opendaylight\




Run the following command:

mvn clean install

If your path is correct Maven should start going. This is the long part. If you read through Brent's tutorial you can see this step is long and will be a possible trip up. It is possible that something doesn't build right, this is active code being changed by many all the time. 

Once built you can change directories to: target\distribution.opendaylight-0.1.0-SNAPSHOT-osgipackage\

As you can see the path is pretty long, but there is a run.bat. Call that file and you can then browse to: http://localhost:8080 , on the machine in which you just built the project.

You can login using username: admin password: admin

And there you have it. 

From there it is up to you. I am still working down this journey as are lots of us. Thanks for following along.

Starting with OpenFlow

So after about 8 months of having Openflow code running on a switch, I was able to get a controller up and going. This is because using Openflow is not a business necessary project.

So why Openflow?

There are a few reasons in which I would like to use it.

• Redirecting a copy of user flows for analysis- Remote packet capture
• Traffic Engineering - Using different paths for flows based on policy
• Management Abstraction 

Now some will say that there are other ways to do these things, yes that is true. When you are using switches  that support Openflow it makes send to look into using it. 

My university is connected to Regional Optical Network - Ilight, which is managed by Indiana University. There will be a day when Openflow services will be sitting at my campus wan edge from them. It would be nice to be able to take advantage of them and also to inter-operate with them. If you look at Brocade's presentation at Network Field Day 5 you will see projects in which Indiana University is involved with OpenFlow.

I will probably post somethings about what I am doing with it as they happen. But I do have a lot of business critical project and task that are higher on the priority list. 

Currently looking at the following:

• Controller: Floodlight
• Switches: HP 6200yl currently running OF code K.15.06.5008 , 5400zl in production without OF code. Have 2 5406zl's in staging.
• Avior - GUI for adding flows to controller. 

For more OpenFlow info look to the Open Networking Foundation and OpenFlowHub.org
Check out Brent Salisbury's blog listed in the blog roll to the right, Brent has some tutorials on getting started, with some code snippets as well.

Complexity & Change Management - A Lesson for the Week

Complexity in IT is common place. No matter how hard we try to reduce complexity some solutions are complex. Sometime over the life-cycle of a system, changes cause the once simple solution to be complex.

Change Management - "Change management is an approach to shifting/transitioning individuals, teams, and organizations from a current state to a desired future " - from wikipedia. Every organization has different practices ranging from the formal to the informal.

My organization has a fairly informal change management process, outside of certain changes going through a form in the help desk system, changes are on the less documented side. Now let me be clear with 7 members of the IT department, most of us are all aware of the changes that are happening. Yes there is room for improvement into the process.

So to the lesson this week, needing to elevate the Windows Domain and Forest to 2008 level instead of the mixed mode we have been in. Being in mixed mode is now limiting us in GPO tasks and other important projects moving forward.  Server admin's task was to demote 6 DC's across two domains this week.

This started off by consulting the Department's - Jedi Holocrons

Image from moddb.com

That starts by yelling over the cube walls "Hey Dave! If we are going to do this what are we going to break? " I rattle off the things that I know that will be effected plus a couple more that are maybes.

Demotion of DC's start, and are moving a quick clip. Server Admin works with our developer to make sure some custom user provisioning process get moved.

Bump number one - Several web apps point at the demoted dc's for authentication. Fix for one was change in web.config and iisreset. Other was a bit more complex as in change authentication.config then push that into the app.

Bump number two - Custom user provisioning code is hard coded to specific DC's. Stood over the developer's shoulder to verify changes were correct, then install all the dependencies for it to run on the server that we moved the code to.

So hopefully the last DC will get demoted this weekend without any trouble and we can move forward.

Luckily only one set of our users were affected by these bumps, and yes that was faculty/staff opposed to students.

So what seems to be a simple process turns into a bigger one with the lack of complete documentation and change management processes. I am not an advocate of ITIL or a strict rigid change management process, but having a process will help.

In this situation we could have avoided the bumps if some more documentation was kept and read through. But by understanding the overall picture and knowing what we needed to get through we were able to work through the bumps.

PacketFence 3.5

So back in August of last year, I wrote about the previous version of PacketFence that I ran for many years, version 1.6.7 [See Here]

In that post I stated that I would write more about the newer version, well months have past, going to change plans about that post due to the fact that version 3.6.1 is out.

So I will highlight some of the big differences between the versions.

• No more ARP spoofing - Multiple options, including use of snmp traps and port-security, to 802.1x mac security.
• Effectiveness of trapping - Once a client has been identified they are switched to a vlan, instead of having the router address spoofed. It works fast and with less overhead.
• Scale of server - Due to the fact that traffic is not all trunked into the server the server can handle more tasks and is more responsive to the admin requests.
• 3.5 has included many of the components under the control of the PacketFence processes- freeradius being a big one.

Problems that still are being addressed: Xbox 360's dhcp fingerprint is not detected correctly, so that they do not auto-register. Some of the reports still need work as they take a long time to run (some of these have been fixed in 3.6.x)

Features that I still need to try or want to implement:

• Guest Access - PacketFence has a guest portal, this would be ideal to the separate SSID/system currently used on my campus.
• Game system registration form - In 3.6.x there is a self-service form to register Xbox's and other systems that my not auto-register. This would help the manual process done now. 

So this is a quick follow-up to the post from August on my PacketFence install.

Xbox Live - Cisco Wireless

So I had this problem creep up ever since the students showed back up on campus for the spring semester.

From the trouble ticket: "My Xbox won't connect to Live"

What? What do you mean it won't connect. I saw xboxes connected to the wireless network all over campus. Then the flood of tickets stated coming in. Most of the students know by now that they need to register there xbox on the network via a helpdesk ticket. This is due to the problems with DHCP fingerprint with our registration system, Packetfence. If you want to know more about that read some of my other posts or encourage me to finish some about the updated version.

Anyway back to the problem at hand, Xboxes not connecting to Live. I have tickets out the wazoo about them, grumble grumble, search up and down. MS points to reboot your router and reconnect. Yeah that will happen.

So last friday, I got an xbox 360 from a student to test with. Connects to Wifi, but that is as far as it will go. Connected to the wired network, tests clean to live, except the NAT, which is a fact of life. Back to the wifi, tests show no response to icmp. Hmmm. Without dancing through some hoops, I do not have the proper tools to capture the wireless traffic. Ok lets do the next best that I can do, track xlate and conns through the firewall and then compare them.

Shortened and simplified
Wired side:

• Client builds xlate
• UDP to Live address
• TCP connection to several other Live addresses

Wireless side:

• Client builds xlate
• UDP to Live address (Same address as wired side)
• Teardown connections - Nothing more

So this behavior has me thinking, am I fragmenting packets? But why? Where?

I finally came across the following article after exhaustive searching: http://revolutionwifi.blogspot.com/2010/07/fragmentation-in-controller_02.html 

Andrew refers to LWAPP Fragmentation in  L3 LWAPP transport - Check doing that. Ok so what I am using. I read down farther to the prevention, and I check my controllers/ap's. Adjust TCP MSS not checked on any of my AP's.

Enabled that on all the ap's on the controllers and then checked back with students. 

Responses back, "It's working, I can connect to Live"

Why did this come up? 

After last semester I upgraded the code from 6.0.xxx to 7.0.235.3 on my WLC's because of the MS Windows 8 issue. So either this was un-done in the upgrade or was not a factor in the previous code. Not sure, but since I have a fix, I am not going back to find out. Time to move forward with the many other projects.

Hopefully this helps someone in the future. Since connecting an xbox to a enterprise wireless network is not really covered in MS help.

Also I would like to thank Andrew von Nagy @revolutionwifi for his blog article, Josh O’Brien @joshobrien77 for pushing me to post and to keep looking. 

Note: This may need some edit but just trying to actually push it out the door before it ends up on the cutting room floor.