The Price of Education - A Follow-UpBy CWNP On 09/01/2008 - 15 Comments
Wow. I heard the mother of all Wi-Fi horror stories today. Believe me, I've heard some doosies in my time, but this one is the worst. Fortunately for the customer, it has a good ending.
My good friend, and CWNA, Jeff works for a company that manufacturers an application that uses Wi-Fi as a transport. Like other such applications (Wi-Fi, RTLS, etc), this application depends on having a dependable and somewhat-optimized Wi-Fi infrastructure in place. And so the story begins...
Jeff was called by the customer, a very large distributor in NJ, to come troubleshoot a problem with his application. His client devices were intermittently disappearing from the server application's monitoring application, and he had to investigate why. Being a CWNA, his first instinct was to take a quick look at the Wi-Fi environment and to ask a few questions of the network administrators. "Roughly 1 million square feet, 240 APs" they said.
In utter disbelief, he fired up his analyzer to find that no matter where he ventured inside the facility, he had a loud signal from over 40 APs. He asked for the site survey document immediately. The network administrators obliged, and pointed out the nice round circles every few feet along every aisle where the site surveying company had indicated the APs should go. Jeff lost it. He was just angry at how anyone could sleep at night having done this to their customer.
Jeff proceeded to power down 205 of the 240 APs and to reassign a few channels. Voila - perfect connectivity for everyone, including his application, with only 35 APs. Up until then, only handheld scanners moving minimal data had ever had to work over this WLAN, so nobody had noticed how horrible it was, and the company performing the site survey got away with the technical equivalent of highway robbery. If the story ended here, you'd just have to cry, but, as it turns out, it doesn't...but there's some points I want to make before I give you the sorta-happy ending.
In addition to the the outrageous cost of an extra 205 APs, 205 extra Cat5e data cable runs, 205 extra PoE 10/100 Ethernet switch ports, 205 AP licenses for the controllers, the extra controllers themselves, and the labor and materials it cost to hang 205 extra APs, there was the time and travel expense for Jeff to come out to troubleshoot and fix the issue, the time it will take to take the 205 APs down, and the time and expense it will take to file a lawsuit against the morons who sold 7 times too many APs to the customer. Wow. Jeff estimated that the entire project end-to-end should've cost less than $100k to start with, but instead was likely somewhere $500k-600k because of this unscrupulous site surveying company. I would venture to say that if the customer had one moderately-experienced CWNA on staff, this catastrophe would've been prevented.
Now for the grand finale. Calling Jeff to come fix this "problem with his application" (which of course turned out to not be his application at all), was just fluke timing. The customer had rolled out the WLAN at this location as the first of many locations. They were getting ready to cookie-cutter this same scenario to many other locations. Now, instead of buying literally hundreds more APs that they don't need from the same survey company, they are installing the 205 "extra" APs they were already sold at the other locations, that I'm sure will be surveyed by someone other than the company that sold them the 240 APs. Jeff's customer got lucky. Is education worth the money? You decide.