The Lessons of Wi-Fi #9: Use Analysts & Audited Financials To Validate Vendor Claims

A loud-talking ranchman applies to a banker for a loan. The banker asks a neighbor if the rancher is a good credit risk. The neighbor ponders for a moment and then replies “Big hat, no cattle.” False bravado is funny when it’s the stuff of fiction, less so in real life – especially for customers snagged by rhetorical barbs.

And yet it happens again and again. Each year the networking world is introduced to “big hat” products with features and specifications so too-good-to-be-true that we let ourselves be reeled in. Why we don’t see through the shiny veneer and ask for proof of pedigree is a wonder. But it happens all the same.

The Lessons of Wi-Fi #9: use analysts and audited financials to validate vendor claims. Neutral independent industry analysts like Burton Group, Canalys, Gartner, IDC, Infonetics, InfoTech, and Yankee Group can quickly assess vendors' technical claims.

And don't forget to check financials - audited financials - because you want your vendor to be in business should you need assistance or spare parts. If a vendor won't give up the numbers - or the numbers are substandard - then you have grounds for real concern.

A quick example will put the discussion in context. In 2008 a “big hat” four-radio 802.11n access point was announced that claimed to deliver 1.2 gigabits-per-second of aggregate capacity. The data sheet claimed that the four radios worked in tandem, enabling users to dramatically reduce the number of access points and additional security sensors, thereby reaping savings on cabling, connection and installation costs.

Still, the press ate it up. A flurry of articles expounded the virtues of delivering multiple HD streams to an entire building, with perfect coverage, at almost no cost. The world would soon be saturated with multi-adio APs, the unwashed masses blanketed with 802.11n. Wow, where do I sign up?

Fast forward to late 2009. The “big hat” super duper access point was no more. It simply vanished from the vendor’s Web site, its demise a secret. Was it ever built? No. But the company received undeserved publicity and that reeled in some unsuspecting customers.

To paraphrase Orson Wells, companies should herd no cattle before their time. Industry analysts can help you separate claims from reality. If an analyst says that a vendor can't excute well, refuses to divulge shipment numbers, and/or lacks technical vision - well, your due diligence is over.

The next time you see or hear about a product that appears to be too good to be true, separate the hats from the herds - kick the tires, test the features, validate the design. Those impressive features might be chimeras or, as with Aruba's AP-105 802.11 Access Point, the genuine article.

The Shoe Drops: Brocade Dumps Single-Channel Architecture

Brocade yesterday announced that it will be reselling a new line of wireless LANs. Foundry (now part of Brocade) has been reselling wireless LANs for some time, so aside from a new OEM supplier where is the news?

Well, it turns out that Brocade didn't just select another vendor. It selected a completely different wireless LAN technology.

For roughly three years Foundry has been reselling Meru's proprietary single-channel / virtual-cell architecture. Meru has long made what we consider to be outrageous claims about its proprietary technology - airtime fairness, high throughput, fewer required access points, and so on. The Foundry team has had years of experience understanding the real strengths and weaknesses of Meru's single-channel networks, their 802.11n technology, their network management. They have deployed wireless LANs with virtual-cell technology across a range of customer types - education, healthcare, business. They, better than any single customer, knew the strengths and weaknesses of the vendor and the technology.

The upshot of yesterday’s news? Single-channel wireless LAN technology was rejected by the customer that knew it best.

Changing an OEM supplier is a decision that is never made lightly because it profoundly impacts the customer base and the company's reputation. Customers who invested in the single-channel / virtual-cell architecture are surely asking why Brocade abandoned a network that was supposed to be so innovative. What are the limitations and deficiencies that caused Brocade to change the underlying architecture as well as the vendor? The Brocade announcement is a watershed because it is a repudiation of the proprietary single-channel architecture. It also serves as a cautionary tale that vendor claims that sound too good to be true generally are.

The technology shift from single-channel to a new architecture will likely be very disruptive to Brocade’s customers. Brocade has already removed the Meru-based products from its Web site, and support-related issues will no doubt be unpleasant. But all is not lost.

Aruba wants Brocade’s wireless LAN customers as our customers, and we have a generous trade-in program to ease the pain of the transition to our award-winning adaptive 802.11n networks. Switching to Aruba will be a real step-up for those customers because we have field-proven adaptive wireless management, wireless infrastructure control, remote networking, wireless intrusion detection, policy-based firewalling, and client-to-core security that were never before available from Meru. Our AirWave Wireless Management Suite will manage their legacy Brocade/Meru network from the same console from which they will manage their brand spanking new Aruba wireless LAN.

The transition will be smooth and Brocade’s customers will be stepping up to a more secure, more stable platform from Aruba. And instead of a story line they'll be getting the real deal.

You Get What You Pay For: Meru Pays Novarum For Performance Not Seen By Customers

Novarum recently published a test report claiming that Meru Networks’ 802.11n wireless LAN delivers higher throughput, better power efficiency, and superior airtime fairness than either Aruba or Cisco. The report is available from Novarum's Web site.

At a high level – setting aside all technical details – the report’s findings are at odds with the experience of many prospects and installed-base customers. Meru deployments have been removed from, or Meru lost head-to-head technical evaluations (“bake-offs”) at, the following schools among many others:

• University of Tennessee – replacement and bakeoff
• C-2 Raytown School District - replacement
• Norwood School - replacement
• Francis Xavier Warde School - replacement
• Drexel - bakeoff

The EDUCAUSE Board (http://ised-l.blogspot.com/2009_01_01_archive.html) has been rife with postings about issues with Meru’s 802.11n network. See for example the posting from Jomar McDonald, Director of Technology, The Frances Xavier Warde School.

Recent press articles have explored the reasons why customers are replacing Meru networks with Aruba adaptive 802.11n networks. One such case is Mike Morisy’s Search Networking article, “From Cisco to Meru to Aruba, school finally finds right WLAN” (http://searchnetworking.techtarget.com/news/article/0,289142,sid7_gci1352631,00.html#).

No one can dispute that performance differences exist between different wireless LANs, however, the dichotomy between the findings of the Novarum report and what customer’s experience in the real world is startling. A little digging into the research methodology employed in the Novarum report highlights casts a bright light on the reasons for this schism. Novarum is a paid consulting firm – a writer for hire, as it were – and given the fact that their findings are completely at odds with what we (along with other vendors) see in actual deployments in the industry, one has to believe that the results they publish are heavily influenced by the source of the funding. For example, a 2007 Novarum report – also commissioned by Meru – saw Aruba’s AP-70 Access Points tested with their antennas closed and in the wrong planar orientation relative to the clients. Novarum claimed that the network was set-up in accordance with Aruba’s guidelines, however, that proved not to be the case with the antenna position and a host of other critical parameters.

Fast forward to the newest Novarum report. The methodology issues are different from the 2007 report but just as significant with respect to their denigration of performance:

• The tests used just one single access point from each vendor - hardly an environment conducive to measuring wireless LAN capacity – and Meru access points were operated at full power but the other access points were not;

• Commercially available software releases were used for the Aruba and Cisco devices (Aruba 3.3.2.10 and Cisco 5.2.178) but Meru used a special test code that is not available to its customers. This inobtainium code was no doubt crafted to perform special tasks, just for the test, that would otherwise be unnatural acts in a commercial deployment;

• Encryption was disabled, despite a mandate by most customers to cipher communications. Encryption has been demonstrated to degrade the performance of Meru wireless LANs;

• Only two client types were used, one being a plug-in adapter, this despite the plethora of clients in real world deployments. The performance of Meru wireless LANs has been previously demonstrated to degrade in the presence of commonly used clients that were excluded from this test;

• Screen shots show major misconfigurations of Aruba’s controller. Aruba utilizes a technology called Adaptive Radio Management (ARM) to optimize wireless LAN performance, and in the test the ARM traffic management profile for fairness was created but not assigned to the Aruba access point under test. The voice traffic DSCP (ToS) tag was also incorrectly set to a value of 56. Other errors abound;

• Meru’s own installation guide states that 3X3 MIMO operation cannot be supported over 802.3af power over Ethernet, and that both radios have to back down to 2X2 MIMO. Therefore it is possible that a single radio was used during power measurements of the Meru access point, and the same was done for the Cisco 1250 access point – providing nothing more than that one radio consumes less power than two.

Occam’s razor - entia non sunt multiplicanda praeter necessitatem – states that the explanation of any phenomenon should be parsimonious with respect to assumptions about observable predictions. The Novarum report is nearly forty pages long, but the most fundamental underlying assumption – that the competing equipment was set-up properly, fairly, and in accordance with the manufacturers’ guidelines – was violated. The results – all of the results – were thereby nullified, the paper wasted.

One assumes one gets what one pays for: Meru got a test report in exchange for paying Novarum. Readers, however, got nothing of value. Caveat emptor.

At Aruba we appreciate and encourage head-to-head testing by our customers before they choose a WLAN. It is only in these real-world scenarios, running the applications and equipment that are intended to be used, that one can best evaluate the performance of a network. We also appreciate the value of thorough testing done by industry experts. However, when you can’t replicate a test in the real world - as is the case with the Novarum report – then the testing procedure is flawed and/or skewed.