More Than Throughput: Next Generation Wi-Fi Testing with Ixia's WaveDevice
by Joshua Ho on March 25, 2016 8:00 AM EST
It’s been quite some time since I first started writing for AnandTech, and without question there’s been a lot of changes that have happened to our testing methodologies in the past few years. One of the main issues that I’ve always been thinking about while working through reviews is how we could improve our testing methodology in a meaningful way outside of simply updating benchmarks to stay current. Internally we’ve been investigating these issues for quite some time now, and these changes have included the addition of SoC power efficiency comparisons and display power efficiency measurements. There are a lot of other changes here and there that I still want to make, but one of the major unexplored areas has been wireless radio performance.
Wireless performance testing is probably one of the hardest things that we could test, and for a time I had almost given up hope on deploying such testing within AnandTech. However life has a way of playing out differently than I expect, and in the past few months we’ve been working with Ixia to make Wi-Fi testing a reality. Ixia, for those of our readers who aren't familiar with the company, is a traditional player in the networking test space. They are perhaps best known for their Ethernet test products, and more recently have been expanding into wireless and security testing with the acquisition of companies like VeriWave and BreakingPoint Systems.
We have done Wi-Fi testing before, but in the past we were mainly focused upon a relatively simple and arguably not particularly interesting test case: maximum throughput in ideal conditions. It was obvious that Wi-Fi in many devices is still not perfect, as subjective differences in reception and reliability can feel obvious. However, without any data or methods of replication it was hard to really prove that what we felt about wireless performance was really the case.
A few months ago, Ixia brought me into their offices to evaluate their WaveDevice system, which a Wi-Fi testing device uniquely suited to solving our testing needs. This system is effectively integrates a number of tools into a single chassis, including: Wi-Fi access points, traffic generators, programmable attenuators to set path loss, channel emulators to simulate a certain kind of RF environment in terms of interference and bandwidth, packet sniffers and analyzers, and signal/spectrum analyzers. These tools are implemented such that each layer of the Wi-Fi protocol can be analyzed, from the physical link layer of raw bits encoded at the carrier frequency of 2.4 of 5 GHz, to the application level where everything is neat bitstreams transmitted or received from specified addresses.
Of course, hardware is just one half of the equation. In order to provide a full solution, Ixia also has a full suite of software to make it possible to run common tests without spending excessive amounts of time developing custom tests. While WaveDevice supports custom tests through its API, WaveDevice out of the box supports a simple throughput test, which is effectively like iPerf but with more advanced statistics. There's also a general data plane test to evaluate throughput when varying the traffic direction, traffic type, frame size, and frame rate. Other than these basic tests, WaveDevice also has tests for rate vs range, roaming latency, and traffic mix testing. In the case of rate vs range, it's possible to run an automated sequence of throughput tests while varying frame rate, transmit power, and physical link rates. Meanwhile in the interesting case of roaming latency, we can test how long it takes for a Wi-Fi device to hop from one access point to another when fading out the signal of one access point and fading in the signal of another. Finally, the traffic mix test allows for a test of throughput when faced with competing traffic that is also transmitting.
Ixia has also made sure to support a range of OSes in order to make sure that almost any device can be tested against WaveDevice. In addition to maintaining iOS and Android applications, it seems that WaveAgent is a simple C application at its core that can be easily used for embedded systems like wearables and Wi-Fi cameras, so it seems that the "device" in WaveDevice really refers to any client with a Wi-Fi chipset.
While at first glance it might seem like these tests are pretty simple, it turns out that there's a huge amount of nuance to them. To try and understand the nuance I'm talking about, it's best to start with a basic primer on how Wi-Fi works.
Facebook
Twitter
RSS
44 Comments
View All Comments
JoshHo - Saturday, March 26, 2016 - link
I was aware before the testing that the Pixel C had something wrong with WiFi, but the goal was to try and separate software issues from hardware ones.It was subjectively obvious but difficult to understand what was causing the problems in a reproducible manner.
at80eighty - Saturday, March 26, 2016 - link
Have to concur with everyone. Happy to see anandtech still keeping that edge to their content. Great work Joshua.mannyvel - Sunday, March 27, 2016 - link
We have one of these systems at work, and you have to be a wifi genius to use it.That said, what our guy said was this: test your throughout with multiple clients at multiple distances. There also some subtleties about wifi that are odd. One is that the anount of airtime that a given adapter uses is more important sometimes than its actual speed. There is only so much airtime, and a badly behaving adapter that's further away will prevent faster devices with better signal from using the AP because of how PHY rates are negotiated (not sure what the term is). Basically an adapter further away will scream louder and more slowly to see if the AP can hear it - but the slow screaming takes airtime away from the faster devices that have better signal.
Also, since it's RF more clients means more interference. I suspect you will find, as we did, that all your wifi testing has been completely wrong for pretty much forever. With the ixia stuff you can tell how bad your AP really performs in, say, and apartment-like configuration.
One more confounding factor is not all chips and drivers behave consistently. An AP that works great with Windows 10 and an atheros chipset may work crappily with the same chip in win7...as you've seen in the iPad Pro tests.
bman212121 - Monday, March 28, 2016 - link
I agree 100% with everything mannyvel said. One of the biggest hurdles people face is range vs density. Everyone thinks that throwing a bigger antenna will give you better wifi and that having an AP with increased range is a good thing. In reality, most APs these days are turned down in power so they match closer to the clients capabilities, and in the case of roaming a minimum RSSI is enforced. Only having "1 bar" hurts overall performance more than having a second AP that can provide higher signal strength. We used to have one high powered Aironet that could cover a huge area. In order to increase performance you might take down that one AP and put up 3 in it's place to get the proper coverage while getting the performance you need.I'd really be interested in some minimum RSSI tests. A great performing antenna might not have the highest power, but you can easily make up for power by having better sensitivity. Even APs of the same power output will have different receiving abilities, and higher end APs can pickup and work with much weaker signals and still maintain solid connections.
skavi - Sunday, March 27, 2016 - link
Holy shit, I can't wait for these WiFi performance and roaming tests for more devices. This is so important, but there is rarely any information specific to devices. My S6 is never able too roam correctly and forces me to reconnect each time I move somewhere, maybe these kinds of tests will pressure OEMs into paying more attention to these issues. It seems Apple at least has gotten it down.Powervano - Monday, March 28, 2016 - link
Very nice and informative article!Could you please test Surface Book and Surface Pro 4 using the WaveDevice? Would be nice to shed some light around the Wi-Fi issues on these devices.
Conficio - Monday, March 28, 2016 - link
Thanks Josh,you are on your way to become a hero for mobile device users. I appreciate the hard work flowing into this article.
I wonder which parts of this stack are software updatable? As there is talk of the Pixel-C having a broken Wifi, is there hope it gets fixed by a software update? In the same interest did I miss the specification of the exact relevant software versions on the devices?
P.S.: The Baseband package on my device influences WiFi or is it only for the WAN portion of my Android device?
cuyapo - Monday, March 28, 2016 - link
Great article, thanks! One thing only:CSMA/CA = Carrier Sense Multiple Access with Collission Avoidance
James5mith - Tuesday, March 29, 2016 - link
Still reading through the article, but it's worth noting that SmallNetBuilder has also switched to this test setup for wireless testing going forward. Interestingly the one thing this doesn't take into account is antenna design. While it's great to test throughput vs. attenuation/power/etc. None of that takes into account if the antenna is poorly designed, oriented incorrectly, etc. It does give a good baseline to work from though. And it should help people track down problems for wifi if they are using a router that tests solidly without taking factors like antenna design into account.James5mith - Tuesday, March 29, 2016 - link
http://www.smallnetbuilder.com/wireless/wireless-f...