Go Mobile! Representative Wireless Usability Tests
Today, many companies expend a great deal of effort to make sure that the participants in a web usability study are similar - in terms of demographics, psychographics, technographics, etc. - to the intended users of the end product. Although this effort is to be commended in its own right, many critical environmental and contextual factors are unfortunately still being ignored.
Have you ever seen a usability test wherein a user is seated at a well lit, uncluttered table (at a specified time and with a detailed agenda) in front of a 19" monitor, the absolute latest browser with no plug-in conflicts, and a T3 network connection? Chances are, it's absolutely nothing like the environment currently around you.
And that fact is important because environmental factors such as lighting, network speed, and desk clutter (among many others) may all influence how well or poorly you perform when using an application.
While many would argue that current web usability testing may be able to avoid the consideration of these types of factors - at the very least you're sitting at a desk, using a mouse, and are somewhat familiar with the basic operation of the chosen web browser - things aren't quite so black-and-white when your users can go mobile.
As more and more wireless devices and applications come into popular use, usability testing will only remain successful if we find ways to incorporate these contextual factors into the experimental environment.
Representative Design of Experiments
Put succinctly, the main reason for conducting a usability test - or any experiment for that matter - is to be able to generalize what was learned in the experimental setting to some target setting. And while usability tests don't always conclude with the interpretation of "statistically significant" results, we still try to understand what we've learned in the test and apply it to the problem being addressed. Many have argued that the closer the test environment is to actually representing the "real world", the more applicable the test results will be.
The concept of designing experiments to more closely represent the target operational setting is embodied by Egon Brunswik's idea of "representativeness," formally established in his Perception and the Representative Design of Experiments, published in 1956.
Representativeness involves presenting the subject with an experimental situation that captures the relevant aspects of a corresponding real-life situation. The closer the design of the experiment is to the situations found in the actual operational setting, the more we can assume the results will be able to be generalized to that target concept. This idea of generalization is closely tied to "external validity" - the degree to which the conclusions of a study hold for other persons in other places and at other times.
Brunswik's ideas go against classical psychological experimentation methods, wherein "all relevant external conditions are to be systematically controlled, and that all internal conditions are to be treated quasi-systematically by computational elimination of random variability." Although his thoughts at the time were considered radical, one quick look around your current environment should make it quite obvious how insightful he was. In your "real world" setting, there is a natural interaction of related and non-related variables and other factors that influence your behavior and performance - presenting a much different environment than the septic usability lab environment described above.
While we believe that even today's web-based usability tests should be designed to capture and incorporate more of these natural interactions, we believe it will be critical that wireless-based usability tests do so.
Which, unfortunately, is where the problems begin...
Scheduled Spontaneity and Your Darned Uniqueness
The biggest challenge facing those attempting representative usability evaluations of wireless devices is the simple fact that wireless systems, untethered by copper wire, coaxial, or fiber optic cables, have been specifically designed to support spontaneous use. They are devices that, according to the marketing hype, go where you go. They're available for your use whenever and wherever you, in your own unique and personal way, need them to be.
What does that mean for usability researchers? Well for one thing, it means their days of collecting video data in a well-controlled laboratory environment are all but over. It means they need to figure out ways to understand and evaluate how someone uses a wireless application based on a spontaneous need and not a predefined set of directions. It means they need to go mobile, to go along with users to see how things really are - out there - in the real world.
So are we suggesting that usability researchers be at the restaurant when a user spontaneously decides to read movie reviews and check showtimes for later that evening? That researchers be there at the airport, just about to board the plane when a user decides to check his email? That a researcher be in the car with a user when she pulls over (hopefully!) to get directions to the nearest gas station?
As a point of argument, yes, that's what we're suggesting. It's not going to be easy, obviously, but we don't think it's entirely impossible, either. It's a challenge that must be faced and overcome in order to accurately test the usability of wireless systems.
For more information on this type of research, see our Coffee Talk on Usability Inspection Methods.
Other, Related Challenges
In addition to the "when and where" challenge for representative wireless usability testing, there are at least 3 other key environmental challenges that should be considered: display size, network reliability, and device differences.
Display Size. Many web-based usability tests employ video recordings of screen activity for future reference - usually running the computer's output through a scan converter to capture it on videotape. For those on a tighter budget, a hand-held camera pointed at the screen accomplishes the same thing, although quality takes a hit and you can see the scan lines on the video.
Video recording a cell phone display, however, is a whole different animal. Phones and other wireless devices typically have tiny displays. Our need for smaller, more compact forms have forced multitudes of users to squint at a display that is oftentimes only a square inch in size. So in addition to the fact that there's currently no way to directly capture the display output, there's not a lot of room to position a hand-held camera and not interfere with how a user typically handles the device.
Network Reliability. Put simply, it's bad. Really bad. If you've ever used a cell phone you're familiar with just how often calls get dropped due to lack of coverage, interference, or (it seems) the price of tea in China.
That type of variability is a huge problem for usability testing. However, it's also completely representative of what happens to cell phone users in the real world, every day. As such, we believe it should be accepted as part of a usability test. After all, if a user can't access an application, can you not - at least on some level - claim that the system is unusable?
Device Differences. Oh for the days when all we had to deal with was the simple, surmountable differences between Netscape Navigator and Internet Explorer!
As of this writing, there are at least:
22 Mobile Phone Manufacturers
6 Different Browser Types
3 Different Development Languages - all complete with proprietary extensions
...and that's just for WAP-based phones. Combine that with at least 4 different types of input methods being used and you start to wonder if choice is always a good thing.
In fact, there have already been a number of wireless usability studies published that state that the primary cause of many initial usability problems are due to the participant's unfamiliarity with the operation of the device. Seems wireless devices aren't as simple as "point and click." Wonderful news, isn't it?
Usability testing in the unwired future requires that our experimental subjects and the tasks they perform, as well as the domain properties in which these tasks are performed, are close to the population, tasks and domains that we want to generalize to, respectively.
But when faced with so many confounding variables, many of which are completely beyond our control, can we ever expect to properly evaluate wireless applications for their usability? Yes, but it requires better planning, decision-making, and acceptance of tradeoffs that in other experimental settings would be deemed unreasonable.
To gain the knowledge you'll need to develop representative usability tests for wireless devices, we suggest you go mobile first. Use the insights you gain from those excursions to help you construct your test environment ... sort of an observational session as a pre-requisite to test development.
Yes, it'll cost more and take more time, but we believe it will turn out to be a critical factor to conducting successful wireless usability tests.
_______________________________
For more information on the work of Egon Brunswik, please visit The Brunswik Society (http://www.brunswik.org/)