Assessing Usability Part 2: Discovery

Any comprehensive usability program must include some discovery sessions that are aimed at evaluating what happens in the real world, with real users.

Formal usability tests, like any research experiment, benefit from taking place in a laboratory environment that makes data collection and recording relatively easy, provided you have the right equipment. However, evaluating user behavior in a laboratory setting has the disadvantage of removing both the user and the product from the context in which they will actually be used.

Therefore, it's important that any comprehensive usability program include some discovery sessions that are aimed at evaluating what happens in the real world. Ideally, by trying to understand how people work, play, and interact in the real world, you can actually avoid creating certain usability problems during product design.

The techniques presented below (Interviews/Focus Groups, Field Observation, and Contextual Inquiry) are typically used at the beginning of a project, when you have a general idea of what the product needs to do, but you need clarification on how to best implement your ideas in later design activities.

Group Interviews/Focus Groups

The primary purpose of conducting Group Interviews or Focus Groups is to capture, on a broad scale, the participants' likes and dislikes of a particular product concept or proposed design. Sessions are moderated by a trained facilitator and should concentrate on only a small number (no more than 5) of high-level design issues. Focus Groups should never be used as a means of "validating" a finished product, for the reasons discussed below.

The open-ended, interactive group setting (typically 4 - 8 participants) used when conducting Focus Groups serves as a great forum for capturing information which is not ordinarily available in an individual interview, such as the examination of how people interact and where/how they agree or disagree. Be warned, however, that it is human nature to just "follow the herd" when placed in a group setting (especially with strangers) and therefore individuals may have a hard time voicing dissenting opinions, instead preferring "acceptance" by implicitly agreeing with the group consensus.

Invariably, the people within the group each have a preconceived notion of what the Moderator is trying to accomplish by holding the session. Further, it is not uncommon for Focus Groups to agree with a particular design, simply because they believe it will spare the Moderator's feelings. It is recommended that the Focus Group Moderator be as objective and as independent from the actual product as possible. The Moderator should also avoid asking many questions and instead try to get the group to open up through more creative techniques (i.e., drawings, role play, visualizations, and word association activities).

In general, Focus Groups are faster, cheaper and easier to conduct than Field Observations or Contextual Inquiries. Unfortunately, the resulting data is often more susceptible to misinterpretation and abuse due to confounding variables such as "group think" and an individual's desire to please the Moderator. Focus Group data should be validated through additional usability assessment techniques.

Field Observation

In a field observation, you essentially visit the environment in which the user will eventually interact with the end product. So whether you visit a user at work, at home, or join them in a short car ride depends entirely upon where they will be using the final product, once it's developed. You should plan to visit with a number of different users, sampling both from the target population and the environments in which the product will be used (i.e., if it's a web site, visit some people at work, some at home, some at the library, etc.).

Field observation is part inquiry and part observation. During your visit, you should actively interview users about their jobs and the ways they do (or will) use your product, and passively observe them using your product in the way they normally would, if you weren't there.

Many Field Observation "How-to" guides liken the practice to anthropology - you're there to collect data on the artifacts (physical things people use when performing tasks), outcroppings (notable characteristics of the environment itself), relationships, and patterns of communication, that potentially influence if, when, and how someone will use your product.

While Field Observation is a good way to initially gain an understanding of how people behave in their natural surroundings, it is difficult to gain true insight as to why a particular action is being performed or what other possible actions were considered and why they were rejected. Interview questions should be targeted at understanding the user's goals, motivations, and assumptions for actions observed.

Contextual Inquiry

As with Field Observations, the intent of a contextual inquiry is to evaluate the context in which a person is going to use your product as a means of better understanding how to design that product to meet users' needs. Further, successful Contextual Inquiries require the establishment of a well-defined, clear focus. It is not enough to just "watch users work." The more detailed and explicit the focus, the better the results.

The primary difference between Field Observations and Contextual Inquiry is the emphasis that the latter places on the relationship between designers and users. Whereas Field Observations have a tendency to be more passive, in Contextual Inquiry the user is considered part of the design process - part of the team. The better the relationship between the designer/evaluator and the user, the better the design team will understand the user's needs.

Often, a successful Contextual Inquiry results from a Master/Apprentice relationship wherein the user "teaches the designer the ropes" - the basic, standard operating procedures as well as any shortcuts, workarounds, learning/memory tools, and "unsolvable" problems that impact task completion.

Building this type of relationship, obviously, takes more time and energy. However, it is only because of this closer relationship that the designer can truly understand the user's needs well enough to design an effective solution to meet those needs.

Ironically, that need for a closer relationship is the main reason why Contextual Inquiry may not work well for website design. Many people use the web in the privacy of their own homes on their own schedule - perhaps at 6am before work, at 2pm in-between meetings, and again at 11pm before going to bed. Much of this use is unscheduled and therefore presents very real challenges to establishing the critical user/designer relationship.

Next Steps: Inspection Methods

The next article, Assessing Usability Part 3: Inspection provides insight as to the types of usability assessment techniques that are applicable during the design phase of your project.