Usability testing APIs: Easier than it sounds

An API has no buttons to click and no screens to record. So when a platform team hears “usability testing,” most assume the term belongs to someone else’s product. Ship the API, watch the support tickets, patch the docs.

I spent about a decade as a product designer on developer platforms, and some of the highest-return research I ever planned was on products with no UI at all. Here are a few ideas and techniques you can use in your own work, all based on industry-standard usability research and best practice.

Test it before you build it

The strongest version of API usability testing happens before the API exists. Jenny Wanger’s DevRelCon talk walks through a four-phase process lifted directly from UX practice (discovery, taxonomy, mocking and prototyping, then usability testing), and her cost argument is the part that should get your attention: “At no point during the UX process do you actually need to code a single line” [1]. She estimates the full cycle at one to two weeks if you dedicate time to it.

Basic terminology recognition & understanding

The index card test. Write the verb and endpoint name for every method on good old-fashioned index cards. Hand the stack to a developer who doesn’t know your API, and ask what each one does and what order they’d call them in. If they can’t reconstruct the workflow from names alone, the names are wrong. Wanger describes a team that had settled on calling user metadata “characteristics,” then watched the term fail with everyone they handed it to. Cards let them cycle through candidates in minutes instead of shipping the wrong noun into a versioned contract [1].

The instant messenger test. Also known as a “Wizard of Oz” test. Put a participant in a room with a chat window and that same stack of cards. In another room, a colleague plays the role of the API: the participant types a request, the human types back a response [1]. It might sound silly, but it actually forces the participant to voice their expectations of every endpoint while they assemble the calls. That’s exactly the data you can’t get from analytics.

The API “wireframe”. For later-stage detail work, Wanger stubs the entire API in Postman or Swagger (still no backend). She then runs standard moderated usability test sessions against it, with tasks written as binary pass/fail outcomes. This might include things like “charge a credit card for $12.50” for a payments API. She also argues for deliberately ugly paper prototypes, because “people are willing to tell you the truth if they don’t think that you’ve put a lot of time into it” [1].

Formal usability lab study

For an API that already shipped, one effective method is captured by Clarke and Becker’s work at Microsoft [2]. The process will be familiar to anyone who’s run these before: Work with the team that owns the library to identify the core scenarios it should support. Turn those scenarios into development tasks. Bring in developers, record them working the tasks, then analyze the recordings for patterns of breakdown across participants.

Their first study fed a redesign, and a second study on version 2 measured the difference. The paper notes that this before-and-after evidence is what convinced other Microsoft library teams the method was worth running [2].

The part most teams skip is the analysis vocabulary. Clarke and Becker adapted the Cognitive Dimensions framework into twelve dimensions for API usability, including Work-Step Unit (how much of a task can be completed in a single step) and Consistency (once part of an API is learned, how much of the rest can be inferred). Named dimensions are what let findings travel: a team that never saw your study can still act on “this library scores badly on Work-Step Unit,” because the vocabulary generalizes where a raw session recording doesn’t [2].

Different methods find different problems

Do you need a lab study at all, or will an expert review do? Grill, Polacek, and Tscheligi tested that question directly by evaluating the same framework API three ways: a heuristic evaluation with four experts, a one-day hands-on workshop with eight developers, and post-workshop interviews [3].

The evaluation surfaced 157 unique usability problems. Heuristic evaluation found 109, the workshop found 44, the interviews found 15. What’s most interesting here is that the findings from the expert evaluation and the workshop barely overlapped. Only two issues were common between them [3].

They also found different kinds of problems. Experts inspecting class definitions caught structural and documentation issues but were not able to analyze run-time behavior [3]. The failures that only exist once code is running, where the API does something the developer had no way to anticipate, showed up in the workshop and nowhere else.

Bottom line: treat the two as different instruments. They measure different things. An expert review is a fast, wide sweep for structural problems (Grill’s team used four experts, on the strength of Nielsen and Molich’s finding that three to five evaluators catch 60 to 75 percent of an interface’s usability problems [3]). Observation is the only one of the two instruments that can reach behavior. Run expert reviews first, then spend your time with users on real tasks.

Scaling past the lab

All that said, much of this might not scale well to a company shipping dozens or hundreds of new APIs and endpoints a year. Google faced exactly this. According to its own researchers, the number of APIs coming out of its business units grew “at an astounding rate,” and the result was an experience full of inconsistencies and usability problems [4].

Their answer was an internal design review program dubbed “Apiness” (API happiness). In this method, nobody actually uses the API. A trained reviewer reads the specification and the sample code as a document, comments asynchronously, and iterates with the owner until sign-off [4]. No recruiting, no scheduling, no session recordings to analyze, and it can run before a line of the implementation exists.

The tradeoff is measured. Macvean and colleagues cite Farooq’s data at Microsoft: lab studies were 16 times more productive at finding usability defects than peer reviews, though reviews were substantially faster and skip the recruiting problem entirely [4]. That ratio is the honest summary of this whole piece. Reviews buy you breadth across a lot of surface area. Observation buys you the problems nobody on the inside can predict.

Where I would start

If you own a developer platform and have never tested it, my advice is to skip the program-building and run one small study. Pick the workflow that decides adoption, which is almost always the first integration. Write three or four tasks with binary pass/fail outcomes. Recruit developers who have never touched your API, stub the responses if the surface is still moving, and watch them work. Everything above says the severe problems will show up fast, and that they will be problems nobody inside your team can see anymore.

A stack of index cards and a conference room is enough to begin.

References