You can’t really validate or invalidate a design idea just by looking at it and declaring it a success or failure because of some best practice or design heuristic that usually works. You’re just talking about theories. Ultimately, no design disputes can be settled convincingly without making a model and testing it out.
But a theoretical debate about the strengths and weaknesses of a design is a critical first step that design teams must pass through before actually going off and testing their designs. To me, a good design critique is a kind of low fidelity user testing: using our imaginations instead of using a lab and test subjects.
Regarding the inspiration for my previous post, the absurd-looking application “Bulk Rename Utility“, the debate was almost predominantly based on “gut opinions” — albeit by many people with expertise in UI design. And I think that’s great. In my particular gut, I suspected that this app would surprise people and do well in testing. Others felt in their guts (and present compelling arguments, too) that this app would fail miserably in a user test.
What’s great about having this kind of hypothetical discussion at all (especially when the debate might seem to be easily and quickly settled by simply testing the application with real users) is that we learn more about the kinds of things we would need to think about and the questions we should be asking when we actually do test the application. Without debating the options we might not have uncovered (for example) these kinds of questions to ask during testing:
- Would different types of users react to the app in different ways?
- Do different apps with the same stated purpose serve different use cases?
- Is efficiency important, or a feeling of efficiency?
- Is preventing error of primary importance, or permitting error correction?
Sometimes the expert critiques and gut reactions are so compelling (whether positive or negative) that an experienced designer will know right away that formal user research would be a waste of time. Sometimes it’s just obvious — but only becomes obvious once you’ve thought it through, especially if you’re talked it through with other people.
And, of course, testing can be dead wrong. Seinfeld was famously rejected by test audiences, and finished last in the ratings in its first season.
Then there are the “unknown unknowns” (a Donald Rumsfeld-ism that I think is entirely valid and sensible). There are some design decisions that seem so obvious, and may even test well, but fail miserably because of an completely unforeseen factor in real-world practice. Our goal as designers considering design options is to try to minimize the number of unknown unknowns. And again, the best way to uncover unknown potential problems is to imagine as many of them as possible through lively debate from diverse viewpoints.
Whether you user-test your product or not, there’s no doubt that a lively, opinionated and adversarial discussion about any complex design decision, especially a user experience design decision, can only help the overall product development process.
Comments
2 responses to “Adversarial Design, Part 2: Testing by Discussing”
I agree. The way I see it, the most fundamental, cheap and quick act of design evaluation is such form of rapid thought experimentation which largely relies on one’s own experience. I at least can testify that when I design, I also wrap the emerging concept with envisioned stories of use many times over, which in turn evaluate the concept. These stories which fuel the imagination can come from experience, from others (during debates), and of course from user research. In the rising popularity of user-centered design it’s nice to see your post balance design experience by giving it validity.
@Jakub: I like that term “thought experimentation”. I’ve heard “thought experiment” before, but it’s an interesting twist to make it sound like a formal process.
Regarding user centered design, thanks for noticing that this post implies that there are design methods that work really well that are decidedly not user centered design. I wrote a post a while back disputing the common assumption that usability specialists are as a group especially possessed with a sense of empathy. I questioned that assumption because so much of the user centered design process seems to be designed to compensate for a methodology and a design team completely and profoundly incapable of conducting effective thought experiments. Put it this way: the less empathy you have (i.e., the less able you are to imagine yourself as another person), the more important user testing and user research becomes.