A Case Against Free Examination..

Users are goal oriented, they complete tasks, perform actions and ultimately seek results."

The mistake of free examination

Free examination happens when a user reviewing your product or design are asked to give open-ended feedback without instruction. As a designer, you've most certainly asked users to freely examine your work (I'm guilty of it myself). More often than not interface designers aren't properly prepared or educated on how to run usability tests. This is an important point because seeking measurable feedback during user testing is critical. Free examination produces less valuable feedback because users don't experience products this way organically.

Ask yourself; when would a user visit a website or an application just to consider the layout? Don't allow yourself to bias this question as the designer. The user isn't interested in your inspirations. To further this point, consider framing the question as: "Why would my user visit our product/service?" Is it to ask themselves, "What are my thoughts on this design?" The answer is, probably not. Remember that users are goal oriented, they complete tasks, perform actions and ultimately seek results. Users do not generally visit products or applications just to experience a layout. No matter how much work we designers put into carefully crafting beautiful UI's, the aesthetics aren't the primary goal for the end-user. Requesting free examination of our work comes naturally to designers because we are programmed to constantly seek creative feedback. This is a difficult habit to break.

Never seek generic feedback

Authentic Results

If you establish your objectives or user goals before beginning your product review/testing, you can avoid the pitfalls of free examination. But be aware that in testing scenarios, participants routinely give more consideration to product functionality and design then they would if they were outside of the testing environment. Some researchers blame this phenomenon on the Hawthorne Effect (more on that here). Regardless of the catalyst - the important takeaway is that the collected results can be misleading. Think of it this way: A user experiences your product design much differently when they elect to interact with it vs when they are participating in a usability test. In the comfort of our own homes, we are far less likely to study the interactions and patterns required for us to take action. Users have a goal and any friction they encounter is dramatically more evident in the real world vs. during testing.

Lessons Learned

On one of our recent website projects, the user testing I conducted provided us with valuable results around an e-commerce ordering process - or so I assumed. The test was conducted with assumptions or expectations about what my findings would be. These assumptions were not revealed to the testers to avoid coaching responses (establishing hypothesis). What I learned after implementing my test findings was that real world users were still confused. During the testing phase, the participants made logical assumptions about the necessary steps needed to complete their actions (in this case: online food reservations and purchase). Once outside of the testing environment and with less context about the product and service they were interacting with, the experience broke down. The process wasn't as evident or clear to users after all. This information was at odds with the findings during the first round of testing.

Goal oriented testing

Focus on Function

The lesson learned from this exercise was that testing needed to involve a follow-up period with enough time for users to forget their expectations of the product. Note: I established a 3 week follow-up period for the product mentioned in this test. When the product was revisited, participants returned with some "innocence" to the experience. This provided cleaner data and results that were less informed by bias (from the first round of testing). The findings were that the product still needed refinement. The results clearly aligned with the real-world user pain points.

Doing it solo

Guess what? We don't all have fancy UX departments. We don't all have the ability to walk over to the user experience design team and ask for mind maps and user flows. Often we designers are left to fend for ourselves when it comes to implementing things like user-testing and persona development. Sure we all could benefit from a having a top-down approach to UX in our organizations. Yet the reality for many professional designers is often never that convenient. The good news is that smaller teams and freelancers can practice some very helpful exercises like avoiding free examination and implementing follow up testing. We designers can have a more measurable impact on the product when we are ambitious enough.