Usability Testing 101

Testing a new design

Test as early and frequently as you can. The further along in development you go, the harder and more costly it is to make changes. Work with your team to identify key issues to prioritize and retest in subsequent rounds to ensure the design makes sense to people before launch.

If you’re developing a new design or interaction, you can test this at different levels of fidelity, from rough sketches through to interactive prototypes to developed features before release.

Use what you learn from each round of testing to improve your design, and then test it again. The more frequently you can iterate, the more confidence you’ll have in your design. Even in later stages of the development process, your team should be prepared to address larger usability problems and make relevant tweaks.

Usability testing should be an iterative process. Each round should be made up of several testing sessions, and it pays to do several rounds to get things right.

Testing an existing product or service

Conducting a usability test on an existing product or service helps you to assess the current solution and identify areas for improvement.

It also gives you an opportunity to benchmark the existing experience, making it easier to measure the impact of changes later down the track.

Once you have a new design in place, you can re-test it using the same tasks as your original test and see how things compare.

Testing your competitors

Reviewing the usability of your competitors is a great way to learn how your product or service compares to others in your market. It can also help glean insight into potential opportunities and gaps in your current offering and feed into strategic decision making.

Conducting usability tests on your competitors is no different to testing your own product. Identify a common set of tasks and scenarios to use during your study to easily compare the performance across different competitors.

A script (or guide) is an outline of all the essential information to cover during your session.

It should include the key questions, scenarios and tasks you want your participant to complete, as well as any additional information you want to communicate to your participant, such as:

• introductions (including any observers present)
• a general overview of the session and the purpose of the research
• privacy and consent information
• recording permissions (if applicable, including information on how these will be used and stored)
• instructions to encourage participants to think aloud and verbalize their thoughts and feelings throughout the session
• a friendly reminder that you’re testing the product or service, not the participant
• any additional information, such as guidance on what to do when the participant gets stuck or has a question.

As a researcher, it’s your job to set clear expectations and make sure the participant feels comfortable. Ensure you cover off all essential information in your script, even if you do usability testing often.

Here’s a great example of a script from renowned usability testing expert Steve Krug.

Before your first session, you’ll want to consider the finer details to help you get your usability tests right:

• Where will your sessions take place?
• Who will be involved in the sessions? What do they need to know?
• What technology do you need?
• What will your incentives be? How will you pay/supply these?
• How will you capture consent for using data and recordings from your sessions?
• Does your research require an NDA?

Work with your team to find the answers to these questions before you conduct your first session to keep your study running smoothly.

If you have access to a pool of participants (like employees if you’re working on an internal product, or your customer mailing list) then reaching out to potential participants via email can be an easy way to source them.

Similarly, you can recruit participants by advertising on social media channels or by running banners or pop ups on your website.

When sourcing participants for usability tests, you may want to direct them to a screener, depending on your needs. This is to ensure you’re reaching people who meet the specific criteria you’re looking for. What you decide to include in your screener will vary depending on your objectives.

Some useful things to include may be demographic information, previous experiences using your product or service, frequency of use, and anything else that will help you match your candidates against any target-user characteristics.

You can also make use of recruitment providers, which can be a good option if you don’t have easy access to the people you’d like to target. Dedicated recruiters will generally have access to a wide variety of potential participants, which can make this a fast and easy way to source relevant people with minimal effort.

In some cases, they’ll also oversee logistics, such as scheduling sessions and paying out incentives on your behalf. Keep in mind that recruiters and panels come at a higher price than sourcing participants on your own.

Make sure you provide your recruiter with a clear understanding of the participant characteristics you’re looking for – similar to a screening questionnaire.

An alternative to using recruiting agencies is sourcing participants on your own from online panels. These self-service panels contain databases of potential participants, allowing you to filter, match and submit orders based on your own criteria without relying on a dedicated recruiter.

To encourage potential participants to take part in your research, you’ll want to consider how you incentivize them. While you can get away with little to no incentive for internal participants (for example, existing users within your organization), external participants may be less obligated to participate if they’re not reimbursed for their time.

Incentives can be monetary in the form of cash or gift cards, or take other forms, such as company merchandise or free access to your product or service.

The amount you decide to pay your participants will depend on a few different factors. Consider the length of the session, the complexity of the tasks and whether you need participants with specific experience or skills. For example, if you’re testing a specialist medical interface with doctors, the incentive you offer should be higher than if you’re testing a banking app that has a wider range of users.

An incentive should be high enough to motivate people to participate in your study. A good starting point is to consider how much a participant would usually receive for an hour of their time and add in extra to cover the additional effort required for participating.

Once you have your first participants scheduled, you can start to create sessions in Reframer. Use the ‘Session information’ field to add any relevant information about your participant during or after your session, such as demographics, their experience using your product, or any other details worth noting.

This is also where you add Segments to each of your sessions. Segments are useful for identifying participant characteristics such as demographics, personas or user types. They come in handy when you’re analyzing your results and wanting to identify patterns in your data across different groups.

For this study, some useful information to capture about our participants could be things such as the device used during the test, their job, age group and how long they’ve been a customer.

Tags allow you to code your observations, helping you identify and organize relevant findings later on.

From the study overview, you can create and manage the tags you intend to use for your study. The Tags tab will display all the tags used by you and your team.

Create tags that will make it easy for you to filter and make sense of your data when you move on to the next stage – analysis. For usability testing, you’ll want to create a set of tags dedicated to the tasks, flows or screens you’ll be testing, as well as tags that will help you identify any problems or issues your participants encounter. This could be as simple as a ‘pass’ or ‘fail’ for each of your tasks, or a group of tags that relate to sentiment, such as ‘delighted’, ‘frustrated’ or ‘surprised’.

Whether you defined tags upfront or not, you can still change or create new tags any time.

Once you move on to analysis, you’ll easily be able to filter your data by task and see at a glance where your participants may have run into issues. This will help you to identify potential improvements.

Tips for tagging observations

• If you’re involving others in your research sessions or analysis, it’s a good idea to agree on a consistent set of tags in advance.
• Add tags for keywords you expect to use in your study to help standardize the spelling and style of tags for you and your team.
• Include tags in your script to remind you what tags to use when capturing observations.
• To encourage better quality observations, leave tagging until after your session.

Consider the environment in which your usability tests will take place so you have what you need for your first session.

Some things to ask yourself:

• Where will the session be held?
• How will you record the session (including any screens)?
• Where should your participant sit to make them most comfortable?
• What technology will your notetakers or observers require?
• Do you need to set up an observation room for anyone else involved?
• If you’re hosting a remote session, how will you set up the call to make it easy for others to observe what’s happening on the screen?

While you want to get as many people involved in your research as possible, try to limit the number of people in the same room as your participant when running in-person sessions. This will create a much more relaxed environment for your participant. Keep the number down to the essentials: the facilitator and a notetaker or observer.

If you’d like to invite others to observe the session, consider setting up an observer room with a live feed from the test room. It’s also a good idea to record your sessions so you can share them with those who are unable to attend on the day.

If you’ve recruited some coworkers or stakeholders to take part in your sessions, you’ll want to brief them to ensure they’re clear on what they can expect. This is crucial if they have never been part of a usability test before. It’s important to explain what a usability test is, how they’re run, what the value is and what sort of findings it will generate.

People you involve in your study could have a few different roles, such as observing, notetaking, asking questions or even being involved with the analysis later on.

No matter who you’re involving or what their roles are, go over the following details:

• when and where are the sessions taking place
• how many sessions they should attend
• what you expect from them during and after the session
• when they will have an opportunity to ask participants questions (many researchers will specify no questions until the end of the session and/or when prompted)
• let them know not to interject during the test, and ask them to hold off an any additional questions until the end
• how to act during the test and what their role is.

Tips for notetakers

Notetakers have an important job that requires a lot of focus. Make sure you teach your notetakers how to take good quality notes and what to listen and look for during the session. Let your notetakers know what happens to their notes after the session and how you’ll be using them.

If there are any tools you’ll be using during the test, let your notetakers know in advance so they can become familiar with them ahead of time.

For more help with taking effective observations, read this article on UX Mastery’s blog.

Tips for observers

While observing is the easiest way to involve a large number of people, it’s important to ensure everyone stays engaged so they can understand the value. Encourage observers (especially key stakeholders) to write down their top insights following each session, and invite them to participate in a debrief afterwards to identify patterns among everyone involved.

Capturing observations as your session is taking place ensures you don’t miss the essential details as they arise. It also takes some of the load off after your session is complete, allowing you to quickly identify key findings.

Observations refer to any relevant things you see or hear during your usability testing sessions. In Reframer, you’ll capture your qualitative data in the form of individual observations.

If you have a dedicated notetaker, they can focus on taking notes while you facilitate the session.

If you’re on your own and don’t have a dedicated notetaker, it’s best to record your session and stay focused on your participant. You can listen back and take notes later.

Information to include in your observations

When capturing observations, it’s important to focus not only on what your participant said but also what they did. There is often a difference between what people say they do, and what they actually end up doing. Capturing behavioral information can give your data an additional level of insight.

The role of a notetaker is not only to ensure key information is being recorded, but also to observe the session and make note of any behaviors and nonverbal cues (pauses, hesitations, looks of frustration or confusion and so on).

Sometimes little things can have a big impact, so be sure to include smaller details in your observations. For example, a participant using an operating system they’re not familiar with may affect their ability to perform tasks.

Tips for capturing observations in Reframer

If you’re a notetaker you can capture your observations directly in Reframer while your session is taking place. Observations can also be added after the session is complete (for example, from a recording). You can also import existing data into Reframer from elsewhere.

Capturing observations directly into Reframer is as easy as writing a note and hitting enter. The observation you’ve written will automatically save and appear in your list of observations. Each of your sessions can have an unlimited number of observations.

You may choose to capture your observations verbatim, or only make note of key information throughout your session. Including quotes is a great way to bring your observations to life.

To help focus on your participant during your session, you may want to refrain from tagging until after your session is over.

You can also:

• use the star to quickly highlight any particularly important observations, or those you want to quickly refer back to later on
• use quotation marks to automatically add a “quote” tag to your observation
• use the hash symbol (#) to quickly tag observations as you capture them, or pick from an existing list.

Once your session is complete, it’s a good idea to conduct a debrief, allowing the team to come together and pull out key insights from what everyone saw and heard during the session.

Host this meeting as soon as possible after the session has finished so that the information is still fresh in your minds — booking in time immediately after each session is ideal!

If you had multiple notetakers and observers, encourage each person to list their most significant outtakes from the session to find patterns among what was observed across the team.

The debrief is an opportunity for the team to look through the session and review relevant findings in Reframer. Use the filter to bring up any starred observations so the team can discuss them together, such as why they were starred or if anything in particular made these observations stand out. You can even use this time together to tidy up some of the data and get rid of duplicates and errors before moving on to analysis.

In some circumstances, you may choose to record your sessions and make notes by reviewing the recordings.

You can use Reframer to make notes from your video or voice recordings in the same way a notetaker would capture observations during live in-person sessions.

This can be the case if you don’t have a dedicated notetaker, in which making notes on your own during the session could be disruptive.

If your notetaker has used a different method for taking notes other than Reframer, you can import this information directly into your individual sessions. If you have information on the time your observations were taken, you can include timestamps for individual observations.

Tagging your observations helps you make sense of your data once you move on to analysis. There, you’ll be able to filter your observations based on tags and combine these to uncover themes. For example, observations tagged with #confusion and #check_out may suggest issues with that particular feature.

Use tags to link observations back to your research objectives, identify sentiment or help to mark any particularly significant findings.

You can add tags directly to your observations using hashtags (for example, #frustration), or using the tag list in the sidebar on the right. You can also multi-select your observations and add several tags at once.

The star feature allows you to quickly highlight key observations from your session, making them stand out from the rest of your findings.

Stars function in a similar way to tags, making it easy to filter key findings, quotes, or notes of interest when you begin to analyze your results across all your sessions.

Use this feature to mark any observations that are particularly significant, or those that provide great quotes or outtakes from your session.

The core of your Reframer analysis lies in the theme builder, which allows you to create themes by grouping observations that all relate to a similar topic or finding.

Using the theme builder, you can filter your observations by combinations of sessions and tags. When you select a tag, you’ll see all observations in your study that have been assigned that specific tag. From here, you can see any other tags that were used in combination with the one you selected, and broaden or narrow your selection to reveal relationships between them.

When you narrow your tag selection you can reveal observations that contain the specific tags you selected (for example, ‘T1_Login’ AND ‘delight’).

Narrowing

Broadening

Broadening your criteria will show you observations that contain at least one of the tags you have selected (for example, ‘frustration’ OR ‘fail’).

Tips for creating themes

To get the most use out of the theme builder, it’s best to start by exploring themes that relate to your key research objectives. You may want to look at your data from the perspective of:

• major issues for each task, or steps in a workflow
• the most significant observations based on those that have been starred during the session
• parts of the test where participants ran into the most pain points
• moments where participants experienced moments of delight
• any suggestions, requests or ideas your participants had
• patterns in existing behaviors reported by participants at the start of the session
• finding quotes to support the outcomes of your study.

When exploring the relationships between different tags, pay attention to tags that have a strong relationship, as well as those that don’t. These can help you validate any assumptions you had prior to your research. For example, you may have had concerns that a new navigation will confuse users and find that, in fact, no one encountered any issues finding what they needed.

When your tag combinations start to reveal interesting patterns in your data, you can save these as a theme. Themes group together observations that relate to a similar topic. You can name your theme and add a description for additional context and help other team members understand what you’ve discovered.

The chord diagram is a visualization that helps you understand the relationships between the tags you’ve used throughout your observations.

You can also exclude any number of tags from the visualization, making it easy to focus on relevant tags.

You can use the chord diagram to explore relationships you may want to delve into in greater detail in the theme builder.

The bubble chart shows you the relative frequency of all the tags used throughout your project. The larger the bubble, the more frequently a tag was used.

The bubble chart makes it easy to identify tags that were used most often and can help in understanding common findings in your research. It’s also a great way to showcase your findings to stakeholders.

Similar to the chord diagram, you can exclude any number of tags from the visualization. To help you focus on your most important findings, consider removing non-specific tags, such as stars or “quote” tags to help you get a sense of the themes in your data.

If you need to export your data for any reason, Reframer makes it easy to download this information in .XLSX format. If you download the spreadsheet you’ll have access to the data from all your sessions, including starred observations, tags, timestamps and any themes you created when analyzing your data.

Once you’ve identified the key themes from your research, it’s time to share your findings with others.

There’s no right or wrong way of sharing your findings with the rest of your team. While in some cases a simple session with your team and stakeholders to run through the themes you identified will suffice, in other cases you may have to compile supporting evidence such as quotes, audio or video clips to get others onboard.

The most important thing about usability testing is making sure the most significant findings are understood, prioritized and actioned.

Remember — usability testing is an iterative process. Each round should lead to improvements, and with time, lead to more confidence in you and your team, and a better experience for your users.