Testing Abroad: Lessons from the Field
by Christina S
Conducting usability studies abroad is an exciting opportunity for researchers. For those who get to travel to new places, you get to see another part of the world, eat new foods, and learn new customs. There are many resources available with tips and suggestions on how to plan an international trip—but planning a vacation abroad and planning a usability study abroad are two different things.
Doing behavioral research in a foreign culture can lead to potential complications that are harder to predict when compared to usability studies done at home. Additionally, when problems do occur, it’s not always clear what resources you have at your disposal to address the situation. Ensuring that an international study goes as smoothly as possible takes additional work and planning. Below are some tips and topics to consider when planning your next international usability study, inspired by our recent experiences in Germany and Japan.
Tip 1: Understand the Local Customs
Tips like learning a few greetings in the local language abound on the internet and in guide books—and for good reason; people generally appreciate that you’ve put in the effort to learn a bit about their culture and language. However, in the context of conducting studies abroad, other situations may come up, and it’s important to be as conscious of local customs as possible to minimize any negative impact on your study.
Address Participants According to Local Customs
Because you’ll likely be meeting with many participants, it’s important to know how to address them in a more formal or business setting. Americans tend to be friendly and open but we can’t assume this is the case everywhere. For example, here in America, it is generally acceptable to address someone by their first name unless they are much more senior. However, in both Germany and Japan, greetings between people who don’t know each other are much more formal. In both countries, people are typically addressed as Mr./Mrs./Ms. and their surname—and not doing so could be a sign of disrespect. If there’s a specific situation where you are unsure, it’s better to err on the side of formality. The person will correct you if necessary.
Tailor Your Root Cause Analysis
Typically, when someone makes a mistake or has difficulty using a product, we’ll (politely) point it out in order to have a discussion on why that mistake was made. In Japan, we had to alter our customary approach and take a more indirect route to the root causes of users’ issues. Saving face is an important sociological concept there, along with many other countries. Respect for social standing and self-perception is a foundational principle underlying social interactions in Japan, and confronting participants about their difficulties during testing can violate this principle and put the session at risk.
In order to understand the root causes of difficulties that participants experienced, we could not simply point out the errors like we normally would, as such directness would have caused them great embarrassment. Instead, we had participants discuss the process to see if they volunteered any information regarding difficulties they had. If not, we focused on the step of interest and asked if they had any difficulties with that step—it’s best to phrase questions so they can say yes. Additionally, we emphasized that any difficulties they had were due to the product and not their own performance. This line of questioning certainly took longer, but it was an effective method to get the information we needed while keeping the participants comfortable. Ensure that your studies account for this extra time.
Tip 2: Work with Local Moderators and Interpreters
Unless you speak the language of the country that you’re visiting—and we don’t mean those few years of high school language classes years ago—then you’ll likely have to rely on local moderators and interpreters. Not only can they help to communicate, but they can also act as a bridge between you and your participants, helping you to navigate through the subtleties and nuances of the local culture.
Work with Trained Moderators and Interpreters
We recommend using moderators and interpreters who are trained on usability studies. Although using someone who can speak both languages can work, you also run the risk of biasing the study. Interpreters are trained to translate the meaning of your statements as closely as possible into the local language. If that person is untrained, he/she may only summarize the main idea without communicating all the details, or worse, use incorrect words that may change the meaning of the original statement.
Communicate the Study Purpose
It’s important to spend time going over the purpose of the study when working with moderators and interpreters. Reviewing the tasks and questions beforehand can save a lot of effort later on, so take the time to make sure you and your moderators are on the same page.
Although providing them with a translated document of the questions is helpful, they need to understand the intention behind the words so they can properly phrase the questions and know how to ask follow up questions that are not planned.
It is often the case that the local moderators have not specifically conducted a usability study before, and they may be unaware of the specific requirements of usability testing. You’ll need to spend time training them on certain specifics, such as asking questions in a neutral manner, and not providing help to the participant.
Tip 3: Have Fun!
While you’re traveling, try to plan in some time where you can explore the local sights and culture. During our recent trip to Germany and Japan, we made time for the Berlin wall and the first sightings of the Tokyo cherry blossoms—among other things.
This post was edited by Matthew Cavanagh.
Share this entry