Designing an Arabic User Experience – Part 3: Usability Testing in Saudi Arabia

Design for your users. Not for you.

We’ve repeated it a few times throughout this series on Arabic UX and Usability.

No amount of experience or familiarity with the region, cultural influences or the language will ever match insights you’ll get from testing designs with real users.

At UXBERT we learn something new every time we run usability testing sessions, and we run them a lot. Whether it’s websites, mobile apps or intranets, client projects or our own products, usability testing sessions are an essential part of our process. It’s the key ingredient to building usable designs.

Usability testing brings us face to face with real users and gives us actionable data to make intelligent design decisions.

Our belief in the value of usability testing combined with being right here on the ground gives us unique insight into conducting user research in general, and usability testing specifically, in Saudi Arabia.

In this final article of our Arabic UX & Usability series, we’re going to cover some of the challenges that come with usability testing in Saudi Arabia.

Who Do We Test With?

A better question would probably be ‘who don’t we test with’?

We’re fortunate enough to work with clients from a range of different industries. This gives us the chance to test and research attitudes and behaviors of users from different walks of life, social status, and educational backgrounds.

While a big proportion of our testing is with Saudi nationals, the large expat population in the country provides opportunities for testing with Arabs from the broader Middle East as well as expats from Europe, America and the rest of Asia.

Quick Summary of the Usability Testing Process

Before we dive into the details, it’s useful to provide a summary of our usability testing process for context into these insights.

To begin with, the key is to test with representative users.

Although testing with any users is better than no testing, to get real value from usability testing you need to be talking to and testing with people who’ll actually end up using your designs.

When recruiting, we make a point of screening any potential participant before we invite them to take part. This consists of going through a series of pre-prepared questions to determine if the potential participant fits the profile of our target user. Screening questionnaires are made up of a combination of demographic, behavioral and attitudinal questions put together with the client.

Once a user is considered eligible, they’re booked in for a session. Sessions either take place at our own usability lab or we’ll go out to the user. It depends on the needs of the project and the availability of the participant.

Sessions start with an explanation of what we’re doing, why we’re doing it, and clarifying that it’s the product/design that’s being tested and not them. It’s important to emphasize to participants that there are no right or wrong answers.

So, quick recap out of the way, here are some of the things that we’ve noticed that are particular to users here in Saudi Arabia.

No One Knows What Usability Testing Is. No One Knows What to Expect.

User research in general and usability testing, in particular, are foreign concepts to both users and businesses here.

People are familiar with certain techniques (surveys, focus groups, etc.) but these are generally associated with marketing efforts, and therefore opinions. The idea of conducting research into behavior as a part of design efforts is a new one.

This provides challenges in the recruitment process as well as the actual research itself.

When we first get in touch with potential participants and begin to explain to them what we’re doing many will immediately assume we’re carrying out a survey over the phone.

The go-to response then is to tell us they haven’t got the time to take part or asking us to just send them our questions in an email, not realizing the questions we want to ask aren’t actually a part of the research but instead a part of the recruitment process.

Even when we’re given time to explain, it usually takes a detailed explanation to communicate exactly what it is that we’re doing.

People associate ‘research’ with market research. But when they hear ‘usability testing’, it’s associated with technical testing. This can be intimidating for non-technical users in particular.

“I don’t think I’m the right person for this” is a statement we’ll hear regularly. We have to reassure them that they are in fact the right person as long as whatever is being tested is something that they’d use.

“We’re testing the product, not you.”

This is followed by a rundown of what will actually happen during the session. They’ll be sat at a desk in front of a computer (or other devices) and will be asked to perform a number of tasks. As they’re conducting the tasks there’ll be a moderator who’ll be asking them questions about what they’re doing.

Most of the time once we’ve gone through the process, people seem to have a better understanding and at the very least are now curious about it.

Participants Treat Sessions Like They’re The Ones Being Tested

Despite being briefed that the aim of the sessions is to test the design and not them, many participants still approach sessions as if it’s them being tested.

They want to make sure that they do the ‘right’ thing or have the ‘best’ answers.

Participants often search for confirmation that they were helpful to the study. At the end of sessions, some will express feelings of guilt that they didn’t do well or will ask if they completed tasks correctly.

Alternatively, it does sometimes go to the other end of the spectrum, something we often observe with younger more tech literate users. They’ll be highly critical of everything trying to be extra-helpful with their feedback.

Because of this, it’s important we emphasize at the beginning of the session as well as throughout the test that there’s no such thing as passing or failing in a usability test.

Also helpful is to remind them that we didn’t design what’s being tested so they shouldn’t worry about hurting our feelings. Not only will we not take criticisms personally, we actually want to hear those criticisms.

Now, this issue isn’t exactly unique to users in Saudi Arabia, it’s something that UX Researchers from all over the world will have experienced at some point. But in our experiences, we’ve encountered it more regularly here than in usability testing sessions we’ve conducted abroad.

Moderating Usability Test Sessions With the Opposite Gender

Segregation between genders in public is a particular feature of Saudi society. And this can play a role in how we have to approach usability testing.

When the moderator is of the opposite gender, participants sometimes demonstrate more reserved behavior (for both male and female participants). In some cases, female participants will only be willing to take part in sessions once they’ve been reassured that the moderator is also female. And in a few instances, we’ve had female users bring in male relative to the session to sit in with them.

As a result, we try to make an extra effort to ensure we have same gender moderators available for each session. Planning and timetabling is important for any usability test project, but even more so here.

Another characteristic that’s particularly unique to Saudi is that a significant number of females who attend test sessions wear the niqab. The niqab is a veil that covers the face revealing just the eyes.

This makes the moderator’s job that little bit more difficult.

Part of the moderator’s role when running usability tests is to take notice of the participant’s reactions and identify opportunities to ask probing questions not just from what people say but also through their reactions and facial expressions.

The niqab effectively creates a barrier to this type of analysis meaning the moderator needs to make an extra effort to probe intelligently and get reactions while getting the user to ‘Think Aloud’.

Participants Can Be Apprehensive About Video Recording

We regularly film usability testing sessions.

It allows the research team to go back and review the session to pick up anything that the moderator may have missed. It also means we can share footage of sessions with team members and clients who didn’t attend.

Nothing convinces clients or developers about usability flaws in a product better than seeing and hearing it for themselves. Even if we report back that a user didn’t like or had difficulty with something, it’ll never have the same impact as them seeing and hearing the user’s complaints for themselves.

While we always encourage clients to be present for at least one or two tests themselves, they can’t be there for all of them. And if it’s a big team, having the recording means we can share highlights of sessions with the whole team.

But not every participant is willing to be filmed.

Participants are already slightly apprehensive about the test sessions because of the newness of it. And when we say we’re going to film the session, it can bring an added level of apprehension.

People here value personal privacy highly. If they’re being recorded, there’s a concern about whom those recordings will be shared with and what they’ll be used for.

When we’re asking for permission to record sessions, we make a point of assuring participants of the confidentiality of the recording, emphasizing that it will never be made public or used for any promotional materials without their permission.

Because having video recordings of sessions is such a benefit, it’s important that we conduct every aspect of the session (from recruitment and scheduling through to welcoming them to the session) with professionalism. That’s what helps the participant to trust us and take us at our word when we emphasize the confidentiality of any recordings.

While apprehension about being recorded occurs with both genders, it’s more prevalent with female respondents. Even when female respondents do agree to be recorded, if they’re wearing a niqab we effectively lose out on a lot of the benefit of seeing facial expressions.

Because of the degree of apprehension about video recording and for general ethical reasons, we make sure to obtain the participant’s written consent to record.

When there’s no specific request for recordings from the client, we’ll inform participants about our desire to record and get their consent at the beginning of the session, when we’re briefing them on how the test will work. Having them in front of us makes it easier to establish trust and allows us to show them that we’re conducting legitimate research.

If they’re still not comfortable with filming, we’ll carry on with the session without it and make a note of it to report back to the client.

For certain projects though, the client will specifically request video recordings.

In such instances, we’ll always let our clients know about the common reservations that people have about being recorded. If however, they’re insistent on filming sessions (which is completely understandable) then it’s important that we inform participants and get their consent when they’re actually being recruited.

Those participants who refuse to consent to recording are then not recruited for these particular projects.

Older Participants Don’t Like To Be Told What To Do

As we mentioned above, the best feedback from testing comes when you test with representative users.

We test with a wide range of users, from different backgrounds, levels of educations, income levels and also of different ages.

In our usability testing sessions, we’ve often found that older users are generally less self-conscious about sharing their feedback.

However, they also have a habit of deviating from task goals meaning the moderator has to redirect them to complete the task. This is an important concern as constant deviation affects the pace of the interview. We have limited time with users. And because we’re usually conducting multiple usability testing sessions in a day, it’s important we stick to the timetable.

With older users, this isn’t always very easy.

In a society where respect for elders is deeply embedded in most interactions with them, it’s sometimes difficult for the moderator to have to continuously interrupt an elderly participant.

The first time, fine. The second time, ok. But when it keeps happening it starts to affect the dynamic between the participant and the moderator.

The moderator starts to feel uncomfortable about having to interrupt. And the elderly participant starts to get annoyed because they’re not really used to being told what to do by someone so much younger than them.

Sure they’ll still listen, but we’ve often felt their demeanor change very rapidly.

Usability Testing in Saudi Arabia. If Your Target Users Are Here, You Need To Test With Them.

The issues we’ve covered above highlight some of the unique attributes of running usability testing in Saudi Arabia.

Some of them are similar to those faced by UX Researchers from around the world if only a little bit more amplified. Others are more particular to Saudi Arabia.

Through a combination of flexibility in the approach, planning ahead when scheduling and having moderators who can adapt to the situation, usability tests can provide exceptionally useful insights into your product’s designs.

And regardless of the obstacles involved, if your target users are in Saudi Arabia, then you need to be testing your designs with them.

Points of Note:

• Nothing can replace testing your designs with real users
• The key to getting the most out of usability testing is to test with the target users of your product
• Usability testing is a new concept in the country. As a result, most participants don’t know what to expect making recruitment more difficult
• Participants often act as if they’re the one being tested and not the product, resulting in them being too eager to give the ‘right’ answer rather than an honest one
• Make an effort to ensure that you use moderators of the same gender as the participant to help make them more comfortable
• Not all participants are comfortable with being recorded. Ensure that you obtain their consent beforehand
• When running tests with elderly users, the moderator must be able to ensure that they can keep the participant on-track without offending them

Other Articles in the ‘Designing an Arabic User Experience’ Series:

• Part 1: What, Why and How

• Part 2: Usability & Arabic User Interfaces

At UXBERT Labs we specialize in UX Research, Design & Development services that deliver world-class experiences. With offices in Riyadh and Dubai, our team of UX Researchers, Designers, and Developers deliver custom designed and built software to help businesses succeed.

Looking for a UX agency in Dubai or Saudi? Email us at hello@uxbert.com and let us show you why we’re an award winning company.