A Beginner’s Guide to Heuristic Evaluation (Part 2)

Here’s part 1 of this blog series in case you haven’t read it yet. I suggest reading it first to get a good look and feel of what a heuristic evaluation is.

A Beginner’s Guide to Heuristic Evaluation (Part 1)

Moving on

I guess by this time you’ve already prepared the materials or even so, the people to do your evaluation. Before we start, I want to be perfectly clear that a heuristic evaluation can be a subjective form of an assessment. Remember, a number 2 score for another person might be a number 1 score for another.  I could not emphasize enough to do a usability test to complement and debunk some of the issues you’ve encountered during your evaluation.

Conducting the Evaluation

There are two ways to conduct the evaluation, an individual evaluation or a group evaluation. I did this evaluation by myself so I have to go with an individual evaluation. Tune in on part 3 of our blog post to learn how to do a group evaluation with your team.

Individual Evaluation – each reviewer will go through your website by themselves. After they are done with the evaluation all the issues seen would be reported to you. You can choose to brainstorm with the other evaluators after they are done or just go through the list of issues by yourself. Conducting a brainstorm together is better though, you guys can dissect and prioritize issues together.

Pros:

  1. Can be done remotely
  2. Allows reviewers to focus on their tasks
  3. Easier to facilitate, you basically brief them and get your results after

Cons:

  1. A lot more issues will be reported
  2. May have a longer turnaround time

Group Evaluation – evaluators review the interface with the team while one person records all the problems. This person can be you or somebody else. The key here is that evaluators must assess the problem together and its severity. They don’t have to agree on the same problem, you guys can vote or leave it to the usability test to see which should be prioritized.

Pros:

  1. One time facilitation
  2. Real time discussion
  3. Issues are prioritized on the spot

Cons:

  1. Harder to prepare and schedule

Starting the individual evaluation

Start the heuristic evaluation by briefing your evaluators. Relay the goals and target market of the website and the main tasks that they have to go through. If your reviewer is not a usability expert, an explanation of the heuristics must be done. Since we have a heuristic dedicated to the home page, I usually start the evaluation there then proceed to the main tasks that I have done.

Here’s a short breakdown of my methodology:

  1. Find out who the website is for, what is its purpose or goal from the facilitator
  2. Receive tasks that I have to do from the facilitator (If you plan to do a usability test after the heuristic evaluation, the tasks for the usability test should be the same)
  3. Start the evaluation with the assessment of the home page
  4. Carry out my tasks while taking note of issues and problems I’ve encountered. At this point I’ll start giving a score on my heuristics.

Once I’m done carrying out my tasks and completed my evaluation. I’ll end up with an overall usability score of the website and a radar chart as well. I would then proceed to the next website I would assess.

FAQs

I don’t know if this heuristic applies to the website

Hey, no worries. Not all heuristics are going to be used in the assessment. Simply skip that question and leave it blank.

I can’t do the task while giving scores on my heuristics

This is basically the problem I had when starting out. Just take it slow and do the tasks that are assigned to you. If you have to go over them again, do so. One trick I would do is to see where the problem I’ve found can be applied. For example Information Architecture, find a specific question tailored to that problem from your excel sheet. If there’s none, take note of it and discuss it with your facilitator and team.

I’ve done all the tasks but have a lot more heuristics to cover

Sometimes there are tasks that won’t have any forms involved; however you have it as one of your heuristics. What you can do is look for a form, if they have one, then assess it.

Can the facilitator be an evaluator as well?

Yes, sometimes the facilitator is just a member of your team or maybe a person who will lead the discussion later on.

Analysing your results

Once every one of your evaluator is done, you can start by reviewing each heuristic individually. Start at the home page and work your way down. Remove any similar or duplicate comments seen by the evaluators if you have more than one evaluator. You can do this by yourself or choose to do a brainstorming session with your evaluators after they are done. Next, tally the overall score of your website. Here’s the formula in case you need it.

Sum of evaluator scores / (100)(# of evaluators) = A

A x 100 = Average Score

For Example:

Evaluator 1: 70% ; Evaluator 2: 55%; Evaluator 3: 60%

70% + 55% + 60% = 185%

(100)(3) = 300%

185%/300% = 0.61

0.61 x 100 = 61% Average Score

So I guess you’re thinking what does the score I’ve tallied mean? Is this a good score or a bad score? Ideally a website should at least get a 75% Usability Score. Each heuristic should at least have a 75% score as well to keep it balanced, something like the table below.

 

The image above shows the overall usability score of a website I've evaluated. This website has an overall usability score of 55%.

 

The image above shows the overall usability score of a website I've evaluated with a 75% score on all heuristics.

The image above shows the overall usability score of a website I’ve evaluated with a 75% score on all heuristics.

Here’s a sample analysis of a website as well.
The image above shows a sample of a heuristic table with an overall score of 55%.

The image above shows a sample of a heuristic table with an overall sccore of 55%.

 

The image above shows a sample of a radar chart with an overall score of 55%.

The image above shows a sample of a radar chart with an overall score of 55%.

The Good

This website has a fairly high score for the home page with a total of a 70% score. Navigation areas on the home page are not over-formatted and users will not mistake them for advertisements however text on the mega menu does need improvement. Excessive use of scripts, applets, movies, audio files, graphics and images has been avoided which means no banner blindness for the users. The site also avoids advertisements, especially pop-ups that may put off other users.

The Bad

Referring to the radar chart, we could clearly see that Task Orientation should be improved first, garnering a percentile score of 36% which is the lowest. A number of common tasks are hard to accomplish and new windows are used to present other data. Forms are not validated before being submitted and some writing are too short to be informative. Overall the website needs improvement on all heuristics.

 Comparing Websites

If you’ve evaluated a single website only, you can skip this part. But hey, a little more knowledge is better than none.

To emphasize each heuristic you have calculated you can overlay one radar plot on another.

The image above shows three websites with an overall score of 55%, 49% and 48%.

The image above shows three websites with an overall score of 55%, 49% and 48%.

See, now you have a clearer view of the results of each website.

As you can see all of them have strong and weak points to their website. All three websites have a relatively high score for their home page. Writing and content quality has the most similar figures out of all heuristics with search being the lowest for all three websites.

Seeing that you’ve pinpointed your competitor’s strengths and weaknesses, it is now up to you how to use this to your advantage.

Ok, now I have my score. What do I do next?

You can start by prioritizing which problems to tackle with your team. Since you have your excel sheet full of notes and scores, you simply have to segregate your data and fix the problems with the lowest value first. In my case the value (-1) is the lowest value on our scoring system.

Maybe now you’re thinking, “But there a lot of -1s in my scores, how do I know which to fix first?” Remember, it’s always easier to do quick fixes first. For example, things like copy changes and colours of call to action buttons. By doing quick fixes first you will immediately minimize the severity of the issues and it might even solve the problem completely. Consult with your team and see what you guys can do immediately.

Wait, there’s more!

Since we love you guys so much, Part 3 of our blog post would be how to conduct a group heuristic evaluation. We’ll walk you through the whole process and show you guys the difference of results with a group vs. an individual evaluation.

For questions just comment down below!

Good luck guys!

References:

Danino, N. (2001, September). Heuristic Evaluation – a Step By Step Guide Article. Retrieved from http://www.sitepoint.com/heuristic-evaluation-guide/ – See more at: http://blog.foolprooflabs.com/2013/11/beginners-guide-heuristic-evaluation-part-1/#sthash.c50AyDyV.dpuf

 



About Christine Balatbat

Have you ever used a website or an application and got confused or frustrated? My job is to make sure that doesn't happen. People sometimes call us User Experience Designers, usability analysts or some other fancy word, but we pretty much do the same thing - make people's lives better. My goal is to share UX to the world, to debunk common myths about it and to inspire our future generations to innovate. You may sometimes see me talking about it. Don't be afraid to say hi! :)

One thought on “A Beginner’s Guide to Heuristic Evaluation (Part 2)

  1. […] 1 and Part 2 of our Heuristic Evaluation for Beginners blog posts. If not, head on there to get an in depth […]

Leave a Reply

Your email address will not be published. Required fields are marked *


*