Introduction

There are two types of usability evaluation techniques, formative and summative.

  • Formative evaluations are carried out at early and intermediate stages.
  • Summative evaluation is carried out at the final stage of development.

Therefore, the usability evaluations used within the analysis stages are formative. The following formative evaluations will be discussed within this answer; cognitive walkthroughs, heuristic evaluations…..

Formative evaluations provide informal, often qualitative indications of usability which can be very useful in determining whether the design needs revision or simply minor amendments. The results of the evaluations can be relatively quickly analysed and the results fed back into the design stage.

Evaluation Methods

  • Analytical
    • Based on mathematical models of the user (examples:- GOMS, TAG)
  • Expert
    • Usability “Experts” work through a checklist identifying problem areas (examples:- heuristic evaluation, cognitive walkthroughs)
  • Observational
    • Users are given tasks to do and their performance is monitored closely, often in a ‘Usability Lab’ (examples:- direct observation, video-based evaluation, interaction monitoring, cooperative evaluation)
  • Controlled Experiment
    • Controlled and monitored experiments using several groups of carefully chosen subjects
  • Survey
    • Questionnaires and interviews to gather user views

TDD (Task Decomposition Diagram)

Definition
A diagrammatic representation of how ‘something’ is done – hierarchical and ordered in levels

  • Top level = task name
  • Next level = broad description of a complex activity
  • Next level = more detailed description or ‘break down’ of the actions necessary
  • …and so on until the task is diagrammed in detail

Methodology
Each diagram consists of:-

  • A series of linked “boxes”, each containing a task
  • Each box is made up of an “activity” and an “object” (i.e. do something with, or to something)
    (a phrase with a verb and a noun – e.g. switch on light)
  • All boxes are numbered in logical order
  • The descending levels indicate increasing detail
  • Moving across left to right across diagram constituents indicates the order in which tasks are carried out

Evaluation
Positives

  • Intuitive for many people

Negatives

  • Mis-match between designer’s model and users application
  • Validity of results “is the analysis correct?” and “Is the analysis appropriate?”

Cognitive Walkthrough

Definition
‘Usability Experts’ put themselves in place of a user to ‘walk through’ a specific task, attempting to predict problems users may face.

Requirements
It is stated by Lewis, C. and Rieman, J. (1994) that the following items are required in order for a cognitive walkthrough to be carried out;

  • Description/prototype of the interface
  • Description of a typical task the system is designed for
  • A written list of ‘ideal’ user actions needed to complete the user goal efficiently
  • The characteristics of the target user

Methodology
The cognitive walkthrough it carried out as follows;

  • The usability expert is provided with the characteristics of the user and also the user goal and associated tasks
  • The usability expert then attempts to complete the user goal whilst recording any potential problems they perceive a user may face when trying to complete the same goal.
  • Due to the nature of the cognitive walkthrough, it is particularly useful for identifying problems that may occur with first time use of a system.

Support the Production of Usable Systems

  • The cognitive walkthrough supports the production of usable systems as it alerts developers as to what elements of the system are likely to cause problems for users.
  • The knowledge gained from the cognitive walkthrough can be used, by the developer, to carry out alterations/ improvements within the system, consequently enhancing the system usability.

Evaluation
Positives

  • The cognitive walkthrough puts all focus totally onto the user
  • The cognitive walkthrough may enable the usability expert to focus their attention on known problem areas
  • The cognitive walkthrough recognises user goals
  • It is cheap to carry out

Negatives

  • The cognitive walkthrough does not cover the entire problem space as it is based around a single user goal
  • The cognitive walkthrough may lead to bias results due to it revolving around one goal
  • The nature of the cognitive walkthrough may mean that it is tedious to carry out

Heuristic Evaluation

Definition
Usability experts evaluate interface specifications, designs or prototypes according to a pre-determined set of ‘heuristics’.

The nine heuristics utilised by Nielsen and Molich are;

  1. Simple and natural dialogue
  2. Speak the users language
  3. Minimise user memory load
  4. Be consistent
  5. Provide feedback
  6. Provide clearly marked exits
  7. Provide shortcuts
  8. Good error messages
  9. Prevent errors

Methodology
The usability expert assesses the system against the set of heuristics, recording any relevant issues raised against the heuristics.

Support the Production of Usable System
The heuristic walkthrough supports the production of a usable system as it brings possible usability issues to the attention of the developer or system owner.

When these issues are noted by the developer/system owner, they will be able to make the relevant alterations to the system.

Consequently a more usable system will be produced.

Evaluation
Positives

  • Can be carried out at very early stages and also throughout development
  • Five Usability Experts can find 75% of usability problems
  • It is not task oriented (as the cognitive walkthrough is) therefore a broader view of the system will be gained
  • It evaluates both the screen layout and also the ‘flow’ of information and interaction through the system
  • It is cheap to carry out

Negatives

  • As it is not task oriented, general information will be gathered which may not be of any use to the developer
  • The web site www.userdesign.com states the validaty of Nielsen and Molich’s nine heuristics have been questioned

Observational Evaluation

Definition
Designers observe users performing tasks with the interface and record the results.

Methodology
A number of users are chosen to complete a user goal and its associated tasks. Any problems they encounter whilst completing this goal are recorded.

Support the Production of Usable System
The use of observational evaluation support the production of a usable system as it involves the recording of problems encountered by actual users using the system. For example, an observational evaluation could be carried out using a prototype for a new system. The evaluation may indicate that a number of users had difficulty logging onto the system. Consequently the logging on section of the system could be amended.

Evaluation

Positives

  • Can be cheap, quick and simple or alternatively it can be carried out so that it is more sophisticated and expensive
  • 5 users find 75% of problems, 15 find 100% (Jakob Nielsen useit.com)
  • Early designs and prototypes can be tested
  • Enables the collection of qualitative and/or quantitative data

Negatives

  • It could potentially cost a lot of money, depending on how many usability testers are hired.
  • To get an effective result, equipment to record the results may be required to be purchased, for example, voice and/or video recorders

Predictive Modelling

Definition
Applied at the early stages of the design process. The GOMS family of models include a number of specifically predictive models that allow designers to predict aspects of users’ interaction with alternative designs. GOMS stands for:

Goals
Operators
Methods
Selection rules

When users work with a computer application, they are attempting to accomplish a series of goals. In order to complete a goal, the user will apply one of a number of possible methods, sequences or discrete actions. The discrete actions are combinations of a small number of operators (such as key presses or mouse movements). In some instances, goals are achieved by using more than one method.

A particular model within the GOMS family is the keystroke level model (KLM). This model predicts the user’s speed of operation in accomplishing a task using a defined method.

Methodology
In the original model (Card et al, 1983) there were six defined operators:

K – Key or button press
P – Moving a mouse pointer to a target object
H – Moving the hands to the ‘home’ position on the keyboard
D – Drawing a straight line with a mouse
M – Mental preparation of an operation
R – The system response time

The time to accomplish a method is calculated by a two-step operation:

  1. Decompose the task into the sequence of component operators
  2. Allocate appropriate times for each operator and calculate the total

A simple example (Newman and Lamming, 1995) illustrates the approach, two alternative methods of making a piece of text bold are:

  • Select the bold option from a menu
  • Use a keyboard short-cut

Method 1:
P, K, P, K, P, K, P, K = 5.2 seconds
Method 2:
K, K, K = 0.6 seconds

Support the Production of Usable System
Therefore, the times for the two methods indicates that keyboard shortcuts is a sound design decision, although it assumes expert knowledge on the keyboard short-cut codes.
Helps designers to incorporate alternative methods of performing particular tasks.

Evaluation

Positives

  • Useful in early stages of design to help choose between design alternatives

Negatives

  • Takes time to produce a set of methods and to get users to test the methods

Pluralistic Walkthroughs

Definition
Users, developers and usability expert’s work together on tasks.

Methodology

  • Scenarios are developed in form of hard-copy screens
  • Evaluators write down sequence of actions to move from one screen to other without conferring
  • Panalists discuss their actions with each other. Users go first, then experts then developers
  • Process is repeated for all screens

Support the Production of Usable System
The purpose of bringing together various stakeholders is that each one brings a certain perspective, expertise, and set of goals for the project that enables a greater number of usability problems to be found.

Evaluation

Positives

  • Strong focus

Negatives

  • Hard to get experts together
  • Limited scenarios due to time consuming