Usability evaluation methods for the web: A systematic mapping study

Abstract

Context

In recent years, many usability evaluation methods (UEMs) have been employed to evaluate Web applications. However, many of these applications still do not meet most customers’ usability expectations and many companies have folded as a result of not considering Web usability issues. No studies currently exist with regard to either the use of usability evaluation methods for the Web or the benefits they bring.

Objective

The objective of this paper is to summarize the current knowledge that is available as regards the usability evaluation methods (UEMs) that have been employed to evaluate Web applications over the last 14 years.

Method

A systematic mapping study was performed to assess the UEMs that have been used by researchers to evaluate Web applications and their relation to the Web development process. Systematic mapping studies are useful for categorizing and summarizing the existing information concerning a research question in an unbiased manner.

Results

The results show that around 39% of the papers reviewed reported the use of evaluation methods that had been specifically crafted for the Web. The results also show that the type of method most widely used was that of User Testing. The results identify several research gaps, such as the fact that around 90% of the studies applied evaluations during the implementation phase of the Web application development, which is the most costly phase in which to perform changes. A list of the UEMs that were found is also provided in order to guide novice usability practitioners.

Conclusions

From an initial set of 2703 papers, a total of 206 research papers were selected for the mapping study. The results obtained allowed us to reach conclusions concerning the state-of-the-art of UEMs for evaluating Web applications. This allowed us to identify several research gaps, which subsequently provided us with a framework in which new research activities can be more appropriately positioned, and from which useful information for novice usability practitioners can be extracted.

Introduction

Usability is considered to be one of the most important quality factors for Web applications, along with others such as reliability and security (Offutt [27]). Web applications are currently the backbone of business and information exchange, and are therefore the initial means to present products and services to potential customers. They are also employed by governments to disseminate relevant information to citizens. The ease or difficulty that users experience with these Web applications determines their success or failure. usability evaluation methods (UEMs) which are specifically crafted for the Web, and technologies that support the usability design process, have therefore become critical (Neuwirth and Regli [25]).

The challenge of developing more usable Web applications has led to the emergence of a variety of methods, techniques, and tools with which to address Web usability issues. Although much wisdom exists on how to develop usable Web applications, many of these applications still do not meet most customers’ usability expectations (Offutt [27]). In addition, many companies have folded as a result of not considering Web usability issues (Becker and Mottay [4]). This issue has been addressed in several studies aimed at studying and/or comparing UEMs for Web development (e.g., Alva et al. [2], Cunliffe [8]). This kind of study often compares a limited number of evaluation methods, and the selection of methods is normally driven by the researcher’s expectations. There is thus a need for a more systematic identification of those UEMs which have been successfully applied to Web development.

A systematic mapping study provides an objective procedure for identifying the nature and extent of the research that is available to answer a particular research question. These kinds of studies also help to identify gaps in current research in order to suggest areas for further investigation. They therefore also provide a framework and background in which to appropriately develop future research activities (Budgen et al. [6]).

In previous work we have presented a preliminary systematic mapping study that was carried out to assess which UEMs have been used for Web usability evaluation and their relation to the Web development process (Insfran and Fernandez [16]). In this paper, we present an extended, improved and updated systematic mapping study with the aim of examining the current use of UEMs in Web development from the point of view of the following research question: “What usability evaluation methods have been employed by researchers to evaluate Web artifacts, and how have these methods been used?”. We have improved our preliminary systematic mapping study by: defining a new search string that allows more papers to be retrieved; searching in more bibliographic sources; applying new data extraction criteria and applying other synthesis techniques in order to present useful information to both researchers and practitioners.

The systematic mapping study has allowed us to outline the issues that are especially relevant to practitioners who conduct usability studies, which are, among others: how the usability evaluation methods are applied in the Web domain, what types of UEMs are most widely used, and which phase of the Web development process they are applied in. We also outline the issue that is most relevant to usability researchers, which is how to improve the current practices of Web usability research.

This paper is organized as follows. Section 2 presents the background of UEMs and introduces readers to the topic of usability evaluation. Section 3 describes the research method that was used to map the UEMs employed in Web development. Section 4 presents the results obtained from the systematic mapping study. Section 5 discusses the main findings and the limitations of this mapping study, along with the implications for research and practice. Finally, Section 6 presents our conclusions and suggests topics for further research.