Recent Advances in Computer Science

Verification Process of Usability Evaluation Model for M-banking Application HAMISU IBRAHIM ABUBAKAR, NOR LAILY HASHIM & AZHAM HUSSAIN School of Computing, University Utara Malaysia, Malaysia 06010, Kedah, Darul Aman MALAYSIA [email protected];[email protected];[email protected]

Abstract: - Several usability factors for mobile application do exist, but they are static, disintegrated and do not adequately capture the complexity of interacting with m-banking application. The model is developed in response to the need by literatures, as it is essential for m-banking application to provide the users with the expected natural sense of interaction, easy to use and also motivate the customers to accept the technology. This paper explains the verification process of a newly proposed model for evaluating the usability of mbanking application. The aim is to symbolize main practices in evaluating the usability of m-banking application. The verification was performed through the use of six usability experts in academia and mobile application developers in examining the model components. The experts completed a verification form and questionnaire that measured the model in terms of consistency, understandable, ease of use, tailorable, verifiable and overall impression. Furthermore, the proposed model has been modified based on the comments and suggestions received from experts. Similarly, the experts’ questionnaire result indicates that the proposed model is original, complete and acceptable. Therefore, the study will provide additional knowledge in both theory and practice towards model verification process, especially for financial applications usability evaluation approach. Key-Words: - Verification Process, Experts Review, Usability Evaluation, M-banking application

such models have not been sufficiently used in evaluating the usability of m-banking application. Consequently, these models did not provide overall descriptions on how to select metrics corresponding to the usability factor or criteria [20]. Therefore, the usability evaluation model for m-banking application is built through the establishment of usability factors, criteria and metrics. However, the identified component of the developed model is within the four usability contextual factors such as user, environment, technology and task [7], [2]. Consequently, these usability measurements are seen sufficient enough to use for evaluating the usability of m-banking application since they are comprehensive and carefully support Human Computer Interaction principles [2], [15], [19]. However, the main part of this paper is devoted to explaining the model verification method. The model itself contributes towards a better understanding of the modelling usability approach. Therefore, the model now reached the final stage of development and the need to verify the model component become imperative.

1 Introduction M-banking can be described as the banking transactions and services that the user can perform via a mobile device at any time and from anywhere at a users’ convenience. Most of the m-banking applications have the same functionalities. It provides a variety of financial transactions which comprise of bill payments, fund transfer, recharge card, investment and insurance. Therefore, evaluating the usability of m-banking application is important as both the designers and developers would identify the strength, weaknesses of the application [20], [15], [8]. Similarly, it captures the efficiency, effectiveness and accuracy of application. However, in order to achieve usability evaluation of m-banking application, there is a need for a comprehensive usability evaluation approach containing appropriate usability measurements [9], [1]. Many usability evaluation models have been proposed such as ISO 9241-11, ISO 9126-1, and mGQM [12], [2], [3]. However, most of these models are based on design and are not specific to a particular mobile application [8], [2]. Additionally,

ISBN: 978-1-61804-297-2

182

Recent Advances in Computer Science

2

main categories: testing and predictive metrics [18], [8]. The testing metrics are used to collect data in order to measure the actual use or function of working application and identify the problem associated with such application. In collecting this type data, it requires a fully functional application or prototype. However, preference metrics deals with the subjective evaluation level of end users satisfaction and performance metrics measure the real performance of the users while accomplishing a task, all are classified under testing metrics. Therefore, the developed model is designed based on testing metrics that comprises of preference and performance metrics. See Fig 1 in Appendix 1 for the first version of the proposed model of mbanking usability evaluation. Fourteen criteria have been generated through the use of the Systematic Literature Review. Each criterion is positioned to its corresponding usability factors based on the supported literatures. These grouping of criteria to usability factors have been used and agreed by many usability evaluations literatures both in mobile application and software domain, such as [18], [12], [20], [9]. Furthermore, to generate considerable usability metrics to the generated criteria, the defined metrics in GQM [3], QUIM model [18] and other usability studies such as mGQM [12], [13], [13] were critically analysed and employed. Therefore, forty three metrics are derived both for subjective and objective measurements. However, seventeen metrics measure objective data and twenty six metrics measure subjective measurements. Table 1 below describes the grouping process of criteria to their corresponding factors.

Problem Formulation

Many researchers on mobile application usability approach, concentrate mainly on one aspect of evaluation without considering proper model verification approach. Therefore, the model may not provide a satisfactory range of accuracy towards validation process. Consequently, little literature does exist that relate to how the model and its components have been verified by the experts [4].

3

Proposed Usability Evaluation Model for M-banking Application

The proposed usability evaluation model constitutes of three key components which include; usability factors, criteria and metrics. The usability factors comprise of efficiency, effectiveness, user satisfaction, learnability and trustfulness [1], [2], [5], [6], [7], [18]. These five usability factors are derived based on prioritization in the reviewed literatures, impotency and relevancy to m-banking functionalities. However, the derived usability factors support task/activities and technology (device) contexts. The usability factors are broken down into measurable criteria (sub-factors). A criterion is directly measurable via a particular metric that is linked to a usability factor [18], [10], [11]. In other words; the measurement of usability factor depends on the corresponding criteria while metrics are used to measure criteria that are assigned to a particular usability factor. Metrics are defined in terms of formula or countable metrics which are extracted from raw data such as video observation or experiment depending on the application type [10], [14]. The usability metrics are classified into two

Table 1 Grouping of criteria to their corresponding usability factors USABILITY FACTORS CRITERIA

Compatibility Loading time Operability Accuracy Presentation Navigation Privacy Reliability Simplicity Familiarity Consistency Content Structured Task User Guide

ISBN: 978-1-61804-297-2

Efficiency Effectiveness Trustfulness Learnability User satisfaction √ √ √ √ √ √ √ √ √ √ √ √ √ √

183

Recent Advances in Computer Science

the model, components of the model relationship that exist between each entity and five dimensions in the experts questionnaire in which the model is judged. Additionally, items included in the expert verification documents are; objectives of the model and instrument for model testing.

4 Methodology This activity was performed as a first evaluation of the developed model. Moreover, the verification process determined whether all the metrics should remain in the proposed model or some needs to be removed. It also examines the capability of collecting data by each metric. [5] cited (Lauesen and Vinter, 2001) that the reliability of using expert decision is very high when put into practice. Similarly, integrating experts in both theory and practice can significantly lead to simple and accurate results [11], [4]. The expert review and verification was used in order to provide empirical evidence from the field of HCI in academia and m-banking application developers in the industry. It will also enhance originality, richness and quality flow of measurement [22]. Verification is a technique for ensuring that the model and the components as well as other entities within the model are sufficient, accurate and complete for its purpose or determine whether the model is being built in an orderly approach [4]. However, verification is used to confirm that all the components of the model possess a satisfactory range of accuracy, completeness and consistent with the intended application. Usability experts provide speedy and valuable comments that will improve the quality of model design and development [16]. The expert may be independent from the development team and willing to give honest opinions and comments [21]

4.2

Data Collection

The experts were contacted and discussion has been made concerning the improvement of the proposed model. Six experts were selected from academics institution and m-banking application developers’ industry respectively. [17] mentioned that three to five experts are sufficient enough to review and verify a newly developed model. To achieve effective model verification process, the designed experts’ instruments were distributed to the selected usability experts in both academia and industry. Though, four of the selected experts was contacted via their emails and followed by telephone calls. While, two experts were contacted directly in their respective offices in order to get additional comments and suggestions.

4.3 Data Analysis Data collected from the experts sorted and stored in SPSS statistical package for analysis. The SPSS version 22 was used to find the MEANS of individual measurement in the experts’ instrument. The result was analysed and presented in Table 6. All the comments and suggestions received from the experts were carefully examined and considered for improvement of the proposed model.

4.1 Instrument Development In order to have a good verification approach, experts’ verification form and questionnaire were designed as instruments for the expert reviewers. The instruments contain three sections, section A is expert profile, section B contains a set of metrics with corresponding criteria and usability factors and section C is questionnaire. The questionnaire is originated from [4] with little modification and it contains five measurable factors with two (2) scale options; “Agree”, or “Disagree”. The measuring factors used to build the questionnaire includes: 1) Consistency, 2) Understandable, 3) Ease of use, 4) Tailorable and 5) Verifiable. However, the overall impression to measure experts’ opinion about the model was also included in the questionnaire. The five dimensions are used for the expert to judge the model in terms of its originality and acceptability in use for usability practitioners and research purposes [4]. The verification form contains details about the model which includes the grammatical structure of

ISBN: 978-1-61804-297-2

5 RESULTS/DISCUSSION The verification provides positive experience and has given some significant contributions for the developed model. Based on the suggestions and comments made by the experts, it appears that some metrics are repeated while few are not relevant to the intending applications. The unrelated and repeated metrics have been removed, to avoid obstacle for the proposed model. Besides, experts commented that the definition usability factors, criteria and many of the metrics are relevant and capable of collecting data. Additionally, the experts agreed that the proposed model is specific to the m-baking application and expected to provide satisfactory results when tested. Please, refer to Table 3 for overall comments given by each expert.

184

Recent Advances in Computer Science

experts rested towards “Agree”. Please, refer to Table 6 for mean scores of individual experts.

Table 3 Comments/suggestions received from the experts Comment/Suggestions

Table 2 Experts Profile Expert

B

C

D

E

F

The metrics are good for the evaluation of usability m-banking applications, but the experts have highlighted the importance of privacy and user guides. Metrics such as “single sign on”, “session timeout”, “alert/warning message”, and authentication should be included. The metrics are relevant to the defined usability criteria, but the metric “satisfaction with help” need not to be overemphasized. Metric “time taken to respond” should be modified to “Response time” and menu name should be changed to “menu item” and link list can be modified to “sub-menus”. Performance speed should be considered. The metrics are generally suitable and relevant to the defined criteria in the proposed model. The experts suggested that metrics such as “Number of errors during key-in user detail” should be under “Accuracy” criteria, “Number of interactions” should be modified to “Number of interactions per unit time”. The word attempt can be modified to “step”. Metric “Time taken to navigate a menu” can be included under “Navigation” criteria. Some mobile devices do not provide keypad anymore. The metrics are suitable, but more aspects need to be explored in each dimension. More metrics should be added under “Navigation” criteria. All the features are relevant and important. Questions on Overall Satisfaction and “Satisfaction of the task” as metrics should be included. Tasks should be selected for the evaluation and the objective of the task need to be explained

Profile

A

Specialization: Mobile application developer Position: Senior Software Engineer Year of experience: 4 years Country: Malaysia Specialization: Software Engineering Position: Senior Lecturer Year of experience: 14 years Country: Nigeria Specialization: Mobile application developer Position: Director General Year of experience: 8 years Country: Nigeria Specialization: Interaction Design Position: Senior Lecturer Year of experience: 14 years Country: Malaysia Specialization: HCI/Interaction Design Position: Associate Professor Year of experience: 14 years Country: Malaysia Specialization: HCI & Software Engineering Position: Senior Lecturer Year of experience: 14 years Country: Nigeria

B

C

D

E

F

Table 4 List of dropped metrics Dropped metrics Number of click to sign-in Number of attempts to sign-in/sign-out Satisfaction with help Satisfaction with menu names Time taken to key-in user details Satisfaction with device keypad Number of error during key-in user data

Based on the experts’ comments, metrics that are not related to the defined criteria have been removed, while other metrics suggested by the experts were added to their corresponding criteria for improvement on the model as shown in Table 4 and 5. Experts were asked to judge the proposed model using five dimensions, namely; consistency, understandable, easy to use, tailorable, verifiable and overall impression. These dimensions measured the originality, completeness and acceptability of the developed model. Two options were given in the questionnaire instrument, “Agree” and “Disagree”. The result shows that all

ISBN: 978-1-61804-297-2

Expert

Table 5 List of added metrics Added Metrics Satisfaction with session timeout if idle Satisfaction with alert message if error occurred Satisfaction with authentication technique Performance speed Time taken to navigate Number of steps during task selection Satisfaction with the task performed Satisfaction with menu items provided

185

Recent Advances in Computer Science

Table 6 Means scores for individual experts Dimension

Expert A

Consistency Understandable Easy to use Tailorable Verifiable Overall impression

0.50 100 100 100 100 100

B

.75 100 100 100 100 100

C

.75 100 100 100 100 100

Four dimensions score relatively high with.75 and 100 by individual experts where as two dimensions; consistency and understandable that scores.50 each by expert A and expert C respectively. As illustrated in Table 6, all the individual dimension scores relatively high (overall scores). The dimensions, “Easy to use”, “Tailorable”, “Verifiable” and “Overall impression” got the highest score (100). Consistency scores.79 and understandable scores.92 respectively. Therefore, these results revealed that the developed model for evaluating the usability of m-banking application is original, complete and acceptable.

DIMENSION

6

E

100 100 100 100 100 100

CONCLUSION

The main objective of this model is to guide usability practitioners and m-banking application developers’ to relate processes in evaluating the usability of m-banking applications. Moreover, this paper had shown how group of experts was used to verify the developed usability evaluation model for m-banking application. Some metrics were removed, while few have been added in the proposed model based on the feedback received from the experts. Furthermore, all the individual dimensions used by the experts to judge the model reached a satisfactory level that is in terms of consistency, understandable, easy to use, tailorable, verifiable and overall impression. This indicates that the proposed usability evaluation model is original, complete and acceptable. Experts’ comments and suggestions have given significant contributions for the developed model. The experts’ responses in the verifications forms and questionnaire pointed out some potential strength and weakness of the proposed model. However, their comments and suggestions are supportive to the study that building the model needs credible and experience expertise from both academia and industry. Therefore, this paper shows a good verification practice and it will serve as a guide to the research community, especially literatures on model verification process of mobile application evaluation are very limited. The verification has justified whether the model reflects the needs of the m-banking application based on usability perspective. In this regards, the verification process, details of the expert's report and questionnaire result are presented in this paper. However, this verification represents the final stage of the first cycle of model development. The future target of this study is to

MEANS

.79 .92 100 100 100 100

Additionally, it indicates that four dimensions acquired the highest score, followed by dimension “Understandable. The “Consistency” dimension got the lowest score (79%). This could be due to the irrelevancy or inconsistency of some metrics identified by the experts. However, those metrics were removed and a few were added based on the comments and suggestions received from the experts. All the comments and suggestion from the experts were carefully examined and given much consideration for improvement of the proposed model. Therefore, an amendment has been made concerning the metrics and their corresponding criteria from the first version of the model based on the experts feedback. For instance, previously the model contains a total of forty nine (49) metrics in which thirty (30) are subjective data whereas nineteen (19) for objective data. Therefore, the revised version of the developed model comprises forty two (42) metrics. Twenty six (29) metrics

ISBN: 978-1-61804-297-2

E

.75 100 100 100 100 100

focus on subjective data and thirteen metrics centred to objective data respectively. See Fig 2 in Appendix 2 for the revised version of the proposed model. The asterisk (*) in the model represents amended metrics

Table 7 Overall scores for individual dimension Consistency Understandable Easy to use Tailorable Verifiable Overall impression

D

100 0.50 100 100 100 100

186

Recent Advances in Computer Science

test the developed model through usability experiment involving real m-banking application users. This will examine the capability of each metric both objective and subjective for collecting data.

[12]

[13] [1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

References: Antunes, P., Herskovic, V., Ochoa, S. F., & Pino, J. A., Structuring dimensions for collaborative systems evaluation. ACM Computing Surveys (CSUR), 44 (2), 8. 2012. Baharuddin, R., Singh, D., & Razali, R., Usability Dimensions for Mobile Applications-A Review. Research Journal of Applied Sciences, 5 (6), p2225-2231, 2013. Basili, V. R., Software modeling and measurement: The Goal/Question/Metric paradigm. University of Maryland, College Pack, 1992. Beecham, S., Hall, T., Britton, C., Cottee, M., & Rainer, A., Using an expert panel to validate a requirements process improvement model. Journal of Systems and Software, 76(3), 251275, 2005. Bertoa, M. F., Troya, J. M., & Vallecillo, A., Measuring the usability of software components. Journal of Systems and Software, 79(3), 427-439, 2006. Cooharojananone, N., Kongnim, P., Mongkolnut, A., & Hitoshi, O., Evaluation Study of Usability Factors on Mobile Payment Application on Two Different Service Providers in Thailand. Paper presented at the 12th International Symposium on Applications and the Internet (SAINT), 2012. Coursaris, C. K., & Kim, D. J., A metaanalytical review of empirical mobile usability studies. Journal of Usability Studies, 6(3), 117-171, 2011. Dubey, S. K., Gulati, A., & Rana, A., Integrated Model for Software Usability. International Journal on Computer Science and Engineering, 4, 429-437, 2012. Emmanuel, E. A., & Nsung-Nza, M. H., Mobile Commerce Interaction Techniques for African Rural Economy Development: A Case Study for Dwesa, 2012 Frøkjær, E., Hertzum, M., & Hornbæk, K., Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? Paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems, 2000. Harrison, R., Flood, D., & Duce, Usability of mobile applications: Literature review and

ISBN: 978-1-61804-297-2

[14]

[15]

[16] [17]

[18]

[19]

[20]

[21]

[22]

187

rationale for a new usability model. Journal of Interaction Science, 1(1), 1-16, 2013 Hussain, A., Metric based evaluation of mobile devices: mobile goal question metric (mGQM). University of Salford, 2012. Hussain, A., & Kutar, M., Usability Metric Framework for Mobile Phone Application. PGNet, ISBN, 978-971, 2009. Hussain, A., Kutar, M., Mutalib, A. A., & Kamal, F. M.: Modeling Subjective Metrics for Mobile Evaluation. Journal of Information System Research and Innovation, ISSN 22891358, 2011. Madan, A., & Dubey, S. K., Usability evaluation Methods: A Literature Review. International Journal of Engineering Science and Technology, 4(02), 590-599., 2012. Nielsen, J., Heuristic evaluation. Usability inspection methods, 24, 413, 1994. Nielsen, J., & Hackos, J. T., Usability engineering (Vol. 125184069): Academic press Boston, 1993. Seffah, A., Donyaee, M., Kline, R. B., & Padda, H. K., Usability measurement and metrics: A consolidated model. Software Quality Journal, 14(2), 159-178, 2006. Simes-Marques, M., & Nunes, I., Usability of interfaces: INTECH open science, Ergonomics-A System Apprach, 2012. Tan, J., Ronkko, K., & Gencel, C., A Framework for Software Usability and User Experience Measurement in Mobile Industry. Paper presented at the Software Measurement and the Eighth International Conference on Software Process and Product Measurement (IWSM-MENSURA), Joint Conference, 2013. Tory, M., & Moller, T. (2005). Evaluating visualizations: do expert reviews work? Computer Graphics and Applications, IEEE, 25(5), 8-11, 2005. Wiegers, K. E., Seven truths about peer reviews. Cutter IT Journal, 15(7), 31-37, 2002.

Efficiency

Appendix 1

ISBN: 978-1-61804-297-2

Loading Time

Navigation

Presentation

Operability

Compatibility

188

O=Time taken to start O=Time taken to select a task O=Time taken to learn S=Satisfaction with help

S=Satisfaction with menu names S=Satisfaction with output display S=Satisfaction with help

O=Number of attempts during task selection O=Number of interaction while performing a task S=Satisfaction with help

Fig 1 First version of the proposed usability evaluation model for m-banking application

O=Time taken to complete a task O=Number of error (s) during a -task O=Number of interactions

Privacy O=Time taken to connect to the network O=Number of attempts to signin/sign out S=Satisfaction with help

Simplicity

S=Satisfaction with menu buttons presentation S=Satisfaction with informationorganisation S=Satisfaction with graphics presentation S=Satisfaction with output format

Familiarity

S=Satisfaction with menu buttons S=Satisfaction with contents S=Satisfaction with pages’ style & format S=Satisfaction with help menu

Consistency

S=Satisfaction with surname/password style and format O=Number of clicks to sign-in O=Number of attempts to be denial access

Metrics

Metrics

User satisfaction

S=Satisfaction with help menu provided S=Easy to make error(s) correction

S=Satisfaction with logicalpresentation of task S=Satisfaction with functions provided S=Satisfaction with update request S=Satisfaction with help

S=Satisfaction with contents design S=Satisfaction with logical design of menu- buttons S=Satisfaction with output display & format S=Satisfaction with application style & format

Content

S=Satisfaction with menu names S=Satisfaction with link list S=Satisfaction with navigation- structure S=Easy to find contents

Reliability Metrics

Learnability

Structured task

S=Mobile Device Support S=Satisfaction with device Keypad

Accuracy Metrics

Trustfulness

Metrics

Criteria

Usability evaluation factors

User guide

O=Time taken to load/initialize O=Time taken to key-in user data O=Time taken to response O=Number of error(s)

Metrics

Effectiveness

Usability evaluation factors for m-banking application

Legend

Recent Advances in Computer Science

Efficiency

Appendix 2

ISBN: 978-1-61804-297-2

Loading Time

Navigation

Presentation

Operability

Compatibility

189

O=Time taken to start transaction O=Time taken to select a task

O=Time taken to finish a task O=Number of error (s) during a –task O=Number of interactions while performing a task O= Time taken to display output S= Satisfaction with task performed *

Privacy S=Satisfaction with output display S=Satisfaction with interface layout

O=Number of steps during a task * S= Easy to select task in the interface S=Easy to use and learn the application

S=Satisfaction with help menu provided S= Satisfaction with alert message if error occurred * S=Easy to make error(s) correction

S=Satisfaction with logicalpresentation of task S=Satisfaction with transaction task provided S=Satisfaction with update request

S=Satisfaction with contents design & format S=Satisfaction with logical design of menu- buttons S=Satisfaction with application style & format

Metrics

User satisfaction

Fig 2 Revised version of the proposed usability evaluation model for m-banking application

S= represent subjective metrics O= represent objective metrics

O=Time taken to connect to the network S=Satisfaction with session timeout if idle *

Simplicity

S=Satisfaction with menu buttons presentation S=Satisfaction with informationorganisation S=Satisfaction with graphics presentation S=Satisfaction with output format

Familiarity

S=Satisfaction with menu buttons S=Satisfaction with contents of the application S=Satisfaction with pages’ style & format

Consistency

S=Satisfaction with user ID/password style and format S= Satisfaction with authentication techniques * S= Satisfaction with access denied support

Metrics

Content

O=Time taken to navigate * S=Satisfaction with navigationstructure S= Satisfaction with menu items provided *

Reliability Metrics

Learnability

Structured task

S=loading the application with this device S=performance speed *

Accuracy Metrics

Trustfulness

Metrics

Criteria

Usability evaluation factors

User guide

O=Time taken to load/initialize O=Time taken to response O=Number of interactions during key-in user ID and password O=Time taken to display transactions page

Metrics

Effectiveness

Usability evaluation factors for m-banking application

Legend

Recent Advances in Computer Science