ELECTRONIC DATA INTERCHANGE (EDI): KEY AUDIT ISSUES by TANOM SURAVONGTRAKUL B.Acc., Chulalongkorn University, 1983 M . B . A . , Michigan State University, 1986

A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE (BUSINESS ADMINISTRATION) in THE F A C U L T Y OF GRADUATE STUDIES The Faculty of Commerce and Business Administration Department of Management Information Systems

We accept this thesis as conforming to the required standard

THE UNIVERSITY OF BRITISH COLUMBIA August 1992 ® Tanom Suravongtrakul, 1992

In

presenting

degree

this

at the

thesis

in

University of

partial

fulfilment

of

the

requirements

British Columbia, I agree that the

for

an advanced

Library shall make

it

freely available for reference and study. I further agree that permission for extensive copying

of

department

this or

thesis for by

his

or

scholarly purposes may be her

representatives.

It

publication of this thesis for financial gain shall not permission.

Department

of

ÔGYY\f?\eXŒ

The University of British Columbia Vancouver, Canada

DE-6

(2/88)

is

granted

by the

understood

that

head of copying

my or

be allowed without my written

ABSTRACT The development of EDI technology has created many concerns and challenges for the auditing profession. Along with its many suggested benefits, the technology brings an important potential to change business information systems and the way businesses operate. As a consequence, it may put auditors in a new audit environment and may thus force significant modification to the established methods of auditing. This study identifies important EDI audit issues as viewed by information systems auditors in the greater Vancouver area. A three round Delphi methodology was used to solicit opinions from a group of IS audit experts. The expert respondents predominantly held managerial positions in internal audit functions while their organizations comprised a wide cross section of sizes and industries. The research findings reveal a consensus set of eleven most important issues. Among these EDI audit issues, "Controls Over EDI Network", "Backup, Disaster Recovery and Contingency Plans", "Auditability and Audit Trail", "Audit Involvement during the System Development", and "Legal and Audit Evidence" are rated in the top five ranks.

ii

TABLE OF CONTENTS Abstract

ii

Table of Contents

iii

List of Tables

vi

Acknowledgements

vii

Chapter 1. INTRODUCTION 1.1 Background and Motivation 1.2 EDI Concepts 1.2.1 The Definition of EDI 1.2.2 The Benefits of EDI 1.2.3 The Growth of EDI 1.3 The Implications of EDI on Auditing 1.4 Statement of Problem and Need for the Research 1.5 Research Objectives 1.6 Application of Proposed Research 1.7 Overview of the thesis

1 1 2 2 3 4 5 6 8 8 9

Chapter 2. LITERATURE REVIEW 2.1 Previous Empirical Research 2.1.1 West [1988] 2.1.2 Holstrum et al. [1988] 2.2 Selected Professional Publications 2.2.1 Staats [1981] 2.2.2 Hinge [1988] 2.2.3 Hansen and Hill [1989] 2.2.4 Sadhwani et al. [1989]

10 10 10 12 14 14 15 17 20

Chapter 3. RESEARCH FRAMEWORK 3.1 Introduction 3.2 EDI'S Key Audit Issues 3.2.1 Audit Evidence 3.2.2 Audit Trail 3.2.3 Audit Involvement during the System Development 3.2.4 Timing of Audit Tests 3.2.5 Audit Reporting (Periodic versus On-Line) 3.2.6 Audit Focus (Substantive versus Compliance Testing) 3.2.7 Pre-determination of Audit Scope (Boundary of Audit) 3.2.8 Audit Tools

22 22 22 23 24 25 26 26 27 29 30

iii

3.2.9 Audit Techniques 3.2.10 Audit Risk Assessment 3.2.11 The Changing Role of Auditors 3.2.12 Audit Responsibility in Evaluating Controls 3.2.13 Relationship Among Company's Auditors 3.2.14 Collaboration Among Auditors of EDI Parties 3.2.15 Auditor Skills (Skills required of auditors) 3.2.16 Auditor education and training RESEARCH DESIGN Research Questions Selection of Research Methodology The Delphi Process - An Overview Instrument Development 4.4.1 Round 1 Questionnaire 4.4.2 Round 2 Questionnaire 4.4.3 Round 3 Questionnaire 4.5 Participant Recruitment 4.6 Data Collection Procedures 4.6.1 Round 1 4.6.2 Round 2 4.6.3 Round 3

31 32 33 34 36 38 39 40

Chapter 4. 4.1 4.2 4.3 4.4

42 42 42 44 46 46 47 49 50 52 53 53 54

Chapter 5. ANALYSIS AND DISCUSSION OF RESULTS 5.1 Introduction 5.2 Round 1 Results 5.2.1 Controls Over EDI Networks 5.2.2 EDI Contracts (Trading Partner Agreements) 5.2.3 Backup, Recovery and Contingency Plans 5.2.4 Third Party EDI Services 5.2.5 EDI Records Retention 5.3 Round 2 Results 5.3.1 The Rating of Round 2 Issues 5.3.2 Additional Issues Identified in Round 2 5.4 The Comparison of Round 1 and Round 2 Results 5.5 Round 3 Results 5.5.1 Rating of the Original 21 Issues 5.5.2 Rating of the Final 25 Issues 5.6 The Interpretation of the Results 5.7 The Comparison of Round 2 and Round 3 Results 5.8 Movement Towards Consensus 5.9 Study Participants 5.9.1 Organizational Category 5.9.2 Position and Primary Area of Responsibility

56 56 56 57 58 58 59 59 60 60 62 62 63 66 66 67 68 70 73 73 74

iv

5.9.3 5.9.4 5.9.5 5.9.6

Chapter 6. 6.1 6.2 6.3 6.4

Professional Designations Areas of Audit Expertise Level of Audit Experience Background in EDI Technology 5.9.6.1 Engagement in an EDI Project 5.9.6.2 Self-report Level of knowledge and Understanding of the EDI Technology 5.9.6.3 Primary Source(s) of knowledge and Understanding of the E D I Technology 5.8.6.4 EDI audit manual or guideline

CONCLUSIONS Summary of Findings and Conclusions Generalizeability of Results Limitations of Research Study Directions for Future Research

74 75 75 75 75 76 76 77 78 78 79 80 81

BIBLIOGRAPHY

82

APPENDIX A - ROUND 1 QUESTIONNAIRE

87

APPENDIX B - ROUND 2 QUESTIONNAIRE

91

APPENDIX C - ROUND 3 QUESTIONNAIRE

98

APPENDIX D - ROUND 1 RESULTS

.

108

APPENDIX E - ROUND 2 RESULTS

115

APPENDIX F - ROUND 3 RESULTS (21 Issues)

119

APPENDIX G - ROUND 3 RESULTS (25 Issues)

123

v

LIST OF TABLES

TABLE 1 - ISSUES IDENTIFIED IN ROUND 1

57

TABLE 2 - RATING OF ROUND 2 ISSUES

61

TABLE 3 - ISSUES IDENTIFIED IN ROUND 2

62

TABLE 4 - ROUND 3 RATING OF 21 ORIGINAL ISSUES

64

TABLE 5 - ROUND 3 RATING OF 25 FINAL ISSUES

65

TABLE 6 - THE TOP E L E V E N ISSUES IN ROUND 2 AND ROUND 3 . . . .

68

TABLE 7 - RESEARCH SUBJECTS: PARTICIPATION PATTERN

73

TABLE 8 - RESEARCH SUBJECTS: ORGANIZATIONAL CATEGORY . . . .

73

TABLE 9 - RESEARCH SUBJECTS: TYPE OF POSITION

74

TABLE 10 - RESEARCH SUBJECTS: PROFESSIONAL DESIGNATION TABLE 11 - RESEARCH SUBJECTS: AREAS OF AUDIT EXPERTISE

...

74

....

75

TABLE 12 - RESEARCH SUBJECTS: L E V E L OF EXPERIENCE

75

TABLE 13 - RESEARCH SUBJECTS: ENGAGEMENT IN A N EDI PROJECT

75

TABLE 14 - RESEARCH SUBJECTS: L E V E L OF KNOWLEDGE OF EDI TECHNOLOGY

76

TABLE 15 - RESEARCH SUBJECTS: PRIMARY SOURCE(S) OF KNOWLEDGE OF EDI TECHNOLOGY 76 TABLE 16 - THE TOP E L E V E N ISSUES IN VANCOUVER

vi

78

ACKNOWLEDGEMENTS

This thesis is dedicated to my parents, Lieang Tang and Kimchang Lee, who always have confidence in me and support me in my endeavour for personal growth and professional development. Sincere thanks go to my thesis advisor, Professor Albert S. Dexter, and the members of my thesis committee, Professor Dan A . Simunic and Andrew W. Trice, for their advice and encouragement. I would also like to thank Ms. Khim Seow at Commerce General Office for her assistance with the final copy of the thesis. Special appreciation is extended to Mr. Alan R. Drinkwater, Membership Director of the EDPAA-Vancouver, Mr. James W. Topham, President of the EDPAA-Vancouver, and Ms. Angela M . Louie, President of the IIA-Vancouver, for their kind assistance which contributes greatly to the achievement of this research study. Finally, I gratefully acknowledge the Thai-Canada Rattanakosin Scholarship which have provided financial support throughout my study in Canada.

vii

Chapter 1. Introduction 1.1 Background and Motivation The recent development in information technology in the form of Electronic Data Interchange (EDI) has created many concerns and challenges for the auditing profession. Along with its many suggested benefits, this technology brings the important potential to change business information systems and the way businesses operate and, consequently, to create a new audit environment and to force significant changes in the established methods of auditing. While the global business community has increasingly paid attention to EDI, and corporations in North America and Europe have been adopting this technology at a noticeable rate, the auditing profession has not been as prompt in its approach to EDI audit concerns.

At present there are no auditing standards or specific guidelines regarding

EDI/EFT [Sadhwani et al. 1989; Cowan, 1990]. Studies indicate that EDI systems will prevail over paper-based systems [Hinge 1988; Schatz 1988; West 1988; Holstrum 1988; Tsay 1989], and that indication, plus the potential direct impact of EDI on auditing, suggest that EDI is a technology that deserves immediate attention from auditors. To date, academic research has provided little insight into this domain. Although EDI is expected to have a profound impact on many aspects of auditing, the nature and extent of such an impact are not specifically known. It is the intent of this project to conduct an exploratory research into EDI to identify and assess key impact issues of audit concerns. The knowledge of audit concerns contributes to the overall success of EDI adoption and technological improvement in the business world.

1

1.2 EDI Concepts 1.2.1 The Definition of EDI Although there are variations in the definition of EDI, Hinge's [1988] definition is adopted for the purposes of this project: Electronic data interchange (EDI) is the intercompany, computer-tocomputer exchange of business documents in standard formats. Through EDI, such common business forms as invoices, bills of lading, and purchase orders are transformed to a standard data format and electronically transferred between trading partners. [Hinge, p. 9] This definition is chosen because it captures the essence of an EDI system and it has the relevant meanings in the context of auditing and of this project. The above definition includes the EDI essential terms "intercompany", "computer-to-computer", and "standard formats" which also meet the criteria for an adequate definition of an EDI system suggested by Powell [1991] . The first two terms, "Intercompany" and "computer1

to-computer", imply that there must be at least two different computer systems involved in the electronic transfers of business data. Furthermore, The term "intercompany" is appropriate for the purpose of this project because although EDI systems can be implemented by non-business organizations for purposes other than trading, and EDI transmissions can occur between different computers of the same company (e.g., between the administrative office and manufacturing plants), only the EDI systems for trading activities among different business enterprises are of interest in this audit issues project . 2

Based upon his extensive reviews of the literature, Powell [1991] suggests that although EDI can be defined in many different ways, an adequate definition of EDI must indicate a transmission of data/information between at least two different computers using a standard format, [p.4] 1

Most of the reviews in the auditing literature express concerns on the audit of open EDI network systems among business enterprises rather than the closed or non-business oriented EDI network systems. 2

2

In addition, the term "computer-to-computer" signifies the automation of business functions and the reduction of paperwork, which have important meanings for auditing activities, because it means that information can flow directly from the sender's application to the receiver's application without paper and without human intervention. Moreover, the term "standard formats" helps differentiate EDI from electronic mail and facsimile transmission. Because EDI standard formats are in coded, machine-readable forms, EDI messages can 3

be created and interpreted by computers. Electronic mail and facsimile transmission, on the other hand, do not have such standard formats and their messages are in free text form which must be created and interpreted by humans. 1.2.2 The Benefits of EDI EDI offers many attractive benefits. Hinge [1988], Gardner [1989], and Wright [1990b] discuss the direct benefits of EDI in terms of speed, accuracy, and savings. Hansen and Hill [1989], basing their findings on a survey by EDI Research, Inc., cite speed, accessibility of information and improved customer services as the most frequently mentioned benefits of EDI. In his doctoral dissertation, Kavan [1991], using Porter's definition of competitive advantage, states that EDI contributes to both cost effectiveness and product differentiation strategies. He also mentions that enterprises are adopting EDI to increase productivity, reduce financial exposure, and gain a competitive advantage in the market place. Others view EDI as an increasingly essential technology for business survival. For instance, Schatz [1988], Emmelhainz [1990], and Powell [1991] point out

Examples of EDI standards currently available are—ANSI ASC X12 for cross industry, AIAG for automotive, TDCC for transportation, USC for retail, and EDIFACT for international. For detailed discussions of EDI standards see Emmelhainz [1990, pp.63-87], Kimberley [1991, pp.97-124]. 3

3

that the reason businesses adopt EDI technology is not solely for competitive advantage but for survival. They note that companies are being forced by both their suppliers and customers to implement the EDI systems, and Schatz cites as an example General Motors' 1984 letter that gave its suppliers until 1987 to get on-line with EDI or go off-line with G M . Furthermore, Tsay [1989] reinforces the point by predicting that those who resist EDI technology could eventually run the risk of losing their business to competitors. 1.2.3 The Growth of EDI Since the emergence of the EDI concept in the late 1960s, the adoption of EDI systems has continued to grow. During the 1970s significant progress was made towards the development of EDI standards, and by the mid-1980s, there was a noticeable expansion of EDI use. With relatively inexpensive supporting software and hardware, EDI links between customers and suppliers became more feasible for many industries. Current information suggests a bright future for EDI growth. For example, based upon her research findings, Emmelhainz [1986] indicates that the use of EDI is likely to become the norm in the purchasing community in the relatively near future and that third party network services will play an important role in the continued growth of EDI. Hinge [1988], an EDI expert, claims that EDI has become a prerequisite for doing business, and she predicts that by 1993 an estimated 70 percent of U.S. businesses will make significant use of EDI. She also notes that the international EDI market is growing, particularly in Canada and Great Britain. In his recent review, Damyanoff [1991] affirms the trend toward the international use of EDI and reports that U.S. Customs has made the combination of EDI link (among parties involved in importing-exporting transactions) and

4

EDIFACT the basis of its proposed Customs Modernization Act of 1990. Because the US 4

has a number of trading partners all over the world, it is reasonable to expect a pervasive use of EDI both domestically, not only in the US but also in its trading partners' countries, and internationally in the near future.

1.3 The Implications of EDI on Auditing In spite of their benefits, the unique characteristics of EDI introduce additional complexities into business transaction processing and the audit environment. Academic and professional studies consider the ramifications of EDI adoption from many perspectives. For example, Hansen and Hill [1989] conclude that EDI necessitates new control and audit considerations and that there are methods and procedures to respond to the changes. They discuss the impact of EDI on internal controls in terms of the absence of source documents, bridging applications, and direct interaction with trading partners. As a computer audit specialist, Wright [1990b] agrees that the greatest direct effect of EDI will be on corporate accountants and internal and external auditors. She suggests three main areas to be considered: controls, contracts, and paper elimination. Cowan [1990] considers information flows and boundaries to data ownership as important audit and legal issues. He concludes: Although the use of EFT/EDI does not alter the essence of audit objectives, it creates new issues and it has changed the information flow that the auditor needs to understand... The problem with EFT/EDI is the sheer pace of transactions and their integration with the accounting functions of an EDI for Administration, Commerce and Transport is the acronym for standards developed within WP4 (Working Party 4 on the facilitation of International Trade Procedures of the Economic Commission for Europe, a commission of the United Nations) [Hinge 1988,p. 76,86]. 4

5

organisation. [Cowan, p. 30] As a lawyer who specializes in electronic trading, Wright [1991, p.38] confirms that companies' adoption of EDI systems will have a profound effect on auditors and states the following points as the major audit concerns: - auditors' responsibility for a system that lacks adequate controls; - auditors' obligation to electronic legal issues; - auditors' duty to provide advice regarding the establishment of necessary controls. The extensive movement toward "paperless" electronic data processing will eliminate much of the traditional audit trail and will radically change the nature of audit control, audit evidence, audit techniques, and the timing of audit tests. As a result, many aspects of the conventional audit process will have to change to suit the new environment.

1.4 Statement of Problem and Need for the Research As discussed in the preceding sections, EDI technology will alter business conduct and, as a consequence, will force the practice of auditing to change. The major concern is to indicate to auditors the importance of this technology. The impact of EDI technology has been sudden but its effect is extensive and widespread. As implied in Wise's article [1989], the challenges of the electronic system apply not only to EDP or information 5

systems auditors but also to all types of auditors. He elaborates that point by noting that when there was always a paper trail of documents, non-EDP auditors could avoid "auditing through" the computer by "auditing around" the computer.

5

Electronic Data Processing

6

However, as businesses

progress toward a "paperless office" there will be fewer documents to enable "auditing around" the computer. Thus, inevitably, all types of auditors are under pressure to adapt themselves to new circumstances. A similar concern applies at the academic and research levels. As educators of future auditors, both instructors and researchers will need to conceive and respond suitably to changes brought about by this new trend in technology.

There are many indicators of a need to respond promptly to the EDI challenges. For instance, as stated by Cowan [1990] and Wright [1991]: There are no auditing standards or specific guidelines (regarding EDI/EFT) at present. What the auditors have achieved to date is the adaptation of existing professional standards to track new developments....[TJhe professional and regulatory bodies will need to adapt more quickly to changes in technology, though this should not be at the expense of rigorous assessment of the precise impact of those changes. [Cowan, p. 31] Electronic transaction technologies present the accounting profession with a daunting task. The audit and control of electronic systems require new methods drawn from the principles of past practices. Accountants must fast educate themselves in these new ways. They will otherwise be swept under the avalanche of electronic data that industry is generating. [Wright, p. 39] Furthermore, Kavan [1991] shows the urgency of the issue in that while auditors have not yet established firm guidelines, the situation has been made more serious because business enterprises have implemented the technology without awareness of the legal and accounting problems: Many organizations, eager to implement EDI, overlook critical controls and safeguards. Because this technology is so new, user documentation, standards, conventions, guidelines, and the law are either not developed or are inadequately implemented. [Kavan, p. 14] The auditing profession needs to have available extensive new research that will 7

provide an insight into the EDI audit issues. At present, very little academic research has been accomplished in the area of EDI [Kavan 1991]. My own review indicates that there has been even less research in the specific area of EDI and auditing. Although a formalized study, which is closely related to this project, was done by West in 1988, and a more general study, a part of which is applicable to this thesis, was done by Holstrum, Mock, and West in 1988, prior to the initiation of this project, there has been no published study of this type.

1.5 Research Objectives In response to the stated problems and needs for research, this study is designed to achieve the following objectives: 1.5.1 To identify and gain consensus on key issues of concern to auditors when auditing in the EDI environment. 1.5.2 To rank the priorities of these issues.

1.6 Application of Proposed Research Knowledge of important EDI audit issues will be useful in helping accountants, managers, information system consultants, and vendors better to understand and build EDI systems that satisfy audit needs and concerns. Systems that are satisfactory from the audit perspective have well built-in controls. Such secure systems would contribute to the overall success of EDI adoption in the business world. In addition, by knowing the significance and the priorities of the EDI audit issues, educators and researchers can direct their efforts

8

toward the most critical areas and, as a consequence, can better satisfy professional needs.

1.7 Overview of the thesis This thesis proceeds as follows.

Chapter two reviews the previous empirical

studies and the selected professional publications which are related to the control and audit of the "paperless" EDI systems. Chapter three outlines the critical issue frameworks used in the research.

Chapter four formulates the specific research questions and

methodological details of the study. Chapter five analyzes and discusses the research's findings and the final chapter presents the conclusion.

9

Chapter 2. Literature Review

2.1 Previous Empirical Research 2.1.1 West [1988] In completing his doctoral dissertation, West conducted a study comprised of two distinct phases: 1) A general investigation, using the Delphi technique, of potential technological changes that may have an impact on the audit environment in the year 2000. 2) A detailed investigation, through a case study, of the impact of one specific technological change upon auditing. The first phase of the study included a panel of 31 highly experienced accounting professionals. He found that the trend toward a "paperless" accounting system was the area of primary concern to the majority (20) of the respondents.

As a result, the

"paperless" or EDI system was chosen to be studied in more detail in the second phase. In the second phase of the study, eight internal auditors (seven were computer auditors), and six external audit managers (all general auditors) evaluated a case study detailing a "paperless" purchasing, accounts payable, and inventory control system. They were asked to evaluate the exposures, key controls, and reliability of the system. The results of this in-depth analysis showed that: 1) An adequate audit trail could be obtained from the paperless system. 2) There is a high level of consensus on which exposures and controls were considered most critical. The three most important controls were:

10

- Controls (separation of responsibilities) over computer program changes - Dual access controls - Computerized matching of invoice, purchase order, and receiving report prior to recording the liability. The auditors also suggested the following as important additional controls: - managerial reviews - programmed controls (e.g., range and limit checks) - frequent (e.g., monthly) testing of perpetual records. 3) There was significant concern about fraud and unauthorized transactions (especially fraudulent payment of accounts payable and unauthorized access to data, programs, and inventory). 4) There was a lack of consensus concerning the adequacy of internal controls. 5) There was an increased emphasis on audit tests of the system, less emphasis on detailed tests of balance, and little use of analytical procedures. West's research is important because it marked the first attempt to identify EDI audit concerns. Although the research is not built upon any existing theories, the resulting predictions of technological trends/events seem to have a high degree of accuracy. Present EDI literature contains evidence that suggests the trend toward a "paperless" system has become a reality. Electronic trading systems are now being used in many major industries in North America, Europe, Asia, and Australia [Emmelhainz 1990; Wright 1990; Baker 1991(b); Damyanoff 1991]. Further, although the case study is simplified in comparison to a real-world company, and the tasks assigned to the subjects are restricted to the

11

evaluation of internal accounting and computer controls, the resulting analysis provides a useful insight into the implications of electronic trading systems on auditing. In addition, the selection of the "paperless" revenue cycle for in-depth analysis is appropriate. Today, purchasing is one of the prime areas for EDI applications in most pioneering organizations. However, the prediction about the shift of audit focus toward system testing remains controversial.

While some current professional literature confirms this shift, others

indicate a move towards more substantive testing. This issue will be addressed in more detail in this study . 6

2.1.2 Holstrum et al. [1988] This study is built upon the findings of West's Delphi survey.

It provides a

detailed analysis of the impact of technological change on audit evidence and control structures and also examines the impact of social, legal, and economic changes on auditing. Among other issues, the study predicts that: By the year 2000, most computers will be able to communicate with one another...the volume of paper documents will be reduced, but the volume of available information will increase significantly, [p.xxi] This projection is supported by the more recent professional reviews which indicate the increased use of automated EDI networks in current businesses [Emmelhainz 1990; Wright 1990; Baker 1991(b); Damyanoff 1991]. In addition, audit software embedded in the audited entity's operating system and interconnections with mainframes and large databases are predicted as likely to become vital audit tools, and audit of the systems development process is viewed as increasingly

6

Please see Chapter 3 under the issue "Audit Focus".

12

important. Further, high-level systems review and evaluation software, database access and modification controls, computer monitoring, and examination of controls over paperless intercompany information network such as Electronic Data Interchanges (EDI) are mentioned as the focus of future control testing, whereas the imbedded audit software for continuous on-line monitoring (auditing) of the system is included as a major substantive testing. The study also predicts that blurred boundaries of the audited entity, continuous online auditing, and expanded responsibility for evaluating the integrity of internal and external databases will influence the changing role of auditing. Moreover, the authors cite "the key overriding skill of being able to readily adapt to rapidly changing information technology, including computer adeptness and interfacing effectively with expert systems" [Holstrum et al., p. 179] as the skills required of future auditors. Supplementarily, they suggest that auditors receive extended education and training and that the auditing curriculum be modified to emphasize computer familiarity, computer modelling, and the behavioral impact of information technology. Finally, the authors note that, although the expert panellists in this study believed that paperless transactions (computer-to-computer input) are technically feasible, and that by the year 2000, more than half of the most common types of business transactions (e.g.payment, invoicing, ordering, payroll time-cards) in large companies would be completed without paper, some experts felt that

the "paperless trading" may have

difficulty in gaining public acceptance, and that auditors would have problems with the significant disappearance of the paper audit trail.

13

In

general,

rapid

advances

in

computer

technology

integrated

with

telecommunication technology show these predictions are being actualized. Today, the technical ability to conduct business without paper is already available.

Further,

substantial improvements in hardware and software technology can facilitate the development of sophisticated systems and tools. In addition, the current environmental movement to reduce the amount of paper used together with other motives such as speed, accuracy, and saving may help improve social acceptance of the "paperless" concept and lead to the common use of EDI systems.

2.2 Selected Professional Publications 2.2.1 Staats [1981] Staats, retired Comptroller of the United States, cites the following duties as the critical challenges confronting the auditing profession in the year 2000: - Auditing paperless transactions, - Auditing to prevent and detect fraud, - Reporting on the adequacy of internal controls. In addition, he predicts that, "Paper transactions will be virtually eliminated, and auditors will have to review transactions as they occur. Moreover they will concentrate more on tests of systems than on testing individual transactions.'' [Staats 1981, p.ll]. His predictions of "paperless" transactions and the review of transactions as they occur are well supported in the studies by West [1988], Holstrum et al. [1988], and Hansen and Hill [1989]. The prediction regarding the shift towards system testing, which 14

will be examined in this study, remains debatable. Although the shift is confirmed by the results from West's case study of the "paperless" purchasing system, it is challenged by such reviews as Jancura et al. [1986], ICAEW [1989], and Brown [1991].

7

2.2.2 [Hinge 1988] This report states that a company's use of EDI will have a profound effect on auditing activities. Besides giving the formal definition of EDI which will be adopted in this thesis project, Hinge suggests the following auditing issues to be considered when designing the information system (p.43-45): 1) Payment Validation Audit Concern

: reconciliation of invoice, purchase order, and receiving documents to assure the correct payment amount.

Effect of EDI

: all these documents are computerized, and the validation process is changed.

Strategy

: automate the validation process to get time saving benefits.

2) Audit TraU of Activity Audit Concern

: tracking data flow within the company; recording authorizations.

Effect of EDI

: information security procedures are altered; paper documents and paper backup files are missing; EDI data flow can now be documented internally, between company and EDI V A N , and between company and trading partner.

7

See Chapter 3, under the "Focus of Auditing" issue.

15

Strategy

: replace signatures with codes and IDs; electronic signatures are also an option. : date/time stamp all activities and all attempts to access the information system; : maintain a specific audit trail database; : require identification of the terminal/pc to track the point of access.

3) Order/payment control Audit Concern

: ensuring only authorized sources can place orders and initiate payments.

Effect of EDI

: no authorization "sign-offs"; less human intervention means less control.

Strategy

: create safeguards parallel to those of paper systems; : require password access to the system; : incorporate "reasonableness checks" into the system; : emphasize user training to reduce system errors.

4) Accounting/transaction correspondence Audit Concern

: insuring that internal company data reflects actual inventory and dollar figures.

Effect of EDI Strategy

: all files are computerized; no paper backup to verify records.

: spot check actual transactions versus system files; : verify assets with different (that is, non-EDI) data.

Hinge advises that EDI data within a company be used to generate accounting

16

reports to facilitate the audit process, and a "control reporting" service offered by EDI VANs be used in tracking data flow between companies. 2.2.3 Hansen and Hill [1989] Hansen and Hill believe that EDI does change the control and audit environment, but methods and procedures exist that are supportive to those changes.

The authors

address EDI's impact on internal controls in terms of: - the absence of source documents (authorization signature) - bridging applications (automatic initiation of transactions) - direct interaction with trading partners (direct initiation of transactions by outsiders implies that system access control is very critical). In addition, the authors cite that EDI has a dramatic impact on control evidence because it is in machine-readable format (electronic documents) and it is distributed at locations that transcend traditional corporate boundaries. Also, the authors note that controls must be exercised beyond the traditional system periphery, and this changes the auditor's evaluation of general controls. Where third-party VANs are used, auditors are urged to evaluate network application features either directly or through the VAN's auditor. The following issues are addressed as important EDI control concerns. The corresponding control strategies were also suggested: 1) Validation of payments Concern

: both (source) document and signature may be missing (cannot be matched for verification).

control strategy

: programmed routines that match control documents before 17

allowing the next transaction process to begin; codes and IDs to replace signatures 2) Audit trail Concern

: a trail of documents that allows tracking of the transaction activities is not necessary to process transactions in an EDI system.

control strategy : depends on the method of EDI data entry 2.1) source documents are batched, then entered via direct entry terminals - a batch number serves as a batch reference. 2.2) source documents are entered as received - a programmed routine assigns electronic documents to batches that are automatically numbered while computercreated source documents are batched and filed by entry station. 2.3) transactions are entered directly without preparation of source documents surrogate documents (computer-generated substitutes for source documents) indicate the person preparing or authorizing the transactions. 3) Order and Payment Control Concern

: signatory authority is removed; opportunities for unauthorized access may increase

control strategies : a file to hold transactions that require managerial approval; levels ofpassword control to restrict access to applications and data files; encryption may be used to prevent data or password pirating; computerized checks to emulate human judgement in detecting fraudulent activity.

18

4) System Boundaries and Flow of Transactions Concern

: VANs have shown some reluctance to allow auditors other than their own to access their facilities.

Strategy

: the client's auditor may benefit from a control evaluation performed by professional computer-audit specialists.

In closing, Hansen and Hill propose the concept of "Continuous Auditing" for audit considerations. They address the followings points as the key characteristics of continuous process auditing: 1) On-line monitoring of the major modules of EDI processing -a supervisory program (programmed control) -Integrated Test Facility (ITF) 2) Systems metrics for key processing functions - software monitors to collect performance measurement data 3) System alarms to call attention to system problems -embedded audit modules to monitor all transaction activity and to notify the auditor of any activities having special audit significance (typically the modules write such information on a file called "the audit log". 4) Functional acknowledgements to capture data flows and errors within moments of their occurrence -some firms consider EDI orders as authentic if there is a record of subsequent payments -If production lead times or payment terms render confirmation of

19

subsequent payment difficult, the firm may consider confirming the existence

of such EDI transmitted

orders

through

"independent

confirmations". 2.2.4 Sadhwani et al. [1989] Sadhwani, Kim, and Helmerci claim that although traditional controls no longer apply to EDI integrated systems (in which a computer-based network enables transactions to be initiated, recorded, approved, and executed electronically), it is viable to maintain adequate control and auditability. This paper provides an outline of some of the methods and procedures that managers and system designers must comprehend and implement in such a system. The authors emphasize that EDI internal controls must be designed to: -promote auditability of data -provide assurance that information is completely and correctly posted -ensure that transactions are authorized and posted on a timely basis. The article suggest that the evaluation of internal controls in a typical EDI network should involve the following three parties, and all parties must provide assurance that the proper controls exist within their individual systems: -the originator of the transactions and documents -the processor (e.g. a third-party network or a bank) -the receiver of the data and documents. The authors urged the auditor to get involved early: "... the auditor must play a significant role during the design and development of EDI systems and must assure management that secure, 20

auditable, and properly controlled systems are developed and that adequately designed programmed procedures are effectively implemented. " [Sadhwani et al. 1989, p. 24]. The authors comment that although SAS No. 48, "The Effect of Computer Processing on Examination of Financial Statement", provides a broad framework for the internal control issue, "Current auditing standards do not provide specific guidelines that pertain to EDI systems. " [p.27]. They recommend that auditors consider the following issues when evaluating internal controls of an EDI system [p.27]: -control boundaries -processor's overall general control environment -data transmission controls -data access controls -audit objectives (how they could be redefined) -restructuring of internal controls to reduce control gaps -new risk exposures when using third-party networks.

21

Chapter 3. Research Framework

3.1 Introduction The purpose of this chapter is to provide a general EDI issue framework upon which research questions explored in this study are based. Because there is no established theory at present, an extensive literature review will identify potential issues of important audit concern regarding businesses' adoption of EDI. The literature consulted consists of previous empirical works, EDP audit professional publications, seminars and conferences, information technology (particularly EDI) sources, and relevant works from auditing publications. The second phase of West's study [1988] provides a useful insight into and a partial framework for EDI audit issues.

However, because the analysis is confined to the

reliability of internal controls and the auditability of the paperless purchasing systems, the issues raised are neither exhaustive nor conclusive. Current professional reviews suggest many other important issues and unanswered questions which need to be considered. This study analyzes and verifies additional issues to those discussed by West.

3.2 EDI'S KEY AUDIT ISSUES The unique attributes of EDI that make it advantageous—e.g. reduction of paperwork and human handling, and direct connection with suppliers and customers-create a number of concerns among auditors. For the purpose of this study, these concerns will be classified and discussed in terms of their consequences on the following aspects of

22

auditing: • Audit Evidence • Audit Trail • Audit Involvement during the System Development • Timing of Audit Tests • Audit Reporting (Periodic versus On-Line) • Audit Focus (Substantive versus Compliance Testing) • Pre-determination of Audit Scope (Boundary of Audit) • Audit Tools • Audit Techniques • Audit Risk Assessment • The Changing Role of Auditing • Audit Responsibility in Evaluating Controls • Relationship Among Company's Auditors • Collaboration Among Auditors of EDI Parties • Auditor Skills (Skills required of auditors) • Auditor education and training

3.2.1 Audit Evidence Rationale: The absence ofpaper documents and signatures in EDI systems implies the absence of important audit evidence such as proof of authorization and legal documentation in paperform. Auditors must assure that equivalent and acceptable forms of audit evidence are established and properly incorporated into EDI systems.

23

In the highly automated EDI environment (application-to-application processing), it is not always necessary to create paper source documents or they may need to be available only for a short time. In the absence of paper documents, signatures and other information which usually appear on the documents as "evidence of authorization" are also missing. The lack of both paper documents and signatures causes important concerns regarding payment validation and order/payment control [Hinge 1988; West 1988; Hansen and Hill 1989; Baker 1991]. This concern may be serious if most of the traditional authorizing procedures and controls are removed when the electronic systems are implemented. From a control and audit viewpoint, equivalent forms of audit evidence must be developed to substitute paper documents and hand-written signatures.

Some

experts recommend using electronic signature and electronic authorization (EA) processes, but the question of the acceptability of such alternatives remains unsettled [Lewis 1989]. Whatever substitute forms are used, they should be identified, agreed upon by the auditor, and incorporated into the system from the early phases of the system development.

3.2.2 Audit Trail Rationale: Because substantial reduction of paperwork means a possible loss of the audit trail and the consequent inability to conduct an audit, auditors must take actions to ensure the auditability of the EDI system and the availability of audit trail in proper forms. In the EDI environment, business transactions are processed in an invisible, electronic form that is heavily coded and almost impossible to monitor, and a trail of paper documents that allows tracking of the transaction activities also no longer exists. Auditors have habitually relied on the paper audit trail to test the reliability of a system. Without 24

paper documents, auditors may have difficulty, or in some cases they may find it impossible to conduct an audit. The potential loss of audit trail and the inability to audit in a paperless environment are the main concerns expressed in many reviews [Hinge 1988; West 1988; Hansen and Hill 1989; Baker 1991]. Actions must be taken to ensure the auditability of the system and the availability of an audit trail in proper forms.

3.2.3 Audit Involvement during the System Development Rationale: Auditors must get involved early in EDI projects to ensure that proper controls and auditability features are designed and incorporated into the systems. For this reason, guidance is needed for auditors to accomplish competently this important task. In order to handle audits of a complex information system, auditors are strongly urged to take a proactive approach and become involved early in the project, especially during the system design and development stage [Bieber 1987; Rhodes 1987; Holstrum et al. 1988; Kothari 1988; Craig-Bourdin 1989; ICAW 1989; Sadhwani et al. 1989; Wise 1990]. Although the concern of "audit independence" keeps the roles and the extent of audit participation indeterminate, there is agreement that auditors perform a function that would be useful at the system design stage, and that audit involvement in the development, testing, and installation of computer-application systems can substantially add value to the process. That claim applies to such a complicated system as EDI.

From an audit

perspective, audit resources can be used productively at this stage because it is the best opportunity for the auditor to assure that proper controls and auditability features (e.g. the creation of audit trails) are designed and incorporated into the system. Further, the auditor 25

can acquire background knowledge and solid understanding of the system which will prove valuable in the subsequent audits. Therefore, sound guidance on audit participation in EDI projects is needed to assist the auditor to accomplish capably this important task.

3.2.4 Timing of Audit Tests Rationale: Because of the high volume of transactions and the velocity of electronic processing, EDI transactions may have to be reviewed as they occur. Consequently, the audit process must be modified and specific audit standards established. In contrast with a conventional paper-based system where audit testing is performed periodically, the high volume of transactions and the velocity of electronic data processing may force auditors to review EDI transactions as they occur. In such a case, the concepts of "Concurrent Accounting" and "Continuous Process Auditing", widely discussed in both academic and professional reviews , become relevant [Staats 1981; West 1988; Holstrum 8

et al. 1988; Kothari 1988; Hansen and Hill 1989; Baker(a) 1991]. Upon the availability of such a supporting tool as audit software embedded in the audited entity's operating system, the concept can be feasible . In such a circumstance, the audit process must be 9

modified significantly and specific standards must be established to guide the practice.

3.2.5 Audit Reporting (Periodic versus On-Line) Rationale: Because the high speed of EDI transactions makes information obsolete within a very short time, there is a need for more frequent

8

See chapter 2, Hansen and Hill [1989] for the discussion of key characteristics of continuous process auditing.

Holstrum et al. [1988] predicted that such an embedded audit software would be available for audit use by the year 2000. 9

26

accounting disclosure. Auditors will have to adjust their reporting procedures to meet the need for timely and accurate information. In concord with continuous process auditing, the reviews indicate a propensity to move away from formal periodic financial reporting towards more frequent accounting disclosure [Holstrum 1988; ICAW 1989; Yang 1990]. 10

The claim is that the high

volume and speed of EDI transaction processing make information obsolete within a very short time. Hence, audit reporting procedures need to be adjusted to reflect that problem and to accommodate the need for timely and accurate information.

According to Yang

[1990], the Securities and Exchange Commission (SEC) had marked the movement toward this direction by introducing EDGAR in May 1984 . He noted that since the start of the 11

pilot project, the idea of EDGAR has gained the enthusiastic attention of filing firms, securities analysts, and the general public. When such an electronic reporting concept is put into practical use, interested parties should be able to promptly access information they desire. However, in such a circumstance, auditors will be subject to greater demandsfrom management, investors, institutions, and the general public-to certify the reliability, security, and integrity of crucial databases on a continuous basis.

3.2.6 Audit Focus (Substantive versus CompUance Testing)

According to the ICAEW's report [1989], although the speed of the move towards this trend is unclear, it is clear that the technology is already available to enable reporting "at a frequency that could, in theory, be 'up to the minute" [p.4]. 10

" Electronic Data Gathering Analysis and Retrieval (EDGAR) system is "An electronic data processing system that is capable of receiving companies' financial reports electronically, allowing for their review by the SEC staff in a similar manner, and permitting computerized dissemination of information to investors, analysts and others capable of receiving information in this way." [Yang 1990, p.49].

27

Rationale: Although there is a general agreement that the focus and the types of audit tests need to change to suit highly automated, paperless intercompany EDI networks, there is no consensus on the direction or specific procedures that should be applied. Therefore, efforts are needed to establish appropriate audit approaches. Although there is a general agreement that the focus and the types of audit tests need to change to suit highly automated, paperless intercompany EDI networks, there is no consensus on the direction or specific procedures that should be applied to attain the audit objectives. Staats [1981] and West [1988] believe that the shift should be away from testing individual transactions towards tests of the system and system security. They also note that auditors should be more concerned with fraudulent or intentional manipulations of the records in such a paperless system. Jancura et al. [1986] suggest (for audit tasks in general) that the possibility of added calculations and statistical analyses without excessive additional costs should enable auditors to perform more analytical reviews. The Information Technology Group of the Institute of Chartered Accountants in England and Wales [ICAEW 1989] observes the discordance between clients' needs and audit focus and notes that: Users with high volume, rapid response transaction processing systems need well controlled computer systems or they run the real risk of going out of business. They have to understand their systems and control them. They also expect their auditors to understand such systems and to be able to provide critical comment thereon. Yet many auditors, faced with increasing complexity of clients' systems, look to substantive testing techniques rather than control-based compliance testing techniques. This response is frequently driven by cost-effectiveness considerations. It can also reflect the inherent difficulties of compliance testing in respect of complex systems. [ICAW, p.3]

28

During a seminar on "Implications of Emerging Technology to Auditors" , a 12

speaker who is a partner responsible for the Computer Audit Support Group of a big six accounting firm in Vancouver B . C . , commented that considering the availability of advanced software to ease the task, auditors should be able to perform more substantiation of data. Brown [1991] quotes Hugh Parkes, general manager of group audit at National Australia Bank, as stating that: / think it will require major changes to audit methodologies, and to the sacred cows of auditing-particularly the issues of substantive and compliance auditing. I have serious questions as to the validity of some of these with advanced large-scale systems, where there is virtually no paper, very little to substantiate it, and it's necessary to have a good understanding of how the transactions are arrived at... these are (issues) of international significance,... [Brown, p. 12]. These questions remain unresolved. Research efforts are needed to establish appropriate audit approaches. Different audit functions, e.g. internal and external audits may be called to have different audit focuses.

3.2.7 Pre-determination of Audit Scope (Boundary of Audit) Rationale: Paperless intercompany transactions create a "boundaryless" information system environment, and auditors may be required to audit beyond the traditional boundaries of clients' systems. Therefore, audit responsibilities need to be pre-determined and agreed upon by the parties involved. Paperless intercompany transactions create a "boundaryless" information system

The lecture was a part of a one-day seminar on "Emerging Information Technologies—An Auditor's Perspective" organized by the EDP Auditors Association, Inc. (Vancouver Chapter) at Simon Fraser University (Harbour Center) on November 8, 1991. 12

29

environment, and auditors may be required to audit beyond the traditional boundaries

13

of clients' systems. In order for auditors to prevent a "boundaryless" audit responsibility from occurring while they continue providing an adequate audit service, the scope and extent of audit task and responsibility need to be pre-determined and agreed upon by the parties involved. For example, in a fully automated EDI system, where electronic data originating at one company is transmitted to a receiving company and incorporated directly into that company's application system (it is to be hoped after some forms of review or computerized edit checks), the auditor of the receiving company may choose to take the responsibility of evaluating the reliability of either or both the network over which the data was transmitted and the quality controls of the sending company [Holstrum et al. 1988, p. 173].

3.2.8 Audit Tools Rationale: The increased complexity of intercorporate automated paperless transactions make it more difficult, or in some cases impossible, for auditors to test and evaluate network systems with existing audit tools. More powerful tools must be developed to match the growth in sophistication of clients' systems. The increased complexity of intercorporate automated paperless transactions make it more difficult, or in some cases impossible, for auditors to test and evaluate network systems with existing audit tools. It is therefore critical that more powerful audit tools be developed to match the growth in sophistication of clients' systems. Auditors need to be

"Boundary defines a system in terms of the degree of control it can exercise, and with EDI, this control is somewhat extended beyond the original focal organization." [Kavan 1991, p.41]. 13

30

equipped with appropriate tools adequately to achieve audit objectives and to satisfy clients' and the general public's expectations. The literature review suggests that such tools as integrated audit networks, portable workstations, interconnections with large databases, audit software embedded in the operating systems, expert systems, multiple input modes, evaluation software, and natural language programming are indispensable future audit tools [West 1988; Holstrum et al. 1988].

3.2.9 Audit Techniques Rationale: Traditional audit techniques may no longer apply in an EDI environment. Auditors need to develop effective techniques to enable them to describe, evaluate, and test an intercompany information network that includes very few paper documents. EDI technology introduces a new era to business transaction processing. The significant cutback on paper may put the end to auditing "around" the computer. The complex interconnections of EDI networks imply that such traditional audit techniques as test data and Integrated Test Facility (ITF) have to be significantly modified. The volume and the sheer pace of transaction processing may make continuous on-line computer monitoring more effective than human observation.

Programmed edit checks and

programmed monitoring systems may have to be used either to prevent or to detect unusual activity for near-immediate follow-up or to do both. The flowcharting technique, which is document oriented, may have to be replaced by another technique which enables auditors to describe, document and evaluate an accounting system that does not include paper documents [West 1988; Wise 1989]. In summary, to demonstrate competence in dealing with EDI systems, auditors need to find effective audit techniques that are capable 31

of describing, evaluating, and testing a "paperless" intercompany information network.

3.2.10 Audit Risk Assessment Rationale: Audit risks in an automated open EDI network are significantly different from those in a closed system because EDI involves more parties and more diverse exposures. A distinctive approach of audit risk assessment is required, and sound guidance must be developed to guide the practice. Certain attributes of EDI technology make an EDI system more secure while others make it more risky. The automation of system functions means high consistency of transaction processing and increased system reliability. Further, because errors that may be generated by such a system are typically systematic (non-random error), they are easier to be detected and corrected. Thus, if there is an assurance that the system functions properly and reliably, business and audit risks may decrease substantially. On the other hand, the EDI notion involves many exposures over which users have little or no control. For instance, the security of an EDI network depends on such a considerable amount of trust among all the involved parties that adequate controls must be maintained over each individual system, and each party must follow the agreed-upon rules. Further, the use of Value Added Network (VAN) creates new risk exposures because of the company's 14

increased reliance on a third-party to provide acceptable controls and services. In addition to that, the company auditor's inability to evaluate directly third party (VAN) performance increases audit risks. Moreover, open intercompany systems are naturally more vulnerable

A communication network over which a third-party vendor performs EDI services beyond transmission of data—for instance, translation, training, encryption, etc. These services add significant value to the basic function of message switching and enable different computers to communicate to each other [Emmelhainz 1990; Baker 1991(b); Kimberley 1991]. 14

32

than the closed singular systems, and the dial-in lines used in the transmission of EDI transactions are more prone to be attacked.

This vulnerability coupled with the

acceleration of transactions, increases business and audit risk substantially because a single failure, if it occurs, can be widespread within a short time. The reviews recommend advanced planning and up-front risk analyses as a vital part of an EDI audit. By recognizing the risks associated with the technology, auditors can help improve the security of the system by establishing an integrated program of risk assessment and monitoring of early warning indicators. [West 1988; Sadhwani et al. 1989; Eckerson 1990; Chalmers 1990; Burns and Sorkin 1991]. However, because EDI involves more parties and different exposures, risk assessment in an EDI environment require consideration of a different set of elements from that required in the conventional system. Thus, sound guidance should be established to assist auditors in performing this task.

3.2.11 The Changing Role of Auditors Rationale: In an EDI environment, the role of auditing will have to change to meet both the new demands of clients and the needs of the profession. Failure by auditors to assume suitable new roles adequate to meet such demands may lead to the decline of the profession. In an EDI environment, the role of auditing will have to change to meet both the new demands of client needs and the needs of its own practices. Like other advances in information technology, EDI should be viewed less as a threat than as an opportunity. Auditors are in a good position to assume many new roles which businesses and the general public expect them to, and incidentally, these roles can be both financially and professionally rewarding. As suggested in the professional literature, auditors are uniquely 33

suited to serve as "IT advisor[s]" or consultants on technology decisions. Auditors can 15

help assure the integrity of internal controls and also control future audit fees. Furthermore, as independent experts, auditors can mediate different technological views that may exist among managers, technologists, and senior executives, and help an organization reach quality decisions. Moreover, by working together, external and internal auditors can advise and assist an organization's executives with the technology's strategic implications [Hogarth 1986; Willits 1990; Brown 1991]. Failure by auditors to develop a strategy to meet clients' changing needs and their technology demands may lead to the decline of the auditing profession.

Thus, auditors must continue to live up to the

expectations of the corporations and the public.

3.2.12 Audit Responsibility in Evaluating Controls Rationale: Auditors, especially internal auditors, may be held responsible for the review and evaluation of external control(s) in open EDI network systems. In such circumstances, it is necessary that guidance be developed and standards be established to assure that the task is performed adequately. 16

Both academic and professional literature indicate that the responsibility of auditors for the review and evaluation of control structures in open automated EDI network 17

systems may have to be expanded. For instance, Holstrum et al. [1988] suggest that

15

Information Technology.

External control is defined as "a domain of factors that operate outside of an organization and may affect the way the organization and management operate. " [Barrett 1990, p. 63]. 16

17

The term "Control Structures" is used here to cover both internal and external controls.

34

"blurred boundaries of the audited entity" demand the expanded responsibility of the auditors to evaluate the integrity of both internal and external databases. Hansen and Hill [1989] recommend that controls be exercised beyond the traditional system periphery and that, where a third-party V A N is used, auditors evaluate network application features either directly or through the VAN's auditor. Sadhwani et al. [1989] have a similar concern, and they assert that the evaluation of controls in a typical EDI network involve the following three parties, each of which must provide assurance that proper controls are maintained within each individual system: --the originator of the transactions and documents; —the processor (i.e. a third-party network or a bank); —the receiver of the data and documents. Moreover, Barrett [1990] believes that "Only a global notion of control is realistic and relevant in the fast-paced global economic environment..." [p.68]. He then proposes that auditors recognize the importance of the concept and the audit of external control . 18

In addition, if the audit responsibility in reviewing controls is to be expanded, authorities seem to suggest that internal auditors will have increased responsibilities. For instance, Staats [1981] states that the "paperless" transaction processing and the increased reliance on the adequacy of the system controls demand more commitments from the internal auditors. He claims "internal accounting control is the area where the corporate internal auditor's depth and breadth of knowledge is superior to everyone else's—inside or

External control was defined as "a domain of factors that operate outside of an organization and may affect the way the organization and management operate. " [Barrett 1990, p. 63]. 18

35

outside the company." [p.7]. Further, West [1988] advises that internal auditors analyze proactively data security and data communication. Moreover, Barrett [1990], who underscores the importance of external control, insists that it is internal auditors who should take responsibility for understanding and auditing external control. He cites the following activities as examples of sources of external control that deserve attention from internal auditors: —assessing the effectiveness of regulation; —evaluating external audit performance; —evaluating material acquisition in a just-in-time setting and EDI transactions; —auditing customers. The auditing profession needs to take this issue of external control into serious account. If auditors are to be held responsible for external control, the task of reviewing, testing, and evaluating this type of control must be performed properly. The profession needs not only to develop solid guidance for but also to establish standards of practice on this added activity.

3.2.13 Relationship Among Company's Auditors Rationale: As business information systems grow in complexity, external auditors may have to rely on internal auditors, and information systems auditors will be requisite members of audit teams. It is crucial to develop suitable audit approaches to promote and make the best use of this interrelationship. Along with the tendency towards increased demands EDI makes on the internal

36

audit function, the reviews encourage close cooperation between internal and external auditors. One reason for that cooperation is that, as business information systems become increasingly complex, external auditors may have to rely more on internal auditors. Brown [1991] illustrates this point by stating, with a quotation from Hugh Parkes, general manager of group audit at the National Australian Bank, that:

The technology gap is beginning to blur the distinction between the roles of internal and external audit Junctions.... fit is found that] even the most experienced external auditor required additional expert advice [from internal auditors] in order to perform an adequate audit... 'the reality is that internal audit environments of banks are so much bigger than the external audit commitment that it is the people who are there all the time, who increasingly have the balance of knowledge about the systems. This does pose questions and some challenges to the relationship between internal and external audit. Knowledge of the client is very important, and external auditors should make sure they have enough of this. '. [Brown, p. 12] The foregoing statement illustrates a typical situation in current businesses. As outsiders, external auditors are forced with a limited time to gain sufficient knowledge and understanding of the client's systems to conduct an adequate audit. By teaming with internal auditors, they can be more efficient and more effective in accessing client computer resources and in identifying the strengths and weaknesses of the systems. Moreover, the same generalized audit software used for internal audits can be useful for preparing and performing the annual external audit. Furthermore, in some cases audit coverage can be expanded and overall audit fees can be reduced because much of the external auditors' substantive testing can be replaced with less expensive compliance testing [Boughton 1987; Brown 1991].

37

In addition, because of the increasing complexity of the "paperless" EDI networks, teamwork between computer auditors and general auditors is strongly recommended. The stake here is that telecommunications is an area requiring the expertise of a technical computer audit expert and technical audit software. Further, it is recommended that to create a good working relationship between financial and information systems auditors, the differences between the two types of auditors must be both communicated and deemphasized [Moeller 1986; West 1988; Dunmore 1989; Utter and Bertram 1989]. It is crucial for the auditing profession to develop audit approaches suitable to promote the workability and to make best use of this inter-relationship.

3.2.14 Collaboration Among Auditors of EDI Parties Rationale: An EDI network involves not only auditors of a company but also auditors of its trading partners and of third parties. Because the tasks of these auditors are inter-dependent, it is vital that the roles of each party's auditors be determined and the rules of collaboration be established. Interconnections of different organization's systems into one large system is one of EDI's key characteristics. A typical EDI network comprises of not only a company's system but also outsiders' systems. These outsiders can be trading partners (i.e., customers, suppliers), service bureaus, VANs, and third-party telecommunication networks.

As a result, auditors of an EDI network consist of auditors (internal and

external) of a company under consideration as well as auditors (internal and external) of the network participating parties. These groups of auditors are inevitably inter-dependent on each other's work because the controls and security of a system depends heavily on those of the connecting systems. Therefore, close collaboration among auditors of all 38

parties in an EDI network is desirable, and each group of auditors can gain substantial benefits from other group's work. For example, as recommended by Hansen and Hill [1989] and by Sadhwani et al. [1989], because most third parties are reluctant to allow auditors other than their own to access their facilities, auditors of a company that uses third-party services may evaluate network application features through the work that has been performed by the third-party's auditors. Further, while urging a company to seek the auditor's reports from third parties, Hansen and Hill [1989] remark that "This practice is not yet commonplace, but should become so as EDI becomes more pervasive." [p.412]. Today, businesses make an extensive use of EDI technology and a large proportion of EDI networks of trading partners uses a third-party service to serve as an intermediate processing agent. It is therefore important that the roles and responsibilities of each party's auditors be determined, and the rules for collaboration among these auditors be established.

3.2.15 Auditor Skills (Skills required of auditors) Rationale: Because of rapid expansion in the extent, scope, and types of information to be audited in an EDI system, auditors need to acquire certain skills that enable them to maintain a high standard of practice. Because of rapid expansions in the extent, scope, and types of information to be audited in an EDI system, auditors need to acquire a specific set of skills that enables them to overcome difficulties and maintain a high standard of practice. Some essential skills cited in the reviews are the computer skills, the ability to adapt readily to rapidly changing

39

information technology , and the high analytical skills to accommodate the shifting 19

composition of audit tasks to high-level analysis [West 1988; Holstrum et al. 1988]. It is important that a comprehensive set of required skills be researched and incorporated into skill development programs for auditors.

3.2.16 Auditor education and training Rationale: Auditors trained today practice in a significantly different environment from that in the past. The educational curriculum and training requirements of auditors must be updated to reflect technological change and to embody the types of knowledge and proficiency required of auditors to maintain the profession. Auditors trained today practice in a significantly different audit environment from that in the past. Therefore, the educational curriculum and training requirements must be updated to reflect technological change and to embody the types of knowledge and proficiency required of auditors to maintain the profession. Certain education and training needs required of information systems auditors in general may also apply to EDI auditors. The reviews suggest the following subjects as important: - continuing education for auditors to gain expertise in data processing and fraud prevention and detection [Staats 1981, p. 11] - proficiency in auditing and data processing, as well as a solid business background [Helms 1986] - knowledge of methods being developed in the study of human information processing and artificial intelligence in addition to a broad base of skills and a high degree of professional commitment [Elliott 1986].

Adaptability which was specified includes: computer adeptness; ability to interface effectively with expert systems; and knowledge of and adeptness with modelling concepts and applications. 19

40

- more use of case studies methods in the educational process to train auditors in sharing audit expertise with organizations adopting new technologies [Gilhooley 1987] - sound understanding of the system (accounting/management) implemented by audited companies to process their business information [West 1988, p. 187] - thorough understanding of computerized information systems (IS) and learning of technical data processing concepts, as well as the traditional computer audit skills [Glynn and Lemieux 1990] In addition, the following ongoing training for audit staff is recommended: - training for computer security auditing with focus on: (1) general security, (2) specific applications, and (3) technological trends. [Gallegos 1987] - "individualized training of staff members" for a small EDP audit department. Audit performance should be evaluated and the needs for staff training be determined in accordance with that evaluation. [Goldner 1987] The reeducation and retraining of auditors is an important issue because failure to educate and train auditors capably to keep pace with business advances in the use of information technology threatens the existence of the auditing profession.

41

Chapter 4. Research Design

4.1 Research Questions This research project is designed to answer the following questions: 1) What are the most important EDI audit issues as seen by leading audit professionals? 2) What is the order of importance of these issues?

4.2 Selection of Research Methodology In the search for an appropriate research methodology to investigate the foregoing research questions, a variety of technology assessment techniques were 20

explored. The techniques which seem to be applicable are Delphi technique, survey technique, expert panels/workshops, and compilation and analysis of all available information. In this study, the technique of compilation and analysis of existing literature is used to form the primary research issue framework in chapter 3. A Delphi survey, however, was used to solicit opinions and consensus of leading information systems auditors in the greater Vancouver area regarding the importance of EDI audit issues. The Delphi methodology is a cost-effective way systematically to solicit and combine the individual judgments of experts in the field and thus to obtain a reasoned consensus. It is capable of yielding answers to the research questions being addressed in By forecasting what the consequences might be if a complex emerging technology is in fact adopted, technology assessment techniques enable us to study complex technologies and their potential impacts upon society. [Fowles i97g p. 146] This type of assessment, therefore, can be well applied to an emerging technology such as EDI and is suitable for the purpose of this study. 20

;

42

this research project, and it has been, and continues to be, widely and successfully used in similar issue researches in the area of Management Information Systems (MIS). Further, in comparison with the survey technique, a research finding shows that the Delphi technique, even when only two rounds are conducted, generates a more reliable and valid result than the survey technique. [Martino 1983] As opposed to the expert panels/workshops or committee meeting approach, the Delphi technique replaces direct debate by a carefully designed program of sequential individual interrogations conducted through written questionnaires. Information and opinion feedback from the earlier parts of the program are used in the later stages. The respondents may, for instance, be asked the reasons for previously expressed opinions, and a compilation of these reasons may then be presented to each respondent with an invitation to reconsider and possibly revise his or her earlier estimates. This inquiry and feedback may stimulate the experts to consider factors they might have inadvertently overlooked or disregarded on first thought. The Delphi technique offers many benefits. It eliminates committee activity and reduces the pitfalls of face-to-face discussion.

21

Because each Delphi panellist is allowed

to give opinions independently and anonymously in the written questionnaires, the influence of undesirable psychological factors, such as specious persuasion, unwillingness to abandon publicly expressed opinions, and the "bandwagon" effect of majority opinion, can be greatly reduced. Moreover, considering the relatively small Vancouver IS auditors

Lanford [1972] states that "face-to-face discussion tends to make the group less accurate, whereas the controlled-feedback procedure [as used in the Delphi approach] makes group estimates more accurate. " [p. 22] 21

43

community, the possibility of personal conflict and social pressures that might occur can be prevented. Also, in spite of the capability to avoid the important drawbacks of open face-to-face discussion, the Delphi approach can still obtain the benefit of such discussion. By assimilating comments from Delphi panel members into the questionnaires in the subsequent rounds, a desirable interaction among several participants expressing their opinions can be facilitated . Furthermore, another benefit is that the Delphi procedures 22

create a well-defined process that can be described quantitatively. Finally, because the Delphifindingsreflect reasoned, self-aware opinions, expressed in the light of the opinions of associate experts, they are claimed to provide a sounder basis for long-range decisionmaking than do unarticulated intuitive judgments. [Lanford 1972; Martino 1983]

4.3 The Delphi Process - An Overview The Delphi approach requires several iterations of questionnaire completions by the expert participants. In general the Delphi process in this specific study proceeds as follows: (1) Potential Delphi panel members are identified from the group of Vancouver's leading IS auditors, and 33 experts are selected . The reason for using this sample size is 23

discussed in section 4.5 of this chapter. (2) The selected experts are contacted and requested for an agreement to participate in the

According to Martino [1983, p.23], the experiment by Salancik [1973] showed that Delphi panels do assimilate the comments from panel members into their aggregate estimates and group interaction does occur. 22

For purposes of conducting the research within a limited budget, Vancouver, a major city in Canada, was chosen as the study site. 23

44

study. Among these, only 12 are asked to participate in the unaided first round survey. The rationale for using only a subset of the entire sample in the first round is discussed in the later section dealing with participant recruitment. (3) An open ended first round questionnaire and a cover letter explaining the research are sent to the 12 selected experts. The purpose is to solicit unbiased views and identification of important issues from the highly experienced IS auditors in the greater Vancouver area. (4) The results from the first round questionnaire are analyzed and integrated with EDI audit issues previously identified in the existing literature. The purpose is to facilitate the rating of the issues and the identification of the most important audit concerns. This integration can also provide information on the perspectives of the Vancouver IS auditors on important EDI audit issues addressed in the North American literature. (5) The second round questionnaire and a cover letter are sent to all 33 Delphi panel members to rate the importance of EDI audit issues. The panellists are asked to rate, rather than rank, each issue because rating is a less stressful and time-consuming procedure. In this round the panellists are also provided with an opportunity to add new issues. (6) The results are analyzed and feedback is incorporated into the third round questionnaire. The incorporation of this feedback is aimed at facilitating the reflection of opinions and the movement toward consensus. A greater depth of insight is expedited by multiple rounds. (7) The third round questionnaire is sent to all 33 panel members regardless of whether or not they responded in the second round. The purpose of still including the panellists

45

who fail to respond in the second round is to improve the response rates. (8) The analysis and feedback procedures will be repeated until a convergence of opinions or the stability of ideas is obtained.

4.4 Instrument Development This study consists of multiple rounds, requiring the development of a separate questionnaire for each round. The following sections outline the procedures in developing these instruments: 4.4.1 Round 1 Questionnaire The primary purpose of the first round survey is to generate an initial list of issues to be rated in subsequent rounds. The format of the questionnaire developed for this first round is therefore open-ended, asking the participants to state objectively what they regard as the key EDI audit concerns. The unaided format is used here to minimize the risk of the researchers biasing the participants' responses and to give the audit experts an opportunity to identify important issues without the distraction of considering issues from other sources. The first-round questionnaire and the cover letter can be found in Appendix A . The instrument consists of three parts: - a cover letter - notes to the respondents (to define terms used in the core questionnaire) - a core questionnaire (a participant identification section (optional), instruction, and space for issue identification)

46

In addition to the instructions for answering the questionnaire, the notes to respondents are added to clarify and define the terms "auditors" and "EDI" used in the core questionnaire.

The notes also differentiate "EDI" from "EFT (Electronic Fund

Transfer)". Because each term may have different meanings and interpretations for each individual audit expert, this procedure is performed to provide common definitions and thus create a common frame of mind when participants respond to the questions. 4.4.2 Round 2 Questionnaire In a conventional Delphi study, the round 2 issue list would be generated solely from the round 1 resulting issues and would therefore reflect only the opinions of the participating Delphi panelists. However, the use of only a subset of the total panelists in the first round and the pressure to complete the research within a reasonable amount of time justify the use of a supplementary technique to generate additional issues for the round 2 survey. The primary purpose of this supplementary issue generation is to assure that, to the greatest possible extent, all the relevant issues are identified and included in the study. The issue list for the second round survey is thus generated using the following two techniques: I.

Analysis of results from round 1 survey

II. Compilation and analysis of existing literature Each of these techniques, together with the issues discovered by the first round survey, are discussed in more detail below. I. Analysis of Results from Round 1 Survey Each of the completed questionnaires was carefully examined by the researcher. The issues identified by the first round experts (as shown in table 1) were analyzed and 47

classified. The following is a list of five new issues, i.e. previously not well articulated in the existing literature, which were identified and clarified by the researcher based upon the analysis of the results of the round 1 questionnaires: • • • • •

Controls Over EDI Network EDI Contracts Backup, Disaster Recovery and Contingency Plans Third Party EDI Services EDI Records Retention

n. Compilation and Analysis of Existing Literature As discussed in Chapter 3 on the EDI audit issue framework, the reasonably current information on important EDI audit concerns is available, and this information should be used, when applicable, to provide a broader consideration of the subject. The 16 issues outlined in chapter 3 are thus incorporated into the initial issue list resulting from round 1 survey. These issues are: • • • • • • • • • • • • • • • •

Audit Evidence Audit Trail Audit Involvement during the System Development Timing of Audit Tests Audit Reporting (Periodic versus On-Line) Audit Focus (Substantive versus Compliance Testing) Pre-determination of Audit Scope (Boundary of Audit) Audit Tools Audit Techniques Audit Risk Assessment The Changing Role of Auditing Audit Responsibility in Evaluating Controls Relationship Among Company's Auditors Collaboration Among Auditors of EDI Parties Auditor Skills (Skills required of auditors) Auditor education and training

The foregoing issues and the results from round 1 survey are then put into 48

random order

24

and combined into a single round 2 questionnaire.

As shown in

Appendix B, the resulting instrument consists of three parts: - a cover letter - a core questionnaire (a participant identification section, instruction, and 21 issues to be rated) - an open-ended section for adding new issues (instruction and space for issue identification) The style and format of the main body of the questionnaire closely imitates the layout established in previous Delphi studies on MIS issues. The open-ended section is included as a control to ensure that the two techniques described above have been sufficient to generate all of the potentially important issues. 4.4.3 Round 3 Questionnaire The round 3 issue list is generated from the twenty-one issues in round 2, together with the four new issues which were added. This final questionnaire provides an opportunity for participants to reflect on their answers in round 2 by supplying, for each original issue, information on both the group's mean response and that particular individual's response. This instrument is similar to the round 2 instrument and is aimed at obtaining the final rating and ranking of each issue. The round 3 instrument can be found in Appendix C. The instrument consists of four parts: - a cover letter - a core questionnaire (a participant identification section, instruction,

24

The Rand Corporation, A Million Random digits with 100,000 Normal Deviates, The Free Press, 1955.

49

21 issues to be rated, the group's mean and the individual's original rating, and open spaces for final rating and rationale if individual final rating is significantly different from the group) - a questionnaire on additional issues (instruction and 4 new issues to be rated) - a questionnaire on background information It should be noted that the 21 original issues to be rated are now presented to the participants in a decreasing order of importance, as determined by the group mean response from round 2. In addition, each participant's questionnaire shows his or her previous individual response as compared to the group mean response for each issue. The analysis of the open-ended section from round 2 reveals four additional important issues. These new issues and their rationals are incorporated into the round 3 questionnaire but are listed separately and without any previous rating.

No further

opportunity to add new issues is provided in this final round. The questionnaire on respondent's background information is added to the round 3 questionnaire so that data can be gathered for analysis and categorization purposes. In general, this section asks for area(s) of audit expertise, level of experience both in information systems auditing and EDI system auditing, and an indication of the firm's preparation for EDI technology. Further, as a way to motivate response, each participant is asked to indicate whether or not he or she would like to receive a copy of the final research findings.

4.5 Participant Recruitment According to Lanford [1972], research by Norman Dalkey of the RAND 50

Corporation shows that most of the limitations of using the Delphi technique can be overcome by working with groups of at least twenty. Further, as cited by Dexter et al. [1992], "In fact, Dalkey (1969) found Delphi studies produce quite accurate results with a group size of approximately thirty individuals. Furthermore, he found that increasing the number of participants does not markedly enhance the accuracy of the findings. " [p. 7] In order to minimize the study's time and costs without sacrificing the values of its results and to take into account the possibility of unexpected drop-outs, this project seeks to identify and obtain cooperation from 33 of the Vancouver leading IS auditors. The participant recruitment procedures began by the researcher's identifying and seeking cooperation from two contact persons from two local accounting professional organizations. The Director Membership of the EDPAA (Mr. Alan Drinkwater) and the President of the IIA (Ms. Angela Louie) agreed to assist in this research project. The researchers personally met with both of the contact persons to promote commitment and to create a better understanding of the nature of the study and of the instruments. Then, with their assistance, the membership databases of the EDP Auditor Association (EDPAA) and of the Institute of Internal Auditors (IIA), Vancouver Chapter, were used to identify and recruit potential participants. Besides their involvement and experience with EDI projects, participants were recruited on the basis of professional qualifications, audit specializations, and peer recommendations. Once appropriate individuals were selected, they were contacted in person and asked for their agreement to participate in this multipleround Delphi study. Care was taken to ensure that this sample group represented various types of organizations (public accounting firm, private or limited company, crown 51

corporation, government, and academic) and included both internal and external IS audit experts.

Because the majority of this group of experts is from the major firms in

Vancouver which are either utilizing or pioneering the EDI technology, this group can be regarded as providing a representative sampling group of the Vancouver EDI audit community. It should be noted that only a subset (12) of the total number of the panelists (33) were selected to participate in the unaided first round survey. The reason is that the unstructured nature of the first round survey would likely make this the most difficult and time consuming iteration in the study. Therefore, in order to reduce turnaround time and complete the research project within a reasonable time frame, the size of the study group was reduced to include only participants who have the most extensive experience in auditing EDI systems. This sampling approach generates a group of qualified and motivated participants, and it is hoped that they could collectively reach a reasonable level of consensus in judging key EDI audit issues.

4.6 Data Collection Procedures The data collection of this research project was conducted in Vancouver, Canada between March 16 and July 10, 1992. The questionnaires for each round were mailed to participants outside the downtown area and hand-delivered to those located downtown. A stamped, self-addressed return envelope was always provided with each questionnaire, although the respondents were instructed either to mail or fax the completed questionnaires

52

to the primary researcher at the University of British Columbia. All follow-up on late respondents was done by telephone calls. 4.6.1 Round 1 The first round questionnaires were forwarded to a subset of the final survey sample. As explained in the participant recruitment section, only 12 experts were asked to complete this open-ended first round questionnaire. The contact persons from the two professional organizations had helped to make initial contact with each participant to increase understanding and commitment. In addition, the cover letter of the questionnaire explained in detail the initial goal, ultimate objective, and nature of the study. Also, participants were encouraged to contact the primary researcher directly should they need any clarification.

The participants were

requested to reply within two weeks. After three weeks, which is a reasonable time period for all the mail to reach the researcher, non-respondents were contacted by telephone to encourage responses. Once the first round questionnaires were returned, the responses were analyzed and classified to isolate the primary EDI audit concerns of Vancouver IS auditors. These issues were then combined with those identified in the literature to prepare a list of issues to be rated in round 2. 4.6.2 Round 2 The analysis of the first iteration results and the inclusion of issues generated by using supplementary techniques led to the creation of a new questionnaire for the second round. The questionnaires were then sent to all 33 participants. As in the first

53

round, the cover letter of the questionnaire explained the goal, ultimate objective, and nature of the study as well as encouraged participants to contact the primary researcher directly should they need any clarification. In this iteration, participants were asked to rate a list of issues in terms of their importance on a scale from 1 to 10. In this manner, the most important issues could be quantitatively identified and the appropriate analysis could be made.

In addition,

participants were given a final opportunity to add new issues to the list to ensure that all the major EDI concerns had been identified.

These issues were then analyzed and

classified before being incorporated into the final round questionnaire. In this round, the participants were requested to reply within two weeks. After three weeks, follow-up telephone calls were made to non-respondents to encourage responses. It was found that these follow-up calls greatly increased the response rate. Some panelists were out of town during the period of two weeks when the questionnaires were sent out. Having thought that it was too late, they did not respond. However, after the follow-up calls they were willing to complete the questionnaires and return them by facsimile so that the researcher received the responses in a timely manner. 4.6.3 Round 3 As discussed in the instrument development section, the construction of the core questionnaire in round 3 was based on the results of round 2, with a separate page to gather respondents' background information. In this round the questionnaires were sent to the same group of thirty three participants as in round 2. As in the first two rounds, the cover letter of the questionnaire encouraged

54

participants to contact the primary researcher directly should they need any clarification. The participants were requested to reply within two weeks. Although the response rate improved from the second round, follow-up telephone calls were still necessary to motivate responses. In this iteration, participants were asked, after considering the feedback in the form of the mean group response, to give a final rating for each issue, using the same scale as in round 2. No further opportunity was provided to add new issues. As suggested by proponents of the Delphi technique [Delbecq 1972; Martino 1983], two iterations are generally enough to establish the list of issues and their relative ordering. The third iteration serves primarily to confirm the ordering and promote a consensus among participants. In this study, the analysis of round 3 results indicates that the Delphi process moved the group toward a consensus on the eleven most important EDI audit issues.

55

Chapter 5. Analysis and Discussion of Results

5.1 Introduction This chapter discusses the results from the three-round Delphi survey. Although all key EDI audit issues identified by the respondents in round 1 [Appendix D] are included in the round 1 results, only those issues which are not addressed in chapter 3 are discussed in detail in this chapter. The rating results from round 2 are stated and analyzed along with the additional important issues revealed in this round. Then, comparison is made of round 2 and round 1 results. Next, the results from round 3 which are the final rating of all the issues are then discussed and the comparison is made with the results from round 2.

The movement towards consensus and the background information on the

respondents are outlined in the last two sections of the chapter.

5.2 Round 1 Results In the first round, open-ended questionnaires were sent out to 12 selected audit experts. However, only 9 questionnaires from 10 experts were returned. This happened because two of the respondents worked together and submitted a single copy of the questionnaire. Two non-respondents were away; however, they participated in the second and the third round. Based on the narrative content and the labelling of issues [Appendix D], the responses from the first round were analyzed and classified into seven major issues. Table 1 shows these key EDI audit issues identified by this group of experts.

56

T A B L E 1 - ISSUES IDENTIFIED IN ROUND 1 Frequency

Issue

12

New

Controls Over EDI Networks

6

New

EDI Contracts (Trading Partner Agreement)

4

Auditability and Audit Trail

4

New

Backup, Recovery and Contingency Plans

3

New

Third Party EDI Services

3 2

Legal and Audit Evidence New

EDI Records Retention

The foregoing issues are presented in order of the frequency of citations given by the respondents.

Of these seven issues, only two were previously addressed in the

compilation and analysis of existing literature in Chapter 3. The issues which were not discussed in the issue framework in Chapter 3 but were identified during this round are designated as 'new' issues. These issues are discussed in more detail in the following section. 5.2.1 Controls Over EDI Networks Rationale: Because weak controls can cause significantfinancialloss to EDI trading partners, auditors must assure that controls over the EDI networks such as access controls, authentication controls, transmission controls, and controls over mailboxes are effective. As shown in Appendix D, there are 12 responses in round 1 that can be classified under the control issue. These responses include audit concerns for controls over (third party) mailbox service, data confidentiality, system security, integrity of data, authentication of trading partners, trading partner's security, access controls to the EDI (network) environment, financial controls, communication (transmission) controls with 57

suppliers, and accurate and complete transmission of data. 5.2.2 EDI Contracts (Trading Partner Agreements) Rationale: Because EDI contracts are the basis for the company's future dealings with and liabilities to EDI partners, auditors must ensure that the terms, services, conditions, and responsibilities of each EDI party are clearly defined and the contracts are inclusive and enforceable. As shown in Appendix D, there are 6 responses in round 1 that can be classified under the EDI contracts issue. The respondents feel that EDI contracts or trading partner agreements are important because they believe these contracts or agreements are the basis for the company's future dealings with its EDI partners. Therefore, they suggest that in order for the agreements to be enforceable all agreements/contracts must be in written form and cover all significant issues. In addition, terms, services, and responsibilities of each party must be clearly defined and agreed upon by all parties involved. 5.2.3 Backup, Recovery and Contingency Plans Rationale: Because of a company's increasing reliance on the EDI network for operational and financial services, auditors must assure that control strategies exist for backup and recovery in case the network fails. These strategies form an indicator of the company's ability to continue as a going-concern. As shown in Appendix D, there are 4 responses in round 1 that can be classified under the issue of backup, recovery and contingency plans. The respondents express concerns over this issue because, in their view, in an EDI environment a company relies heavily on the EDI networks. In case of network failure where a company does not have adequate plans and strategies for backup and recovery, the company's assets in the form of valuable data may be lost, and the company's ability to continue as a going-

58

concern may be in jeopardy. 5.2.4 Third Party EDI Services Rationale: Different types ofEDI networks have different implications for the participating companies and their auditors. Auditors must evaluate their clients' third party EDI service firms in terms of responsibilities, resources, and abilities to provide, on an ongoing basis, reliable and secure services per contract terms. As shown in Appendix D, there are 3 responses in round 1 that can be classified under the issue of third party EDI services. In case where a company uses third party EDI VAN(s), the respondents think that the types of networks and the reliability, availability, and security of EDI services need to be evaluated. As certain of the respondents stated, such an evaluation may be based on the service firm's "financial resources to provide service per contract terms on an ongoing basis" and "adequate provision for trouble shooting, client communication, system upgrading capabilities". The respondent auditors believe such an evaluation of EDI service qualities should be done because different types of EDI networks have different implications for the network participating companies and their auditors. 5.2.5 EDI Records Retention Rationale: Because EDI uses electronic source documents, guidance must be established to ensure that EDI records are maintained properly and securely for an appropriate amount of time to suit tax, audit, backup, and management purposes. Ineffective records management can lead to exposures such as the loss of critical data files and major litigation costs and penalties. Although this issue is mentioned together with the responses under the issue of auditability and audit trail, it has different implications and is important enough to be an issue on its own. As shown in Appendix D, there are 2 responses that can be directly 59

classified under this issue. The respondents affirm that the system access logs and complete records of all EDI transactions in the form of electronic data files need to be maintained for the complete fiscal period to satisfy audit purposes.

The professional

literature confirms that an electronic record management system needs to be established to satisfy legal, audit, and management requirements. As Decker [1991] warns, ineffective records management can lead to such risks as loss of critical data files and major litigation costs and penalties.

5.3 Round 2 Results In the second round, a twenty-one item questionnaire was sent to 33 participants. Each participant was asked to rate each issue in terms of its importance using the scale from 1 to 10. In this round, 32 questionnaires were returned, resulting in the response rate of 97 percent. It should be noted that both the 2 non-respondents and the 10 respondents in round 1 responded in round 2 (i.e. all selected participants in round 1 responded in round 2). The only one non-respondent in the second round was not selected to participate in the first round. This particular participant had to go overseas before having the opportunity to answer the round 2 questionnaire and was not scheduled to return until after the round 2 cut-off date. 5.3.1 The Rating of Round 2 Issues Whereas Appendix E shows detailed round 2 results, Table 2 summarizes the rating of key EDI audit issues by this group of experts. The group's mean responses of

60

each issue shown in the second column are used in ranking the importance of the issues in the first column. Moreover, because the spreads or standard deviations (stdev) of the mean scores should also be taken into account when considering the importance of the issues, they are provided in the third column. T A B L E 2-RATING O F ROUND 2 ISSUES Rank

Mean

Stdev

1

9.34

0.96

Controls Over EDI Network

2

8.72

1.35

Backup, Disaster Recovery and Contingency Plans

3

8.47

1.66

Auditability and Audit Trail

4

8.06

1.69

Audit Involvement during the System Development

5

7.81

1.70

Legal and Audit Evidence

6

7.72

1.55

EDI Contracts

7

7.69

1.67

EDI Records Retention

8

7.41

1.77

Third Party EDI Services

9

7.31

1.47

Auditor Education and Training

10

7.09

1.76

Audit Techniques

11

6.94

1.66

Auditor Skills (Skills required of auditors)

12

6.63

2.03

Audit Focus

13

6.53

2.34

Audit Scope (Boundary of Audit)

14

6.50

2.06

Audit Risk Assessment

15

6.44

1.78

Audit Responsibility in Evaluating Controls

16

6.31

2.05

Collaboration Among Auditors of EDI Parties

17

6.28

1.89

Changing Roles of Auditors

18

6.25

2.19

Timing of Audit Tests

19

6.22

1.34

Audit Tools

20

5.50

1.82

Relationship Among Company's Auditors

21

4.28

2.00

Audit Reporting (Periodic Versus On-Line)

Issue

61

5.3.2 Additional Issues Identified in Round 2 As a procedure to obtain as complete an issue list as possible, the questionnaire in round 2 provided a final opportunity for participants to contribute additional issues of importance. Four additional issues were identified at this stage. These issues and their rationale are stated in Table 3. TABLE 3 - ISSUES IDENTIFIED IN ROUND 2 Issue

Rationale

Form of Audit Assurance

The closer interrelationships established between trading partners in an EDI network will affect business and financial risks. Therefore, auditors should reevaluate the types of assurance required by the public and formulate suitable audit procedures and related opinions to satisfy these needs.

Professional Support for Practising Auditors

The auditor has to face many new issues when auditing an EDI system. Thus, professional organizations such as EDPAA, IIA, and CICA should take a proactive approach to providing reference materials and training opportunities to help practising auditors understand and deal effectively with the EDI environment, its risks and control measures.

Inconsistent EDI Approaches

Inconsistent EDI approaches (used by various EDI trading partners) can lead to operational and administrative problems resulting in missed business opportunities, additional costs, and weakened internal controls. Therefore, auditors must be aware of inconsistencies and provide direction to management.

The Network and Ownership of Data

As EDI systems develop, the sharing of common data/programs will increase and the information flow that the auditor needs to understand will change. Auditors must take part in defining information flows and boundaries to data ownership. (This will also help draw legal boundaries among parties in a large integrated EDI system).

5.4 The Comparison of Round 1 and Round 2 Results Note that all seven issues identified by the respondents in the first round are in the top eight rank in the second round [Table 2]. For example, "Controls Over EDI 62

Network", the issue stated most often by the respondents in the first round, was the most important issue in the second round. This result indicates a strong consistency between the first expert group and the larger panel in reporting the issues of importance. Also, four issues from the literature review, "Audit Involvement during the System Development", "Auditor Education and Training", "Audit Techniques", and "Auditor Skills" round out the top eleven issues in the second round. Therefore, the first round can be counted as being successful in revealing most of the potentially important EDI audit issues in the Vancouver area, and the supplementary issue generation technique is useful in making the list more inclusive.

5.5 Round 3 Results In the third round, the questionnaires with a list of 21 original issues and 4 additional issues from the second round were sent to the sample of 33 participants. As in the second round, each participant was asked to rate each issue in terms of its importance using the scale from 1 to 10. No further opportunity was provided for respondents to add new issues. In this round, 33 questionnaires were returned resulting in the response rate of 100 percent. It should be noted that the only one participant who was not selected to participate in the first round but was selected and did not respond in the second round, did respond in the third round. As a result, this participant rated the issues only once in the final round without rating in the previous two rounds. In summary, among 33 respondents in the third round, 10 had responded in all three rounds, 22 had responded in both the second and the third round, and 1 responded only in the final round.

63

T A B L E 4 - ROUND 3 RATING O F 21 ORIGINAL ISSUES Rank

Mean

Mean diff

Stdev

1

9.44

-

0.65

Controls Over EDI Network

2

8.82

0.62

1.09

Backup, Disaster Recovery and Contingency Plans

3

8.70

0.12

1.09

Auditability and Audit Trail

4

8.15

0.55

1.52

Audit Involvement during the System Development

5

7.83

0.32

1.30

Legal and Audit Evidence

6

7.50

0.33

1.44

EDI Records Retention

7

7.42

0.08

1.33

EDI Contracts

8

7.30

0.12

1.49

Audit Techniques

9

7.26

0.04

1.57

Third Party EDI Services

10

7.21

0.05

1.32

Auditor Education and Training

11

7.02

0.19

1.31

Auditor Skills (Skills required of auditors)

12

6.45

0.57

1.71

Audit Risk Assessment

12

6.45

0.00

1.42

Audit Responsibility in Evaluating Controls

14

6.39

0.06

2.00

Audit Scope (Boundary of Audit)

15

6.38

0.01

1.46

Audit Focus

16

6.24

0.14

0.99

Audit Tools

17

6.14

0.10

1.80

Timing of Audit Tests

18

6.09

0.05

1.82

Collaboration Among Auditors of EDI Parties

19

6.05

0.04

1.56

The Changing Roles of Auditors

20

5.27

0.78

1.50

Relationship Among A Company's Auditors

21

3.91

1.36

1.40

Audit Reporting (Periodic Versus On-Line)

Issue

64

T A B L E 5 - ROUND 3 RATING O F 25 FINAL ISSUES Mean

Mean diff

Stdev

1

9.44

-

0.65

Controls Over EDI Network

2

8.82

0.62

1.09

Backup, Disaster Recovery and Contingency Plans

3

8.70

0.12

1.09

Auditability and Audit Trail

4

8.15

0.55

1.52

Audit Involvement during the System Development

5

7.83

0.32

1.30

Legal and Audit Evidence

6

7.50

0.33

1.44

EDI Records Retention

7

7.42

0.08

1.33

EDI Contracts

8

7.30

0.12

1.49

Audit Techniques

9

7.26

0.04

1.57

Third Party EDI Services

10

7.21

0.05

1.32

Auditor Education and Training

11

7.02

0.19

1.31

Auditor Skills (Skills required of auditors)

12

6.55

0.47

1.92

Professional Support for Practising Auditors

13

6.45

0.10

1.71

Audit Risk Assessment

13

6.45

0.00

1.42

Audit Responsibility in Evaluating Controls

15

6.39

0.06

2.00

Audit Scope (Boundary of Audit)

16

6.379

0.011

1.46

Audit Focus

17

6.375

0.004

1.76

Inconsistent EDI Approaches

18

6.28

0.095

1.98

Network and Ownership of Data

19

6.24

0.04

0.99

Audit Tools

20

6.14

0.10

1.80

Timing of Audit Tests

21

6.09

0.05

1.82

Collaboration Among Auditors of EDI Parties

22

6.05

0.04

1.56

The Changing Roles of Auditors

23

5.69

0.36

2.18

Form of Audit Assurance

24

5.27

0.42

1.50

Relationship Among Company's Auditors

25

3.91

1.36

1.40

Audit Reporting (Periodic Versus On-Line)

Rank

Issue

65

5.5.1 Rating of the Original 21 Issues Whereas Appendix F shows detailed round 3 rating results of the original 21 EDI audit issues, Table 4 summarizes the final rating and ranking of these issues. The group's mean responses of each issue are used to determine the ranking of importance of the issues in the first column. These mean responses are shown in the second column while their corresponding standard deviations (stdev) are shown in the fourth column. The mean differences between adjacent pairs of issues are shown in the third column. It should be noted that a tie occurs in the final ranking of the 12th issue. 5.5.2 Rating of the Final 25 Issues While Appendix G shows detailed round 3 results for the final 25 EDI audit issues, Table 5 summarizes the final rating and ranking of these issues. As in the previous table, the group's mean responses of each issue are used to determine the ranking of importance of the issues in the first column. These mean responses are shown in the second column while their corresponding standard deviations (stdev) are shown in the fourth column. The mean differences between adjacent pairs of issues are shown in the third column. The 4 issues added by the respondents in the second round are shown in bold type. It should also be noted that a tie occurs in the final ranking of the 13th issue. The rating results in Table 4 and Table 5 show that the four issues which were added in the second round have no effect on the judgment of the eleven most important EDI audit issues. The reason is that none of the four issues takes place in the top eleven rank in the final round. However, these issues do have an observable effect on the rating and ranking of the 12th to 20th issues. The effect of these additional issues is discussed

66

in more details in Section 5.7 of this Chapter.

5.6 The Interpretation of the Results Caution should be exercised in interpreting the absolute order of issues obtained by the Delphi approach. As presented in both round 2 and round 3 of this study [Table 2,4, and 5], differences in mean ratings are negligible in many cases, given the size of the corresponding standard deviations (stdev) or the spreads of individual scores from the group mean. Consequently, additional revisions of the ordering can be anticipated if the study is repeated or the sample size is enlarged. Therefore, it would be prudent not to have absolute confidence in the current ordering of the issues. In a traditional method, statistical tests on the significance of the ordering would be useful. However, in the Delphi approach the statistical assumptions of normality and independence are violated making such an analysis inappropriate. As argued by Kiudorf [1991], "Despite the lack of an appropriate statistical test, it is possible to make the general statement that 'as the distance between issues increases, the likelihood of incorrect ordering decreases. '" [p. 70]. Using this argumentation and the mean group responses of the issues, we can be quite certain that issue 1 through 11 are in the group of highest priorities [Table 5]. 25

Although we can not be absolutely certain about the ordering of issues 7 through 10, we can be confident that issue 1 should be placed first and issue 2 and issue 3 should be placed before issue 4, 5, and 6. On the other hand, with comparable sizes of standard deviations, the mean scores of issues 23 through 25 are observably lower than those of the The fact that the top eleven issues remain the same in round 2 and round 3 also affirms the importance of these eleven issues. 25

67

preceding issues, making us quite confident that these are issues of lower priorities. In short, although it must be acknowledged that the exact ordering of the issues may lack strong statistical support, the eleven most important issues as determined by the group mean ratings are acceptable because they are resulted from the study which follows the well-established protocol in issues research.

5.7 The Comparison of Round 2 and Round 3 Results The round 3 results show a high degree of consistency with the findings from round 2. In both rounds, the eleven most important issues remain the same. Although the ranking orders of issues 6 through 10 change slightly, the five most important issues remain identical. In addition, the two issues of least importance, "Audit Reporting" and "Relationship Among A company's Auditors", remain the same in both rounds. T A B L E 6 - T H E TOP E L E V E N ISSUES IN ROUND 2 AND ROUND 3 EDI Key Audit Issues Controls Over EDI Network Backup, Disaster Recovery and Contingency Plans Auditability and Audit Trail Audit Involvement during System Development Legal and Audit Evidence EDI Records Retention EDI Contracts Audit Techniques Third Party EDI Services Auditor Education and Training Auditor Skills (Skills required of auditors)

Rank in Round 2 1 2

Group Mean 9.34 8.72

Rank in Round 3 1 2

Group Mean 9.44 8.82

3 4

8.47 8.06

3 4

8.70 8.15

5 7 6 10 8 9 11

7.81 7.69 7.72 7.09 7.41 7.31 6.94

5 6 7 8 9 10 11

7.83 7.50 7.42 7.30 7.26 7.21 7.02

68

Table 6 illustrates the change pattern of issue ranking and rating for the eleven most important issues in round 2 and round 3. It can be observed from Table 6 that an opportunity to reflect on round 2 group ratings results in minor changes to the rating and ranking of the eleven most important issues in round 3. For all top five issues, the mean group responses increased while the standard deviations decrease. The reason may be that, being confirmed by peer ratings, the respondents felt more confident of the importance of these issues and gave higher ratings in the third round. The more noticeable change occurs in the rating and ranking of the top 20 issues. In round 3, three of the four issues which are added in round 2 are rated in the 12th to 18th rank. As shown in Table 5, "Professional Support for Practising Auditors" is ranked 12th, "Inconsistent EDI Approaches" is ranked 17th, and "The Network and Ownership of Data" is ranked 18th, surpassing the importance of some of the original issues in round 2. These indicate that opinions and concerns expressed by expert peers do have significant effects on the rating of the top 20 issues in the subsequent round. The respondents had the opportunity to be reminded of certain important issues which they did not think of in the first place. Further, these results may also indicate that, in addition to its own advantages, the Delphi process utilized in this study captures some benefits similar to those of an open discussion or committee approach. Also, because none of the four issues which were added in round 2 was ranked in the top eleven, it may be concluded that the first round open-ended questionnaire and the compilation and analysis of existing literature have been successful in identifying the eleven most important issues.

69

5.8 Movement Towards Consensus One of the advantages of using the Delphi technique is that it encourages participants to reach a consensus on the issues of greatest importance. Measuring the change in the standard deviations of mean ratings between subsequent rounds is an appropriate method for showing movement towards consensus. A declining mean standard deviation indicates that participants are reflecting on the issues and revising their ratings to correspond more closely with their colleagues. For the 21 original issues measured for importance, the mean standard deviation in round 2 is 1.75 whereas that in round 3 is 1.42. This decline in spread means the scores are clustered more closely about the center and thus indicates a movement towards consensus. A further indicator of the trend towards consensus is illustrated in figure 1 - the statistical summary and graphical display of the participants' rating of the eleven most important EDI audit issues in round two and round three. The box plot is used because it is useful for identifying quickly the median, hinges, and outside values of the issue rating in each round and it makes enables comparison of rating of issues in two different rounds on the same scale. The median and interquartile range (IQR) are not distorted by extreme scores like the mean and the standard deviation and thus are suited for summarizing spotty numbers. The plot shows the median and the IQR of the first, fourth, and eleventh pair of issues were unchanged between round two and round three. Moreover, the unusual responses (*) which appeared in round two of the first and fourth issues disappeared in

70

round three, indicating that the respondents which gave unusual low scores in relation to the group norm in round two increased their scores to meet the group norm in round three. In addition, although some unusual responses remained, the IQR and the whisker length of the third, fifth, seventh, eighth, and tenth pair of issues became noticeably shorter in round three. Collectively, these incidents indicate a movement towards group consensus on the importance of EDI audit issues. While it is possible that additional rounds might have improved the degree of consensus, it is highly unlikely that perfect agreement would ever be attained because the study respondents would continue to maintain certain independent views.

As happened

in the third round, two respondents maintained exactly the same rating as they did in the second round for every original issue. In addition, such factors as their position, type of audit (internal, external), level of experience, industry, and other organizational aspects may influence the respondents' views of the importance of a given issue.

71

ZL

5.9 Study Participants This section discusses the various characteristics of the 33 information systems audit experts recruited to participate in this study. Among the 33 participants, 12 were requested to participate in all three rounds and 21 were requested to participate in round 2 and round 3 only. Table 7 displays the actual participation of these 33 individuals. T A B L E 7 - R E S E A R C H SUBJECTS: PARTICIPATION PATTERN Number of Rounds Responded 3 2 1

Round Responded Round Round Round Round Round Round Round

1 1 1 2 1 2 3

Number of Respondents

2 3 2 3 3

10 0 0 22 0 0 1 33

TOTAL

During the final round, an effort was made to collect some background information and descriptive data about the individual respondents and organizations reflected in this research project. The questionnaire for this purpose is exhibited at the end of Appendix C. Because the round 3 findings are the final research results, it is appropriate to use the information obtained from the respondents in this round as a basis in providing a better understanding of selected characteristics of the research subjects. 5.9.1 Organizational Category T A B L E 8 - R E S E A R C H SUBJECTS: ORGANIZATIONAL C A T E G O R Y Category Public Accounting Firm

Number of Respondents 7

Percent of Total 21.21

Audit Services Bureau

1

3.03

Incorporated Company Crown Corporation

13

39.40

6

18.18

Government Agency

3

9.09

Academic Institutions

3

9.09

33

100.00

Total

73

The 7 respondents in the accounting firm category are from 6 firms which are ranked in the eight largest accounting firms in Greater Vancouver . The respondents in 26

the incorporated company category are from companies in variety of industries (financial, food, insurance, manufacturing, and retail). 5.9.2 Position and Primary Area of Responsibility T A B L E 9-RESEARCH SUBJECTS: T Y P E O F POSITION Type of Position

Number of Respondents

Percent of Total

Internal Auditor

24

72.73

External Auditor

9

27.27

33

100.00

Total 5.9.3 Professional Designations

T A B L E 10 - R E S E A R C H SUBJECTS: PROFESSIONAL DESIGNATIONS Number of Respondents

Designations Chartered Accountant (CA)

Percent of 33 Total Respondents 27

20

60.61

Certificate in Data Processing (CDP)

1

3.03

Certified General Accountant (CGA)

7

21.21

Certified Internal Auditor (CIA)

6

18.18

Certified Information Systems Auditor (CISA)

14

42.42

Certified Management Accountant (CMA)

4

12.12

Others (CIPS's ISP and F C C A L

2

6.06

As reported by Peter Brow in Business in Vancouver as of June 4, 1991 (p. 29). Firms were ranked on number of chartered accountants employed by the firms. 26

Because an individual respondent may possess more than one designation, the sum of the percent numbers in this column may be greater than 100. 27

74

5.9.4 Areas of Audit Expertise T A B L E 11 - R E S E A R C H SUBJECTS: AREAS O F AUDIT EXPERTISE Number of Respondents 20

Area of Expertise External Audit

26

Internal Audit Information Systems Audit

20 11

General Audit Others (Systems Analyst, IS Security, V F M and Fraud Audit)

3

Percent of 33 Total Respondents 60.61 78.79 60.61 33.33 9.09

5.9.5 Level of Audit Experience T A B L E 12 - R E S E A R C H SUBJECTS: L E V E L O F EXPERIENCE Years of Experience

Auditing: No. of Respondents

Information Systems Auditing : No. of Respondents

None

2 2

6

1-3 4-5 6-10

2 8

6 7 7

11-15

5 10 4

3 3 1

16-20 Over 20

5.9.6 Background in EDI Technology 5.9.6.1 Engagement in an EDI Project T A B L E 13 - R E S E A R C H SUBJECTS: E N G A G E M E N T IN A N EDI PROJECT Engagement in EDI Project

Number of Respondents

Percent of 33 Total Respondents

Yes

11

33.33

No No answer

21

63.64

1

3.03

Total

33

100.00

Because an individual respondent may specialize in more than one area of auditing, the sum of the percent numbers in this column may be greater than 100. 28

75

One of the respondents who report as have never been engaged in an EDI project states that he will be involved in his first EDI project in the next few months. Further, the respondents who have the experience of being engaged in EDI projects report their capacities as "(being involved in) system development", "(is responsible for) information forum", "review of a pilot project in purchasing department", "preliminary discussion with vendors to streamline account payable process", "organize and monitor the progress (of the EDI project)", "part of information system strategic plan", "presently involved in planning stages with a few clients", "auditor", "general control review for financial statement", "general review of completed work by internal audit", and "internal auditor". 5.9.6.2 Self-report Level of knowledge and Understanding of the EDI Technology T A B L E 14 - R E S E A R C H SUBJECTS: L E V E L O F K N O W L E D G E O F EDI T E C H N O L O G Y Level

Number of Respondents

Good Working Knowledge

2

Average Knowledge

24

72.73

Little Knowledge No answer

6 1

18.18

Percent of 33 Total Respondents 6.06

3.03 100.00

33

Total

5.9.6.3 Primary Source(s) of knowledge and Understanding of the EDI Technology T A B L E 15 - R E S E A R C H SUBJECTS: P R I M A R Y SOURCE(S) O F K N O W L E D G E O F EDI T E C H N O L O G Y Source

Number of Respondents 7

First hand experience Professional Literature

30

Oral Communication/Discussion Group on EDI

19

Course and Seminar on EDI

8

Others (EDI Software Vendors)

2 76

5.8.6.4 EDI audit manual or guideline In the background information section, a question was asked to obtain information on the use and development of EDI audit manual or guideline. As can be expected from the number of organizations in Vancouver which have EDI systems in operation at this point of time, 3 firms have the EDI audit manuals in use, 4 firms are in process of developing ones, and the rest neither has one in use nor is in the process of development. This information is useful in understanding the progress of the respondents' firms in standardizing audit procedures for EDI systems.

77

Chapter 6. Conclusions

6.1 Summary of Findings and Conclusions The primary goal of this research is to identify the most important EDI audit issues as viewed by the Vancouver IS audit community. The resulting eleven most important issues are summarized in Table 16. Table 16 - The Top Eleven Issues in Vancouver Final Rank

Issue of Importance

1

Controls Over EDI Network

2

Backup, Disaster Recovery and Contingency Plans

3

Auditability and Audit Trail

4

Audit Involvement during the System Development

5

Legal and Audit Evidence

6

EDI Records Retention

7

EDI Contracts

8

Audit Techniques

9

Third Party EDI Services

10

Auditor Education and Training

11

Auditor Skills (Skills required of auditors)

The most important issues for this group of audit experts concern controls and security of EDI systems. This is not unexpected because by nature the fundamental goal of auditing is to evaluate and ensure that clients'systems are secure and adequately controlled. These security and control issues are significant for auditors in many aspects.

78

They play an important part in determining a company's ability to continue as a goingconcern and as a consequence, they greatly affect business and audit risks. Also, they form the main basis for planning the entire audit work in each engagement. The researcher would also like to make a personal observation here that this theme of audit concerns seems to be in concordance with general public concerns.

Having

opportunities to discuss the EDI technology with people in different occupations, the researcher noticed that the first and major concern expressed by these people are the security and reliability of EDI systems. The researcher thus views the resulting audit concerns as being in the right direction and regards this circumstance as an opportunity for auditors to keep up with the shifting trends and to continually dignify their profession by capably fulfilling their important roles as the 'public watchdog' on this relatively new information technology.

6.2 Generalizeability of Results Whereas the results of this study provide an indication of the EDI audit priorities of the Vancouver's IS auditors, the findings may be applicable in other North American cities. The rationale is that there is no significant diversity in this continent on both general dimensions such as language, politics, economy, educational systems, and specific dimension such as organizational culture, industrial activities, advancements in computer and information technology, and accounting and auditing systems. Therefore, one may induce that the concerns of Vancouver IS auditors may be comparable with those of their counterparts in other Canadian and American cities. In addition, as suggested by Kiudorf

79

[1991], in determining whether the most important issues in one specific setting apply in other settings, it is useful to consider the issues themselves. Of all the EDI audit issues measured for importance in this study, none appears to be rigidly unique to the Vancouver audit environment. Most of the issues tend to be universal in nature and can be seen as concerns for IS auditors in many communities of the auditing profession. These considerations provide some measures of face validity for the claim that the results may be generalizeable. In summary, this study identified the most important EDI concerns in a single Canadian city. The nature of the findings suggests that these concerns probably apply to other industrialized environments, but additional research would be required to establish the actual validity of generalizing the results.

6.3 Limitations of Research Study Although attempts were made to preserve the rigour of the Delphi research, several constraints did exist. The limitations of this research are summarized below: M Non-Random Participant Recruitment: there were no pre-existing lists or databases of qualified candidates for random sampling. Therefore, participant recruitment was accomplished largely through peer recommendations and personal contacts. iii/ The number of auditors who have first-hand experience with EDI systems: the limited number of organizations in Vancouver which already have EDI systems in operation make it difficult to obtain audit experts with high levels of specific experience in auditing EDI systems. iv/ Statistical Significance of Issue Ordering: as is true in previous Delphi issues studies, the specific ordering of consecutive issues of importance may not have statistical significance.

80

y I Comparisons with other studies: this project is the first attempt to conduct a Delphi survey research of this topic and nature. Therefore, data from previous studies is generally not available for comparative purposes. 6.4 Directions for Future Research Whereas this study is a first attempt to identify the priorities and concerns of IS auditors in a Canadian context, there are a number of additional research projects that would be useful in improving our understanding of the topic. In particular, it would be useful to solicit views from IS auditors in industrialized settings other than Vancouver; this would clarify whether auditors in Canada and other industrialized nations face a common set of concerns.

EDI technology has been largely directed in companies in North

America, Europe, and Australia. Therefore, these areas would be of particular interest in a supplementary study. A repetition of the Vancouver study in a few years would also be useful. The continuing advancements of computer and telecommunication technologies will change the face of the EDI technology. Also, when EDI systems are more common in Vancouver, and IS auditors have more experience with EDI systems, it would be of interest to track the evolution of EDI audit priorities over time. This will be useful in maintaining a current understanding of important issues and trends. Also, future researchers should be encouraged to utilize the Delphi methodology in order to preserve the comparability of the results.

81

BIBLIOGRAPHY Baker, Carol, (a) "EDI in Business." Accountancy (UK) 107 (Apr 1991): 121124. Baker, H . Richard, (b) EDI: What Managers Need to Know about the Revolution in Business Communications. PA: TAB Professional and Reference Books, 1991. Barrett, Michael J. "External Control." Internal Auditing 6 (Summer 1990): 6268. Boughton, Charles M . "Combining the Efforts of Internal and Independent Auditors: Using EDP to Maximize Audit Resources." Woman CPA 49 (January 1987): 26-27. Brown, Janet. "Alarmed by Inadequate IT Training." The EDP Auditor Journal 1 (1991): 11-12. Burns, David C. and Sorkin, Horton Lee. "EDI Security and Controls." Bank Management 67 (Feb 1991): 27-31. Chalmers, Leslie S. "New Technology Introduces New Risks." Accounting & EDP 5 (Winter 1990): 28-30.

Journal of

Cowan David, "EFT/EDI-Electronic Age Poses New Legal Problems.", Euromoney (UK). Corporate Finance Supplement. July 1990, pp.27-30. Craig-Bourdin, Margaret. "The Here and Wow!" C A Magazine (Canada) 122 (Aug 1989): 20-30. Damyanoff Dan, "EDI and EDIFACT: The Future's Cornerstones.", Global Trade, vol: 111, Issue 6, June 1991, pp. 35,41. Decker, David L . "Record Retention - A Critical Internal Control." The EDP Auditor Journal 1 (1991): 61-68. Delbecq Andre L . , Van de Ven Andrew H . , and Gustafson David H . Group Techniques for Program Planning: A guide to Nominal Group and Delphi Processes. Glenview, Illinois: Scott, Foresman and Company, 1975. Dexter Albert S., Marius Janson A . , Kiudorf Enn, and Laast-Laas Juri. "Key Information Technology Issues in Estonia: Definition and Measurement." Working Paper 92-MIS-001, University of British Columbia. March 1992. 82

Dunmore, David B. "Farewell to the Information Systems Audit Profession." Internal Auditor 46 (Feb 1989): 42-48. Eckerson, Wayne. "EDI Susceptible to Costly Order Errors. " Network World 7 (Sep 17, 1990): 23-24. Elliott, Robert K . "Auditing in the 1990s: Implications for Educational and Research." California Management Review 28 (Summer 1986): 89-97. Emmelhainz, Margaret A . The Impact of Electronic Data Interchange on the Purchasing Process. Ph.D. diss., The Ohio State University, 1986. Emmelhainz, Margaret A . Electronic Data Interchange: A Total Management Guide. New York: Van Nostrand Reinhold, 1990. Fowles Jib. Handbook of Futures Research. Westport, Connecticut: Greenwood Press, 1978. Gallegos, Frederick and Bieber, Douglas W. "Emerging Technology and Information Systems Auditing." Journal of Accounting & EDP 3 (Summer 1987): 47-56. Gardner Elizabeth, "A direct line Between Buyer and Supplier.", Modern Healthcare, vol: 19, Issue 11, Mar 17, 1989, pp: 26-28. Gilhooley, Ian. "Emerging Technologies and Auditing: IIA's Advanced Technology Forum." Internal Auditor 44 (Feb 1987): 50-54. Goldner, Gary. "EDP Auditing with a Small Staff." Journal of Accounting & EDP 2 (Winter 1987): 36-42. Hansen James V . and Hill Ned C , "Control and Audit of Electronic Data Interchange.", MIS Quarterly. December 1989, pp. 403-413. Helms, Glenn L . "Career Opportunities for Information Systems Auditors." Journal of Accounting & EDP 2 (Fall 1986): 9-12. Hinge Kathleen C. Electronic Data Interchange. Management Association, 1988.

83

New York: American

Holstrum Gary L . , Mock Theodore J., and West Robert N . The Impact of Technology on Auditing—Moving Into the 21st Century. Altamonte Springs, Florida: The Institute of Internal Auditors Research Foundation, 1988. Holstrum Gary L . , Mock Theodore J., and West Robert N . , "Information Systems in the 1990s.", Internal Auditor, vol 47, February 1990, pp. 32-37. Hogarth, Dennis. "How Does IT All Tie Together?" CA Magazine (Canada) 119 (Dec 1986): 75-77. ICAEW (Institute of Chartered Accountants in England and Wales). IT and the Future of the Audit. London: Progress Fine Art Graphic Services Limited, 1989. Jancura, Elise G.; Lehman, John; Baab, John G.; Gilges, Robert D . ; Kinard, James C ; Overbey, John T.; Robins, Richard S.; Stewart, Trevor R.; Wasserman, Arnold. "Widespread Computerization and Automation of Business Operations-Part II- Impact of New Development on the Profession." Woman CPA 48 (Oct 1986): 26-31. Kavan Bruce C. The Adoption of Inter-organizational Systems: The Example of Electronic Data Interchange. Ph.D. diss., University of Georgia, 1991. Kimberley, Paul. Electronic Data Interchange. New York: McGraw-Hill, Inc, 1991. KiudorfEnn. Key MIS Issues for Management: An Eastern European Perspective. M.Sc. Thesis, University of British Columbia, 1991. Kothari, Nick. "Auditing's Role in Systems Development." (Canada) 121 (Oct 1988): 55-60. Lanford H . W. Technological Forecasting Methodologies. Management Association, Inc, 1972.

C A Magazine N.p.: American

Lewis, Barry. "Electronic Authorization- The Next Wave In Automation." Journal of Systems Management (March 1989): 28-32. Lipsett Carol, "EDI implications for security and audit controls.", CIPS Review. August/september 1989, pp.20-21. Martino Joseph P. Technological Forecasting For Decision Making. New York: Elsevier Science Publishing Co., Inc, 1983.

84

McDonald, Hal. "EDI Implementation Consideration." The EDP Auditor Journal 1 (1990): 43-46. Moeller, Robert R. "Using a CPA Firm as an Internal Auditor." Journal of Accounting & EDP 2 (Fall 1986): 20-24. Norris Daniel M . and Waples Elaine, "Control of Electronic Data Interchange Systems.", Journal of Systems Management, vol: 40, Issue 3, March 1989, pp.21-25. Powell Kevin D. Security and Control of Electronic Data Interchange Systems. M.Sc. Research Project: California State Polytechnic University, Pomona, June 1991. Rhodes, Wayne. "The Audit Experience." Infosystems 34 (July 1987): 18-20. Sadhwani Arjan T, Kim Ill-Woon, and Helmerci John, "The Impact of Electronic Data Interchange on Internal Controls.", Journal of Accounting and EDP. Fall 1989, pp. 23-31. Sadhwani, Arjan T, Ill-Woon Kim, and John Helmerci. "EDI's Effect On Internal Controls." EDPACS XVII no. 1 (July 1989): 1-11. Schatz Willie. "EDI: Putting the Muscle in Commerce & Industry. " Datamation. 15 March 1988: 56-64. Staats, E. "Auditing as We Enter the 21st Century-What New Challenges Will Have to be Met." Auditing: A Journal of Practice and Theory 1 (Summer 1981): 1-11. Tsay Bor-Yi, "System Controls for Electronic Data Interchange.", CPA Journal, vol: 59, Issue 6, June 1989, pp. 70-73. Utter, Allen C. and Bertram, Timothy R. "Revisiting "A Farewell to the Systems Audit Profession." Internal Auditor 46 (Jun 1989): 70-72. West Robert N . The Impact of Paperless Systems and Other Technological Changes Upon Auditing. Ph.D. diss., University of Southern California, 1988. Willits, Stephen D. "Information Technology: Decisions, Decisions, Decisions." CA Magazine (Canada) 123 (Aug 1990): 51-54.

85

Wise

Timothy M . , "EDI: Progressing Toward the Paperless Office.", Internal Auditing, vol: 5, Issue 1, Summer 1989, pp. 75-81.

Wise, Timothy M . "Looking at the Systems Development Audit." Internal Auditing 6 (Summer 1990): 69-74. Wright, J. Benjamin, (a) EDI and American Law : A Practical Guide. Alexandria, Verginia: TDCC:The Electronic Data Interchange Association, 1990. Wright, J. Benjamin. The Law of Electronic Commerce. Boston: Little, Brown and Company, 1991. Wright Margaret, (b) "Accounting in a Paperless Office.", Australian Accountant (Australia). vol:60, Iss:7, August 1990, pp. 44-48. Yang David C , "The Effect of EDGAR on Auditing Practice.", Ohio CPA Journal. Vol: 49, Issue 4, Winter 1990, pp.49-50.

86

APPENDIX A ROUND 1 DELPHI QUESTIONNAIRE FOR AUDIT EXPERTS

87

tacuity of Commerce & Business Administration

March 16, 1992

The University of British Columbia 2053 Main Mall Vancouver, B.C. Canada V6T 1Z2

James Topham Peat Marwick Thome P.O. Box 10426 Pacific Center 777 Dunsmuir Street Vancouver, B.C. V7Y 1K3

Tel: (60-;) 822-8500 Tax: (604) 822-8489

Dear Mr. Topham: UBC, in conjunction with the Vancouver Chapters of the EDPAA and the IIA, is conducting a Delphi study to investigate the impact of Electronic Data Interchange (EDI) on the audit process. The goal of our project at this point is to identify on the basis of expert opinions the key issues of EDI audit impact. You have been selected to participate in this study because of your considerable expertise in information systems auditing and interest in EDI control and audit issues. The ultimate objective of this Delphi study is to obtain consensus (or response stability) from the panel of experts on important EDI audit issues. The issues indicated by you and your peers will be combined with those stated in the literature to form the list of important issues to be rated by a group of Vancouver's IS auditors in the next questionnaire. Attaining the stated objective normally requires two or three iterations. Your participation therefore will be limited to two or three questionnaires. We believe you will benefit from participation in this research project. As you are aware, EDI is an increasingly popular technology which has the potential to significantly change business information systems and the practice of auditing. This potential impact makes this study worthy of your time and attention. Furthermore, you will benefit by being able to compare and contrast your opinions with those of other experts in your field. In addition, the knowledge of important EDI audit issues will assist you and your firm in directing efforts and resources to the most critical areas. Therefore, we ask your assistance in identifying important EDI audit issues by completing the attached open-ended questionnaire. May we have your reply by March 27? Please be assured that your individual responses will be kept strictly CONFIDENTIAL. You will of course be able to receive the eventual results of the study which will not identify specific participants. If you have any questions, please feel free to contact Professor AI Dexter at 8228380. We thank you very much for your cooperation.

Sincerely,

Sincerely,

Sincerely,

James W. Topham President, EDPAA Vancouver Chapter

Angela M . Louie President, IIA Vancouver Chapter

Albert S. Dexter Associate Professor UBC Faculty of Commerce

88

NOTES TO T H E RESPONDENTS: 1) The term "auditors", on its own, is used to signify all types of auditors. References to internal, external, and specific types of auditors, are made explicitly in the questionnaire. 2) Although there are variations in the definition of EDI, the following definition is adopted for the purposes of this research project: Electronic data interchange (EDI) is the intercompany, computer-tocomputer exchange of business documents in standard formats. Through EDI, such common business forms as invoices, bills of lading, and purchase orders are transformed to a standard data format and electronically transferred between trading partners. 3) EDI should be differentiated from Electronic Fund Transfer (EFT) which refers to the transfer of value electronically from buyer to seller as assisted by a financial intermediary, usually a bank. Because of the complementary objectives of EDI and EFT (the elimination of paper in business transactions), many organizations involved in electronic payments are proponents of EDI.

89

1992 DELPHI STUDY O F K E Y EDI AUDIT ISSUES Round 1 Your Top Five EDI Audit Issues Your Name (Optional) Please list what you feel are the five most important issues in auditing EDI systems. Please list the audit issues in order of their relative importance and kindly give a brief rationale/description of each issue. The importance of the issue should be considered in terms of its impact on different aspects of the audit process. 1) Issue: Rationale:

2) Issue: Rationale:

3) Issue: Rationale:

4) Issue: Rationale:

5) Issue: Rationale:

Thank you very much for your cooperation.

90

APPENDIX B ROUND 2 DELPHI QUESTIONNAIRE FOR AUDIT EXPERTS

91

h acuity of Commerce & Business Administration

April 15, 1992

The Universily of British Columbia 2053 Main Mall Vancouver, BC. Canada V6T 1Z2

Andy Campbell Internal Audit MacMillan Bloedel 925 West Georgia Street Vancouver, B.C. V6C 3L2

Tel. (604) 822-8500 Fax: (604) 822-8489

Dear Mr. Campbell: UBC, in conjunction with the Vancouver Chapters of the EDPAA and the HA, is conducting a Delphi study to investigate the impact of Electronic Data Interchange (EDI) on the audit process. You have been selected to participate in this study because of your considerable expertise in information systems auditing. The objective of this Delphi study is to obtain consensus from the panel of experts on the key issues of EDI audit impact. Attaining this objective normally requires two or three iterations. Your participation therefore will be limited to two or three questionnaires. The next questionnaire will incorporate the results from this questionnaire, and will be sent to you in about one week after the results of this round have been received and analyzed. In order to have your responses included in the upcoming round, please have your responses mailed or fax (822-8489 attn. Al Dexter) to us by April 30. We believe you will benefit from your participation in this research project. As you are aware, EDI is an increasingly popular technology which has the potential to significantly change business information systems and the practice of auditing. This potential impact makes this study worthy of your time and attention. Furthermore, you will benefit by being able to compare and contrast your opinions with those of other leading experts in your field. In addition, the knowledge of important EDI audit issues will assist you and your firm in directing efforts and resources to the most critical areas. Please be assured that your individual responses will be kept strictly CONFIDENTIAL. You will of course be able to receive the eventual results of the study which will not identify specific participants. If you have any questions, please feel free to contact professor AI Dexter at Tel. 822-8380. We thank you very much for your cooperation. Sincerely,

Sincerely,

Sincerely,

James W. Topham President, EDPAA Vancouver Chapter

Angela M . Louie President, HA Vancouver Chapter

Albert S. Dexter Associate Professor UBC - Commerce

9Z

1992 DELPHI STUDY OF KEY EDI AUDIT ISSUES Round 2 Your Rating of EDI Audit Issues Your Name

The issues listed below were obtained by combining the results from round 1 survey with the issues frequently discussed in previous studies and professional reviews. We are interested in determining the degree of importance of these issues based on your opinion. Please indicate your opinion by rating each issue on a scale of 1 to 10, where 10 indicates that the issue deserves the highest priority from the auditing profession and 1 indicates that the issue has the lowest priority. Please use the space provided in front of the statement of each issue to assign your ratings. The more important the issue, the higher the rating. You can assign the same number to more than one issue.

RATING SCALE: Not Important 1 2 3

Moderately Critically Important Important 4 5 6 7 8 9 10

Your Rating Key EDI Audit Issues and their Rationale: Backup, Disaster Recovery and Contingency Plans Rationale: With a company's increasing reliance on EDI network for operational and financial services, auditors must assure that adequate measures for backup and disaster recovery exist. These measures form an indicator of the company's ability to continue as a going-concern. Controls Over EDI Network Rationale: Because weak controls can cause significant financial loss to EDI trading partners, auditors must assure that controls over the EDI network such as access controls, authentication controls, transmission controls, and controls over mailbox are in place. Relationship Among A Company's Auditors Rationale: In an EDI environment, information systems grow in complexity, external auditors may have to rely more on internal auditors, and information systems auditors will be requisite members of audit teams. Suitable audit approaches must be developed to promote and make the best use of this inter-relationships.

93

Your Rating Key EDI Audit Issues and their Rationale: Audit Tools " Rationale: The increased complexity of intercompany automated paperless EDI transactions makes it more difficult, or in some cases impossible, for auditors to test and evaluate network systems with existing audit tools. More powerful tools must be developed to match the growth in sophistication of clients' systems. Auditor education and training " Rationale: Auditors practice today in a significantly different environment from that in the past. The educational curriculum and training requirements for auditors must be updated to reflect technological change and to embody the types of knowledge and the proficiency required of auditors to maintain the profession. Audit Reporting (Periodic versus On-Line) Rationale: Because the high speed of EDI transactions makes information obsolete within a very short time, there is a need for more frequent accounting disclosure. Auditors will have to adjust their reporting procedures to meet the need for timely and accurate information. Legal and Audit Evidence " Rationale: The absence of paper documents and signatures in EDI systems implies the absence of important legal and audit evidence such as proof of authorization and other documentation in paper form. Auditors must assure that equivalent and legally acceptable forms of evidence are established and properly incorporated into EDI systems. The Changing Roles of Auditors Rationale: In an EDI environment, the role of auditing will have to change to meet both the new demands of clients and the needs of the profession. Failure by auditors to assume suitable new roles adequate to meet such demands may lead to the decline of the profession. Audit Focus (Substantive versus Compliance Testing) Rationale: Although there is a general agreement that the focus and the types of audit tests need to change to suit highly automated, paperless intercompany EDI networks, there is no consensus on the direction or specific procedures that should be applied. Therefore, efforts are needed to establish appropriate audit approach. Auditor Skills (Skills required of auditors) Rationale: Because of rapid expansion in the extent, scope, and types of information to be audited in an EDI system, auditors need to be specially trained to acquire certain skills that enable them to maintain a high standard of practice in such an environment.

94

Your Rating Key EDI Audit Issues and their Rationale: Audit Scope (Boundary of Audit) Rationale: Paperless intercompany transactions create a "boundaryless" information system environment, and auditors may be required to audit beyond the traditional boundaries of clients' systems. Therefore, audit scope and responsibilities need to be pre-determined and agreed upon by the parties involved. Collaboration Among Auditors of EDI Parties Rationale: An EDI network involves not only auditors of the company but also auditors of its trading partners and of third parties. Because the security of an EDI system depends on those of others in the network, these auditors are inter-dependent. It is vital that the roles of each party's auditors be determined and the rules of collaboration be established. Audit Involvement during the System Development Rationale: Auditors must take a proactive approach and get involved early in EDI projects to ensure that proper controls and auditability features are designed and incorporated into the systems. Guidance must be established to assist auditors to accomplish this task. Third Party EDI Services Rationale: Different types of EDI networks have different implications for the participating companies and their auditors. Therefore, auditors must evaluate their clients' third party EDI service firms in terms of responsibilities, resources, and abilities to provide, on an ongoing basis, reliable and secure services per contract terms. Audit Risk Assessment Rationale: Audit risks in an automated open EDI network are significantly different from those in a closed system because EDI involves more parties and more diverse exposures. A distinctive approach of audit risk assessment is required, and guidance must be established. Audit Responsibility in Evaluating Controls Rationale: In open EDI networks, auditors, especially internal auditors, may be held responsible for the review and evaluation of external controls (in addition to internal controls). In such circumstances, it is necessary that guidance be developed and standards be established to lead the practice.

95

Your Rating Key EDI Audit Issues and their Rationale:

Auditability and Audit Trail Rationale: Because substantial reduction of paperwork means a possible loss of the audit trail and the consequent inability to conduct an audit, auditors must take actions to ensure the auditability of the EDI system and the availability and the adequacy of an audit trail in proper form. Audit Techniques Rationale: Traditional audit techniques may no longer apply in an EDI environment. Auditors need to develop effective techniques to enable them to describe, evaluate, and test an intercompany information network that includes very few paper documents. Timing of Audit Tests Rationale: Because of the high volume of transactions and the velocity of electronic processing, EDI transactions may have to be reviewed as they occur. Consequently, the audit process must be modified and specific audit standards established. EDI Contracts Rationale: Because EDI contracts are the basis for the company's future dealings with and liabilities to EDI partners, auditors must ensure that terms, services, and responsibilities of each EDI party are clearly defined and the contracts are inclusive and enforceable. EDI Records Retention Rationale: Ineffective records management practices can lead to exposures such as the loss of critical data files; therefore, possible litigation costs and penalties could result. Guidance on EDI records retention must be established and organizational and individual accountability must be clearly defined.

96

Additional Issues In the space provided, please feel free to indicate any additional EDI audit issues which you think important and deserve consideration from the study group. This will help us ensure that the list of issues which we will send to you in the next round is most comprehensive. Your Rating Additional EDI Audit Issues and their Rationale: Issue: Rationale:

_ Issue: Rationale:

_ Issue: Rationale:

Thank you very much for your cooperation. 97

APPENDIX C ROUND 3 DELPHI QUESTIONNAIRE FOR AUDIT EXPERTS

98

Faculty of Commerce & Business Administration

, 1992

The University of British Columbia 2053 Main Mall Vancouver, B.C. Canada V6T 1Z2 Tel: (604)822-8316

Gait Arthur Partner, Deloitte & Touche PO Box 49279, Four Bentall Centre 200-1055 Dunsmuir Street Vancouver, B.C. V7X lp4

Fax:(604) 822-8521

Dear Mr. Arthur: Thank you very much for your participation in the previous rounds of the Delphi study on EDI Key Audit Issues. This is the THIRD AND FINAL ROUND of the study. We have now established a preliminary list of the most important EDI audit issues based upon the responses from you and your audit expert peers. In order to complete this study, we need your assistance one final time so that we can determine the final ranking of the issues. Detailed instructions for this round are provided in the attached questionnaire. Could you please have your responses mailed or faxed (822-8489 attn. A l Dexter) to us by4he" J22fltT33f"M«y'. We would like to reassure that all individual responses will be kept strictly CONFIDENTIAL. If you have any questions, please feel free to contact Professor A l Dexter at Tel. 822-8380. We sincerely wish to thank you in advance for your support and cooperation. We hope that you have found your participation in this research project to be a meaningful experience. If you are interested in having a copy of the final results of the study, please so indicate on the questionnaire. We will be pleased to forward it to you upon completion. Again, thank you very much for making this research project feasible. Sincerely,

President, EDPAA Vancouver Chapter

Angela M . Louie President, IIA Vancouver Chapter

99

Albert S. Dexter Associate Professor UBC

1992 DELPHI STUDY O F K E Y EDI AUDIT ISSUES Round 3 Your Rating of EDI Audit Issues Your Name

The following list of EDI audit issues is presented in the order of importance as determined from the previous round of Delphi questionnaires. The average group rating and your original rating are also provided. Please review these ratings and the accompanying rationale for each issue. Make a final rating decision and record it in the blank space. Please remember that each issue is to be rated on a scale from 1 to 10, where 10 indicates that the issue deserves the highest priority from the auditing profession and 1 indicates that the issue has the lowest priority (the more important the issue, the higher the rating).

RATING SCALE: Not Important 1 2 3

Average Group Rating

Your Original Rating

Moderately Important 4 5 6

Your Final Rating

7

8

Critically Important 9 10

Key EDI Audit Issues and their Rationale:

9.3

Controls Over EDI Network Rationale: Because weak controls can cause significant financial loss to EDI trading partners, auditors must assure that controls over the EDI network such as access controls, authentication controls, transmission controls, and controls over mailbox are in place.

8.7

Backup, Disaster Recovery and Contingency Plans Rationale: With a company's increasing reliance on EDI network for operational and financial services, auditors must assure that adequate measures for backup and disaster recovery exist. These measures form an indicator of the company's ability to continue as a going-concern.

100

Average Group Rating

Your Original Rating

Your Final Rating

Key EDI Audit Issues and their Rationale:

8.5

Auditability and Audit Trail Rationale: Because substantial reduction of paperwork means a possible loss of the audit trail and the consequent inability to conduct an audit, management and auditors must take actions to ensure the auditability of the EDI system and the availability and the adequacy of an audit trail in proper form.

8.1

Audit Involvement during the System Development Rationale: Auditors must take a proactive approach and get involved early in EDI projects to ensure that proper controls and auditability features are designed and incorporated into the systems. Guidance must be established to assist auditors to accomplish this task.

7.8

Legal and Audit Evidence Rationale: The absence of paper documents and signatures in EDI systems implies the absence of important legal and audit evidence such as proof of authorization and other documentation in paper form. Auditors must assure that equivalent and legally acceptable forms of evidence are established and properly incorporated into EDI systems.

7.7

EDI Contracts Rationale: Because EDI contracts are the basis for the company's future dealings with and liabilities to EDI partners, auditors (especially internal) should be involved in negotiation process to help ensure that terms, services, and responsibilities of each EDI party are clearly defined and the contracts are inclusive and enforceable.

7.7

EDI Records Retention Rationale: Ineffective records management practices can lead to exposures such as the loss of critical data files; therefore, possible litigation costs and penalties could result. Guidance on EDI records retention must be established and organizational and individual accountability must be clearly defined.

101

Average Group Rating

Your Original Rating

Your Final Rating

Key EDI Audit Issues and their Rationale:

7.4

Third Party EDI Services Rationale: Different types of EDI networks have different implications for the participating companies and their auditors. Therefore, auditors must evaluate their clients' third party EDI service firms in terms of responsibilities, resources, and abilities to provide, on an ongoing basis, reliable and secure services per contract terms.

7.3

Auditor education and training Rationale: Auditors practice today in a significantly different environment from that in the past. The college and university educational curricula and training requirements for auditors must be updated to reflect technological change and to embody the types of knowledge and the proficiency required of auditors to maintain the profession.

7.1

Audit Techniques Rationale: Traditional audit techniques may no longer apply in an EDI environment. Auditors need to develop effective techniques to enable them to describe, evaluate, and test an intercompany information network that includes very few paper documents.

6.9

Auditor Skills (Skills required of auditors) . . Rationale: Because of rapid expansion in the extent, scope, and types of information to be audited in an EDI system, auditors need to be specially trained to acquire certain skills that enable them to maintain a high standard of practice in such an environment.

6.6

Audit Focus (Substantive versus Compliance Testing) Rationale: Although there is a general agreement that the focus and the types of audit tests need to change to suit highly automated, paperless intercompany EDI networks, there is no consensus on the direction or specific procedures that should be applied. Therefore, efforts are needed to establish appropriate audit approach.

102

Average Group Rating

Your Original Rating

Your Final Rating

Key EDI Audit Issues and their Rationale:

6.5

Audit Scope (Boundary of Audit) Rationale: Paperless intercompany transactions create a "boundaryless" information system environment, and auditors may be required to audit beyond the traditional boundaries of clients' systems. Therefore, audit scope and responsibilities need to be pre-determined and agreed upon by the parties involved.

6.5

Audit Risk Assessment Rationale: Audit risks in an automated open EDI network are significantly different from those in a closed system because EDI involves more parties and more diverse exposures. A distinctive approach of audit risk assessment is required, and guidance must be established.

6.4

Audit Responsibility in Evaluating Controls Rationale: In open EDI networks, auditors, especially internal auditors, may be held responsible for the review and evaluation of external controls (in addition to internal controls). In such circumstances, it is necessary that guidance be developed and standards be established to lead the practice.

6.3

Collaboration Among Auditors of EDI Parties Rationale: An EDI network involves not only auditors of the company but also auditors of its trading partners and of third parties. Because the security of an EDI system depends on those of others in the network, these auditors are interdependent. It is vital that the roles of each party's auditors be determined and the rules of collaboration be established.

6.3

The Changing Roles of Auditors Rationale: In an EDI environment, the role of auditing will have to change to meet both the new demands of clients and the needs of the profession. Failure by auditors to assume suitable new roles adequate to meet such demands may lead to the decline of the profession.

103

Average Group Rating

Your Original Rating

Your Final Rating

Key EDI Audit Issues and their Rationale:

6.3

Timing of Audit Tests Rationale: Because of the high volume of transactions and the velocity of electronic processing, EDI transactions may have to be reviewed as they occur. Consequently, the audit process must be modified and specific audit standards established.

6.2

Audit Tools Rationale: The increased complexity of intercompany automated paperless EDI transactions makes it more difficult, or in some cases impossible, for auditors to test and evaluate network systems with existing audit tools. More powerful tools must be developed to match the growth in sophistication of clients' systems. . . .

5.5

Relationship Among A Company's Auditors Rationale: In an EDI environment, information systems grow in complexity, external auditors may have to rely more on internal auditors, and information systems auditors will be requisite members of audit teams. Suitable audit approaches must be developed to promote and make the best use of this inter-relationships.

4.3

Audit Reporting (Periodic versus On-Line) Rationale: Because the high speed of EDI transactions makes information obsolete within a very short time, there is a need for more frequent accounting disclosure. Auditors will have to adjust their reporting procedures to meet the need for timely and accurate information.

Finally, if your final rating for a specific issue is significantly (5 to 9 scores) different from the group average, please briefly describe your rationale for the final rating on a separate sheet and return with the questionnaire. For example, suppose the group average on a particular question was 8.4, but your rating of the question was 2.5, then this would be significantly different from the average.

104

ADDITIONAL ISSUES The following are four additional issues from the previous Delphi round. Please rate their importance by using the same scale as the above issues. RATING S C A L E : Not Important 1 2 3

Moderately Critically Important Important 4 5 6 7 8 9 10

Form of Audit Assurance Rationale: The closer interrelationships established between trading partners in an EDI network will affect business and financial risks. Therefore, auditors should reevaluate the types of assurance required by the public and formulate suitable audit procedures and related opinions to satisfy these needs. Professional Support Rationale: The auditor have to face many new issues when auditing an EDI system. Thus, professional organizations such as EDPAA, IIA, and CICA should take a proactive approach to providing reference materials and training opportunities to help practising auditors understand and deal effectively with the EDI environment, its risks and control measures. Inconsistent EDI Approaches Rationale: Inconsistent EDI approaches (used by various EDI parties) can lead to operational and administrative problems resulting in missed business opportunities, additional costs, and weaken internal controls. Therefore, auditors must be aware of inconsistencies and provide direction to management. The Network and Ownership of Data Rationale: As EDI systems develop, the sharing of common data/programs will increase and the information flow that the auditor needs to understand will change. Auditors must take part in defining information flows and boundaries to data ownership. (This will also help draw legal boundaries among parties in a large integrated EDI system).

105

1992 DELPHI STUDY OF KEY EDI AUDIT ISSUES Background Information Your Name (Optional)

The following information is needed to help us with the statistical analysis of the data you will provide us in the questionnaires and in making comparisons among different groups of auditors. 1) What is responsibility?

your

present

job

title

and

primary

2) Are you certified for the following professional designations? CA

CDP

CGA

CIA

Other(please specify) 3) Your area(s) of audit expertise: External audit Internal audit Information systems audit General audit Others (please specify) 4) Years of experience in auditing information systems auditing Other computer related positions 5) Have you ever been engaged in an EDI project? No Yes. Please specify in what capacity:

106

CISA

CMA

area

of

6) How do you judge your level of knowledge and understanding of the EDI Technology? Good working knowledge Average knowledge Little knowledge

7) Please indicate the primary source(s) of your knowledge and understanding of the EDI Technology: First hand experience Professional literature Popular literature Oral communication _Others(please specify)

8) Does your firm or audit department have an EDI audit manual or guideline in use, or is it in the process of developing one? EDI audit manual or guideline in use Yes In process of developing Yes

No No

9) Would you like to have a copy of the results? Yes

No

Thank you for your help in completing this study!

107

APPENDIX D ROUND 1 RESULTS

108

Table 1.1 - Controls Over EDI Networks Frequency 12

Issue

Rationale

Control over mailbox

"If just one-to-one relationship with partner, must be certain that 3rd party mailbox service is controlled. Once you start dealing with many suppliers, how do you know all the different mailboxes are secure. Audit reports issued on security/control of sources are limited by their very nature, i.e., if a report is issued to-day, the same controls may not exist or be overridden tomorrow. Can you ever be sure of control- a matter of trust?"

Data confidentiality

"Need to be sure data are sent (only) to correct destination/transmitted timely and data held in service firm backup file are secure. "

Financial Controls

"Weak controls can cause financial loss to participants. "

System security not compromised?

"Need assurance that the transmission channel cannot be used as a backdoor into (company) system."

Assess (access) controls to the EDI (network) environment

"Controls over such areas as approval of payment and receipt of goods will be dependent on access to the EDI systems. For example, transactions for receipt of goods could be fraudulently approved if the access controls to the system are weak." "Controls should ensure adequate password control to prevent unauthorized access to the system (purchasing, receiving)."

Communications security

Concern over competition reviewing transactions. If you have a value added network, and communication lines go down, who is Liable, if purchase order not filled, or data lost? Who is liable if business loses result from unauthorized changes to data? Can unauthorized access be identified?

109

Frequency

Issue

Rationale

Integrity of data

"Transmission errors-Data or transactions may be lost, duplicated, inaccurately transmitted or altered during transmission. Application errors-our EDI partner may omit, duplicate or inaccurately send or receive data or transactions. "

How do we confirm that orders received are valid, complete and accurate?

"How do we know that the order was originated by a legitimate and authorized person/entity? How do we know transaction details are complete and accurate (i.e. shipped to legitimate location)."

Authentication of trading partners

"Errors in authentication could result in misappropriation of funds through transfers to fraudulent partners. Goods shipped to fraudulent trading partners could be misappropriated in the same manner."

Trading Partner's security

"The degree to which a trading partner secures his end of the network has implications regarding confidentiality, accuracy and completeness of data. Our clients rely on their trading partners' security."

Adequate communication controls with supplier

"The controls should ensure that the transmission of the description, quantity and price are correct."

Accurate, complete transmission of accurate, complete data

"Essential that only accurate, complete data are put into the pipeline and equally essential that those data are transmitted accurately and completely. "

110

Table 1.2 - EDI Contracts (Trading Partner Agreement)

Frequency 6

Issue

Rationale

Contract with supplier /partner

"The basis for all future dealings with your EDI partner. The contract must be all inclusive and allow for technology changes. "

Clarity of the trading agreement

"Who is responsible for what and when, what standards are to be followed; recourse available to any partner-will identify the exposures to my clients and my exposures as an auditor."

Organizational responsibility

"Responsibility for controls has to be defined and agreed to by the trading partners and the network supplier. "

Partnership Agreements

"Procedures agreed as to cohort constitutes an offer, acceptance, receipt and acknowledgement of documents. "

Agreement on contractual arrangements

"Disagreement on responsibility for loss or assumption of risks may impose unfair difficulties on one or more of the partners if there is no agreement in place. "

Written agreement to cover all significant issues

"To be enforceable, terms, service, responsibilities must be defined."

Ill

Table 1.3 - Auditability and Audit Trail

Frequency 4

Issue

Rationale

Auditability

"An issue within your company, your partner and with the third party service. Must be absolutely certain documents/data are sent and received accurately. "

Adequacy of audit trail

"If an adequate audit trail does not exist it is difficult to determine if the application controls are working and it may expose the Railway to potential legal liability (no backup to support a transaction received or sent)."

Audit evidence and records

"The maintenance of a complete record of all transactions and an access log of who has been on the system.

Access to and retention of EDI data files

"External auditors will require access to EDI data files for attest purposes. Access to the data may be limited if the files are maintained on a third party network (VAN or WAN). As paper forms and hardcopy printouts are phased out the electronic data files will be the only form of supporting documentation. The transactions in these files should be retained for the complete fiscal period."

112

Table 1.4 - Backup, Disaster Recovery and Contingency Plans Frequency 4

Issue Backup, Disaster Recovery and contingency planning

Rationale "With the Railway's increasing reliance on EDI for operational and financial services, operations will be greatly hindered without adequate backup for the network. " "Assets in the form of account receivable data may be lost if the data is not backed-up on a regular basis and stored in a secure location." "A company's ability to continue as a goingconcern may be in jeopardy if the EDI network is lost. It will be imperative to have an alternate hot-site for recovery of the system and data to allow continuation of the normal business operations." "Contingency plans if EDI not available, or computer not available. "

Table 1.5 - Third Party EDI Services Frequency 3

Issue

Rationale

Third party EDI services

"Third party EDI companies need to provide a secure, reliable and available service. "

Reliability of service

"Need to know service firm has the financial resources to provide service per contract terms on an ongoing basis and has made adequate provision for trouble shooting, client communication, system upgrading capabilities. "

Type of EDI network

"Different types of EDI networks have different implications for an auditor. Some are more secure than others, some involve more intermediaries than others."

113

Table 1.6 - Legal and Audit Evidence Frequency 3

Issue

Rationale

Court acceptance/Dispute mechanism

"Will the electronic signatures be accepted in the courts? Only time will tell! A dispute mechanism must be set up to deal with issues that come up."

Legal implications of transmitting waybills, bills of lading, and purchase orders electronically. Are these documents enforceable?

"Current contract law does not address EDI transactions."

Do electronic contracts impose the same rights and obligations as a written contract.?

"In the absence of the normal contract process which involves offer and acceptance, how will disputes over amounts, quantities, terms etc. be resolved (i.e. no signatures). When is a contract formed?

Table 1.7 - EDI Records Retention Frequency 2

Issue

Rationale

Access to and retention of EDI data files

"External auditors will require access to EDI data files for attest purposes. Access to the data may be limited if the files are maintained on a third party network (VAN or WAN). As paper forms and hardcopy printouts are phased out the electronic data files will be the only form of supporting documentation. The transactions in these files should be retained for the complete fiscal period."

Retention of electronic or hard copy information

"What will stand up in a court of law (evidence)? What are legal and regulatory requirements (e.g. tax department)? Who should keep this information (sender or receiver) and for how long?"

114

APPENDIX E ROUND 2 RESULTS (THE ORIGINAL 21 ISSUES)

115

(

Rank Total Score Respondent Mean Score Stdev.

sa8 10 9 10 9 10 10 10 9 10 10 8 8 6 10 9 10 10 10 10 8 10 10 10 10 9 9 10 10 10 9 8

1 299 32 9.34 0.96

Backup 10 9 9 10 8 9 10 9 9 10 8 10 9 8 9 6 10 9 7 7 10 10 10 9 9 9 6 10 8 10 7 5 2 279 32 8.72 1.35

Trail

Involve

8 9 6 10 10 10 10 8 7 10 8 10 9 8 9 7 10 9 8 10 9 9 10 10 8 8 8 5 9 10 6 3

9 8 9 10 8 7 10 9 6 8 8 9 7 7 9 8 9 8 9 1 10 6 10 8 8 9 8 6 8 10 8 8

3 271 32 8.47 1.66

4 258 32 8.06 1.69

116

Evidence 9 5 7 6 9 9 9 7 6 10 6 9 6 8 9 6 8 8 7 10 10 8 10 10 7 8 9 6 8 10 3 7 5 250 32 7.81 1.70

Con-

tract 8 6 5 10 8 7 10 8 5 10 8 5 8 . 8 8 8 9 7 6 5 6 7 10 8 8 8 9 6 10 8 9 9

6 247 32 7.72 1.55

Record 9 8 6 5 9 7 10 9 4 10 6 10 9 7 8 7 7 8 7 5 10 7 10 8 7 8 9 5 10 7 8 6 7 246 32 7.69 1.67

partes 7 8 6 10 7 8 10 7 5 8 6 8 7 4 9 8 7 8 9 7 3 10 10 8 7 8 6 4 8 8 6 10 Rank Total Score Respondent Mean Score Stdev.

8 237 32 7.41 1.77

Educate 7 8 6 7 7 6 9 6 5 7 9 8 8 8 8 5 6 8 6 7 8 8 10 8 6 8 8 7 8 10 3 9 9 234 32 7.31 1.47

Technique 7 8 6 5 9 4 10 8 6 3 7 6 9 8 8 6 7 8 7 7 10 7 10 9 7 7 8 8 7 5 3 7

Skill

Focus

Scope

Risk

7 7 8 7 7 6 7 7 5 3 10 5 8 7 8 5 8 8 6 6 7 8 10 8 8 8 8 5 5 8 3 9

6 6 7 5 8 3 9 3 3 5 9 5 9 5 8 6 8 8 7 5 4 9 10 10 7 5 8 4 8 8 6 8

6 6 6 5 8 4 8 8 4 2 10 8 7 6 9 3 10 5 2 3 9 8 10 9 5 7 7 7 5 8 4 10

2 6 6 5 7 2 10 4 6 5 3 5 7 7 9 8 6 7 7 7 6 7 10 9 8 6 8 3 8 9 7 8

10 227 32 7.09 1.76

11 222 32 6.94 1.66

12 212 32 6.63 2.03

13 209 32 6.53 2.34

14 208 32 6.50 2.06

117

Response 4 5 5 7 5 5 10 7 5 3 5 5 7 8 8 8 6 7 8 5 4 7 10 8 8 6 5 5 9 9 5 7 Rank Total Score Respondent Mean Score Stdev.

15 206 32 6.44 1.78

Colla

Roles

Timing

Tools

5 8 7 5 8 3 8 2 4 4 8 7 7 6 9 8 8 8 4 4 8 7 6 6 7 6 6 6 8 8 1 10

7 5 5 5 7 2 5 5 5 3 9 7 7 7 8 5 7 7 5 5 8 8 4 7 4 6 9 8 8 10 4 9

7 7 6 5 4 2 10 1 6 3 8 5 8 8 8 6 6 7 8 5 9 7 10 9 6 7 5 7 5 5 2 8

7 7 6 5 5 7 8 6 5 3 7 8 8 7 9 5 6 6 6 5 6 7 6 9 7 5 6 6 5 7 4 5

16 202 32 6.31 2.05

17 201 32 6.28 1.89

18 200 32 6.25 2.19

118

19 199 32 6.22 1.34

Relation 3 6 8 5 6 2 8 3 4 5 9 5 8 6 8 3 4 4 3 5 8 8 6 6 5 4 6 5 5 7 4 7

Report

20 176 32 5.50 1.82

21 137 32 4.28 2.00

3 5 3 5 3 2 8 1 4 1 3 3 6 6 7 2 7 4 2 2 6 6 6 8 3 6 6 5 5 5 2 2

APPENDIX F ROUND 3 RESULTS (THE ORIGINAL 21 ISSUES)

119

ÇioT 9 10 9 10 9 10 10 10 9 10 9 9 8 8 10 9 9 10 10 10 8 10 10 10 10 9 10 10 10 9 9 10 9 Rank Total Score Respondent Mean Score Stdev.

1 311.5 33 9.44 0.65

Backup 9 9 9 10 9 9 10 9 9 10 8 10 9 8 9 6 9 9 8 7 10 10 9 9 9 9 10 9 9 8 5 9 9 2 291 33 8.82 1.09

Trail

Involve

8 9 6 10 10 10 10 9 8 10 8 10 9 8 9 7 9 9 8 9 10 10 9 9 8 8 6 9 9 7 8 9 9

9 8 9 10 8 7 10 9 8 8 8 8 7 8 9 8 9 8 9 1 10 7 10 8 8 8 7 8 9 8 8 9 8

3 287 33 8.70 1.09

4 269 33 8.15 1.52

120

Evidence 9 8 7 7 8 9 9 7 6 10 7 8 7 8 8.5 7 8 8 7 9 10 8 10 8 8 9 7 8 8 3 7 8 8 5 258.5 33 7.83 1.30

Record 8 8 6 6 8 7 9 8 4 9 6 9 9 8 8 7 8 8 7 5 10 7 10 8 8 8 6 10 7 8 6 7 5 6 247.5 33 7.50 1.44

tract 8 7 5 9 8 7 8 8 5 8 8 6 8 8 8 7 9 7 6 5 5 8 8 6 8 8 6 10 8 9 9 6 9 7 245 33 7.42 1.33

Rank Total Score Respondent Mean Score Stdev.

Technique 7 8 6 6 9 4 10 8 6 6 7 7 9 8 8 6 7 8 7 7 10 7 10 8 7 8 8 8 6 3 7 8 7

par^yS

8 241 33 7.30 1.49

9 239.5 33 7.26 1.57

7 8 6 8 7 8 10 7 5 8 6 8 7 5 8 8 7 8 8 7 4 10 10 7 7 3 5 8 8 7 9 9 7

Educate 7 8 6 T s

7 6 9 6 5 7 8 8 8 8 8 6 7 8 6 7 8 7 10 7 6 8 7 8 9 3 9 6 8 10 238 33 7.21 1.32 121

Skill

Risk

7 7 8 7 7 6 7 6 5 5 9 6 8 7 7.5 6 8 8 6 6 6 7 10 8 7 8 6 8 8 3 8 8 8

2 6 6 5 7 3 9 4 6 5 5 5 7 7 8 8 6 7 7 7 7 7 10 8 7 7 3 8 8 7 8 7 6

11 231.5 33 7.02 1.31

12 213 33 6.45 1.71

Response 5 6 5 7 5 5 9 6 5 4 5 6 7 8 7.5 8 7 7 8 5 5 7 10 7 8 6 5 9 6 5 7 7 6

Scope

12 213 33 6.45 1.42

14 211 33 6.39 2.00

6 7 6 6 7 4 8 7 4 3 9 6 7 6 9 3 9 5 2 3 9 7 10 7 5 7 7 6 7 5 10 7 7

1

Rank Total Score Respondent Mean Score Stdev.

Focus 6 6 7 5 6 3 7 5 3 5 8 6 8 6 8 6 7 8 7 5 4 7 10 7 7 8 5 7 7 6 7 8 6

Tools 7 7 6 5 6 7 7 5 5 5 7 7 8 7 8 6 7 6 6 5 8 7 6 7 7 6 6 6 6 4 5 6 5

Timing 7 7 6 5 5 3 8 3 6 3 7 5 8 8 7.5 5 7 7 8 5 8 7 10 7 6 6 7 7 5 1 7 6 5

Colla 5 7 7 5 6 3 7 3 4 4 7 6 7 6 8 8 8 8 4 4 8 7 6 6 7 6 6 8 6 1 10 7 6

Roles 5 5 5 6 6 3 5 5 5 3 8 6 7 7 7.5 6 7 7 5 5 8 7 4 7 4 8 8 8 8 4 9 6 5

15 211 33 6.38 1.46

16 206 33 6.24 0.99

17 202.5 33 6.14 1.80

18 201 33 6.09 1.82

19 199.5 33 6.05 1.56

122

Relation 3 6 8 5 6 2 6 4 4 4 8 5 7 6 7 3 4 4 3 5 8 7 6 6 5 7 5 5 5 4 6 5 5 20 174 33 5.27 1.50

Report 3 5 3 5 3 2 5 2 4 2 3 3 6 6 5 2 6 4 2 2 5 6 6 5 3 4 5 5 4 3 2 5 3 21 129 33 3.91 1.40

APPENDIX G ROUND 3 RESULTS (THE FINAL 25 ISSUES)

123

9 10 9 10 9 10 10 10 9 10 9 9 8 8 10 9 9 10 10 10 8 10 10 10 10 9 10 10 10 9 9 10 9 Rank Total Score Respondent Mean Score Stdev.

1 311.5 33 9.44 0.65

Backup 9 9 9 10 9 9 10 9 9 10 8 10 9 8 9 6 9 9 8 7 10 10 9 9 9 9 10 9 9 8 5 9 9 2 291 33 8.82 1.09

Trail

Involve

8 9 6 10 10 10 10 9 8 10 8 10 9 8 9 7 9 9 8 9 10 10 9 9 8 8 6 9 9 7 8 9 9

9 8 9 10 8 7 10 9 8 8 8 8 7 8 9 8 9 8 9 1 10 7 10 8 8 8 7 8 9 8 8 9 8

3 287 33 8.70 1.09

4 269 33 8.15 1.52 124

Evidence 9 8 7 7 8 9 9 7 6 10 7 8 7 8 8.5 7 8 8 7 9 10 8 10 8 8 9 7 8 8 3 7 8 8 5 258.5 33 7.83 1.30

Record 8 8 6 6 8 7 9 8 4 9 6 9 9 8 8 7 8 8 7 5 10 7 10 8 8 8 6 10 7 8 6 7 5 6 247.5 33 7.50 1.44

Sacf 8 7 5 9 8 7 8 8 5 8 8 6 8 8 8 7 9 7 6 5 5 8 8 6 8 8 6 10 8 9 9 6 9 7 245 33 7.42 1.33

Rank Total Score Respondent Mean Score Stdev.

Technique 7 8 6 6 9 4 10 8 6 6 7 7 9 8 8 6 7 8 7 7 10 7 10 8 7 8 8 8 6 3 7 8 7

par^3

8 241 33 7.30 1.49

9 239.5 33 7.26 1.57

7 8 6 8 7 8 10 7 5 8 6 8 7 5 8 8 7 8 8 7 4 10 10 7 7 3 5 8 8 7 9 9 7

Educate 7 8 6 7 7 6 9 6 5 7 8 8 8 8 8 6 7 8 6 7 8 7 10 7 6 8 7 8 9 3 9 6 8 10 238 33 7.21 1.32 125

Skill

Support

Risk

7 7 8 7 7 6 7 6 5 5 9 6 8 7 7.5 6 8 8 6 6 6 7 10 8 7 8 6 8 8 3 8 8 8

6 7 7 8 6 2 8 6 na 4 7 7 8 7 6 5 9 7 4 6 10 6 6 7 6 8 7 8 9 6 8 8 7

2 6 6 5 7 3 9 4 6 5 5 5 7 7 8 8 6 7 7 7 7 7 10 8 7 7 3 8 8 7 8 7 6

11 231.5 33 7.02 1.31

12 216 33 6.55 1.92

13 213 33 6.45 1.71

Response 5 6 5 7 5 5 9 6 5 4 5 6 7 8 7.5 8 7 7 8 5 5 7 10 7 8 6 5 9 6 5 7 7 6 13 213 33 6.45 1.42

Rank Total Score Respondent Mean Score Stdev.

Scope

Focus

6 7 6 6 7 4 8 7 4 3 9 6 7 6 9 3 9 5 2 3 9 7 10 7 5 7 7 6 7 5 10 7 7

6 6 7 5 6 3 7 5 3 5 8 6 8 6 8 6 7 8 7 5 4 7 10 7 7 8 5 7 7 6 7 8 6

15 211 33 6.39 2.00

16 211 33 6.38 1.46

Owner

Tools

Timing

Colla

4 8 6 7 8 8 8 9 6 3 3 9 7 6 7 8 4 7 9 5 5 6 7 4

8 7 8 7 6 2 5 7 na 4 7 7 7 8 8 8 7 8 4 3 8 7 5 7 8 5 5 9 5 6 6 4 5

7 7 6 5 6 7 7 5 5 5 7 7 8 7 8 6 7 6 6 5 8 7 6 7 7 6 6 6 6 4 5 6 5

7 7 6 5 5 3 8 3 6 3 7 5 8 8 7.5 5 7 7 8 5 8 7 10 7 6 6 7 7 5 1 7 6 5

5 7 7 5 6 3 7 3 4 4 7 6 7 6 8 8 8 8 4 4 8 7 6 6 7 6 6 8 6 1 10 7 6

17 204 32 6.38 1.76

18 201 32 6.28 1.98

19 206 33 6.24 0.99

20 202.5 33 6.14 1.80

6 5 8 6 7 3 8 8

126

21 201 33 6.09 1.82

Rank Total Score Respondent Mean Score Stdev.

Roles 5 5 5 6 6 3 5 5 5 3 8 6 7 7 7.5 6 7 7 5 5 8 7 4 7 4 8 8 8 8 4 9 6 5

Assure na 6 8 7 6 3 4 4 na 5 5 7 6 7 6 3 6 6 4 5 9 7 8 7 5 2 5 9 8 3 7 5 4

22 199.5 33 6.05 1.56

23 176.5 31 5.69 2.18

Relation 3 6 8 5 6 2 6 4 4 4 8 5 7 6 7 3 4 4 3 5 8 7 6 6 5 7 5 5 5 4 6 5 5 24 174 33 5.27 1.50

127

Report 3 5 3 5 3 2 5 2 4 2 3 3 6 6 5 2 6 4 2 2 5 6 6 5 3 4 5 5 4 3 2 5 3 25 129 33 3.91 1.40