Scientific fraud in 20 falsified anesthesia papers

Trends und Medizinökonomie Anaesthesist 2012 · 61:543–549 DOI 10.1007/s00101-012-2029-x Published online: 15. Juni 2012 © Springer-Verlag 2012 J. Hei...
Author: Karl Busch
0 downloads 2 Views 691KB Size
Trends und Medizinökonomie Anaesthesist 2012 · 61:543–549 DOI 10.1007/s00101-012-2029-x Published online: 15. Juni 2012 © Springer-Verlag 2012

J. Hein1 · R. Zobrist2 · C. Konrad3 · G. Schuepfer3 1 Institute of Anesthesiology, Kantonsspital Nidwalden, Stans 2 Institute of Anesthesiology, Spital Langenthal 3 Institute of Anesthesiology, Intensive Care, Emergency Medicine and Pain Therapy,

Luzerner Kantonsspital, Lucerne 16 Redaktion

E. Martin, Heidelberg M. Bauer, Göttingen

Scientific fraud in 20 falsified anesthesia papers Detection using financial auditing methods

Introduction Fraudulent scientific publications are not new [8]. Renowned medical journals are more likely to be affected by scientific misconduct [26, 29]. Appropriate measures, such as comparison with a large text database, can detect plagiarism and retracted falsified publications [29]. Fabricating or distorting data for publication are fraudulent actions that may have different motivations, such as enhancement of prestige, influence or income. Financial auditors and tax authorities use statistical methods for data analysis to detect fraud based on the observations described by Newcomb and Benford [2, 11, 12, 16] and numbers from natural sources show a counter-intuitive frequency distribution, which has also been shown for medical data [21]. According to Benford’s law the digit 1 appears as the leading number to the left of the decimal point more often compared to digits 2–9. In 2009 a total of 21 papers which the author admitted were fictitious [23] were retracted from several anesthesia and other scientific journals. The present study investigates for the first time, whether these studies can be detected as false using the method described by Benford and Newcomb. To the best of the authors’ knowledge, although this law was formulated in the late nineteenth century it has not yet been used to assess data from medical studies except medical abstracts [21]. Therefore, an attempt was made to iden-

tify known falsified examples of medical papers by mathematical analyses based on Benford’s law. It could be argued that this paper is not suitable for publication in an anesthesiology journal and would be better suited to an applied statistics journal; however, anesthesiology journals missed these cases of fraud and will need to learn to identify other cases of fraud in the future [22, 25].

where di=(1, 2, …, 9) is the leading digit left of the decimal point and p is the probability. For example, 3.5/4.0/31 counts the digit 3 twice and digit 4 once. The law also describes the frequencies of pairs of leading digits (. Fig. 1):

where, dji= (10, 11, 12, … 98, 99).

Materials and methods The text and tables of the 21 retracted articles ([23]; . Tab. 2) were manually screened for leading digits and numbers. Extracted numbers were transferred to an Excel spreadsheet and the occurrence of each number determined using the builtin Excel functions (Microsoft® Office Excel 2003). In order to reduce keyboard errors, extractions were performed 3 times and the results compared to each other. The frequencies of digits 1–9 as the leading digit to the left of the decimal point and digits 0–9 as the digit in the second leading position were determined. Digits related to citations or references, such as years and pages, were excluded. The numbers were also used as a grouped ­data set. The frequency of the first digit in datasets from natural sources can be calculated using the following equations [2, 6, 11, 12, 16, 20]:

where pn (d) is the probability of the occurrence of digit d to the base B in the nth position [10, 12]. Equation 3 allows the calculation of the probability of a digit regardless of its position. To apply Benford’s law, the base B of the logarithm has to be 10. Suitable additions may provide the expected probability of a certain digit in the second or third position to the left of the decimal point (. Tab. 1). The law cannot be applied to data from man-made sources, such as postal codes and credit card numbers (i.e. assigned numbers). Nigrini successfully propagated the statistical analysis of differences between expected and observed frequencies for financial auditing [18]. The observed frequencies of digits were compared to the probabilities expected according to Benford’s law. DeJ.H. and R.Z. contributed equally to the article. Der Anaesthesist 6 · 2012 

| 543

Trends und Medizinökonomie Tab. 1  Expected frequencies for digits to the left of the decimal point according to Benford’s

law Digit 0 1 2 3 4 5 6 7 8 9 Total

1st Position   0.30103 0.17609 0.12494 0.09691 0.07918 0.06695 0.05799 0.05115 0.04576 1

2nd Position 0.11968 0.11389 0.10882 0.10433 0.10031 0.09668 0.09337 0.09035 0.08757 0.08500 1

3rd Position 0.10178 0.10138 0.10097 0.10057 0.10018 0.09979 0.09940 0.09902 0.09864 0.09827 1

4th Position 0.10018 0.10014 0.10010 0.10006 0.10002 0.09998 0.09994 0.09990 0.09986 0.09982 1

Tab. 2  Comparison of the observed and expected frequencies of digits in the first and sec-

ond positions left of the decimal point extracted from the 20 falsified papers   Digit 0 1 2 3 4 5 6 7 8 9 Sum χ2 df P

First digit Observed (n)   1548 994 590 457 371 279 241 206 176 4862      

Expected (n)   1464 856 607 471 385 325 282 249 223 4862 58.27 8 (0.975)=1.96 (in accordance with Posch [20]). A meta-analysis that included complex mathematical tools was chosen as a control [5]. Data were extracted in the same

544 | 

Der Anaesthesist 6 · 2012

way. Non-natural numbers, such as brand names or statistical descriptions (e.g. 95% confidence interval) were excluded.

Results Of the 21 withdrawn papers 20 were available as PDF files and 1 abstract was obviously permanently deleted from the abstract database of the American Society of Anesthesiology (ASA) and unavailable for analysis. From the remaining 20 papers, a total of 4,862 first digits and 2,646 second digits were extracted. The observed frequencies differed significantly from the expected distribution (. Tab. 1). A first digit was observed a mean of 243.1 times (SD±118.2, range: 30–592) and a second digit 132.3 times (SD ±72.2, range: 15– 383) per article. Thus, every paper provid-

ed 243 and 132 digits in the first and second positions left of the decimal point, respectively, for analysis. Article 11 (an abstract) provided only 30 first digits and 15 second digits. The observed and expected distributions for the leading digits in each of the 20 available papers is shown in . Fig. 2, where 17 had a first-digit distribution that deviated significantly from the expected distribution and 18 had a second-digit distribution that deviated significantly from the expected distribution (. Tab. 2, 3, 4). Only article 10 had patterns in complete agreement with the expected distribution. No differences were found between the observed and expected patterns in the control paper ([5]; . Fig. 3). The detailed analysis of the frequencies of digits 1–9 in the first position left of the decimal and Z-test statistics is shown in . Tab. 2.

Discussion The Newcomb-Benford law was initially used for financial audits and by tax authorities to detect fraud in filed statements or declarations [15, 17, 18]. Deviations from the expected digit frequencies provoke detailed analysis of data by tax authorities [17, 18, 20, 27]. Until now this law has not been used for the statistical review of data extracted from medical papers although abstracts were recently investigated [21]. In 2009 the editor of the journal Anesthesia and Analgesia published a notification regarding the retraction of 21 studies by one main author that were known to be based on falsified data or were completely fictitious [22, 23, 24]. The author was sentenced to prison for fraud [13, 14]. Therefore, these data were used to perform an analysis according to Benford’s law as a proof of concept; this study examined material that is known to be faked (true positive). The procedure found anomalies in 19 of the 20 papers, therefore the approach seems to be sensitive [1]. Obviously, final statements of specificity (true negative) cannot be drawn from this study. The problem with such a statement is that it has to be certain that the presented data is correct and not faked. Therefore, a paper dealing with a complex meta-analysis [5]

Abstract · Zusammenfassung Anaesthesist 2012 · 61:543–549  DOI 10.1007/s00101-012-2029-x © Springer-Verlag 2012 J. Hein · R. Zobrist · C. Konrad · G. Schuepfer

Scientific fraud in 20 falsified anesthesia papers. Detection using financial auditing methods Abstract Data from natural sources show counterintuitive distribution patterns for the leading digits to the left of the decimal point and the digit 1 is observed more frequently than all other numbers. This pattern, which was first described by Newcomb and later confirmed by Benford, is used in financial and tax auditing to detect fraud. Deviations from the pattern indicate possible falsifications. Anesthesiology journals are affected not only by ghostwriting and plagiarism but also by counterfeiting. In the present study 20 publications in anesthesiology known to be falsified by an author were investigated for irregularities with respect to Benford’s law using the χ2-test and the Z-test. In the 20 retracted publications an average first-

digit frequency of 243.1 (standard deviation SD ±118.2, range: 30–592) and an average second-digit frequency of 132.3 (SD ±72.2, range: 15–383) were found. The observed distribution of the first and second digits to the left of the decimal point differed significantly (p