R Exact Testing Procedures in SAS for Categorical Data Analysis

Guogen Shan, University of Nevada Las Vegas, Las Vegas, NV Darren Liu, University of Nevada Las Vegas, Las Vegas, NV

ABSTRACT In categorical data analysis, the majority of methods are based on asymptotic approaches. As we all know, asymptotic approaches are very easy to calculate, however, they may have an unsatisfied type I error control. Actual type I error rates of testing procedures could be underestimated or over-estimated for the discrete data. In addition, when power of the study is overestimated, sample size is definitely under-estimated. The sample size in a study is actually not enough to attain the pre-specified power. In order to avoid such problems, exact approaches can be considered as alternatives in data analysis. SAS provides the option EXACT in several procedures for producing exact p-values as well as exact confidence intervals. We will review the commonly used conditional exact approach (Fisher’s test) by using examples in the SAS help files. We will also compare the EXACT statement in SAS to exact testing procedures in other statistical software packages. INTRODUCTION In categorical data analysis, asymptotic approaches are often used for statistical inference. The first topic in statistical inference would be the testing of hypotheses, and the second topic is to construct confidence intervals for the parameter of interest. PROC FREQ provides the EXACT statement to calculate exact p-values based on Fisher’s conditional approach. From SAS 9.3, Barnard test (Barnard, 1947) (an unconditional approach) is available in the EXACT statement for comparing proportions (see, What’s New in the Base SAS 9.3 Statistical Procedures). There are still some efficient testing procedures not included in SAS (Lloyd, 2008; Shan, Ma, Hutson, & Wilding, 2012, 2013; Shan, 2013a). Asymptotic confidence intervals are often the default option for most statistical software packages. These confidence intervals do not guarantee the nominal level, where infimum coverage probabilities of these asymptotic approaches may be much less than the coverage size. For this reason, PROC FREQ presents exact confidence limits using the ClopperPearson method. Recently, Wang (2010) proposed a method for optimal exact confidence limits for the difference between two independent or dependent proportions with the use of an R package, ExactCIdiff, which was developed by Shan and Wang (2013) to implement the new optimal confidence limits.

1

Table 1: Output from PROC FREQ for comparing two independent proportions Statistic DF Value Prob Chi-Square Likelihood Ratio Chi-Square Continuity Adj. Chi-Square Mantel-Haenszel Chi-Square Phi Coefficient ContingencyCoefficient Cramer’s V

1 1 1 1

4.9597 5.0975 3.1879 4.7441 0.4644 0.4212 0.4644

0.0259 0.0240 0.0742 0.0294

ASYMPTOTIC AND EXACT P-VALUES We use an example from the PROC FREQ procedure in the SAS help file to compare asymptotic and exact p-values. This is the example used in for analyzing a 2 by 2 contingency table (Example 36.5 in SAS 9.3). proc f r e q data=FatComp o r d e r =data ; f o r m a t Exposure ExpFmt . Response RspFmt . ; t a b l e s Exposure ∗Response / c h i s q r e l r i s k ; exact r i s k d i f f binomial p d i f f ; w e i g h t Count ; t i t l e ’ Case−C o n t r o l Study o f High Fat / C h o l e s t e r o l D i e t ’ ; ods o u t p u t c h i s q = c h i s q ; run ; The data is from a case-control study of patients with high cholesterol diet (n1 = 15) and low cholesterol diet (n2 = 8). The numbers of response are r1 = 11 and r2 = 2 in the high and low cholesterol diet, respectively. The response rate in the high cholesterol diet group (11/15=0.7333) is much higher than that in the low cholesterol diet group (2/6=0.25). The asymptotic p-value for testing the equality of these two proportions is 0.0259, see Table 1. One has to reject the null hypothesis of equal proportions. The exact test based on Fisher’s approach by fixing both marginal totals, also shows a significant difference between two proportions with p-value 0.0393. The exact test based on Fisher’s approach may be conservative due to the limited sample space in small to medium sample settings. An alternative to the Fisher’s exact test is one that based on an unconditional framework, see Barnard (1947). The Barnard option is added in the EXACT statement of the PROC FREQ in SAS 9.3. While the Barnard test is a general approach, it is only available for testing risk difference in the current SAS procedures. Barnard test is computational intense as compared to the Fisher’s exact test. The discussion between Barnard test and Fisher’s test is almost as old as statistical history. Another unconditional test developed by Lloyd (2008) is based on the parametric bootstrap and Barnard’s test. This new approach has been shown to be generally more powerful as compared to Barnard’s test in many statistical problems (Shan et al., 2012; Shan, 2013b, 2013a; Shan et al., 2013). 2

Table 2: Output from PROC FREQ for comparing confidence intervals. Risk ASE (Asymptotic) 90% (Exact) 90% Confidence Limits Confidence Limits 0.7333 0.2500 0.5652 0.4833

0.1142 0.1531 0.1034 0.1910

0.5455 0.0000 0.3952 0.1692

0.9211 0.5018 0.7352 0.7975

0.4892 0.0464 0.3754 0.1074

0.9033 0.5997 0.7418 0.7510

0.90 0.85 0.80

Infimum Coverage

0.95

1.00

Row 1 Row 2 Total Difference

−0.5

0.0

0.5

1.0

θd

Figure 1: Actual coverage probability of upper confidence intervals for n1 = n2 = 15 using an asymptotic approach at a 0.05 nominal level. ASYMPTOTIC AND EXACT CONFIDENCE INTERVALS The confidence intervals for the proportion difference are displayed in Table 2. PROC FREQ produces asymptotic (Wald-type) and exact (Clopper-Pearson) confidence limits by default. There are another three asymptotic confidence limits available from PROC FREQ. For instance, the 90% exact confidence limits based on Clopper-Pearson is (0.1074,0.7510). Wang (2010) developed exact smallest (optimal) one-sided confidence limits for the difference of two independent and dependent proportions. These confidence limits can be implemented by an R package ExactCIdiff (Shan & Wang, 2013). The associated 90% confidence limits based on this implemention are (0.09202,0.75105). Actual coverage probability of upper confidence intervals based on an asymptotic approach and the exact approach are shown in Figure 1 and Figure 2, respectively. The coverage probability of a confidence interval is the proportion that the calculated interval would contain the true value of the parameter of interest. As can be seen, the actual coverage is below the nominal level for majority of the proportion difference θd . Similar resultes are observed for other asymptotic confidence intervals. Exact one-sided confidence limits in Wang (2010)

3

0.99 0.98 0.97 0.95

0.96

Infimum Coverage

−0.5

0.0

0.5

1.0

θd

Figure 2: Actual coverage probability of upper confidence intervals for n1 = n2 = 15 using the exact approach by Wang 2010 at a 0.05 nominal level. respect the nominal level. Shan and Wang (2013) has shown that the new exact confidence limits outperform existing asymptotic and exact confidence limits with regards to the guarantee of coverage and short coverage length. More details about this new confidence limits, see Wang (2006, 2010); Shan and Wang (2013). CONCLUSION Exact tests and exact confidence limits should be used in practice for small to medium sample settings. It takes some time to compute exact p-values, and needs more efforts to calculate exact confidence limits. Shan and Wang (2013) provided the first available package in R to calculate exact confidence intervals for the difference of proportions. One of our ongoing project will be coming up with a method using SAS macros for calculating exact p-value and exact confident limits. REFERENCE Barnard, G. A. (1947). Significance tests for 2 × 2 tables. Biometrika, 34(1/2), 123–138. Lloyd, C. J. (2008). A new exact and more powerful unconditional test of no treatment effect from binary matched pairs. Biometrics, 64(3), 716–723. Shan, G. (2013a, February). More efficient unconditional tests for exchangeable binary data with equal cluster sizes. Statistics & Probability Letters, 83(2), 644–649. Shan, G. (2013b). A Note on Exact Conditional and Unconditional Tests for Hardy-Weinberg Equilibrium. Human Heredity, 76(1), 10–17.

4

Shan, G., Ma, C., Hutson, A. D., & Wilding, G. E. (2012). An efficient and exact approach for detecting trends with binary endpoints. Statistics in Medicine, 31(2), 155–164. Shan, G., Ma, C., Hutson, A. D., & Wilding, G. E. (2013, January). Some tests for detecting ˘ SWeißâ trends based on the modified BaumgartnerâA ¸ A˘ SSchindler ¸ statistics. Computational Statistics & Data Analysis, 57 (1), 246–261. Shan, G., & Wang, W. (2013). ExactCIdiff: An R Package for Computing Exact Confidence Intervals for the Difference of Two Proportions. The R Journal. Wang, W. (2006, December 01). Smallest confidence intervals for one binomial proportion. Journal of Statistical Planning and Inference, 136(12), 4293–4306. Wang, W. (2010, April). On construction of the smallest one-sided confidence interval for the difference of two proportions. The Annals of Statistics, 38(2), 1227–1243. ACKNOWLEDGMENTS Shan’s research is partially supported by Faculty Opportunity Awards from UNLV. INFORMATION Your comments and questions are valued and encouraged. Contact the author at: Guogen Shan Department of Environmental and Occupational Health School of Community Health Sciences University of Nevada Las Vegas, Las Vegas, NV 89154 [email protected] SAS and all other SAS Institute Inc. product or service names are registered trademarks R indicates USA registraor trademarks of SAS Institute Inc. in the USA and other countries. tion. Other brand and product names are trademarks of their respective companies.

5