Session 3. Dealing with Heterogeneity & Meta-regression Michael A. Stoto, PhD ICSA June 4, 2008 © Michael A. Stoto
Plan for the day 1. Introduction to systematic reviews and meta-analysis – Identify/clarify the question – Identify studies to be included
2. Analyze the data & draw conclusions – Basic statistical methods
3. Dealing with heterogeneity – Meta-regression
4. Meta-analysis for drug safety
1
Benefits of heterogeneity Berlin: “heterogeneity is our friend” • Heterogeneity is common – always some unexplained heterogeneity
• Consistency indicates robustness • Can yield insights – use an EDA (Exploratory Data Analysis) perspective – interpret with caution
Heterogeneity is common • Sources of heterogeneity – – – – –
Populations studied Nature and level of exposure/treatment Study methods and exposure/outcome measures Health outcomes studied, how assessed Dimensions of study quality
• Meta-analyses published in 1991-2 (Berlin) – 14/55 tested for heterogeneity – |z| [95% Conf. Interval] ---------+-------------------------------------------------------------------lat | -.0315724 .0061726 -5.115 0.000 -.0436704 -.0194744 _cons | .303035 .2108751 1.437 0.151 -.1102727 .7163427 -----------------------------------------------------------------------------. gen w=1/s^2 . gen fit=.303-.0316*lat . scatter x lat [fw=w], s(Oh) || line fit lat, ytitle("log OR")
17
0.500
BCG data - meta-regression
-1.000
log OR -0.500
0.000
Latitude 10 33.46 42 55
log OR -0.013 -0.754 -1.024 -1.435
Protective effect 0.01 0.53 0.64 0.76
-1.500
Log OR = .303 - .0316*lat
10
20
30 40 Degrees latitude from equator log OR
50
60
fit
Effect of meta-regression weights 0.700
FE
log OR
0.000
RE -0.700
Unweighted
Latitude log OR Random effects model 10 -0.013 33.46 -0.754 42 -1.024 55 -1.435 Fixed effects model 10 0.064 33.46 -0.713 42 -0.995 55 -1.426
Protective effect 0.01 0.53 0.64 0.76 -0.07 0.51 0.63 0.76
RE: Log OR = .303 - .0316*lat FE: Log OR = .395 - .0331*lat -1.600 0
20 40 Degrees latitude from equator
60
18
BCG data - meta-regression (default - REML approach) . metareg Iteration Iteration Iteration Iteration Iteration Iteration Iteration Iteration Iteration Iteration Iteration Iteration Iteration
logrr lat, wsse( sd) 1: tau^2 = 0 2: tau^2 = .00669243 3: tau^2 = .01929125 4: tau^2 = .03200278 5: tau^2 = .04179016 6: tau^2 = .04847299 7: tau^2 = .0527127 8: tau^2 = .05528125 9: tau^2 = .05679435 10: tau^2 = .05767105 11: tau^2 = .05817415 12: tau^2 = .05846126 13: tau^2 = .05862459
Meta-analysis regression
No of studies = tau^2 method tau^2 estimate =
13 reml .0587
Successive values of tau^2 differ by less than 10^-4 :convergence achieved -----------------------------------------------------------------------------| Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------lat | -.0293213 .0066082 -4.44 0.000 -.0422732 -.0163694 _cons | .2641798 .2267088 1.17 0.244 -.1801613 .7085209 ------------------------------------------------------------------------------
BCG data - meta-regression Comparison of approaches Method of moments (mm) -----------------------------------------------------------------------------| Coef. Std. Err. z P>|z| [95% Conf. Interval] ---------+-------------------------------------------------------------------lat | -.0315724 .0061726 -5.115 0.000 -.0436704 -.0194744 _cons | .303035 .2108751 1.437 0.151 -.1102727 .7163427 -----------------------------------------------------------------------------Restricted maximum likelihood (default - reml) -----------------------------------------------------------------------------| Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------lat | -.0293213 .0066082 -4.44 0.000 -.0422732 -.0163694 _cons | .2641798 .2267088 1.17 0.244 -.1801613 .7085209 -----------------------------------------------------------------------------Empirical Bayes (eb) Successive values of tau^2 differ by less than 10^-4 :convergence achieved -----------------------------------------------------------------------------| Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------lat | -.0285879 .0090572 -3.16 0.002 -.0463397 -.010836 _cons | .2238424 .3159682 0.71 0.479 -.395444 .8431287 ------------------------------------------------------------------------------
19
BCG validity (quality) scoring • Vaccine trials based on – method of vaccine assignment – availability for follow-up – equality of surveillance – TB diagnosis criteria – preparation of BCG vaccine • 0 – 20 scale • Assessed prior to analysis of results
BCG meta-regression by validity Variables in model Latitude Validity Both
0.693
log OR
0.000
approx. R2 0.41 0.30 0.66
Log OR = 1.0 - .16*validity -0.693
-1.609
5
10
Validity score
15
20
20
Control rate regression • Is the effect size related to baseline risk? – Measured by outcome rate in the control group – Statistical model: X = α + βXc – Statistical problems • X is correlated with Xc • Xc measured with error – Statistical solutions • Egger Chapter 10 • Schmid CH et al., 1998, Statistics in Medicine 17:1923-42
BCG control rate regression . metan a b c d, or random sortby(r0) label(namevar=study) xlabel(0.1, 0.2, 0.5, 1, 2, 5, 10) Odds ratio (95% C I)
Study
% W eight
G eorgia 33 1969
1. 56 (0. 37, 6.55 )
4.1
G eorgia 33 1976
0. 98 (0. 58, 1.66 )
8.4
P uert o R ic o 18 1974
0. 71 (0. 57, 0.89 )
9.7
Ma dras 13 1980
1. 01 (0. 89, 1.15 )
10. 0
S out h A f ric a 2 7 1968
0. 62 (0. 39, 1.00 )
8.7
Ma danapall e 13 1973
0. 80 (0. 51, 1.26 )
8.8
Ha it i 19 197 3
0. 20 (0. 08, 0.50 )
6.2
UK 52 1 977
0. 23 (0. 18, 0.31 )
9.6
Ch ic ago 42 1961
0. 25 (0. 14, 0.42 )
8.4
Ch ic ago 42 1960
0. 25 (0. 07, 0.91 )
4.6
No rt hern U S A 44 194 8
0. 39 (0. 12, 1.26 )
5.1
Ca nada 55 1949
0. 19 (0. 08, 0.46 )
6.4
No rt hern U S A 44 195 3
0. 38 (0. 32, 0.47 )
9.8
O v erall (95% CI )
0. 47 (0. 32, 0.69 )
.1
.2
.5
1
2
5 10 Odds ratio
21
BCG control rate regression . metareg Iteration Iteration Iteration Iteration Iteration Iteration
logrr logrr0, wsse(sd) 1: tau^2 = 0 2: tau^2 = .08253536 3: tau^2 = .20633473 4: tau^2 = .18628138 5: tau^2 = .18918129 6: tau^2 = .18875165
Meta-analysis regression
No of studies = tau^2 method tau^2 estimate =
13 reml .1888
Successive values of tau^2 differ by less than 10^-4 :convergence achieved --------------------------------------------------------------------------| Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+------------------------------------------------------------logrr0 | -.270705 .0981441 -2.76 0.006 -.4630639 -.0783461 _cons | -1.828303 .4363756 -4.19 0.000 -2.683583 -.9730223 ---------------------------------------------------------------------------------------
BCG control rate regression
.5
. sort logrr0 . gen w=1/sd^2 . gen fit=-1.828 -.2707*logrr0 . scatter logrr logrr0 [fw=w], msymbol(Oh) ytitle("Log RR") || line fit logrr0
-1
Log RR -.5
0
CR = 0.0013
-1.5
CR = 0.26
-6
-4 Log control rate Log RR
-2
0
fit
22
BCG vaccine by control rate Model
Fixed Random
Fixed Random Fixed Random
Group by Control rate
Hign Hign Hign Hign Hign Hign Hign Hign Hign Low Low Low Low Low Low Low Low Overall Overall
Study name
Odds ratio and 95% CI
Aronson, 1948 Ferguson & Simes, 1949 Rosenthal, 1960 Hart & Sutherland, 1977 Stein & Aronson, 1953 Vandiviere, 1973 Rosenthal, 1961
Frimodt-Moller, 1973 Madras, 1980 Coetze & Berjak, 1968 Comstock, 1974 Comstock & Webster, 1969 Comstock, 1976
0.1
0.2
0.5
1
BCG effective
2
5
10
BCG not effective
BCG conclusions • Heterogeneity in BCG results associated with – latitude – study validity (quality) – control rate (may reflect problems with diagnosis) • BCG vaccine efficacy in the US (with a “perfect” study) is likely to be substantially greater than 50%
23
BCG conclusions • Overall, BCG vaccine has a substantial protective effect (34 - 51%) – fixed effects RR = 0.66 (95% C.I. 0.61 – 0.72) – random effects RR = 0.49 (95% C.I. 0.34 – 0.70)
• Substantial heterogeneity: vaccine more effective – Higher latitudes (~ 64% in Boston) – Higher quality studies – With harder outcomes
Vitamin E and all-cause mortality Miller et al., Annals of Internal Medicine, 2005 • Vitamin E and other anti-oxidants thought to have major benefits for prevention of chronic disease – Benefits not clearly established to date – β-carotene shown to be harmful – Vitamin E won’t hurt and might help, so why not take it?
• Many recent trials tested vitamin E supplementation to prevent chronic diseases – Prevention trials tend to have large n
• Three recent meta-analyses found no overall effect on survival – Did not consider dose-response relationship
24
Vitamin E and all-cause mortality Miller et al., Annals of Internal Medicine, 2005 • Purpose – To perform a meta-analysis of the doseresponse relationship between vitamin E supplementation and total mortality • Search strategy – Medline (PubMed) 1966 - August 2004 – Supplemented with • Cochrane clinical trials database • Review of citations and published reviews and meta-analyses
– No language restrictions
Inclusion criteria • RCTs • Vitamin E alone or combined with other vitamins or minerals • Only men or non-pregnant women • Duration of supplementation > 1 yr and > 10 deaths
25
Flow diagram of the trial selection process
Clinical Trials of Vitamin E Supplementation and Risk for All-Cause Mortality, Ordered by Dosage of Vitamin E
26
Statistical methods • Focus on 2-way analyses of factorial trials – Vitamin E vs. no vitamin E, regardless of other arm
• 2-level hierarchical logistic regression model – Within-study: logistic regression for probability of death as a function of assignment to vitamin E arm • Metameter = β in logistic regression for vitamin E – Between-study: • Indicator variable for vitamin E > 400 IU/d • Quadratic spline (X = log dose) – Quadratic below 400 IU/d – Linear above 400 IU/d
• Results translated to risk difference and risk ratios
Risk difference in all-cause mortality Low- vs. high-dosage vitamin E, 2-way analyses
27
Results: 2-way factorial analysis Dose Overall
RD (per 95% RR 10,000) CI 10
Low
-16
High
39
-18, 38 -41, 10 3, 74
1.01 0.98 1.04
95% Q CI test 0.98, 1.04 0.96, 1.01 1.01, 1.07
0.02
0.18
Risk difference in all-cause mortality Low- vs. high-dosage vitamin E, 4-way analyses
28
Results: 4-way factorial analysis Dose
RD (per 10,000)
Overall Low High
95% CI
2-level 10 -18, 38 -16 -41, 10 39
3, 74
RD (per 10,000)
95% CI
4-level 8 -23, 39 -33 -60, -5 34
5, 63
Q test
0.039
0.2
All-Cause Mortality Risk Differences and Risk Ratios for Vitamin E Trials Adjusted for Study-Specific Variables
• 2-level hierarchical logistic regression model – Within-study: logistic regression for probability of death as a function of assignment to vitamin E arm • Metameter = β in logistic regression for vitamin E – Between-study: • Indicator variable for vitamin E > 400 IU/d • Quadratic spline (X = log dose) – Quadratic below 400 IU/d – Linear above 400 IU/d
29
Dose-response relationship between vitamin E supplementation and all-cause mortality in randomized, controlled trials
Pooled All-Cause Mortality Risk Differences and Risk Ratios for Selected Vitamin E Dosages
30
Discussion • Identified a dose-dependent relationship between vitamin E supplementation and allcause mortality – Mortality progressively increased for > 150 IU/d
• Similar results in anti-oxidant meta-analysis – Mortality analyses exploratory & incomplete
• Obvious public health importance • Editorial – Despite uncertainty of results, high-dose vitamin E supplementation is unjustified
Letters to the Editor • > 40 received, 11 published on July 19, 2005 • Natural and synthetic vitamin E are not equivalent – Subgroup analysis of 4 high-dose trials • Natural RR = 1.05 (95% CI 0.97, 1.13) • Synthetic RR = 1.04 (95% CI 1.00, 1.07)
• Generalizeability to healthy populations – Prevention trials enroll high-risk subjects to increase power – Women’s Health Study results support findings
• Some CV risk factors in some studies higher in vitamin E group – Not likely to be imbalanced in 135,000 subjects
31
Letters to the Editor • Statistical dose-response model – Should be constant-linear • Biologically implausible • Sharp change at knot, depends where set
• Increased risk not evident until 400 IU/d – Hard to determine • Women’s Health Study suggests threshold < 400 IU/d
– Hierarchical regression vs. meta-regression • Produces same results as if individual patient data were available • Results are similar to meta-regression
Letters to the Editor • Exclusion of trials with < 10 deaths – Studies < 1 yr and < 10 deaths designed for intermediate outcomes and may not collect or report mortality data systematically – Mortality studies not likely to have < 10 deaths – No a priori reason to expect bias in results • In 11 studies with < 10 deaths, treatment > placebo deaths
• Two trials showed beneficial effect on surrogate markers for atherosclerosis – But mortality results were consistent with MA findings
• Total mortality outcomes masks positive results based on physiological variables related to oxidative stress – All-cause mortality has unambiguous clinical relevance
32
Un-answered points • No evidence of dose response effect in studies that looked for it • Suggestion of publication bias in high-dose trials • Should have searched EMBASE for European trials • β-carotene may be a confounder • Data problems
Authors’ response • 19 RCTs with > 135,000 participants failed to document a survival benefit with vitamin E supplementation • Dose-response analysis “provided evidence that high-dosage vitamin E supplementation may increase total mortality” • “While future trials will refine the estimates of the effect .. and the dose at which the relative risk for death exceeds 1, we stand by our conclusion that use of high-dosage vitamin E supplementation should be avoided.”
33
Take Away Points • “Heterogeneity is our friend” – Heterogeneity is common – Consistency indicates robustness – Can yield insights - use EDA perspective and interpret with caution • Dealing with heterogeneity – Random effects model - Q test, I2 statistic – Subgroup/sensitivity analysis – Regression approach • continuous or indicator values
34