Session 3. Dealing with Heterogeneity & Meta-regression Michael A. Stoto, PhD ICSA June 4, 2008 © Michael A. Stoto

Plan for the day 1. Introduction to systematic reviews and meta-analysis – Identify/clarify the question – Identify studies to be included

2. Analyze the data & draw conclusions – Basic statistical methods

3. Dealing with heterogeneity – Meta-regression

4. Meta-analysis for drug safety

1

Benefits of heterogeneity Berlin: “heterogeneity is our friend” • Heterogeneity is common – always some unexplained heterogeneity

• Consistency indicates robustness • Can yield insights – use an EDA (Exploratory Data Analysis) perspective – interpret with caution

Heterogeneity is common • Sources of heterogeneity – – – – –

Populations studied Nature and level of exposure/treatment Study methods and exposure/outcome measures Health outcomes studied, how assessed Dimensions of study quality

• Meta-analyses published in 1991-2 (Berlin) – 14/55 tested for heterogeneity – |z| [95% Conf. Interval] ---------+-------------------------------------------------------------------lat | -.0315724 .0061726 -5.115 0.000 -.0436704 -.0194744 _cons | .303035 .2108751 1.437 0.151 -.1102727 .7163427 -----------------------------------------------------------------------------. gen w=1/s^2 . gen fit=.303-.0316*lat . scatter x lat [fw=w], s(Oh) || line fit lat, ytitle("log OR")

17

0.500

BCG data - meta-regression

-1.000

log OR -0.500

0.000

Latitude 10 33.46 42 55

log OR -0.013 -0.754 -1.024 -1.435

Protective effect 0.01 0.53 0.64 0.76

-1.500

Log OR = .303 - .0316*lat

10

20

30 40 Degrees latitude from equator log OR

50

60

fit

Effect of meta-regression weights 0.700

FE

log OR

0.000

RE -0.700

Unweighted

Latitude log OR Random effects model 10 -0.013 33.46 -0.754 42 -1.024 55 -1.435 Fixed effects model 10 0.064 33.46 -0.713 42 -0.995 55 -1.426

Protective effect 0.01 0.53 0.64 0.76 -0.07 0.51 0.63 0.76

RE: Log OR = .303 - .0316*lat FE: Log OR = .395 - .0331*lat -1.600 0

20 40 Degrees latitude from equator

60

18

BCG data - meta-regression (default - REML approach) . metareg Iteration Iteration Iteration Iteration Iteration Iteration Iteration Iteration Iteration Iteration Iteration Iteration Iteration

logrr lat, wsse( sd) 1: tau^2 = 0 2: tau^2 = .00669243 3: tau^2 = .01929125 4: tau^2 = .03200278 5: tau^2 = .04179016 6: tau^2 = .04847299 7: tau^2 = .0527127 8: tau^2 = .05528125 9: tau^2 = .05679435 10: tau^2 = .05767105 11: tau^2 = .05817415 12: tau^2 = .05846126 13: tau^2 = .05862459

Meta-analysis regression

No of studies = tau^2 method tau^2 estimate =

13 reml .0587

Successive values of tau^2 differ by less than 10^-4 :convergence achieved -----------------------------------------------------------------------------| Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------lat | -.0293213 .0066082 -4.44 0.000 -.0422732 -.0163694 _cons | .2641798 .2267088 1.17 0.244 -.1801613 .7085209 ------------------------------------------------------------------------------

BCG data - meta-regression Comparison of approaches Method of moments (mm) -----------------------------------------------------------------------------| Coef. Std. Err. z P>|z| [95% Conf. Interval] ---------+-------------------------------------------------------------------lat | -.0315724 .0061726 -5.115 0.000 -.0436704 -.0194744 _cons | .303035 .2108751 1.437 0.151 -.1102727 .7163427 -----------------------------------------------------------------------------Restricted maximum likelihood (default - reml) -----------------------------------------------------------------------------| Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------lat | -.0293213 .0066082 -4.44 0.000 -.0422732 -.0163694 _cons | .2641798 .2267088 1.17 0.244 -.1801613 .7085209 -----------------------------------------------------------------------------Empirical Bayes (eb) Successive values of tau^2 differ by less than 10^-4 :convergence achieved -----------------------------------------------------------------------------| Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------lat | -.0285879 .0090572 -3.16 0.002 -.0463397 -.010836 _cons | .2238424 .3159682 0.71 0.479 -.395444 .8431287 ------------------------------------------------------------------------------

19

BCG validity (quality) scoring • Vaccine trials based on – method of vaccine assignment – availability for follow-up – equality of surveillance – TB diagnosis criteria – preparation of BCG vaccine • 0 – 20 scale • Assessed prior to analysis of results

BCG meta-regression by validity Variables in model Latitude Validity Both

0.693

log OR

0.000

approx. R2 0.41 0.30 0.66

Log OR = 1.0 - .16*validity -0.693

-1.609

5

10

Validity score

15

20

20

Control rate regression • Is the effect size related to baseline risk? – Measured by outcome rate in the control group – Statistical model: X = α + βXc – Statistical problems • X is correlated with Xc • Xc measured with error – Statistical solutions • Egger Chapter 10 • Schmid CH et al., 1998, Statistics in Medicine 17:1923-42

BCG control rate regression . metan a b c d, or random sortby(r0) label(namevar=study) xlabel(0.1, 0.2, 0.5, 1, 2, 5, 10) Odds ratio (95% C I)

Study

% W eight

G eorgia 33 1969

1. 56 (0. 37, 6.55 )

4.1

G eorgia 33 1976

0. 98 (0. 58, 1.66 )

8.4

P uert o R ic o 18 1974

0. 71 (0. 57, 0.89 )

9.7

Ma dras 13 1980

1. 01 (0. 89, 1.15 )

10. 0

S out h A f ric a 2 7 1968

0. 62 (0. 39, 1.00 )

8.7

Ma danapall e 13 1973

0. 80 (0. 51, 1.26 )

8.8

Ha it i 19 197 3

0. 20 (0. 08, 0.50 )

6.2

UK 52 1 977

0. 23 (0. 18, 0.31 )

9.6

Ch ic ago 42 1961

0. 25 (0. 14, 0.42 )

8.4

Ch ic ago 42 1960

0. 25 (0. 07, 0.91 )

4.6

No rt hern U S A 44 194 8

0. 39 (0. 12, 1.26 )

5.1

Ca nada 55 1949

0. 19 (0. 08, 0.46 )

6.4

No rt hern U S A 44 195 3

0. 38 (0. 32, 0.47 )

9.8

O v erall (95% CI )

0. 47 (0. 32, 0.69 )

.1

.2

.5

1

2

5 10 Odds ratio

21

BCG control rate regression . metareg Iteration Iteration Iteration Iteration Iteration Iteration

logrr logrr0, wsse(sd) 1: tau^2 = 0 2: tau^2 = .08253536 3: tau^2 = .20633473 4: tau^2 = .18628138 5: tau^2 = .18918129 6: tau^2 = .18875165

Meta-analysis regression

No of studies = tau^2 method tau^2 estimate =

13 reml .1888

Successive values of tau^2 differ by less than 10^-4 :convergence achieved --------------------------------------------------------------------------| Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+------------------------------------------------------------logrr0 | -.270705 .0981441 -2.76 0.006 -.4630639 -.0783461 _cons | -1.828303 .4363756 -4.19 0.000 -2.683583 -.9730223 ---------------------------------------------------------------------------------------

BCG control rate regression

.5

. sort logrr0 . gen w=1/sd^2 . gen fit=-1.828 -.2707*logrr0 . scatter logrr logrr0 [fw=w], msymbol(Oh) ytitle("Log RR") || line fit logrr0

-1

Log RR -.5

0

CR = 0.0013

-1.5

CR = 0.26

-6

-4 Log control rate Log RR

-2

0

fit

22

BCG vaccine by control rate Model

Fixed Random

Fixed Random Fixed Random

Group by Control rate

Hign Hign Hign Hign Hign Hign Hign Hign Hign Low Low Low Low Low Low Low Low Overall Overall

Study name

Odds ratio and 95% CI

Aronson, 1948 Ferguson & Simes, 1949 Rosenthal, 1960 Hart & Sutherland, 1977 Stein & Aronson, 1953 Vandiviere, 1973 Rosenthal, 1961

Frimodt-Moller, 1973 Madras, 1980 Coetze & Berjak, 1968 Comstock, 1974 Comstock & Webster, 1969 Comstock, 1976

0.1

0.2

0.5

1

BCG effective

2

5

10

BCG not effective

BCG conclusions • Heterogeneity in BCG results associated with – latitude – study validity (quality) – control rate (may reflect problems with diagnosis) • BCG vaccine efficacy in the US (with a “perfect” study) is likely to be substantially greater than 50%

23

BCG conclusions • Overall, BCG vaccine has a substantial protective effect (34 - 51%) – fixed effects RR = 0.66 (95% C.I. 0.61 – 0.72) – random effects RR = 0.49 (95% C.I. 0.34 – 0.70)

• Substantial heterogeneity: vaccine more effective – Higher latitudes (~ 64% in Boston) – Higher quality studies – With harder outcomes

Vitamin E and all-cause mortality Miller et al., Annals of Internal Medicine, 2005 • Vitamin E and other anti-oxidants thought to have major benefits for prevention of chronic disease – Benefits not clearly established to date – β-carotene shown to be harmful – Vitamin E won’t hurt and might help, so why not take it?

• Many recent trials tested vitamin E supplementation to prevent chronic diseases – Prevention trials tend to have large n

• Three recent meta-analyses found no overall effect on survival – Did not consider dose-response relationship

24

Vitamin E and all-cause mortality Miller et al., Annals of Internal Medicine, 2005 • Purpose – To perform a meta-analysis of the doseresponse relationship between vitamin E supplementation and total mortality • Search strategy – Medline (PubMed) 1966 - August 2004 – Supplemented with • Cochrane clinical trials database • Review of citations and published reviews and meta-analyses

– No language restrictions

Inclusion criteria • RCTs • Vitamin E alone or combined with other vitamins or minerals • Only men or non-pregnant women • Duration of supplementation > 1 yr and > 10 deaths

25

Flow diagram of the trial selection process

Clinical Trials of Vitamin E Supplementation and Risk for All-Cause Mortality, Ordered by Dosage of Vitamin E

26

Statistical methods • Focus on 2-way analyses of factorial trials – Vitamin E vs. no vitamin E, regardless of other arm

• 2-level hierarchical logistic regression model – Within-study: logistic regression for probability of death as a function of assignment to vitamin E arm • Metameter = β in logistic regression for vitamin E – Between-study: • Indicator variable for vitamin E > 400 IU/d • Quadratic spline (X = log dose) – Quadratic below 400 IU/d – Linear above 400 IU/d

• Results translated to risk difference and risk ratios

Risk difference in all-cause mortality Low- vs. high-dosage vitamin E, 2-way analyses

27

Results: 2-way factorial analysis Dose Overall

RD (per 95% RR 10,000) CI 10

Low

-16

High

39

-18, 38 -41, 10 3, 74

1.01 0.98 1.04

95% Q CI test 0.98, 1.04 0.96, 1.01 1.01, 1.07

0.02

0.18

Risk difference in all-cause mortality Low- vs. high-dosage vitamin E, 4-way analyses

28

Results: 4-way factorial analysis Dose

RD (per 10,000)

Overall Low High

95% CI

2-level 10 -18, 38 -16 -41, 10 39

3, 74

RD (per 10,000)

95% CI

4-level 8 -23, 39 -33 -60, -5 34

5, 63

Q test

0.039

0.2

All-Cause Mortality Risk Differences and Risk Ratios for Vitamin E Trials Adjusted for Study-Specific Variables

• 2-level hierarchical logistic regression model – Within-study: logistic regression for probability of death as a function of assignment to vitamin E arm • Metameter = β in logistic regression for vitamin E – Between-study: • Indicator variable for vitamin E > 400 IU/d • Quadratic spline (X = log dose) – Quadratic below 400 IU/d – Linear above 400 IU/d

29

Dose-response relationship between vitamin E supplementation and all-cause mortality in randomized, controlled trials

Pooled All-Cause Mortality Risk Differences and Risk Ratios for Selected Vitamin E Dosages

30

Discussion • Identified a dose-dependent relationship between vitamin E supplementation and allcause mortality – Mortality progressively increased for > 150 IU/d

• Similar results in anti-oxidant meta-analysis – Mortality analyses exploratory & incomplete

• Obvious public health importance • Editorial – Despite uncertainty of results, high-dose vitamin E supplementation is unjustified

Letters to the Editor • > 40 received, 11 published on July 19, 2005 • Natural and synthetic vitamin E are not equivalent – Subgroup analysis of 4 high-dose trials • Natural RR = 1.05 (95% CI 0.97, 1.13) • Synthetic RR = 1.04 (95% CI 1.00, 1.07)

• Generalizeability to healthy populations – Prevention trials enroll high-risk subjects to increase power – Women’s Health Study results support findings

• Some CV risk factors in some studies higher in vitamin E group – Not likely to be imbalanced in 135,000 subjects

31

Letters to the Editor • Statistical dose-response model – Should be constant-linear • Biologically implausible • Sharp change at knot, depends where set

• Increased risk not evident until 400 IU/d – Hard to determine • Women’s Health Study suggests threshold < 400 IU/d

– Hierarchical regression vs. meta-regression • Produces same results as if individual patient data were available • Results are similar to meta-regression

Letters to the Editor • Exclusion of trials with < 10 deaths – Studies < 1 yr and < 10 deaths designed for intermediate outcomes and may not collect or report mortality data systematically – Mortality studies not likely to have < 10 deaths – No a priori reason to expect bias in results • In 11 studies with < 10 deaths, treatment > placebo deaths

• Two trials showed beneficial effect on surrogate markers for atherosclerosis – But mortality results were consistent with MA findings

• Total mortality outcomes masks positive results based on physiological variables related to oxidative stress – All-cause mortality has unambiguous clinical relevance

32

Un-answered points • No evidence of dose response effect in studies that looked for it • Suggestion of publication bias in high-dose trials • Should have searched EMBASE for European trials • β-carotene may be a confounder • Data problems

Authors’ response • 19 RCTs with > 135,000 participants failed to document a survival benefit with vitamin E supplementation • Dose-response analysis “provided evidence that high-dosage vitamin E supplementation may increase total mortality” • “While future trials will refine the estimates of the effect .. and the dose at which the relative risk for death exceeds 1, we stand by our conclusion that use of high-dosage vitamin E supplementation should be avoided.”

33

Take Away Points • “Heterogeneity is our friend” – Heterogeneity is common – Consistency indicates robustness – Can yield insights - use EDA perspective and interpret with caution • Dealing with heterogeneity – Random effects model - Q test, I2 statistic – Subgroup/sensitivity analysis – Regression approach • continuous or indicator values

34