Managing Operational Challenges for Interim Analyses

P021 Managing Operational Challenges for Interim Analyses Mei Dey, Merck & Co., Inc Lisa Pyle, Merck & Co., Inc ABSTRACT Interim analysis plays an i...
Author: Virgil Price
24 downloads 3 Views 146KB Size
P021

Managing Operational Challenges for Interim Analyses Mei Dey, Merck & Co., Inc Lisa Pyle, Merck & Co., Inc

ABSTRACT Interim analysis plays an important role in clinical trial studies. As a result of interim analysis, a study may be terminated early if results at the end of the trial are likely to be negative. More recently, interim analysis is seen as a powerful tool to help reduce cycle time and therefore 'time to market' by allowing key decisions to be made earlier regarding dose finding, sample size re-estimation, clinical benefit or safety and efficacy monitoring. Unlike analysis done after the official close of the study, interim analysis poses some unique challenges for programming. This paper will examine the roles of programmers and statisticians in interim analyses and how to manage operational and programming challenges so data analysis results can be generated in the most effective and efficient manner to achieve the goal of interim analyses.

INTRODUCTION What is an interim analysis? Any analysis performed on unblinded data prior to the completion of the data collection in a trial. Treatment arms are compared in terms of safety or efficacy. Interim analyses are typically pre-planned analyses and are performed for any number of ethical or economic reasons. Some reasons for conducting an interim analysis might be to stop the development of ineffective treatments (futility analysis), to stop development of toxic treatments, to plan additional studies, to plan for product launch, to make regulatory decisions among many other reasons. Per ICH (International Conference on Harmonization) guidelines, all interim analyses should be planned in advance and specified in the protocol or protocol amendment. This includes rationale for interim analysis, statistical methods, timing and number of interim analyses, whether or how the conduct of the study will be affected(i.e., early termination, dosage adjustment, modification of study design).

BUSINESS CASE Our study is a randomized, double-blind, placebo-controlled oncology trial. The primary objective of the study is to compare the overall survival between treatment and placebo. Three Interim analyses are planned at three pre-specified time-points. Results of an interim analysis are reviewed by sponsor-independent personnel, a Data Safety Monitoring Board (DSMB), to minimize trial biases and to maintain trial integrity. The DSMB serves as an independent reviewer of safety and efficacy data and is charged with interpreting interim analysis results in order to make periodic recommendations on whether to continue, modify or terminate the study. In this business case, only ONE unblinded statistician has access to unblinded interim analyses data and results. Neither programmers, protocol statistician, data management nor clinical personnel have access to unblinded interim analyses data. The unblinded statistician executes and provides the final table/listing/graph package to the DSMB.

All results are unblinded at interim analyses, including identification of the two treatment groups in all tables and figures. The treatment groups are labeled as "Treatment A" and "Treatment B" to maintain partial blinding.

ROLES AND RESPONSIBILITIES FOR INTERIM ANALYSES Prior to unblinding for an interim analysis, a decision must be made by the project team and approved by the DSMB as to who will be the unblinded statistician and to what extent the data and associated databases will be unblinded and whether or not the protocol statistician will perform any analysis. These would be specified in the DSMB charter. A careful effort is made to ensure that no other staff member or investigator sees unblinded analyses. The protocol statistician is responsible for issuing a data analysis plan (DAP) prior to the unblinding of the data for the first interim analysis. The DAP contains detailed information on the number of planned interim analyses to be expected along with the criteria for termination of the trial. A second statistician, the unblinded statistician is primarily responsible for identifying and extracting data for statistical analysis and reporting at unblinding, and executing the programs. Lastly, they are responsible for writing the final unblinded report and providing the results to the external independent DSMB. The programmer's responsibility includes coordinating with the protocol statistician to determine programming requirements, develop statistical programs in accordance with programming SOPs (standard operationg procedures) and monitor program development and testing to ensure the programming and analysis is in place for the unblinded statisician.

Roles and Responsibilities by Functional Role Protocol

Unblinded

Project

Statistician Statistician Programmer Roles Data analysis plan √ Develop DSMB charter √ Determination of Interim Analysis √ Reports Report Specifications √ √ Tracking document tool √ Program development (Extract data, √ prepare analysis data sets and programs for statistical analysis and reporting). Validation and Testing √ √ √ Program execution user help file √ Perform blinded analyses (Run data √ √ setup and analysis programs in the production environment) Unblind data for statistical analysis and √ reporting for interim analysis Perform unblinded analyses (Run data √ setup and analysis programs in the production environment) Program execution support for interim *√ √ analysis. Write the statistical reports and √ deseminate to DSMB members *Primary support is provided by the project programmer. Backup support is provided by an independent unblinded programmer, if required.

OPERATIONAL CHALLENGES Planning and Timelines Prior to any programming development, a meeting is scheduled to review the DAP and to determine the needs for the interim analysis. An agreement is made between the protocol statistician and the clinical team on the subset of clinical study report (CSR) statistical analyses required for interim review. As a result, a mock-up table package for interim analysis is prepared by protocol statistician and handed off to programmers so that programming can begin. The protocol statistician and programmer will jointly develop program specifications. Once a collection of interim reports is available, a tracking document tool can be used to monitor the programming development and validation process for the interim analysis reports. Planning for program development at this point, utilizes the DSMB meeting date as a reference point for having development work completed. Development timelines, for this business case, were determined based on the pre-defined protocol critieria in combination with the DSMB meeting date rather than the CSR completion date. Consequently timelines are very compressed therefore time needs to be allocated for to allow the unblinded statistician to become acquainted with the programming and unblinding process so that they can deliver results in a timely fashion. Timelines can become even more compressed because they also depend on the timing of frozen file or data ready. Should the timing change for interim database lock be either earlier or later than expected then programming timelines would be impacted. An earlier frozen file would require programming development to be completed even quicker so that information could be passed to the unblinded statistician and a later frozen file would place the unblinded statistician at risk for having results ready. For this business case, a month was allowed prior to the DSMB meeting date for the unblinded statistician to have adequate time for unblinding, execution of programs and reporting results. Data Collection and Analysis Since interim analysis may lead to the termination of a trial, the data on which this decision is based should be as clean as possible. Prior to interim database lock, processes were developed to ensure that interim data have undergone consistency checks to identify, query and correct anomalies. There is a need to determine what subset of data is required to be 100% accurate for the interim analysis reports and analysis and to ensure that it is available, clean and complete. For example, key efficacy endpoints and safety parameters required a 100% data review for analysis. Although the programmer is not typically involved with the data cleaning process, in this case programs were developed to check and report data inconsistency for key safety and efficacy endpoint to facilitate the data review and the efficacy analysis. In addition, a cutoff date for data collection was pre-specified. Data up to the cutoff date was collected for all patients and subsequently the database was "frozen" up to the pre-specified date on a per patient basis. This ensured that data up to this time point was clean and accurate. This included data collected on cumulative forms. Insisting that data on cumulative forms be entered completely up to the point of analysis ensures actual data is available and that no estimation or derivations are required for missing data. The same cutoff date pre-specified for data collection is in turn used for data analysis, specifically in any survival analyses that may be performed. This cutoff date is globally coded in programming so that with every interim look, it can be updated. When the primary endpoint is a morbidity or mortality endpoint, patient status at interim analysis is important. Furthermore, distinguishing and identifying which patients have discontinued or completed the trial versus those patients who are continuing is also important. In oncology trials where the

sample sizes are small, patient status may be integrated in the statistical results to indicate that patients are still ongoing and still responding versus those patients who have responded but discontinued the trial and this distinction may impact the interpretation of the data. Program Development Considerations 1) Maintain Blinding Special programming consideration should be given in order to maintain confidentiality of interim analysis results. Some ways to achieve this is to use a mock treatment file, restrict the location of actual result generation and partial blind final reports. ƒ Mock treatment file

Accidentally unblinding data beyond the "prespecified" individual(s) can comprise the integrity of the trial results and company confidentiality. In our case, a mock treatment group file was developed to facilitate programming so all programmers remained blinded during development. We used mock treatment assignment coding to develop data analysis programs, which generated the sham unblinded interim reports. The individual treatment assignment or allocation schedule was maintained by the unblinded statistician who would merge data files with the actual individual treatment assignment at time of interim analysis unblinding. Startup.sas /** toggle between mock and actual treatment assignment coding files **/ %let an = datadir.trtcodes;

Trt_mock_codes.sas /**mock treament assignment coding, used during development**/ data datadir.trtcodes (keep=an trtcd); set rawdir.s_demos(rename=(an_num=an)); if an ^= . ; trtcd = mod(an,2) + 1 ; run;

Trt_act_codes.sas /**actual treament assignment coding, used during production, Coding developed by unblinded statistician**/ data datadir.trtcodes2 (keep=an trtcd); . . . run;

ƒ Restrict the unblinded statistician to executing results on a secure drive on his/her password-

protected PC. ƒ As an additional precaution, all unblinded analyses have treatment assignments labled by "A" and

"B", rather than the actual name of treatment. While this approach is not a fullproof approach to

maintaining blinding when there is not equal randomization of treatment groups, it can be used as an added measure when there are equal treatment groups. The same letters are used throughout the report for each treatment and are controlled by a SAS® proc format. With this coding schema, the DSMB can be unblinded, but reports remained blinded so that if someone unrelated to the study protocol reviewed the reports, they would be unable to determine the outcome. At the closed DSMB meeting during which interim analysis reports are discussed, members will receive the actual treatment codes. proc format ; value trtfmt 1 = 'Treatment A' 2 = 'Treatment B' 3 = 'Total'; run; 2) Batch submit To assist the unblinded statistician in executing interim analysis programs, SAS® programs using %include statements with redirected input/output for the log and list files is created. Alternatively, a batch executable file can be developed to perform the same function. Either can be chosen depending on the unblinded statistician's preference as he/she is the ultimate user executing the final reports. run_safety.sas /* Approach 1:

utilizes %include statements

*/

dm log "clear"; dm output "clear"; dm odsresults "clear; pgm"; proc datasets nolist kill memtype=data; quit; %inc "&pgmdir.\patient-population.sas"; dm output "file &listdir.\patient-population.lst replace"; dm log "file &logdir.\patient-population.log replace";

dm log "clear"; dm output "clear"; dm odsresults "clear; pgm"; proc datasets nolist kill memtype=data; quit; %inc "&pgmdir.\baseline-characteristics.sas"; dm output "file &listdir.\baseline-characteristics.lst replace"; dm log "file &logdir.\baseline-characteristics.log replace"; . . . and so on for additional programs

run_safety.bat Rem Approach 2 utilitizes ms-dos batch executable utility rem SAS batch file located in \\sharedrive1\Product1\Indic1\utility\run_safety.bat rem Each line represents a sas file to be submitted. rem 02MAR06 C:\PROGRA~1\SASINS~1\SAS\V8\SAS.EXE -sysin \\sharedrive1\Product1\Indic1\pgm_analysis\patient-population.sas -print \\sharedrive1\Product1\Indic1\out_listings\patient-population.lst -log \\sharedrive1\Product1\Indic1\out_logs\patient-population.log -config C:\PROGRA~1\SASINS~1\SAS\V8\SASV8.CFG -nologo -autoexec \\sharedrive1\Product1\Indic1\utility\startup.sas C:\PROGRA~1\SASINS~1\SAS\V8\SAS.EXE -sysin \\sharedrive1\Product1\Indic1\pgm_analysis\baseline-characteristics.sas -print \\sharedrive1\Product1\Indic1\out_listings\ baseline-characteristics.lst -log \\sharedrive1\Product1\Indic1\out_logs\ baseline-characteristics.log -config C:\PROGRA~1\SASINS~1\SAS\V8\SASV8.CFG -nologo -autoexec \\sharedrive1\Product1\Indic1\utility\startup.sas

3) Build portable programming The unblinded statistician in this case worked on a different network in a different physical location so the programming needed to be location independent. In addition S-PLUS® graphic software was required and as a result, dynamic programming was necessary so that the programs accessed the user's graphic software on their local PC to develop graphic reports. When accessing software other than SAS®, code should be developed to point to the software on the person's machine. Similar considerations would be needed if the unblinded statistician was working on a different platform, i.e. platform independent coding. %macro spls; %global splusloc; %if "&sysuserid" = "user1" %then %let splusloc = c:\ProgramFiles\Insightful\splus61network\cmd\;; %if "&sysuserid" ne "user2" %then %let splusloc = c:\ProgramFiles\Insightful\splus6\cmd\;; %mend; %spls; 4) Create user help file

This is helpful to assist the unblinded statistician to perform each step for generating the interim analysis package. This document provides instructions for the unblinded statistician to follow regarding the execution of the programs and analysis reports. The instructions include details of copying final programs and using treatment assignment file. It removed program dependencies by specifying the order that the programs needed to be executed.

User_help.doc Please follow the steps below when running programs for interim analysis: Note: It’s preferable to have a dedicated SAS Version 8.1 session for running these programs. 1. Copy all development files to production area. (Specifics depend on individual setup and location). Create subfolders if necessary for output reports and graphics. 2. Open/run startup.sas; Please change the reference in startup.sas before submitting it to access the actual treatment assignment file. Example: %let an = datadir.trtcodes2; 3. Open/Run run-setup.sas •

The setup programs or analysis datasets need to be executed in a specific order to develop permanent analysis datasets. The order and program dependency is specified in the runsetup.sas program created by the programmer for the unblinded statistician. This file should not be altered.



If for any reason a data set needs to be reexecuted then the program dependencies outlined in the run-setup.sas file needs to be maintained in order for the data set to be successfully generated.



For example, to reexecute mkt2eitv.sas requires that mkdemo.sas, mkoresp1.sas and lastly, mkt2eitv.sas be executed in this order to successfully regenerate the corresponding data set. mkdemo.sas Æ mkoresp1.sas Æ mkoresp2.sas Æ mkbresp.sas | ---------> mkt2eitv.sas Æ mkt2ergt.sas

4. Run tables/figures by executing programs below, not in any particular order • • •

Run run-safety.sas Run run-efficacy.sas Run run-plots.sas

Validation and Testing The validation process for interim analysis follows the same process as validation for a CSR and the same SOPs. The differences being, only some subset of the CSR analysis and reports are pre-

specified for interim analyses and additionally, since data is blinded this process of validation focuses on confirming program logic and statistical methods, not necessarily confirmation of unblinded results. The other difference is that this work needs to be done on a compressed timeline since interim analyses are usually far in advance of CSR timelines. If timelines are at risk at any point during development other blinded programmers can be involved to assist with development and validation. The unblinded statistician tests the program execution on an independent secure share drive in advance of unblinding interim analysis. Part of this testing required mock testing the "unblinding" procedure as it differed from unblinding data for a CSR. It was critical that the unblinded statistician could perform this process independently and without issues. This additional step of confirming the unblinding is unique to performing interim analyses. Execution at Interim Analysis At the time of unblinding, a blinded execution of all reports and analyses is performed by the blinded programmer for the clinicians' benefit. A set of blinded reports for this business case were delivered to them for medical monitoring purposes. Concurrently, the unblinded statistician following the guidelines outlined in the readme documentation executed all the unblinded analyses and reports. Although no issues surfaced at this time for our programming setup, it is possible that there could be some unexpected programming issues that would need to be dealt with at this time. Our recommendation would be to have some backup plan in place to accommodate any surprises. Being that there is only ONE unblinded statistician, risk is involved with meeting timelines if problems arise. If an issue with results was detected at unblinding, by the unblinded statistician they would first contact the blinded programmer and the unblinded statistician would try to communicate the concern without compromising the unblinded results. Hopefully, this would allow the programmer to update the programming if required. Had this been unsuccessful, a provision would have been made to allow an independent programmer to be unblinded to assist in executing the reports. This approach should have been used only as a last resort. CONCLUSION Given all the operational challenges that need to be considered when performing an interim analysis, a standard operating procedure for interim analyses is strongly recommended that delineates all of the roles and responsibilities with all stakeholders involved along with guidelines on planning/timelines, data collection, program development, validation and testing and lastly, execution. REFERENCES Anbar, Dan, Issues in the Planning and Design of Interim Analyses, PERI Course: Basic Training Course for new Clinical Statisticians (April 1998). Facey, K.M. and Lewis, J.A. The management of interim analyses in drug development, Statistics in Medicine, 17, 1703-1714 (1998). U.S. Department of Health and Human Services Food and Drug Administration Center for Drug Evaluation and Research (CDER), Center for Biologics Evaluation and Research (CBER), ICH Guideline E9 Statistical Principals for Clinical Trials (September 1998) U.S. Department of Health and Human Services Food and Drug Administration Center for Drug Evaluation and Research (CDER), Center for Biologics Evaluation and Research (CBER), ICH Guideline E3 Structure and Content of Clinical Study Reports (July 1998) ACKNOWLEDGEMENTS The author would like to thank Mary Varughese and Ying Wan for their careful review and helpful

comments. The author also thanks the management team for their review of this paper and help to go through the company publication process. CONTACT INFORMATION Your comments and questions are valued and encouraged. Contact the authors at: Mei Dey Merck & Co., Inc. P.O. Box 1000 351 North Sumneytown Pike North Wales, PA 19454 Phone: (267) 305-6814 Fax: (267) 305-6474 Email: [email protected] Lisa Pyle Merck & Co., Inc. P.O. Box 1000 351 North Sumneytown Pike North Wales, PA 19454 Phone: (267) 305-6887 Fax: (267) 305-6474 Email: [email protected] SAS® and all other SAS® Institute Inc. product or service names are registered trademarks of SAS® Institute Inc. in the USA and other countries. ® indicates USA registration. Other brand and product names are trademarks of their respective companies.