WBF 107 S. Southgate Dr. Chandler, AZ 85226-3222 (480) 403-4610
[email protected] www.wbf.org
Presented at the WBF Make2Profit Conference Austin, TX, USA May 24-26, 2010
Electronic Workflow for a Bioreactor Christie Deitz Sr. Principal Engineer Emerson Process Management 12301 Research Blvd Austin, TX 78759 USA 512-832-3240 512-832-3199
[email protected]
Joe Maguire Automation Engineer Bristol-Meyers Squibb 15 Queenstown Street Devens, MA 01434 USA 267-250-7266
[email protected] KEY WORDS
Electronic Workflow, MES, Bioreactor, Release by Exception
ABSTRACT Automating workflow and eliminating paper batch records can provide many benefits, including reducing deviations, expediting batch review and release, improving real-time inventory management, and utilizing industry and corporate standards. BMS is currently in the commissioning stage of its new state-of-the-art facility in Devens, Massachusetts. BMS’ objective was to create a paperless manufacturing environment. To meet this objective, automation for the facility includes a process control system (PCS) and a manufacturing execution system (MES) system. The project is unique because, to date, it is BMS’ most extensive automation of workflow; that is, the manual instructions that might be traditionally done using paper. The project team learned some valuable lessons with regard to team organization and approach to testing. They also made some key technical decisions around prompting, phase boundaries and recipe design. This paper will explain many of the lessons learned using the bioreactor area of the project as an example.
Copyright ©2010 WBF. All rights reserved.
Page 1
PAPER INTRODUCTION The Bristol-Myers Squibb Large Scale Cell Culture (LSCC) facility is located on an 89-acre site in Devens, Massachusetts, 45 miles west of Boston. The facility will support the production of ORENCIA® (abatacept), the company's biologic therapy for rheumatoid arthritis, as well as other biologic compounds currently in development. BMS invested $750 million in the construction of the facility. It has 120,000 liters of bioreactor capacity that will be capable of concurrent multi-product production. The vision for the project was to provide a state-of-the-art, fully automated process. The automation goal was to be completely paperless and to support release by exception. In an FDA-regulated environment, batch release requires someone to verify that all exceptions have been investigated and signed off. Release by exception generally means that the approval and release of a batch is based on the review and approval of ONLY the exceptions and the required remedial action taken. The reviewer does not have to spend time sorting through the entire record to identify exceptions. Review by exception also implies that exceptions can be reviewed and dealt with in real-time, not only when the batch is completed. Additionally, a system that supports review by exception can often reduce the number of exceptions by enforcing rules in a real-time environment. For example, the system may not allow the operator to enter an invalid response or may force the operator to sign before proceeding. A system that both minimizes errors and allows real-time exception review can dramatically reduce the time that product is warehoused, waiting to be released for sale, and therefore can reduce the cost of inventory. Using an S95/S88 structured approach and an electronic workflow together with process control enables review by exception. For the LSCC project automation, BMS selected DeltaV as its process control system (PCS) and Syncade as its manufacturing execution system (MES). They also selected several other systems such as SAP business software, SmartLab for lab information management, and Maximo for instrument asset management, all of which would need to communicate with the MES. BMS broke ground for the facility in March 2007. It was operationally complete in 2009, and is currently in the process of validation. The FDA submission will be filed in 2010. ELECTRONIC WORKFLOW Electronic workflow is paperless production, where electronic recipes handle both the manual and the automated functions. For manual functions, the system provides electronic instructions and access to electronic SOPs (standard operating procedures). As operators enter responses and data onto the screen, the system provides error checks and enforces sequencing. For the LSCC project, the functionality was divided between the PCS and MES components of the system. Table 1 shows how the functions were divided.
Copyright ©2010 WBF. All rights reserved.
Page 2
Table 1: PCS and MES Functionality PCS
MES
Instrumentation
Document Control
Continuous Control
Material Management
Batch Control (up to Units/Phases)
Order Management Equipment Tracking Recipe Authoring Workflow Execution
Electronic workflow automates the steps in manufacturing product and generating the batch record, many of which were historically manual steps. Table 2 describes the steps and the role of electronic workflow in each step. Table 2: Electronic Workflow Examples Manufacturing Step
Examples of Role of Electronic Workflow
How it ensures quality and/or expedites review and release
Qualify Equipment and Facilities
Operator scans the barcode of equipment. The system checks the status of the equipment and allows the operator to proceed only if it is the appropriate equipment with the correct status.
Prevents inadvertent use of incorrect equipment.
Verify Materials
Operator scans the barcode of material. The system checks the status of the material and allows the operator to proceed only if it is the appropriate equipment with the correct status.
Prevents inadvertent use of incorrect material.
Batch Execution
Electronic work instructions (EWIs) prompt operators to perform manual tasks and enter data as appropriate. These EWIs provide links to Standard Operating Procedures (SOPs) and Material Safety Data Sheets (MSDSs).
Enforces correct sequence of action.
Electronic work instructions kick off automated process operations such as filling vessels.
Prompts operators if data is out of range to help prevent data entry errors. Generates notifications for QA review in real-time for process deviations and outof-spec data.
Performs calculation. Process deviations and out-of-spec data generate notifications for QA review.
Copyright ©2010 WBF. All rights reserved.
Page 3
Manufacturing Step
Examples of Role of Electronic Workflow
How it ensures quality and/or expedites review and release
Operations Review
When all workflow is complete, the data is compiled and a notification for operations review is launched.
Notification is launched in real-time.
QA Review
The Electronic Batch Record (EBR) is presented in a checklist form for QA review
Notification is launched in real-time.
QA e-signature is collected to approve and archive EBR and disposition the batch.
Disposition occurs real-time and EBR is automatically archived. Any changes to EBR generate a new version that is forwarded to document management.
Batch Disposition
View is provided that displays information relevant to operators. View is provided that displays information relevant to QA in checklist format.
LSCC is currently in the process of executing the electronic workflows on the equipment. Table 3 shows the recipes that have been successfully run at site to date. Table 3: Executed Electronic Workflows Upstream Processes Inoculation Lab
Downstream Processes
150L Bioreactor
Hydrophobic Interaction Chromatography (HIC) Column
750L Bioreactor
Centrifuge
Basal Media Vessels
Buffer Preparation and Hold
(All of the above were cleaned, steamed, batched and transferred.) BIOREACTOR WORKFLOW The development of the automation for the bioreactors started with the development of the units and phases. A similar approach was followed throughout the plant for two reasons: (1) the equipment requirements had already been defined, which logically led into equipment automation requirements, and (2) the use of MES and electronic work instructions was relatively new to BMS. The project team had much more experience with traditional S88-base process control system-level control. Therefore, a bottom-up approach came naturally. A classic S88 approach was used for the development of the bioreactor automation, Figure 1. First, control modules such as valves, motors, indicators and graphic displays were created. Next, the equipment module layer was added. Examples of equipment modules include pH control, dissolved oxygen control, transfer line control, and steam supply line control. Finally, unit-level control, including
Copyright ©2010 WBF. All rights reserved.
Page 4
Figure 1: Example Bioreactor Graphic
phase logic, was added. Example phases include Media Fill, Equilibrate, Inoculate, Growth, Harvest, CIP and SIP. The expectation was that electronic work instructions would kick off phases. Also, in many cases, the phase and EWI need to communicate within a phase. One example is addition of bagged media to a bioreactor. An electronic work instruction in the bioreactor recipe starts the unit phase in the process control system. The unit phase puts equipment into the proper position and then waits for an electronic work instruction to prompt the operator to scan the material barcode. The system checks to ensure that the material was good, and then the electronic workflow walks the operator through the manual steps to connect the bag to the pump. Finally, the unit phase runs the pump to complete the charge. The electronic workflow efforts had not been started at the time; however, the interactions were mostly known based on process requirements. Around the time that the work on unit phases was completing, efforts were ramping up for the electronic workflow. Since electronic workflow was new for BMS, the project team spent some time up front working out the general approach to the project. To start with, the process engineers needed a way to Copyright ©2010 WBF. All rights reserved.
Page 5
communicate to the automation team what the operators needed to do to make product. In addition, they needed a framework to help them get started more quickly than a blank page. To meet these needs, BMS created a flowcharting tool, Figure 2. The flowcharting tool provided some templates to show standard functions such as material checks and equipment checks. It also allowed free-format electronic work instructions (EWIs) for the process engineers to use to communicate requirements. In addition, the flowcharting tool allowed the process engineers to show functions happening in parallel on different process units by using different vertical columns, or swim lanes, for each unit. Figure 2 shows an example of the flowcharting of bioreactor requirements using the tool.
Figure 2: Example of Bioreactor Workflow Requirements Flowchart Workflow for Wave Bioreactor
Workflow for 150L Bioreactor
Free-format EWIs Standard Construct for CIP Sequence EWIs
After requirements were established, the electronic workflow was approached in a bottom-up fashion. The first step was creating the basic building blocks, which are called “instructions” or “manual phases.” Examples of these include prompting an operator with a message and requiring a signoff, checking hygienic status of a vessel, and providing an operator with a link to an SOP. A library of common manual phases was built for the entire project. This enabled the team to take advantage of a modular, object-oriented coding approach. Next, the recipes were built up from the manual phases and the automation phases using the recipe authoring tool. Figure 4 shows an example of recipe authoring. The manual phases and the automation
Copyright ©2010 WBF. All rights reserved.
Page 6
phases can be built into operations, operations built into procedures, procedures into process segments, and process segment into master recipes. The project team made a decision that, for this project, automation phases would be grouped into automation operations – without any manual work instructions. Examples of automation operations include Media Charge, Equilibrate, Growth, Harvest, CIP and SIP. In most cases, one automation phase was wrapped in one automation operation. However, in some cases, such as CIP, the several phase instances required to completely CIP a vessel were included. Then, manual and automation operations were built into unit procedures, as shown in figure 4. Unit procedures are the highest level recipe that executes on a single process unit: for example, a 20K production bioreactor. Examples of unit procedures are Production Bioreactor Equilibrate, Production Bioreactor Growth, and Production Bioreactor Harvest. These recipes contained all of the automation and manual steps required to perform their function. Unit procedures were built into procedures, which can execute across multiple units. An example of a procedure is the 20K Production Bioreactor, which does the preliminary equipment checks, prompts the operator to do the required manual equipment assembly, and executes the Production Bioreactor Media Charge, the Production Bioreactor Equilibrate, Production Bioreactor Growth, and Production Bioreactor Harvest unit procedures – complete with instructions for all of the manual interactions required such as sampling. Finally, these procedures were built into process segements and master recipes. For this project, a master recipe contained a single process segment, and a process segment contained a single procedure.
Figure 3: Example Manual Phase Library
A combined team of BMS and their automation supplier Emerson wrote the recipes for the project. Engineers from the supplier created the manual instructions to use as building blocks. Then BMS automation engineers, with some consultation from the supplier, led the team for each process area workflow, such as the bioreactor area. Supplier engineers, most of whom had experience developing the automation phases, helped build the operations, unit procedures, procedures, process segments and master recipes. The combined team performed software testing of the workflow recipes prior to deployment on site. This approach leveraged the best skills of both companies. LESSONS LEARNED Overall, the executing and testing of the workflows at site has gone smoothly. As would be expected for a project of this magnitude, some changes and corrections have been required at site. Mostly, these changes have been related to conforming to the equipment, now that it is installed and in-use, or changing the order of instructions. Some examples of changes to the bioreactor workflows include: •
Sampling instruction details
•
Timing of when to install base bag for pH adjustment
Copyright ©2010 WBF. All rights reserved.
Page 7
•
Instructions for standardizing pH and DO probes
•
Valve differences (different manual valve arrangement than design assumptions) for steam-on and -off sample apparatus
Executing a project of this size and magnitude was a learning experience for the entire project team. Some recommendations the project team suggest to others starting an electronic workflow project include: 1) Finalize vision and requirements early and document them. On a large project, this helps get and keep everyone marching in the same direction. 2) Make key decisions early. The earlier the team can set the direction on, for example, standard approaches to common problems, the more quickly the entire team can start making progress toward completion. 3) Integrate the development of electronic workflow and traditional automation. For the LSCC project, the automation work was nearly completed before the electronic workflow engineering was really started. Doing the work more in parallel would force the team to think through more of the issues while making changes is still relatively easy. 4) Balance perfecting software against getting it done. Sometimes finding a workable solution quickly is better than taking weeks or months to perfect a solution. 5) Be open to execution strategy adjustments. On LSCC, many schedule and planning decisions were placed in the hands of the engineers to realistically estimate time lines. 6) Bring in plant operations and manufacturing personnel early. They have critical insight into the operators’ perspective and how the process will work. At the time this paper was written, LSCC was still in the process of executing the electronic workflows on the equipment. The recipes that have already been run at site have been successful; although, as is always the case, some minor modifications have been required. Based on efforts to date, the Devens site seems to be on-target for using electronic workflow to achieve its goals of paperless production and review by exception.
Copyright ©2010 WBF. All rights reserved.
Page 8
Figure 4: Recipe Authoring of Manual and Automation Components
Automation Components in Blue
Manual Components in Green
Copyright ©2010 WBF. All rights reserved.
Page 9
