Private Cloud PIM Email Service Category Digital Government: Government to Government

State of Illinois Central Management Services Bureau of Communication & Computer Services 120 W. Jefferson Springfield, IL 62702-5172

State of Illinois Private Cloud PIM/EMAIL Service B. Executive Summary In 2005 the State of Illinois began an IT Rationalization and Consolidation initiative to improve state IT systems and efficiencies. Personal Information Management (PIM), including: email, calendaring, contacts, mobile messaging, list servers, etc., was chosen for standardization and shared service. Standardizing email systems could provide efficiencies and significant cost savings. The project was implemented in two phases: 1. Phase I - common email address scheme and directory for all agencies. Over 50,000 email users and more than 50 agencies, boards, and commissions were converted. This phase included a common mail relay, spam, virus, and content filtering for all email. 2. Phase II - private cloud email service capable of supporting all State email users. This system includes all email functions: archiving, e-discovery, mobile device access, web client, full client, off-site disaster recovery, large file transfer, and list servers. A single unified directory replaced the aging email direct gateways and a secure perimeter solution was put in place for Spam trapping and email virus scan protection. A highly scalable and fully-redundant email hosting system based on Microsoft’s Exchange server was designed and implemented to support the email of all State agencies, boards, and commissions. The Research in Motion (RIM) Blackberry Enterprise Server was selected as the standard supported mobile messaging service and device. The solution included an enterprise email vaulting/archiving service and full e-discovery capabilities. An off-site disaster recovery (DR) component for selected email users, supporting both email and mobile messaging was established to back-up the entire solution. Phase I of the project began in 2006 and was completed in 2008. As of December 2009, a total of fifty six (56) agencies and 56,000 users utilize these services. Phase II of the project began in 2006 and was completed at the end of November 2009. As of the end of November 2009, the service is being utilized by forty-four (44) State of Illinois agencies, boards and commissions with a total of 22,800 users. A total of 2400 Blackberry devices are also managed and supported. The entire project has resulted in significant tangible and intangible benefits. The net cost benefit/savings to the state between Fiscal Years 2005 and 2010 has been over $13,000,000, with a total Return on Investment (ROI) of over 86%.

Page 1 of 6  

State of Illinois Private Cloud PIM/EMAIL Service C. Description of the Project Business Problem Initial analysis was conducted by Accenture in 2004. That analysis determined that there were numerous shortcomings in the State of Illinois email systems. There were over 20 separate email systems managed independently with little or no common address standards, numerous redundant components, and very little standardization. The study recommended the following email service components going forward: • • • •

A consistent and common email address and directory across the enterprise; A common PIM/EMAIL service for agencies; Email Archiving and e-discovery tools; Common gateways, Spam/Virus, content filtering, and mobile messaging

A detailed analysis and design of the State’s IT PIM/email-related systems was conducted in 2005-2006 to define the problem, design a solution, develop bid specifications, project implementation and communications plans. The business case analysis showed that the general maintenance of the email systems was a costly and inefficient process. A project manager was assigned to the project and each of the phases and subprojects. Traditional project management methodologies were utilized throughout the project. A scope was defined for both the development of the systems and migration of agencies. Milestones were tracked and reported. The project manager reported metrics of progress, risks, and issues via weekly status reports to the Enterprise Project Management Office. Any major issues were escalated to the management team for assistance and resolution. Numerous meetings and other communications were initiated with customers, especially during conversion. The Project Manager also created a “dashboard” that showed stakeholders the status of a subproject. This kept the team focused on any issues and problems. At that time, more than twenty different email systems were in use across state agencies. All of the agencies used a different email-addressing scheme with no common address book. Sitting in the middle of all of these email systems was an aging and unsupported email gateway software solution whose purpose was to translate attachments and route the email from one disparate email system to another. There were also numerous incompatible and redundant email related sub-systems, such as gateways, spam filters, virus scanners, mobile messaging systems and list servers. The systems were difficult to manage and not very cost effective. Page 2 of 6  

State of Illinois Private Cloud PIM/EMAIL Service Solution Description A detail design was completed in 2005 and a recommendation was made for common addressing and directory utilizing Active Directory, mail/relays, spam/virus scanning, and mobile messaging. This would be a multi-year project so, scalability and robustness were key factors in the solution. Microsoft Exchange was selected as the primary email back-end system. Cost benefit analysis was completed before the project began. The system was projected to cost $8.2M over 5 years with a $7M net savings. Phase I began when Illinois.gov was selected as the most identifiable email domain name with “fname.lname” as the beginning address scheme. Rules were developed to ensure uniqueness of email address in instances where more than one person has the same “fname.lname”. We also implemented a custom unified directory to replace the aging directory gateway. This has helped reduce the number of delivery failures. The perimeter solution had to be scalable and accommodate 100,000 email messages per hour in 2005. Automated SPAM and Virus signatures had to be obtained from the vendor in timely intervals. A Quarantine option was required so that customers could determine whether trapped items were SPAM or legitimate. Phase II began in 2006 with the design and implementation of the highly scalable and clustered Microsoft Exchange 2003 environment. This system was designed to be centrally managed and to scale to tens of thousands of users. The various mail systems and sub-systems are redundant clustered servers on H.P. blade servers. Many server components are running on virtualized server hardware. The first agency migrated in June 2006 with 1700 clients. There were 44 agencies converted to Exchange. The conversion totals 57 including native and Alias conversions. Agencies using legacy Notes, GroupWise, and Exchange systems were converted to the system between 2006 and 2009. Over 177 email-related servers were decommissioned. Quest email migration tools were chosen. The state required a tool that could migrate all three email types to Exchange. The tool had to be flexible and configurable to accommodate each agency’s unique email addressing scheme. The tool had to have the ability to automate as much of the migration process as possible to keep customer disruption at a minimum and keep over time costs low. The original analysis determined that the email system should have a vaulting, journaling, and e-discovery tool that could resolve issues of records retention and compliance, and lower the overall cost of storing emails. RFP requirements were developed for the software and hardware needed. Symantec Enterprise Vault and EMC Centera were selected after lengthy open competitive RFP processes. The Enterprise Email Vault and e-discovery tools were implemented in 2008. Archive data Page 3 of 6  

State of Illinois Private Cloud PIM/EMAIL Service already residing on the Enterprise Email system was moved to the email vault first to free up file server space. Today, any agency added to the enterprise Email system is vaulted at migration time. An enterprise Exchange upgrade from 2003 to Exchange 2007 was implemented in late 2009 to coincide with the move of the first cluster. This included an optional DR component which allowed access to mailbox data, vaulted messages and attachments, as well as, continued email operability of mobile messaging devices The Disaster Recovery (DR) component selected was EMC’s Recovery Point. Recovery Point provides: real time data replication; an RPO of four hours and an RTO of twelve hours; email and mobile messaging functionality in a disaster scenario; and access to messages stored in the email vault. EMC’s Centera is the backend storage device providing redundancy and de-duplication of messages. D. Significance Email is widely used as a communication medium these days. A common enterprise List Server was implemented to more easily communicate with large groups of external customers. State entities that have completed the email migration are better able to serve and communicate with their constituents. Email addresses are now shorter, consistent, and easier to remember. Communication between agencies that have migrated is much easier and portable. Email attachments are easily transferred across the email system and meetings are scheduled and tracked using the Outlook client calendaring function. Contacts are more easily maintained. Email is securely accessible via the web. SPAM and viruses are captured at the perimeter before they enter the network. All of this combined means faster, more efficient communication with and between agencies and their constituents. Stakeholder Participation Even though this project was a governor’s mandate for the agencies, we needed cooperation and assistance from the agencies in order to effectively perform the work. Each agency was treated like a valued customer and asked to partner with us to complete the migration. Engaging the migrating agency benefitted the entire project. Communication was a key component to the successful implementation. The communication plans included: initial face-to-face meetings with the agency’s CIO and technical team; agency representation in weekly meetings; documentation for notification of and announcements to users; quick source instruction and “train the trainer” guides for agency users. A test group was rolled out initially and a pilot group migrated first. Following a successful pilot, the entire agency was migrated. Forty-four state agencies, boards and Page 4 of 6  

State of Illinois Private Cloud PIM/EMAIL Service commissions have opted to migrate their legacy email systems to the Enterprise Exchange. E. State and Agency Benefits The redundancy built into the Illinois Enterprise Exchange Email system allows for a reliability factor averaging 99.997% over the last few years. This is further enhanced by recent implementation of the Disaster Recovery options for hot recovery at a geographically diverse recovery site. From 2007-2009 we have seen a 365% increase in email traffic utilizing the new service. We now see a total volume of 11,709,000 messages per month. The Email Enterprise vault complies with the Illinois State Records Act. Messages meeting established criteria are automatically pushed to the Email vault on a nightly basis. The Discovery tool allows for timely and efficient response to FOIA and litigation requests, avoiding the inefficiencies of tape restores. In 2009 there were over 500 ediscovery requests processed. Prior to this system, an e-discovery search would routinely take days and weeks to accomplish. Most requests can now be handled in minutes. In addition, archived email data previously stored on Tier One storage is now stored on Tier 3 storage at a significant cost savings. The Enterprise Perimeter Solution traps virus infected documents and SPAM. On average, this spam and virus solution blocks 69% of invalid or inappropriate inbound messages. In 2009 this system stopped over 64 million spam messages and 1.3 million messages with viruses. This system results in increased user productivity and more efficient operation of the mail service; mailbox size is much smaller while preventing virus-infected email from reaching the network. Customers are no longer required to wade through tons of SPAM messages in their Inbox to find valid messages. The Quarantine component of the perimeter solution notifies the customer of any email message that requires action to delete, white list or release. To date, forty-four (44) agencies, boards and commissions have migrated to the Illinois Enterprise Email service. New email migrations are being scheduled on a yearly basis. Our goal is to quickly decommission as many legacy email systems as possible to avoid replacing aging hardware. There are three agency email migrations scheduled for calendar year 2010 with more in the discussion phase Savings and Cost Avoidance We have realized a number of cost savings with each agency migration: ¾ To date, 177 aging legacy agency e-mail post offices, Exchange or Lotus Notes servers have been decommissioned. Result: $1,400,000 in annual savings. Page 5 of 6  

State of Illinois Private Cloud PIM/EMAIL Service ¾ Software costs have been reduced. There is no longer a need to purchase Operating System, virus scan, backup, and proprietary email clients for servers. ¾ Agencies no longer need to maintain individual perimeters. ¾ Staff has been re-allocated to other areas of shared services. Result: 55 % reduction in PIM/Email support staff. $1,800,000 in annual savings. Remaining staff is trained in one e-mail system to support. ¾ Greatly reduced e-discovery costs. Multi-Agency e-discovery search requests have been reduced from several weeks to a few minutes or hours. ¾ Storage costs reduced as a result of vaulting. Messages are unduplicated and attachments moved to slower and cheaper Tier 3 storage. Result: 50% savings in storage costs for archived data. Delayed or avoided need to purchase additional storage for servers. ¾ After hours technical support costs at the agencies for email and mobile messaging eliminated. Result: The Enterprise email service provides 24x7 technical support at no additional cost to the agency. ¾ Return on Investment The State of Illinois has already seen a huge return on investment in the PIM Service. At an initial total cost of $6,990,353, the 2005 analysis projected that the E-mail consolidation project would net $2.8 million in annual savings by the end of fiscal year 2009. The project has greatly exceeded those original expectations. As of June 2010, we have seen the following net benefits: •

Total net benefit/savings for the project in Fiscal Years ’05 through ’10 is $13,239,392.

•

The last 2 years have averaged $3,965,848 annually.

This number will continue to grow as more and more email systems are migrated to the Enterprise Email system and legacy email systems are decommissioned. As it becomes increasingly important for state government to continue to serve the needs of Illinois with dwindling funds, we continue to look for similar opportunities for efficiency.

Page 6 of 6