Cray Technical Enterprise Solutions for Weather, Climate and Ocean Sciences Cray Inc.
WP-XE02-0612
www.cray.com
Table of Contents Introduction ............................................................................................................................................................................ 3 Cray Technical Enterprise Solutions for Weather, Climate and Ocean Sciences ................................................ 4 Danish Meteorological Institute (DMI)........................................................................................................................ 4 MeteoSwiss ......................................................................................................................................................................... 5 Finnish Meteorological Institute (FMI) ........................................................................................................................ 5 Mongolian National Agency of Meteorology and Environmental Monitoring (NAMEM) ......................... 6 TerraMe ................................................................................................................................................................................ 6 Arctic Region Supercomputing Center (ARSC) ........................................................................................................ 7 Cray Worldwide Applications Expertise ......................................................................................................................... 7 Center of Excellence for Earth System Modeling..................................................................................................... 7 Cray Centre of Excellence for EPSRC ........................................................................................................................... 7 Conclusion............................................................................................................................................................................... 8 References ............................................................................................................................................................................... 9
WP-XE02-0612
Page 2 of 9
www.cray.com
Introduction The earth’s environment has far-reaching socioeconomic impacts. Atmosphere and oceans contribute to shaping economies and infrastructures, and touch upon nearly every aspect of our daily lives, including food supplies, energy resources and recreation. Given the delicate relationship between earth systems and the well-being of humankind, the ability to understand and predict shifts in weather, climate and ocean is critical and drives the usage of high performance computing (HPC) in earth system modeling. An increased societal reliance on technology and all manner of infrastructures, combined with increased severe weather vulnerability, calls for the most appropriate tools to support decision-making capabilities. Cray supercomputers are synonymous with solving mission-critical scientific problems and have been systems of choice for years in the earth system research and operational communities. Cray solutions play an enabling role in harnessing the significant advances in science and technology. Earth system modeling is a key application area that spans nearly the entire spectrum of Cray’s customer base, ranging from those whose core business is earth system modeling to multidisciplinary HPC centers. Technologies by themselves do not make world-class supercomputers, but the integration of all the various hardware and software technologies into a production HPC system does. It takes generations of supercomputer system developments and installations to be a successful supercomputer system developer and provider. Cray has an intimate understanding of the unique requirements of production HPC – from architecture, infrastructure and scalability, all the way to the nuances of application performance and long-term supportability. WRF Simulation of Typhoon Man-Yi
Image Courtesy of KMA From model performance to system administration and data management, weather, climate and ocean sciences present unique HPC challenges. The computational requirements for simulations of appropriate spatial and temporal scales are immense and require maximum scalability to execute in practical timeframes. Cray builds innovative HPC systems that deliver exceptional performance – from entry-level systems to the largest and most powerful systems in the world. Cray’s goal is to maximize sustained application performance by combining cost-effective commercial technologies with custom-designed innovations that significantly increase performance, reliability and usability.
This balanced system design approach delivers high-efficiency computing and maintains this efficiency at scale, whether jobs require tens or thousands of processors. The focus on high-efficiency computing allows scientists and forecasters to produce results in the shortest time possible while investigating increasingly complex phenomena. It also allows scientists to solve problems too difficult for commodity clusters or systems designed for commercial applications.
WP-XE02-0612
Page 3 of 9
www.cray.com
Cray Technical Enterprise Solutions for Weather, Climate and Ocean Sciences Cray solutions have been chosen by operational and research centers worldwide to solve their challenging numerical weather prediction (NWP) problems. Cray systems are installed at several operational meteorological and hydrological services and leading climate research centers worldwide. Examples include the U.S. National Oceanography and Atmospheric Administration (NOAA), the National Center for Atmospheric Research (NCAR), Korea Meteorological Administration (KMA), Brazilian National Institute for Space Research (INPE) and Center for Weather Forecasts and Climate Studies (CPTEC), Danish Meteorological Institute (DMI), the U.S. Naval Oceanographic Office (NAVO), U.S. Naval Research Laboratory, MeteoSwiss, the Finnish Meteorological Institute (FMI), the Arctic Region Supercomputing Center (ARSC) and the Mongolian National Agency of Meteorology and Environmental Monitoring (NAMEM). Of particular importance to operational centers, Cray delivers custom-designed and fully integrated solutions to meet operational and site-specific workflow, space, electrical and cooling requirements. Several installations feature a fully redundant system architecture featuring separate operational and research/backup systems with a globally shared file system. Cray also enables the adoption of accelerators with a comprehensive hardware and software approach that addresses the challenges in adopting these technologies into mainstream scientific applications. Building on the scalability, reliability and efficiency of Cray’s largest systems, Cray’s technical enterprise systems are optimized to support application workloads requiring between 500 and 10,000 cores of processing power. Cray technical enterprise systems are installed at leading operational and research centers worldwide.
Danish Meteorological Institute (DMI) DMI serves the meteorological needs of the kingdom of Denmark including territorial waters and airspace. This work entails monitoring weather, climate and environmental conditions in the atmosphere, on land and at sea. The primary aim of these activities is to safeguard human life and property, as well as to provide a foundation for economic and environmental planning – especially within the armed forces, aviation, shipping and road traffic. Applied research, development and exploitation of new knowledge within DMI’s field of activity are preconditions if DMI is to maintain rational production and fulfill its obligations to Danish society. Research and development along with the institute’s other departments carry out the work, encompassing weather, climate and oceanographic models and serving as the basis for the daily forecasts issued by the forecasting services department. Through the use of its Cray XT5™ system first installed in 2008, DMI improved the resolution of the HIRLAM weather forecast model from 5 to 2 kilometers, more than doubling the detail of their operational model. In addition, DMI ran improved operational oceanographic forecasting models that predict water levels, wave heights and storm surges for the maritime and aviation sectors as well as for clients in defense, shipping and agriculture.
WP-XE02-0612
Page 4 of 9
DMI Operational Domains Credit: Bent Hansen Sass, DMI: 18th SRNWP Meeting, 10-13 October 20111
www.cray.com
DMI required a fully redundant configuration to meet an operational failover requirement. They selected two identical Cray XT5 systems along with a global shared Lustre file system equally accessible by each system. Under normal circumstances, DMI uses one system for operations and the other for research. The total performance of the original two systems was 38 teraflops/s. DMI recently upgraded their Cray XT5 systems to a newer processor, bringing the total performance to 58.8 teraflops/s.
MeteoSwiss MeteoSwiss, the Swiss national weather service, provides accurate forecasts up to a few days ahead using techniques that rely heavily on NWP computational methods. The service carries out their predictions using the Swiss National Supercomputing Centre (CSCS) facilities. In February 2007, MeteoSwiss began production weather forecasting using the CSCS’s Cray XT3™ system. Using the Cray system, Switzerland became the first European country to implement a 2 kilometer resolution required to make detailed forecasts of Switzerland's intricate Alpine topography. Since 2007, MeteoSwiss has transitioned to a Cray XT4™ system, and most recently to a Cray XE6™ system. The new Cray XE6 system is a three-cabinet machine partitioned in two logical domains with a total peak performance of 48.3 teraflops/s.
MeteoSwiss Operational Domains Credit: Credit: Philippe Steiner, MeteoSwiss: 18th SRNWP Meeting, 10-13 October 20112
Finnish Meteorological Institute (FMI) The Finnish Meteorological Institute (FMI) gathers and reports Finland’s weather data and forecasts 24 hours a day, 7 days a week. Additionally, FMI conducts research across a range of atmospheric science activities, including air quality, space research, climate change and polar research. Using a Cray XT5m™ system, FMI provides forecasts covering the European continent with a 7.5 kilometer resolution and Finland with a 2.5 kilometer resolution.
FMI Operational Domains Credit: Carl Fortelius et al., FMI: 18th SRNWP Meeting, 10-13 October 20113
WP-XE02-0612
Page 5 of 9
www.cray.com
FMI required a fully redundant configuration to meet an operational failover requirement. Two identical Cray XT5m systems are installed, each with its own Lustre file system. Each system is configured with 3,936 processor cores in two cabinets with a peak performance of 17.3 teraflops/s, providing a twenty times increase in computing capacity over FMI’s previous system.
Mongolian National Agency of Meteorology and Environmental Monitoring (NAMEM) Mongolia’s National Agency for Meteorology and Environmental Monitoring (NAMEM) provides weather, climate and hydrological forecasts and warnings to protect human life and property from natural disasters and enhance the country’s socioeconomic development. The Mongolian terrain consists of steep mountains and rolling plateaus and the country's weather is characterized by extreme variability in both short-term weather and multi-year averages. With an economic sector that relies heavily on weather and climate for livestock and arable farming, a compelling need exists for a supercomputer capable of producing high-resolution weather forecasts. The Cray XE6m™ system provides NAMEM the supercomputing resources necessary for increasing the resolution and accuracy of its weather forecasts and prediction of high-impact, extreme events using the weather research and forecasting (WRF) model. In addition to superior performance and reliability, NAMEM selected the Cray XE6m system over conventional commodity clusters due to the full-featured system software and application development environment. The NAMEM Cray XE6m has a peak performance of 6.46 teraflops/s. The new system has enabled high resolution three kilometer and one kilometer regional forecasts over the central part of Mongolia and around Ulaanbaatar, respectively.
NAMEM Operational Domains Source: NAMEM presentation, “Numerical Weather Prediction and Computing Technology”
TerraMe The TerraMe-Galileu Laboratory is located at the Brazilian National Institute for Space Research (INPE). The new laboratory partners with the other INPE facilities focused on high performance supercomputer and modeling processing – the Weather Forecasting and Climate Studies Center (CPTEC), the Center for Science of the Terrestrial System (CST) and the Coordination of Terrestrial Observation (OBT) – to work on new developments in computational modeling and simulations with models of increasing complexity. To support this work, the institute installed a Cray XE6m with 2,688 cores and a peak performance of 22.6 teraflops/s.
WP-XE02-0612
Page 6 of 9
www.cray.com
Arctic Region Supercomputing Center (ARSC) The Arctic Region Supercomputing Center (ARSC) is an HPC research unit of the University of Alaska Fairbanks. ARSC supports high performance computational research in science and engineering with an emphasis on latitudes and the Arctic. ARSC is home to a Cray XK6m™ called “Fish,” named for Alaska’s fisheries and ocean and water resources. Fish will be a valuable tool for researchers working to understand and predict changes in arctic systems, including weather and climate, oceans and ice, permafrost and other materials. ARSC is one of the first organizations to purchase a Cray XK6m, a hybrid system composed of conventional x86 microprocessors and GPGPUs to accelerate scientific applications. The Cray XK6m system is configured with 1,152 x86 cores and 48 NVIDIA® Tesla™ GPU processors. The total theoretical peak system performance is 41.75 teraflops/s. Initial applications targeted to leverage the accelerator technology include the WRF model.
Cray Worldwide Applications Expertise Cray’s worldwide environmental applications team is engaged in a number of activities concerned with porting and tuning weather and climate models. In many of these efforts, particularly with the community models, Cray works with the application developers so that optimizations will be incorporated into the released code. Efforts include, but are not limited to: work on WRF, UM, HadGEM, NEMO, CCSM, MOM4, IFS, HIRLAM, ALADIN, AROME, HARMONIE, COSMO-LM and COSMOS. Cray has also established Centers of Excellence (CoE) with several key customers, including ORNL, KMA, NERSC and EPSRC. Here, scientists and computer scientists collaborate on models that will advance the state of the science, execute at continuously increasing system sizes, and harness petascale facilities in a production capacity as soon as they become available.
Center of Excellence for Earth System Modeling Cray established a Center of Excellence for Earth System Modeling (CoE-ESM) in early 2009. Based in Seoul, South Korea, the CoE-ESM builds on the expertise developed through the Earth System Research Center (ESRC), a cooperative venture between Cray and KMA. ESRC was established in 2005 to advance earth system modeling over the East-Asia Pacific region and to foster collaborative research and development that maintains KMA as a leading-edge facility.
Cray Centre of Excellence for EPSRC The Cray Centre of Excellence for EPSRC focuses on the optimization and scaling of applications for the U.K. academic community. Its goal is to develop true capability computing and to address the largest scientific problems by scaling applications to reach higher resolutions and larger models in shorter times. The Cray Centre of Excellence for EPSRC works closely with one of the largest HECToR users, U.K. National Center for Atmospheric Science (NCAS). For example, the “HiGEM Decadal Predictions” project was jointly executed between NCAS, the University of Anglia and the Cray CoE at HECToR. The report is available at www.hector.ac.uk/coe/pdf/HiGEM_FinalReport_Jun2010.pdf.
WP-XE02-0612
Page 7 of 9
www.cray.com
Conclusion An increased societal reliance on technology and infrastructure, combined with greater severe weather vulnerability, calls for the most powerful tools to support decision-making capabilities. Cray supercomputers are synonymous with solving mission-critical scientific problems and have been the systems of choice for years in the earth system research and operational communities where production capability computing is of strategic importance. Cray solutions play an enabling role in harnessing the significant advances in science and technology. Cray’s balanced system design approach delivers high-efficiency computing and maintains this efficiency at scale, whether jobs require tens or thousands of processors. The focus on high-efficiency computing allows scientists and forecasters to produce results in the shortest time possible while also investigating increasingly complex phenomena.
WP-XE02-0612
Page 8 of 9
www.cray.com
References 1
– “Danish Meteorological Institute NWP news 2011”, Bent H Sass (DMI), 33rd EWGLAM and 18th SRNWP meetings, 10-13 October 2011, http://srnwp.met.hu/Annual_Meetings/2011/index.html
2
– “Numerical Weather Prediction at MeteoSwiss” Philippe Steiner (MeteoSwiss), 33rd EWGLAM and 18th SRNWP meetings, 10-13 October 2011, http://srnwp.met.hu/Annual_Meetings/2011/index.html 3
– “SRNWP at FMI”, Carl Fortelius,Erik Gregow and Ekaterina Kourzeneva (FMI), 33rd EWGLAM and 18th SRNWP meetings, 10-13 October 2011, http://srnwp.met.hu/Annual_Meetings/2011/index.html
© 2012 Cray Inc. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior permission of the copyright owners. Cray is a registered trademark, and the Cray logo, Cray XK6, Cray XK6m, Cray XE6, Cray XE6m, Cray XT6, Cray XT5, Cray XT4 and Cray XT3 are trademarks of Cray Inc. Other product and service names mentioned herein are the trademarks of their respective owners.
WP-XE02-0612
Page 9 of 9
www.cray.com
