Numerical ocean modeling in the cloud

• Product name: Numerical ocean/weather modeling platform as a service
• Customer groups: Governmental agencies, universities, research and consulting companies (including small and medium size companies). Actual end users of the service are scientists or software engineers with strong scientific background, employed by these organizations.
• Business drivers:
  • Desire to reduce on-premise infrastructure and associated costs
  • Avoiding delays/costs caused by job scheduling
  • Demand for infrastructure specifically optimized for numerical ocean/weather modeling
  • Enhancement of scientific progress by overcoming capacity limitations. While the size of a cluster in the public cloud may be limited (e.g. to 512 cores), the number of simultaneously running clusters is only limited by the cloud providers capacity.
• Product details:
  • Production/customization of software that automates the provisioning of a cloud-based compute cluster optimized for numerical ocean/weather modeling. Automation includes:
    • Provisioning of compute, storage and networking infrastructure (”infrastructure as code”)
    • Possibly, deployment of the numerical model itself, and the software stack for pre- and post-processing of model data, visualization, etc.
  • Flexibility. Customers maintain 100% control over the provisioning process, and can modify it according to their needs. The academic/scientific environment highly values
    • Independence
    • Transparency
    • Reproducibility
    • Accessibility
  • Portability. The configuration tools are portable between cloud providers, to avoid ”provider lock-in”. The software works in combination with most public cloud providers and popular open-source software platforms for cloud computing (OpenStack, etc.)

• Product name: Numerical ocean/weather modeling and consulting
• Customer groups: Enterprises and governmental agencies which require geophysical data enhanced by numerical ocean/weather modeling. These organizations may not employ scientific staff in the field of oceanography/meteorology.
• Business drivers:
  • Demand for scientific consulting combined with custom-tailored, regional (localized), cost-effective and rapid numerical ocean/weather simulations by enterprises (including, but not limited to small and medium size enterprises) and governmental agencies.
  • All of the above listed for Module 1.
• Product details: The main service is comprehensive oceanographic/meteorological scientific consulting. It can be viewed as an additional service to the Module 1, optionally including full management of the computational modeling platform.

Since this proposal is for a short-term (6 months) feasibility study, it focuses mainly on Module 1, whose implementation we regard as an enabling, necessary driver for Module 2.

Numerical ocean/weather models are mathematical ocean/weather models, formulated on discrete computational domains according to the fundamental laws of physics ( Haidvogel and Beckmann 1999; Kantha and Clayson 2000). When a model is run on a computer, prescribed initial- and boundary conditions (temperature, salinity, flow velocity, etc.) are processed, and new (”synthetic”) data is generated. To link this terminology with the context of Förderrichtlinie ”Modernitätsfonds”, one may think of the prescribed data as observed data, and the synthetic data as refined data. The amount of observed data fed into a model depends on the complexity of the model. In highly idealized models, observed input data can consist of little more than a single number, for example the average observed density difference between two fluids in different geographical locations (e.g. Riha and Peliz 2013). More realistic models incorporate a variety of observed data, such as wind stress, radiative heating, freshwater/salinity changes caused by evaporation and precipitation, freshwater input from rivers, etc. In the most realistic models used for forecasting, real-time observed data is continuously assimilated into the model (Chassignet and Verron, 2006). The predictive skill of ocean modeling systems heavily depends on the quality and density of the observed data fed into the model. Note that in ocean models used for biological or biogeochemical studies, additional (biological or biogeochemical) data is necessary to drive the respective sub-model.

Numerical ocean/weather modeling requires large computational resources. Simulations are typically performed in distributed-memory computing environments, consisting of many connected individual computers or nodes. In each node, one or more CPUs access internal memory which is inaccessible to all other nodes. The numerical grid carrying the model variables is partitioned into blocks, and each node performs a computation on a single block. We refer to the connected system as cluster. Synchronization and communication of individual processes in a cluster is usually achieved by using a standardized communication protocol (MPI, see Lusk et al., 2009). Increasing the spatio-temporal resolution of a numerical model by a factor of two, typically increases the computational cost of running the model by a factor of about 8. Sufficiently high resolution is desirable in many numerical modeling applications, but bounded in practice by the capacity of the available computing infrastructure.

The acquisition, maintenance and management of a cluster is expensive. Universities and governmental agencies sometimes under-provision their IT departments with high-performance computing (HPC) infrastructure for economic reasons. That is, the maximum throughput of the acquired HPC infrastructure cannot meet the effective demand during peak utilization. If the number of computing tasks submitted from individual scientists/projects exceeds the capacity of the infrastructure, the tasks are simply ”queued” (i.e. they have to wait in line) until previously submitted tasks are completed. This is referred to as job scheduling or batch queuing. While it allows the respective institution to provision according to the average workload instead of the peak workload, it necessarily causes waiting times for individual projects. To date, this is simply accepted as an inconvenient consequence of limited on-premise computing resources.

Several companies provide access to cloud-based, on-demand supercomputing for fields such as genomics, machine learning, simulation, and scientific computations as a service. Here we specifically consider a business model which aims to facilitate delivery of cloud-based HPC solutions in the form of a paid ”add-on” service on top of general services provided by the mainstream cloud providers such as Amazon Web Services (AWS), Microsoft Azure and Google Compute Engine (see Fig. 1).

Established companies which use this model are, for example, Cycle Computing and Alces Flight . The former company has been cited by media as a ”truly disruptive” innovator, creating a market that never existed before, by providing HPC compute power to small companies/departments or researchers that until recently have never had access to supercomputers (McKendrick, 2016).

As mentioned above, numerical ocean/weather modeling requires large computational resources. To give an example, the DKRZ in Germany is a national service provider operating a supercomputer to enable numerical ocean/weather/climate simulations. Their current (2016) supercomputer features a total of 100.000 processor cores with InfiniBand®-connected compute nodes. Compared to this standard, the maximum capacity of cloud clusters assembled from resources offered by mainstream cloud providers is modest. To give a concrete example from the related field of engineering, Tambe and Kaiser (2015) report that a computational fluid dynamics solver of the ANSYS® software suite deployed on Amazon Web Services scales very well to 256 cores, and on larger problems even up to 512 cores, using 16 to 32 compute nodes interconnected with a 10 Gbit/s network, each of which has 16 cores and a minimum of 60GB of memory. This comparatively modest maximum (efficient) cluster size is partly due to an inferior networking bandwidth of 10 Gbit/s network offered by most cloud providers, as compared to recent InfiniBand® specifications featuring a bandwidth on the order 100 Gbit/s or more. Another performance limiting factor is hardware virtualization, which is employed by most mainstream public cloud providers. The essential point made by Tambe and Kaiser (2015), however, is that the currently available capacity of mainstream cloud infrastructure is perfectly sufficient for a large number practical applications. This finding is of central importance for the current proposal.

Very recently, Vance et al. (2016) described how cloud-based HPC is increasingly used in numerical ocean/weather modeling. They provide an introduction to the use of cloud computing in the atmospheric and oceanographic sciences, with an emphasis on scientific applications and examples rather than on the infrastructure of cloud computing.

1. Assess the demand for cloud-based ocean/weather modeling. Due to the strong influence of scientific staff on decision making processes in universities and research institutions, it is sensible to clearly distinguish the motivation for investment into economic reasons on the one hand, and scientific incentives on the other hand.
   • The economic advantages of cloud computing have been extensively discussed in other areas of computing. How do they apply in the specific context of numerical ocean/weather modeling?
   • How does science benefit? It has been argued that the possibility to perform a parameter sensitivity study (ensemble experiment) on 100 different 512-core clusters simultaneously may fundamentally change the way in which ocean, weather and climate research is conducted. Is this reasonable? Vance et al. (2016) discuss the advantages of cloud computing from a scientific perspective.
   • Several topics within market research overlap with engineering issues. Typically, the most important concern of scientists is how large (computationally intensive) numerical problems can become, under the condition that cloud-based cluster computing remains efficient.
2. Regarding the symbiosis/competition with established companies (e.g. Cycle Computing ; Alces Flight ), assess whether it is viable to focus on the ”niche” market of numerical ocean/weather modeling. How large is this market segment? What is the achievable market share within this segment?

During the 6 month period, we focus on defining and creating a minimum viable product by restricting attention to

• a single cloud provider. We choose Amazon Web Services, which we assume to have the biggest market share and representative technology. Limiting our attention to a single cloud provider allows us to use ”proprietary” (in the sense of ”non-portable”) helper tools to simplify the provisioning process. Although this necessarily leads to non-portable code, it provides a proof of concept and yields a concrete hardware/networking topology pattern, which can later be adapted to other cloud providers.
• a single numerical ocean model. We use the Regional Ocean Modeling System (ROMS) ( Shchepetkin and McWilliams 2005; Haidvogel et al. 2008; Shchepetkin and McWilliams 2009), due to its focus on regional-scale applications widespread adoption by the international ocean modeling community permissive free software license (Open Source Initiative, 2016) comparatively well-maintained and publicly accessible discussion forum and documentation ( The ROMS/TOMS Group 2016a; The ROMS/TOMS Group 2016b).

The main engineering tasks in this feasibility study are as follows:

1. Configure a test suite for benchmarking computational efficiency and scalability. The test suite will answer the question of how large (computationally intensive) numerical problems can become, under the condition that cloud-based cluster computing remains efficient. Is the 512-core limit accurate? The ROMS source code includes pre-configured benchmarking tests, which may be adapted to address questions specifically arising in the context of cloud computing (virtualization, networking, I/O, etc.). These modifications will incorporate opinions and previous experiences of members of the ocean modeling community, in particular the ROMS user community. In the course of this process, the role of the modeling community as a stakeholder in the project will become apparent. The range of involvement by members of the community may range from sporadic feedback on the one end of the spectrum, to continuous active involvement in discussions and/or contributions of code on the other end of the spectrum.
2. Produce software code representing ”infrastructure as code”, meeting the specific demands of numerical ocean/weather modeling. Provide the following functionality:
   • Automated provisioning of compute, storage and networking infrastructure of the cluster
   • Automated deployment/configuration of the scientific software stack, including pre- and post-processing software
   • In the early stage of this project we use Amazon’s CfnCluster (Amazon Web Services, 2016) tool for cluster provisioning, which facilitates proof of concept deployments and which can quickly be extended to support different clustered applications. A particularly attractive feature is the pre-configured integration of third-party scheduler systems (also referred to as batch-queuing systems, such as Sun Grid Engine, OpenLava, etc.) into Amazon’s cloud infrastructure, which automatically allocates compute nodes as a function of the number of pending jobs in the scheduler.
3. A major task (and challenge) is to determine a suitable work-flow for cloud-based numerical ocean/weather modeling studies. Since the scientists workstation is connected to the cluster via the internet (Fig. 1), the user experience may suffer if bandwidth is low and/or latency is high. Application design must be guided by the objective to (1) minimize transferral of large amounts of data, and (2) avoid unnecessary user interaction which could be perceived as unresponsive due to high latency. In this area the proposed project may greatly benefit from collaboration with established service providers in similar fields, such as ANSYS, Inc .
4. For our particular cloud provider, what is the right combination of instance types (CPU and memory) and networking infrastructure for numerical ocean/weather modeling?

A long term objective is to sell expertise in setting up infrastructure for numerical ocean/weather modeling. End users are either scientists or software engineers with strong scientific background. It is therefore necessary to gain a reputation within the scientific community. In this early stage of development, we use our preliminary results to

• exchange opinions and ideas with the scientific community.

The obtained results are not only valuable for end users, but also for technology companies such as cloud-based HPC providers (Cycle Computing; Alces Flight) and cloud providers (Amazon, Google, Microsoft, etc.). We use the obtained results to

• exchange opinions and ideas with cloud providers in general, and cloud-based HPC providers in particular.

In this proposal we focus on offering services on top of public cloud providers, which enables small- and medium sized companies to perform HPC modeling studies without having to invest in on-premise infrastructure. However, we point out that a large part of the obtained results is equally relevant for private clouds. This is significant, given that large research facilities will likely continue to operate private on-premise clouds, be it for economic reasons or for reasons of data protection. Hence, it may be profitable to develop HPC services on top of open-source cloud software platforms such as OpenStack. To further evaluate this idea, we use our results to

• exchange opinions and ideas with developers of open-source cloud software platforms .

Innovative oceanographic/meteorological scientific consulting services result from technological advances obtained through Module 1. Cloud-based modeling studies are less expensive, more flexible and provide results more quickly.