Prometheus and Ares supercomputers took positions on the TOP500 list

Prometheus – fast and energy saving

Prometheus with its 2,65 PFlops of theoretical performance has been listed at 373^rd position. This top-efficient supercomputer located in Poland has been built by Hewlett-Packard according to requirements and partial design provided by Cyfronet.

Prometheus consists of more than 2,200 servers based on HP Apollo8000 platform, combined with super-fast InfiniBand network with 56 Gbit/s capacity. Its energy saving and high-performance Intel Haswell latest-generation processors offer 53,748 cores. These are accompanied by 283.5 TB DDR4 RAM in total, and by two storage file systems of 10 PB total capacity, and 180 GB/s access speed. Prometheus has been also equipped with NVIDIA Tesla GPGPUs.

The Prometheus architecture responds to the diverse needs of researchers by providing computing resources organized in partitions:

• classical cluster of computing servers with highly efficient CPU nodes equipped with Intel Xeon Haswell and Intel Xeon Gold processors,
• cluster of servers equipped with graphic accelerators GPGPU NVIDIA Tesla K40 XL,
• acceleration partition including GPGPU NVIDIA K80 cards as well as Intel Xeon Phi 7120P and Nallatech FPGA accelerators,
• partition dedicated to calculations related to artificial intelligence, equipped with GPGPU NVIDIA Tesla V100 graphics accelerators.

Prometheus has been installed in a high-tech computing room, exclusively adapted for its operation. The supercomputer's proper functioning is additionally supported by the accompanying infrastructure, including such systems as guaranteed power supply with an additional generator, modern air-conditioning and gas extinguishing.

Prometheus - the fastest and the greenest

The Prometheus supercomputer, built according to the assumptions developed by Cyfronet experts, at the time of launch in 2015 was the first in Europe and one of the largest installations in the world based on direct liquid cooling technology. After several years of operation, it is still one of the most energy-efficient computers of its class (143^rd place on the Green500 world list, June 2021).

Thanks to the innovative technology of direct liquid cooling, Prometheus is also one of the most energy-efficient computers in its class in the world. This was achieved by using the cooling liquid having a temperature of 28°C. To cool down the liquid to such a temperature in our climate it is enough to use cheap in use dry-coolers, instead of ice water generators, consuming relatively large amounts of electricity and classic technological air conditioning systems. The liquid cools both CPU and GPGPU processors as well as memory modules using a special hermetic system of heat pipes and contact heat transfer between the servers and the system in which the liquid circulates. The servers themselves remain "dry" – they can be removed at any time without the risk of liquid leakage in the cooling system. Liquid cooling has also made it possible to achieve a very high installation density – up to 144 computing servers in a single rack.

The Prometheus PUE factor is comparable with some of the largest data centres in the world, such as Google, Amazon or Facebook. Furthermore, liquid cooling allows for extremely high installation density, therefore Prometheus, weighing of more than 40 tons, covers 18m² area and is placed on 20 racks only. To achieve the same computing power in case of Zeus (Prometheus' predecessor), it would have to take about 160 racks. Baribal, the predecessor of Zeus, with computing power of 1,5 TFlops was placed on 8 racks. To achieve the computing power of Prometheus it would take as many as 12,000 Baribal's racks – points Cyfronet's Director.

Zeus supports Prometheus

For less demanding computing tasks, supercomputer Zeus still offers its computing power. The heterogeneous architecture of this supercomputer consisting of four dedicated hardware partitions provides 25,468 computing cores and 204 GPGPUs with total theoretical computational power of 374 TFlops. This is accompanied by 60 TB of RAM and 2.3 PB of disk storage. Zeus is built of over 1,300 individual blade servers, interconnected via high-speed 40 Gb/s InfiniBand network.

It is worth noting that even the highest positions on the TOP500 list, or the latest technologies used to build high-performance computers, do not fully reflect the importance of this type of computing resources for the Polish scientific community. The usefulness of supercomputers made available by ACC Cyfronet AGH as a tool for conducting scientific research is best demonstrated by statistical data on their use. In 2020 the two most powerful supercomputers in ACC Cyfronet AGH – Zeus and Prometheus, executed 5 696 919 jobs with a total duration of 41 761 years of CPU time.

Even better support for calculations for artificial intelligence

In 2019, Prometheus was equipped with the fastest in Poland academic system dedicated to artificial intelligence, thanks to which it became possible to effectively combine large-scale simulations with machine learning methods and conduct research in the same computing environment and on shared data sets. The system consists of HPE Apollo 6500 servers, each of which is equipped with two Intel Xeon Gold 5220 processors, eight NVIDIA Tesla V100 accelerators and 384 GB of RAM. The system has the computing power of over 4 PFlops (petaflops) for tensor operations and 256 TFlops (TeraFlops) for standard calculations performed on double precision numbers. The system is mainly used for research in the field of chemistry, biology, medicine and the development of algorithms for autonomous vehicles.

Prometheus supercomputer used in the fight against coronavirus

Since 2020 computing grants dedicated to research on SARS-CoV-2 coronavirus and the COVID-19 disease it causes have had priority in the access queue for Prometheus resources and have been given the highest priority. Over 53 thousand computing cores combined in one machine and accompanying infrastructure allow for fast processing of large amounts of medical, biological and chemical data. The research include antibodies present at the time of infection, molecules with potential for inhibiting infection, and opportunities for vaccine development. Part of the computing power of Prometheus is made available as part of the European PRACE partnership to conduct a pan-European hackathon aimed at developing new solutions in the fight against coronavirus. Cyfronet also makes resources available within the EGI federation - a distributed computing infrastructure that brings together resources of over 250 units from around the world. The EGI Federation and the American Open Science Grid (OSG) have joined forces to support research projects on COVID-19. Additionally, the Haddock tool created at the University of Utrecht, which is used by biomolecular modeling, is available through the EOSC (European Open Science Cloud) portal developed by Cyfronet. ACK CYFRONET AGH also participates in the EOSC Synergy project, which made available cloud computing resources to fight the virus.

Scientific calculations from different fields

Prometheus and Zeus are part of the European cloud and grid infrastructure within the European Grid Infrastructure (EGI). At the same time, Prometheus and Zeus are important supercomputers in the PLGrid nationwide computing infrastructure for scientific research in silico (performed with the help of computers).

Thanks to the PLGrid infrastructure, scientists can intensively use the computing power of Prometheus and Zeus. Dedicated computing environments, so-called domain grids, and specialised IT platforms enable conduction of increasingly complex research problems.

Scientists conducting research with use of the Cyfronet infrastructure represent many fields. Advanced modeling and numerical calculations are used mainly in: physics, chemistry, biology, medicine and material technology, as well as in astronomy, geology and environmental protection. Research topics include: dynamic analysis of ceramic composite materials, modeling and optimization of the prototype of an energy-saving hydrogen fuel cell vehicle, simulations of molecular dynamics of electrolytes for ion cells or use of quantum methods for image analysis. Researchers also conduct research on gene expression in cod from the Baltic Sea, study of the interaction of influenza virus fusion peptides with lipid membranes, analyses of the impact of underground fluid injection on drops of stress of induced seismic tremors or research on microphysical processes in space plasma.

Scientific computations do not include simulations only. Computing power is utilised by Polish scientists also within international projects, including experiments like CTA, LOFAR, EPOS, Large Hadron Collider in CERN and the recently discovered gravitational waves in LIGO and VIRGO detectors.

As part of the PLGrid infrastructure, Cyfronet supercomputers are also used for the purposes of high energy physics (ATLAS, LHCb, ALICE and CMS projects), astrophysicists (CTA, LOFAR), Earth Sciences (EPOS), European Combustion Source (ESS), biologists (WeNMR) and humanities and social scientists (CLARIN).

New supercomputer in Cyfronet - Ares

In 2021, the Ares supercomputer was launched in Cyfronet. It is built of a set of computing servers with a total theoretical performance of over 3.5 PFlops and the necessary disk subsystem with a capacity of over 11 PB. The supercomputer has 37,824 computing cores (Intel Xeon Platinum processors) and 147.7 TB of RAM. It was also equipped with a liquid cooling system and GPU cards. The new supercomputer will be used primarily to carry out the computational tasks within the PRACE-LAB project.

Use the power of supercomputers in scientific research

Supercomputers enable to significantly reduce time of computations that using a single computer would often take many years (in specific cases more than 150, 700 or even 1000 years). Here they may be usually performed within a few days. Cyfronet users can benefit from the professional support – starting from training, to individual consultations with experts. Information concerning the use of the Prometheus and Zeus' resources by scientists is available on http://www.cyfronet.pl and http://www.plgrid.pl. It is worth to register in the PLGrid portal https://portal.plgrid.pl and use many unique services.

Supercomputers from Poland compared to Europe and the world

Supercomputer Fugaku from Japan took the first position on TOP500, with more than 537 PFlops of computational power. The list is dominated by supercomputers from China (187 units) and USA (122 units).

The European countries having supercomputers on the list (15 countries including Russia) occupied a total of 113 positions. The fastest European supercomputer is JEWELS from Germany, ranked 8^th. The top ten list also includes another supercomputer from Italy, HPC5, in the 9^th position. The countries in Europe with the largest number of supercomputers on the TOP500 list are: Germany (23 units), France (16 units), and the Netherlands (16 units).

The June 2021 edition of TOP500 spots also other supercomputers installed in Polish computing centres and research institutes. The full list of Polish supercomputers on TOP500 is as follows:

• 116 – Altair, PSNC, Polish Academy od Sciences, Poznan,
• 216 – Ares, ACC Cyfronet AGH,
• 360 – Tryton Plus, TASK, Technical University Gdańsk,
• 373 – Prometheus, ACC Cyfronet AGH.