PRACE Change of Pace
Michaela Barth, PDC
PRACE is a non-profit organisation which has 25 member countries whose representative organisations have created, and now maintain, a pan-European supercomputing infrastructure that provides access to large-scale computing and data management resources and services for research. The PRACE consortium currently consists of 30 partners from 26 countries. Its budget is 30.1 million EUR with more than 85% of the funding being contributed by the European Commission (EC). To date, the member countries have made two major agreements to provide nationally-funded high-performance computing (HPC) resources to European researchers. The first agreement, PRACE 1.0, ran from 2010-15, and PRACE 2.0 went from 2015-22. In parallel with these agreements, there have been preparatory and implementation phase projects (funded by the EC) which complement the national investments in HPC infrastructure. The sixth of those implementation phases will end in December this year, so let us reflect proudly on the notable achievements of PRACE so far.
The non-profit PRACE organisation can claim its roots both in the HPC in Europe Task Force (HET) and the Distributed European Infrastructure for Supercomputing (DEISA). HET ran from 2004 to 2007 and was all about the big players in European HPC at the time (Finland, France, Germany, Italy, the Netherlands, Spain and the United Kingdom) trying to make Europe competitive within the HPC research arena, establish peer-review processes and obtain EC funding for top-level “Tier-0” national HPC systems. DEISA ran from 2001 to 2011 and, in contrast to HET, consisted of a series of projects which fostered collaboration between national HPC centres running the next level of systems (“Tier-1”). The centres involved in the DEISA projects worked on a joint benchmark suite, had a dedicated network for a distributed shared file system and established a common operational environment which eventually resulted in the DEISA Extreme Computing Initiative (DECI), which was later incorporated into PRACE.
The best ideas from HET and DEISA were used as a basis to form the non-profit PRACE infrastructure organisation in 2010. There was a short preparatory phase (PP) project. Some of you may remember PDC’s system Povel, which was the PRACE-PP prototype for energy-efficient computing. Since then, there have been six project implementation phases: 1IP-6IP, during which PRACE has matured and become more structured, gradually establishing itself as the leading name for providing HPC research services in Europe. Its processes and services have evolved into a well-oiled machine, and PRACE has become the organisation uniting European HPC centres providing services for research.
Some of PRACE’s key achievements to date include the establishment of a peer-review process based on scientific excellence, and the increase in the competence of the researchers who participated in the extensive training programmes. Other achievements that may be less obvious at first glance, but that are definitely not less important, include the creation of the unified European application benchmark suite (UEABS), and all the applications that have been enabled (95 research projects were supported for more than three months), as well as the many best-practice guides that have been developed. Highlights were documented in hundreds of white papers. Overall, the PRACE operational services were able to support regular calls twice a year (typically for two billion core hours from the 14th call onwards), adding up to a total of 24 regular project access calls. Those calls were supplemented by extra calls on the DECI track and the calls supporting more than 70 small to medium-sized enterprises (SMEs) through the SME HPC Adoption Programme in Europe (SHAPE). PRACE can also be proud of its activities related to providing information and guidance for decision makers at different levels, including technology monitoring, and giving input on procurements, and prototyping. Those efforts have made important contributions to improving the design and operation of large energy-efficient HPC centres in Europe. These contributions by PRACE have not gone unnoticed: for example, PRACE received the HPCwire Readers’ Choice Award “Best Use of HPC in Energy” for its strong interest in improving the energy efficiency of computing systems and reducing their environmental impact. Recently, PRACE’s incredibly quick reaction in deploying a COVID-19 fast track to provide access to resources for urgent computing research in Europe stood out positively. I have also witnessed PRACE bringing awareness of diversity, equity & inclusion (DEI) into public discussions about HPC human infrastructure at a time when other infrastructure players have not even considered the need to do so.
Looking more closely at the most recent project implementation phase, PRACE-6IP started in May 2019, and the initial plan was for it to run until December 2021; it was extended to June 2022, with a second extension for only a minority of the activities going to December 2022. During the pandemic, PRACE-6IP developed two new Massive Open Online Courses (MOOCs), the training portal was given a major revamp, six technical reports were produced in the area of market and technology watch ( prace-ri.eu/infrastructure-support/market-and-technology-watch ), operations groups worked towards a federated approach for the PRACE authentication and authorisation infrastructure (AAI), enabling of applications continued as usual along with benchmarking for 13 application codes (including GROMACS, NAMD, CP2K and Quantum Espresso) with an updated presentation format and on nine different systems. A new version (2.2) of the Unified European Application Benchmark Suite (UEABS) should be available in the PRACE repository very soon ( repository.prace-ri.eu/git/UEABS/ueabs ). For five systems, the energy to solution (that is, the total amount of energy consumed in solving a computational problem) was measured and, for the systems that were in the Green500 list, PRACE’s findings were mostly in line with the Green500 results for those systems. All the benchmarking has led us to the conclusion that, so far, there is no single system that gives the best results on all the benchmarks. So, when applying for allocations, researchers are advised to choose the system that will perform best for their particular applications.
During the final extension of PRACE-6IP, which runs to the end of this year, the best-practice guide for the Nordic pre-exascale system, LUMI, will be completed. There have been delays getting access to LUMI (mostly due to pandemic-related global supply shortages), and PRACE wants to be sure to stick close to its initial paradigm for good best-practice guides, namely “We’ve tested it and it works”, so everything in the guide is based on real hands-on experience. The LUMI guide will be the 29th best-practice guide published by PRACE, and since parts of LUMI are GPU-accelerated, it will focus especially on how to port scientific codes to use GPU accelerators. There are also plans to cover how to write code that is adapted, from the start, for GPU systems. It is vital that researchers who intend to use exascale systems in the future start adapting their code to use GPUs now! While LUMI is a stepping stone towards a true exascale system, further effort is needed to adapt software so it is ready for exascale systems. For that reason, PRACE has been investigating forward-looking software solutions. The high-quality software that has been developed is intended to lay the groundwork beyond PRACE-6IP.
The final PRACE-6IP deliverables are due in December 2022, and the final review for PRACE-6IP is set for February 2023. Those deliverables, including the best-practice guide, are expected to be available on Zenodo soon at zenodo.org/communities/prace . Also, the UEABS data sets will be moved to Zenodo. The PRACE CodeVault, an open repository for various HPC code samples, will be migrated to GitHub and the PRACE organisation will keep hosting PRACE documents for a certain period after 6IP ends.
The PRACE Distributed European Computing Initiative (DECI) programme provides access to Tier-1 level resources across Europe via a series of competitive calls. This popular programme and the associated tools (such as the portals hosting the proposals, reviews or the accounting data) will come to a halt by the end of June this year, along with the DECI project management data base (DPMDB) and the directory information service containing user and machine account details (LDAP). After supporting 728 projects in 28 countries with a total of 1.6 billion core hours through 17 calls, DECI is coming to the end of its life in this current form; there are no plans for a direct follow-on. However, reviewers have been presented with a document explaining the benefits of DECI, with a view to carrying the most valuable aspects of DECI forward into future calls supported by EuroHPC. (The European High-Performance Computing Joint Undertaking or EuroHPC JU, which started in 2018, is a joint initiative by the EU and European countries to develop a world-class supercomputing ecosystem in Europe.)
In contrast to other European HPC calls, DECI was set up as a resource exchange programme using pre-existing national panels as reviewers. Spanning a wide range of subject areas, DECI has become well known to European computational scientists while naturally supporting collaboration and pooling of competence across Europe and beyond. Between 15% to 30% of the resources were typically made available to “external” projects which came from countries not contributing resources. For the last three calls, 25% of the projects fell into this category, allowing projects of outstanding scientific excellence to be awarded resources irrespective of differences in levels of national funding, and thus supporting excellence in European research. DECI has also played a role in making access to multiple sites easier for users, as well as building up trust between European HPC centres. The resources offered were of the national (Tier-1) type and with a variety of different architectures, leading to a diverse array of available systems, which made it possible to match researchers with the architectures most appropriate for their simulations, while providing a large range of HPC expertise or filling the gap in between one national system and the next. It would be beneficial if something similar is offered under EuroHPC in the future.
As PRACE 6IP and DECI come to a close, PRACE has been looking into how it can best continue contributing to European HPC use for research. In the EuroHPC work plan, the access and allocation of computing resources, HPC training activities, and the envisaged user forum all play right into PRACE’s strengths; PRACE aims to be using its well-renowned peer-review process to allocate computing time to researchers on the forthcoming pre-exascale and petascale systems of EuroHPC and is considering how to bid in various upcoming calls for EuroHPC activities. PRACE has collected a list of interested partners for future collaborations to participate in those calls and also plans to provide high-level strategic support teams (HLSSTs) for EuroHPC systems; their tasks would include benchmarking, code optimisation and scaling-out applications.
In the future, PRACE aims to deploy and operate a platform for federating supercomputing (including HPC and quantum computing, as well as secure cloud-based services) and data resources for public and private users across the European Union. This platform will provide the framework to deploy basic federated infrastructure services on top of different technology implementations of resources, and enable the evolution, adaptation and combination of such basic services in order to facilitate the development of specific services according to users’ needs, including the research and scientific community, industry (including SMEs) and the public sector.
The draft work plan envisions that funding for facilitating access to the HPC ecosystem through national HPC competence centres will be continued and also proposes that centres of excellence and national HPC competence centres network with each other in the future under the umbrella of the CASTIEL 2 project. One of the first EuroHPC calls in this area involves new algorithms for applications on European exascale supercomputers (see eurohpc-ju.europa.eu ). In order to achieve a significant increase in the level of technology readiness for exascale computing, PRACE plans to answer this call with a project-based proposal where competence centres would propose short projects to develop novel exascale-ready algorithms and apply for funding for two full-time employees for a year to do the work. Then a PRACE consortium, consisting of the competence centres, would decide which of the projects would be funded.
PRACE envisions a future 3.0 phase where it would develop into a European association for computational science and – with a user-centric approach – change role from computing cycle facilitator to a user-focused service provider. By hosting a European user platform which joins HPC users and communities in Europe into an independent voice, PRACE wants to further develop its strengths in supporting HPC user communities to maximise the impact of their use of the HPC infrastructure. PRACE 3.0 plans to be a bottom-up organisation centred around the HPC user: “Give the HPC users a voice”. Provision of training would also be a high priority. The idea would be for user communities to be more actively involved, for example, by being given the power to openly participate in making decisions about PRACE’s activities and about directions for the future of HPC in Europe. The building blocks of such an undertaking would include inviting all the HPC users in Europe to be involved, and establishing a highly recognised cross-disciplinary scientific journal with a major focus on computational science, supported by an underlying high-quality peer-review process to ensure scientific excellence. Existing established rewards, like the PRACE Ada Lovelace reward and the new PRACE HPC Excellence Award, which recognise ground-breaking research relying on HPC, would continue as attractive and living elements of PRACE 3.0.
The organisational details for the next phase of PRACE are not yet set in stone, but the idea is to have different “chapters” of the infrastructure for major areas of computational research (such as biochemistry, bioinformatics, life sciences, physiology and medicine, chemical sciences and material sciences) and for membership to be extended so individual people, and not just organisations, can be members of PRACE.
Even though not all the details of the EuroHPC calls are known at this point, PRACE sees itself in a future where it will definitely be responsible for different activities as part of the European HPC ecosystem, especially in user support. PRACE aims to be in a coordination role, acting as a one-stop shop for the European HPC users. In addition, keeping the peer-review processes in place and enabling access to world-class computing and data management resources in Europe are significant parts of the PRACE 3.0 pillars and principles. Fostering international collaboration, continuing to monitor technology, and a thriving HPC ecosystem are also on the map. PRACE 3.0 would also aim to provide a platform for national computing centres to exchange best practices and to discuss ways to improve HPC services. A continued series of infrastructure workshops would make sense too. It is expected that successful and popular exchange and training programmes, like the “Summer of HPC” (SoHPC) would also be revitalised in the future.
At the latest PRACE-6IP all-hands meeting, discussion focused on how to avoid competence loss and keep the whole European HPC infrastructure structured while including both existing and new institutions in PRACE, and doing so in such a way that it will be easy to prepare applications for the upcoming EuroHPC calls expected to be opened later this year. Many voices were in favour of having a reviewed PRACE journal where technical white papers and reports could be published with a DOI, as envisioned in the PRACE 3.0 concepts. In the meantime, you can find details of open calls, including the EuroHPC access calls, at pracecalls.eu . It is also a good idea to keep an eye on the HPC in Europe portal ( hpc-portal.eu ), which collects all the HPC services offered in Europe into a single database. In addition, you can check out the training being offered by the different national competence centres: www.eurocc-access.eu/services/training .