Various notes from the Accelerating Devices Lab (X-Dev) of Jülich Supercomputing Centre

Carolin Penke

Since early 2022, the Accelerated Devices lab has been involved in the OpenGPT-X project.

 1

OpenGPT-X trains large language models to enable new data-driven business solutions and specifically address European needs. As of January 2025, the project has published its main results and is set to wrap up in early 2025.

Wrapping up OpenGPT-X: Using Supercomputers to Train a European Language Model

Poster in institute repository: http://dx.doi.org/10.34734/FZJ-2023-04519

Jayesh Badwaik

table td {width: 50%;} .pub { line-height: 40px; height: 40px; background-color: #e2e2ec; border-left: 6px solid #8b8bb6; margin-bottom: 20px; padding: 4px 12px; }

 
 
 Poster in institute repository:

http://dx.doi.org/10.34734/FZJ-2023-04519 At the ISC High Performance Conference 2023 a little while ago, we presented a project poster on the SCALABLE project, embedded at the end of this post.

ISC23 Project Poster: SCALABLE: Scalable Lattice-Boltzmann Leaps to Exascale

Chelsea John

Environment Setup           Enabling UCC in OpenMPI       Enabling NCCL in UCC (Team Layer Selection)       All The Variables           Results           1. Plain OpenMPI       2. OpenMPI with UCC       3. OpenMPI with UCC+NCCL           Scaling Plots           Average Latency       Bus Bandwidth                   Comparing MPI, UCC, UCC+NCCL           Comparing UCC+NCCL, NCCL                           Summary   Technical Details  This post

MPI as API: Using UCC’s NCCL Backend for MPI’s Allreduce

Andreas Herten

For a recent talk at DKRZ in the scope of the natESM project, I created a table summarizing the current state of using a certain programming model on a GPU of a certain vendor, for C++ and Fortran. Since it lead to quite a discussion in the session, I made a standalone version of it with some updates and elaborations here and there. I present, the

 GPU Vendor/Programming Model Compatibility Table

! .pub { line-height: 40px;

GPU Vendor/Programming Model Compatibility Table

Albert Kahira

A few days ago, OPTIMA announced the release of deliverable 3.5, to which I contributed. This deliverable is part of a set of five deliverables under work package 3. But first, let’s talk about

 OPTIMA

. OPTIMA is an EU-funded project whose goal is to prove that several HPC applications can take advantage of the future highly heterogeneous FPGA-populated HPC systems.

OOPS Version 1 Release

At the end of May, Forschungszentrum Jülich got a new website. Finally, the group also has a dedicated page there, listing the things we do, the project we are part of, and all the members in the group. And thanks to a photo shooting by Sebastian, we have great portraits and an amazing group picture.

X-Dev Website at fz-juelich.de

At the end of 2023, the contract was signed to build JUPITER, a new European supercomputer and the first to reach 1 ExaFLOP/s HPL performance.

Paper: Application-Driven Exascale: The JUPITER Benchmark Suite

** Poster publication:** http://hdl.handle.net/2128/32006  The 14th JLESC workshop (JLESC: Joint Laboratory for Extreme-Scale Computing) was hosted by the National Center for Supercomputing Applications (NCSA) in Urbana, Illinois from 28th September to 30th September. We had the opportunity to present the OpenGPT-X project in form of a poster.

Poster: OpenGPT-X - Training Large Language Models on HPC Systems

On June 21 and 22, we held a workshop looking back on the last ten years of our lab together with NVIDIA – the NVIDIA Application Lab at Jülich (or

 NVLab

, as I sometimes abbreviate it). The material can be found in the agenda at Indico. We invited a set of application owners with which we worked together during that time to present past developments, recent challenges, and future plans.

10 Year Anniversary Workshop of NVIDIA Application Lab at Jülich

1

This blog is an experiment. We want to share bits and pieces of our work; the reports we write, the presentations we hold, or the little discoveries we make, or even some first, water-testing investigations; and all the rest. It’s a

 documentation

of what we do. Little bits of science, collected in the open, and sometimes even not that little.

DOIng it Right! (DOIs for This Blog)

In November of 2022, I created a table comparing GPU programming models and their  support on GPUs of the three vendors (AMD, Intel, NVIDIA) for a talk. The audience liked it, so I beefed it up a little and posted it in this very blog.

Paper: Many Cores, Many Models – GPU Programming Model vs. Vendor Compatibility Overview

Poster publication: http://hdl.handle.net/2128/34532

.pub { line-height: 40px; height: 40px; background-color: #e2e2ec; border-left: 6px solid #8b8bb6; margin-bottom: 20px; padding: 4px 12px; }

 
 
 Poster publication:

http://hdl.handle.net/2128/34532 The ISC High Performance Conference 2023 was held at Hamburg, Germany from 21st May to 25th May.

ISC23 Project Poster: OpenGPT-X – Training Large Language Models on HPC Systems

The Supercomputing Conference 2023 took place in Denver, Colorado, from November 12th to 17th. For the

 Women in HPC

workshop, we submitted a paper, which focused on benchmarking different accelerators for AI. The paper was accepted and I was invited to hold a lightning talk to show the work, spun off our OpenGPT-X project.

SC23 WHPC Workshop Paper: OpenGPT-X – Novel Architecture Exploration

TL;DR: I held a HPC intro talk. Slides are below.

In MAELSTROM, we connect three areas of science:

 🌍Weather and climate simulation

with

 🤖Machine Learning methods and workflows

using

 📈HPC techniques and resources

. Halfway into the project, we held a boot camp at JSC to teach this Venn diagram to a group of students a few days ago. Some were ML experts, but had never used a HPC system.

Talk: Introduction to HPC

From November 17th to 22nd, 2024, HPC professionals and researchers gathered in Atlanta, Georgia, for the Supercomputing Conference 2024. We presented a paper at the 2024 International Workshop on Performance, Portability, and Productivity in HPC where we introduced CARAML, a reproducible AI benchmarking framework, and jpwr, a custom energy assessment module. The presentation slides are embedded at the bottom.

Advancing AI Benchmarking in HPC: Introducing CARAML and JPWR at SC24

Sebastian Achilles

MSA Concept   MSA Software Building Blocks   Workshop           Exercises                   1: Hello World! 2. GPU Hello World! 3: CPU-GPU Ping Pong                       Slides          On May 29, we held a workshop about using the Modular Supercomputing Architecture (MSA) together with project partners from ParTec.

MSA Introduction Workshop

Together with our colleagues from Helmholtz-Zentrum Dresden-Rossendorf (HZDR) and in collaboration with HIDA and OpenHackathons, we hosted the

 Helmholtz GPU Hackathon 2023

in Jülich in May.

 I’ve blogged about the event for the
 
 Zweikommazwei
 
 blog of Forschungszentrum;

Helmholtz GPU Hackathon 2023

Stepan Nassyr

sup.wayback {         font-size: 0.6em; color: gray; }     .imagegrid {       display: grid; grid-template-columns: auto auto; grid-gap: 20px; }     .imagegrid a {       display: flex; align-items: center; justify-content: center; }     .imagegrid a img {       margin: 0;     }     table#asm-highlight, table#asm-highlight figure code {       font-size: smaller;

Poster: GEMMERATOR – GEMM Kernel Generator

A few months ago, we extended the JURECA Evaluation Platform

 1

at JSC by two nodes with AMD Instinct MI250 GPUs (four GPUs each). The nodes are Gigabyte G262-ZO0 servers, each with a dual socket AMD EPYC 7443 processor (24 cores per socket, SMT-2) and with four MI250 GPUs (128 GB memory). OSU Bandwidth Micro-Benchmark A100 Comparison GPU STREAM Variant Data Size Scan Threads and Data Sizes Conclusion Technical Details OSU

First Benchmarks with AMD Instinct MI250 GPUs at JSC

About    This blog post is based on a presentation I held at the “New Trends in Computational Science in Engineering and Industrial Mathematics” workshop in Magdeburg on 01/07/2022. My goal is to give a brief introduction to the state of current large language models, the OpenGPT-X project, and the transformer neural network architecture for people unfamiliar with the subject. About What is a language model?

JSC Accelerating Devices Lab