JNS I Bengaluru: “Param Pravega”, one of India’s most powerful supercomputer, has been successfully deployed at Indian Institute of Science (IISc), Bengaluru.
Designed by the Centre for Development of Advanced Computing (C-DAC) “Param Pravega” is the largest supercomputer that has been installed under the National Supercomputing Mission (NSM) having a supercomputing capacity of 3.3 petaflops (1 petaflop equals a quadrillion or 1015 operations per second).
Under the Make in India initiative most of its components have been manufactured and assembled in the country, while and software stack were designed and created right here in India by C-DAC.
Notably, the National Supercomputing Mission was launched to enhance the research capacities and capabilities in the country by connecting them to form a Supercomputing grid, with National Knowledge Network (NKN) as the backbone.
The NSM is setting up a grid of supercomputing facilities in academic and research institutions across the country. Part of this is being imported from abroad and part built indigenously, it reported.
The Mission is being jointly steered by the Department of Science and Technology (DST) and the Ministry of Electronics and Information Technology (MeitY) and implemented by the Centre for Development of Advanced Computing (C-DAC), Pune, and the Indian Institute of Science (IISc), Bengaluru.
PARAM Shivay was the first supercomputer assembled indigenously, was installed in IIT (BHU), followed by PARAM Shakti, PARAM Brahma, PARAM Yukti, PARAM Sanganak at IIT-Kharagpur IISER, Pune, JNCASR, Bengaluru and IIT Kanpur respectively.
“Under the Make in India initiative most of its components have been manufactured and assembled in the country, while and software stack were designed and created right here in India by C-DAC; Its having a supercomputing capacity of 3.3 petaflops (1 petaflop equals a quadrillion or 1015 operations per second).”
India is fast emerging a leader in high power computing with the National SuperComputing Mission (NSM) boosting it to meet the increasing computational demands of academia, researchers, MSMEs, and startups in areas like oil exploration, flood prediction as well as genomics and drug discovery.
“These systems have greatly helped faculty members and students carry out major R&D activities, including developing platforms for genomics and drug discovery, studying urban environmental issues, establishing flood warning and prediction systems, and optimising telecom networks,” IISc statement reveals.
According to the reports, Param Pravega is a supercomputer part of the High-Performance Computing class of systems, which is a mix of heterogeneous nodes, with Intel Xeon Cascade Lake processors for the CPU nodes and NVIDIA Tesla V100 cards on the GPU nodes. This machine hosts an array of program development tools, utilities, and libraries for developing and executing High Performance Computing (HPC) applications.
The IISC Bengaluru already has a cutting-edge supercomputing facility established several years ago. In 2015, the Institute procured and installed SahasraT, which was at that time the fastest supercomputer in the country. SahasraT, which was at that time the fastest supercomputer in the country. The facility is being used by students and faculty to carry out research in various impactful and socially-relevant areas.
The research conducted by the SahasraT supercomputer revolved around Covid-19 and other infectious diseases. This included modelling viral entry and binding, studying interactions of proteins in bacterial and viral diseases, and designing new molecules with antibacterial and antiviral properties.
“Researchers have also used the facility to simulate turbulent flows for green energy technologies, study climate change and associated impacts, analyse aircraft engines and hypersonic flight vehicles, and many other research activities. These efforts are expected to ramp up significantly with Param Pravega,” it said in the statement.
About “Param Pravega”
The Param Pravega supercomputer consists of 11 DCLC racks of compute nodes, 2 Service racks of Master/Service nodes, and 4 Storage racks of DDN storage. The node configuration includes 2 Master nodes, 11 Login nodes, 2 Firewall nodes, 4 Management, 1 NIS Slave and 624 (CPU+GPU) compute nodes. The compute nodes are of three categories, namely, Regular CPU, High-memory CPU, and GPU nodes. All the nodes in the system are connected using Mellanox high-speed HDR-Infiniband interconnection network using a FAT-tree topology with a 1:1 subscription ratio. The system is also augmented with a 4 Petabyte parallel storage from DDN for parallel filesystem access.
Regular CPU nodes: CPU nodes are built using Intel Xeon Cascade Lake 8268 2.9 GHz processors in a 2-socket configuration with 48 cores, 192GB RAM (4 GB per core), and 480GB SSD local storage per node. There are a total of 428 such nodes on PARAM Pravega constituting 20,544 cores for CPU-only nodes for computations resulting in 1.9PF peak capability.
High-Memory CPU nodes: The system also hosts High-memory CPU-only nodes that are similar in configuration to the CPU-only nodes except that these high-memory nodes have higher RAM of 768 GB per node (16 GB per core). There are a total of 156 such nodes on this system yielding a maximum of 7488 cores for high-memory computations giving 0.694PF of peak computing capability.
GPU nodes: PARAM Pravega also hosts 40 GPU nodes. The CPU of each node consists of Intel Xeon G-6248 2.5 GHz processor in a 2-socket configuration with 40 cores, 192GB RAM and 480GB SSD local storage. The GPU of each node is made up of two Nvidia V100 Tesla 16GB (HBM2 device memory) GPU cards. Thus, the 40 GPU nodes consist of a total of 1600 host CPU cores and 80 Nvidia V100 cards. The accelerator nodes contribute a total of 0.688 PFs (0.128 (host cpus) + 0.560 (GPUs)) computational capability.
High-Speed Parallel Storage: 4 PetaBytes of usable space is provided by a high-speed parallel Lustre filesystem with a throughput of 100 GB per second. The storage subsystem is connected to the machine using Infiniband interconnection.
High-Speed Interconnect: The system is integrated with a BullSequana XH200 Mellanox HDR Infiniband interconnection using FAT-Tree topology for MPI communications. The line speed of the interconnection is 100Gbps. Apart from this the system also has a secondary 10Gbps Ethernet connection for login and storage communication.
Software environment: The entire system is built to operate using Linux OS based on CentOS 7.x distribution. The machine hosts an array of program development tools, utilities, and libraries for ease of development and execution of HPC applications on the heterogeneous hardware of the machine. Common compilers from GNU and Intel are accessible on the system for MPI and OpenMP parallel libraries. For use on the GPU nodes the system has CUDA and OpenACC SDKs installed. Further, popularly used parallel mathematical, scientific, and application libraries like Intel-MKL, GNU Scientific library, HDF5, NetCDF, range of Python-based mathematical and data manipulation libraries etc. are installed on the system. Added to these, the machine also hosts system monitoring and management tools developed by CDAC team.
The system is accessible to users through the login nodes with domain name parampravega.iisc.ac.in and can be used for launching jobs in batch execution mode. User is expected to ssh into the login nodes and create appropriate job scripts and submit these scripts to the SLURM batch scheduling software. The parallel file system is accessible to users through job scripts and is meant for use during job execution.
