Hardware, Hebb

In January 2007, PDC received and installed a 1024 node, 2048 processor Blue Gene/L system with 1 TB of memory and a peak capability of 5.7TF for the Stockholm Brain Institute (SBI).

The BG/L system has a disk system with 96 x 146.8 GB (13.7 TB) of Fiber Channel 2Gbps/15kRPM FC (40K6808) disks, residing in DS4700 controllers and DS4000 disk-drawers. The disks are divided into several Raid5 sets. With assignment of hot spares the effective file system size is 10TB. GPFS is used as the clusterwide (Blue Gene) file system. GPFS is mounted by the front-end-node (Linux), the control-node (Linux) as well as all of the Blue Gene compute and I/O-nodes (custom Linux.)

The BG system has its own file servers, four IBM 9110-51A running AIX (dual Power5 processors, 2GB memory), a front-end, a IBM 9131-52A running Linux (8-way Power5, 16GB memory), and a control-node (aka master-node,) also an IBM 9131-52A running Linux (8-way Power5, 8GB memory.)

The interconnect between the GPFS server-nodes, the Blue Gene, the front-end and the control-node is based on GigE, though the switches for this interconnect support 10GE (2 Cisco Catalyst 4948 10GE switches.)

The GPFS file system delivers an aggregate of 1001 MB/s READ and 1126 MB/s WRITE. When doing 1024 simultaneous writes, from the 1024 nodes into 1024 unique files, it delivers over 1 MB/s/file; sustained, until the file system is full.