Computing cluster
 |  |  |
Computing cluster
HPC cluster is based on blade systems from IBM and Dell. Dell blade servers were purchased within VAVINET project, blade servers from IBM within projects of FR CESNET and FRVŠ. Together there is now 102 modules, each with two 4 to 28 core CPUs and 8 to 256GB of RAM. On all blade servers we use 64bit CentOS operating system. To store large datasets there are 12 storage servers with total capacity of over 500TB.
All computations on the cluster are maintained by the Sun Grid Engine (SGE) job management system. The tasks have to be described with a script (usually a shell script) and this script is submitted to SGE. Based on current state of computational nodes as well as storage and other conditions SGE decides when and on which node to assign each task. Maximal theoretic size of a parallel computation is dynamic, depends on the current state of all nodes and currently it could be as high as 2400 processes.
Blade servers
| Chassis | Modules | Hardware |
|---|---|---|
 blademm1 |
 blade001 - blade012 |
IBM BladeCenter E, switch BNT 14x1Gb/s+3x10Gb/s PSU 4 x 2kW |
| 12 modules HS22 (7870) 2 CPU Intel Xeon 5670 2,93 GHz 6/12core 32K/256K/12MB RAM DDR3 36GB (3 channels) - blade001-005,009,010,012 RAM DDR3 30GB (3 channels) - blade006-008,011 HDD SAS 2,5" 146 GB | ||
blademm2 |
blade013 - blade024 |
IBM BladeCenter E, switch BNT 14x1Gb/s+3x10Gb/s PSU 4 x 2kW |
| 12 modules HS22 (7870) 2 CPU Intel Xeon 5670 2,93 GHz 6/12core 32K/256K/12MB RAM DDR3 36GB (3 channels) HDD SAS 2,5" 146 GB - blade013-018 HDD SAS 2,5" 300 GB - blade019-024 | ||
blademm3 |
blade025 - blade029 |
IBM BladeCenter E, switch BNT 14x1Gb/s+3x10Gb/s PSU 4 x 2kW |
| 5 modules HS22 (7870) 2 CPU Intel Xeon 5670 2,93 GHz 6/12core 32K/256K/12MB RAM DDR3 36GB (3 channels) HDD SAS 2,5" 300 GB | ||
 blade030 - blade036 |
7 modules HS21XM (7995) 2 CPU Intel Xeon E5345 2.33GHz 4core 8MB L2 RAM FB-DDR2/5300 20GB (2 channels) HDD SAS 2,5" 300 GB | |
blademm4 |
blade037 - blade048 |
IBM BladeCenter E, switch BNT 14x1Gb/s+3x10Gb/s PSU 4 x 2kW |
| 12 modules HS21XM (7995) 2 CPU Intel Xeon E5345 2.33GHz 4core 8MB L2 RAM FB-DDR2/5300 8 GB (2 channels) HDD SCSI 73 GB | ||
 dellmm1 |
 blade055 - blade070 |
Blade chassis Dell M1000e, switch M8024-k 24 x 10Gb/s |
| 6 modules Blade Dell M630 2 CPU Intel Xeon E5-2680 v3 2,4 GHz, 14/28core, 35 MB cache RAM DDR4-2133 128 GB (4 channels) HDD 2,5" 300 GB SAS 15k LAN Broadcom 57810-k 2 x 10Gb/s KR | ||
|
10 modules Blade Dell M620 2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache RAM DDR3-1600 64 GB (4 channels) HDD 2,5" 300 GB SAS 10k LAN Broadcom 57810-k 2 x 10Gb/s KR | ||
dellmm2 |
blade071 - blade086 |
Blade chassis Dell M1000e, switch M8024-k 24 x 10Gb/s |
| 16 modules Blade Dell M620 2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache RAM DDR3-1600 64 GB (4 channels) HDD 2,5" 300 GB SAS 10k LAN Broadcom 57810-k 2 x 10Gb/s KR | ||
dellmm3 |
blade087 - blade102 |
Blade chassis Dell M1000e, switch M8024-k 24 x 10Gb/s |
| 14 modules Blade Dell M620 2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache RAM DDR3-1600 64/128 GB (4 channels) HDD 2,5" 300 GB SAS 10k LAN Broadcom 57810-k 2 x 10Gb/s KR | ||
|
2 modules Blade Dell M620 2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache RAM DDR3-1600 256 GB HDD 2,5" 300 GB SAS 10k LAN Broadcom 57810-k 2 x 10Gb/s KR |
HPC servers
 welda1, welda2, ulrika, svatava |
Supermicro SC113TQ MB Supermicro X10DRW-E CPU Intel Xeon E5-2620v3 2,4 GHz 6/12core, 20 MB cache RAM DDR4-2133 32GB (4 channels) HDD Crucial SSD 250 GB PSU 600W |
GPU servers
 supergpu1,supergpu2 |
Supermicro 7047GR-TPRF MB Supermicro X9DRG-QF 2 CPU Intel Xeon E5-2630v2 2,6 GHz, 6/12core, 15 MB cache RAM DDR3-1600 64GB (4 channels) HDD WD RE4 500GB Ethernet 10 Gb/s Intel X520-DA2 4 GPU NVidia GTX 980 4GB GDDR5 PSU 2x1600 W |
supergpu3,supergpu4 |
Supermicro 7048GR-TR MB Supermicro X10DRG-Q 2 CPU Intel Xeon E5-2620v3 2,3 GHz, 6/12core, 20 MB cache RAM DDR4-2133 64GB (4 channels) SSD Crucial 250GB Ethernet 1 Gb/s 4 GPU NVidia GTX 1080 8GB GDDR5 PSU 2x2000 W |
superphi1,superphi2 |
Supermicro 7048GR-TR MB Supermicro X10DRG-Q 2 CPU Intel Xeon E5-2620v3 2,3 GHz, 6/12core, 20 MB cache RAM DDR4-2133 64GB (4 channels) SSD Crucial 250GB Ethernet 1 Gb/s 1 GPU NVidia GTX 980 4GB GDDR5 3 Intel Xeon Phi PSU 2x2000 W |
 dellgpu1,dellgpu2 |
Dell T620 2 CPU Intel Xeon E5-2670 2,6 GHz, 8/16core, 20 MB cache RAM DDR3-1600 64GB (4 channels) HDD WD RE4 500GB LAN Broadcom 57810S 2 x 10Gb/s SFP+ 4 GPU NVidia GTX 980 4GB GDDR5 PSU 2x1100 W |
File servers
 matylda1-6 |
Supermicro SC113TQ MB Supermicro X10DRW-E 2 CPU Intel Xeon E5-2620v3 2,4 GHz 6/12core, 20 MB cache RAM DDR4-2133 64GB HDD Intel SSD 80 GB Ethernet 10 Gb/s Intel X520-DA2 72 TB RAID-6 3Ware (LSI) 9750SA-8e External JBOD Supermicro SC836E16-R1200B |
 scratch1-6 |
Supermicro SC815TQ MB Supermicro X9DRW-iF CPU Intel Xeon E5-2630, 2,3 GHz, 6/12core, 15 MB cache RAM DDR3-1600 16 GB (4 channels) Ethernet 10 Gb/s Intel X520-DA2 10 TB RAID-6 3Ware (LSI) 9750SA-8e External JBOD Supermicro SC826E16-R500LP |
Network interconnect 10Gb/s
 cluster switches |
Extreme Networks Summit X650-24x 4 x 24 ports Ethernet 10 Gb/s SFP+ SummitStack 256 Gb/s ring |