Yet *more* RAID speed comparisons, now having an LSI SCSI RAID card (LSI Megaraid 320-1 card) as well as the Adaptec RAID add-on and plain Linux md software RAID. This is largely repitition of the 2006-07-* tests but with the new LSI controller too. /* THE DISKS: * * This is obtained from the megamgr utility of * the LSI Megaraid card, which card was not used * during the loading tests. * Adapter Type : MegaRAID SCSI 320-1 * Adapter-1ts * {Rbld Rate = 100% * Fw Ver = 1L47 Drive * BIOS Ver = G121 Drive * DRAM Size = 64MB * Logical Drives Info */ SCSI-ID 0 Device Identification Device Type : DISK Capacity : 70135MB Vendor ID : FUJITSU Product ID : MAW3073NC Revision : 0104 SCSI Standard : SCSI-3 SCSI-ID 1,2 Device Type : DISK Capacity : 140013MB Vendor ID : SEAGATE Product ID : ST3146707LC Revision : 0005 SCSI Standard : SCSI-3 SCSI-ID 3 Device Type : DISK Capacity : 140013MB Vendor ID : SEAGATE Product ID : ST3146707LC Revision : D701 SCSI Standard : SCSI-3 SCSI-ID 4,5,6 Device Type : DISK Capacity : 140297MB Vendor ID : MAXTOR Product ID : ATLAS10K5_147SCA Revision : JNZH SCSI Standard : SCSI-3 SCSI-ID 7 Device Type : DISK Capacity : 140013MB ========================================================================== Using on-board Adaptec controller, mdraid. ========================================================================== Gentoo LiveCD livecd ~ # uname -a Linux livecd 2.6.12-gentoo-r10 #1 SMP Sun Nov 13 20:03:23 UTC 2005 x86_64 AMD Opteron(tm) Processor 248 AuthenticAMD GNU/Linux /dev/sda: Timing cached reads: 3512 MB in 2.00 seconds = 1756.27 MB/sec Timing buffered disk reads: 254 MB in 3.01 seconds = 84.51 MB/sec Timing cached reads: 3500 MB in 2.00 seconds = 1750.27 MB/sec Timing buffered disk reads: 254 MB in 3.01 seconds = 84.48 MB/sec /dev/sdb: Timing cached reads: 3336 MB in 2.00 seconds = 1666.59 MB/sec Timing buffered disk reads: 228 MB in 3.00 seconds = 75.91 MB/sec Timing cached reads: 3516 MB in 2.00 seconds = 1757.39 MB/sec Timing buffered disk reads: 228 MB in 3.00 seconds = 75.91 MB/sec /dev/sdc: Timing cached reads: 3444 MB in 2.00 seconds = 1722.26 MB/sec Timing buffered disk reads: 226 MB in 3.01 seconds = 75.14 MB/sec Timing cached reads: 3532 MB in 2.00 seconds = 1765.39 MB/sec Timing buffered disk reads: 228 MB in 3.00 seconds = 75.96 MB/sec /dev/sdd: Timing cached reads: 3388 MB in 2.00 seconds = 1691.72 MB/sec Timing buffered disk reads: 228 MB in 3.02 seconds = 75.58 MB/sec Timing cached reads: 3596 MB in 2.00 seconds = 1797.38 MB/sec Timing buffered disk reads: 228 MB in 3.02 seconds = 75.43 MB/sec /dev/sde: Timing cached reads: 3556 MB in 2.00 seconds = 1775.61 MB/sec Timing buffered disk reads: 254 MB in 3.02 seconds = 84.17 MB/sec Timing cached reads: 3496 MB in 2.00 seconds = 1746.52 MB/sec Timing buffered disk reads: 254 MB in 3.02 seconds = 84.09 MB/sec /dev/sdf: Timing cached reads: 3420 MB in 2.00 seconds = 1710.26 MB/sec Timing buffered disk reads: 254 MB in 3.00 seconds = 84.54 MB/sec Timing cached reads: 3408 MB in 2.00 seconds = 1704.26 MB/sec Timing buffered disk reads: 254 MB in 3.01 seconds = 84.45 MB/sec /dev/sdg: Timing cached reads: 3444 MB in 2.00 seconds = 1721.40 MB/sec Timing buffered disk reads: 254 MB in 3.01 seconds = 84.48 MB/sec Timing cached reads: 3612 MB in 2.00 seconds = 1804.47 MB/sec Timing buffered disk reads: 254 MB in 3.01 seconds = 84.51 MB/sec /dev/sdh: Timing cached reads: 3480 MB in 2.00 seconds = 1739.39 MB/sec Timing buffered disk reads: 204 MB in 3.02 seconds = 67.65 MB/sec Timing cached reads: 3376 MB in 2.00 seconds = 1686.57 MB/sec Timing buffered disk reads: 202 MB in 3.01 seconds = 67.10 MB/sec mdadm /dev/md1 --create --auto=yes -l 6 -n 6 -z 10000000 /dev/sd[bcdefg] Personalities : [raid6] md1 : active raid6 sdg[5] sdf[4] sde[3] sdd[2] sdc[1] sdb[0] 40000000 blocks level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU] /dev/md1: Timing cached reads: 3512 MB in 2.00 seconds = 1756.27 MB/sec Timing buffered disk reads: 292 MB in 3.02 seconds = 96.54 MB/sec Timing cached reads: 3796 MB in 2.00 seconds = 1897.34 MB/sec Timing buffered disk reads: 288 MB in 3.02 seconds = 95.44 MB/sec (this is very similar to the 98MBps or so obtained from full size of disks) mdadm /dev/md1 --create --auto=yes -l 5 -n 6 -z 10000000 /dev/sd[bcdefg] Personalities : [raid5] [raid6] md1 : active raid5 sdg[5] sdf[4] sde[3] sdd[2] sdc[1] sdb[0] 50000000 blocks level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU] /dev/md1: Timing cached reads: 3660 MB in 2.00 seconds = 1829.36 MB/sec Timing buffered disk reads: 510 MB in 3.00 seconds = 169.91 MB/sec Timing cached reads: 3496 MB in 2.00 seconds = 1747.39 MB/sec Timing buffered disk reads: 488 MB in 3.00 seconds = 162.64 MB/sec (failed sdb, added sdh[a little slower]; got 167 and 169 MBps still) mdadm --create /dev/md0 --auto=yes -l 6 -n 6 -z 10000000 /dev/sd[bcdefg] mkfs.ext3 /dev/md0 mount -t ext3 /dev/md0 /t ; chmod a+rwx /t bonnie++ -n 16 -s 4g -u 500:100 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP livecd 4G 289 97 83479 36 34501 13 1106 94 88445 17 871.8 22 Latency 117ms 4488ms 558ms 19711us 61335us 650ms Version 1.93c ------Sequential Create------ --------Random Create-------- livecd -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 1603 96 +++++ +++ +++++ +++ 1775 97 +++++ +++ 8792 92 Latency 118ms 36us 61us 78555us 11us 412us bonnie++ -n 16 -s 4g -u 500:100 -c 10 -d /t/ (actually, done after rebuilding raid6 having had raid5[below]) Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 10 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP livecd 4G 244 98 87780 42 33374 13 1114 97 87404 17 882.7 23 Latency 118ms 4070ms 663ms 17268us 62067us 635ms Version 1.93c ------Sequential Create------ --------Random Create-------- livecd -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 1528 96 +++++ +++ +++++ +++ 1494 96 +++++ +++ 8021 92 Latency 82823us 39us 66us 99351us 11us 290us mdadm --create /dev/md0 --auto=yes -l 5 -n 6 -z 10000000 /dev/sd[bcdefg] mkfs.ext3 /dev/md0 mount -t ext3 /dev/md0 /t ; chmod a+rwx /t bonnie++ -n 16 -s 4g -u 500:100 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP livecd 4G 266 98 102589 43 39674 16 1127 97 147053 28 1002 22 Latency 119ms 3973ms 275ms 34205us 56801us 409ms Version 1.93c ------Sequential Create------ --------Random Create-------- livecd -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 2380 95 +++++ +++ +++++ +++ 2528 95 +++++ +++ 8487 91 Latency 15872us 34us 56us 103ms 12us 283us bonnie++ -n 16 -s 4g -u 500:100 -c 10 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 10 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP livecd 4G 250 97 109516 52 41330 17 1174 98 147083 29 1017 24 Latency 130ms 538ms 366ms 18941us 50125us 756ms Version 1.93c ------Sequential Create------ --------Random Create-------- livecd -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 1738 97 +++++ +++ +++++ +++ 1816 96 +++++ +++ 8131 94 Latency 72295us 34us 64us 91089us 11us 329us Now installed RHEL5 (kernel 2.6.18-8.el5) on /dev/sda[123], and used md raid with the on-board adaptec scsi controller to do the same tests. mdadm --create /dev/md0 --auto=yes -l 6 -n 6 -z 10000000 /dev/sd[bcdefg] mkfs.ext3 /dev/md0 mount -t ext3 /dev/md0 /t ; chmod a+rwx /t for d in /dev/sd? /dev/md0 ; do hdparm -tT $d ; hdparm -tT $d ; done /dev/sda: Timing cached reads: 3132 MB in 2.00 seconds = 1566.21 MB/sec Timing buffered disk reads: 256 MB in 3.02 seconds = 84.78 MB/sec Timing cached reads: 2968 MB in 2.00 seconds = 1484.22 MB/sec Timing buffered disk reads: 220 MB in 3.01 seconds = 73.14 MB/sec /dev/sdb: Timing cached reads: 3248 MB in 2.00 seconds = 1624.42 MB/sec Timing buffered disk reads: 228 MB in 3.02 seconds = 75.50 MB/sec Timing cached reads: 3112 MB in 2.00 seconds = 1556.42 MB/sec Timing buffered disk reads: 228 MB in 3.00 seconds = 75.95 MB/sec /dev/sdc: Timing cached reads: 3284 MB in 2.00 seconds = 1642.12 MB/sec Timing buffered disk reads: 224 MB in 3.01 seconds = 74.42 MB/sec Timing cached reads: 2632 MB in 2.00 seconds = 1316.24 MB/sec Timing buffered disk reads: 230 MB in 3.02 seconds = 76.10 MB/sec /dev/sdd: Timing cached reads: 2992 MB in 2.00 seconds = 1496.64 MB/sec Timing buffered disk reads: 226 MB in 3.01 seconds = 75.08 MB/sec Timing cached reads: 3456 MB in 2.00 seconds = 1727.65 MB/sec Timing buffered disk reads: 228 MB in 3.01 seconds = 75.71 MB/sec /dev/sde: Timing cached reads: 3364 MB in 2.00 seconds = 1683.08 MB/sec Timing buffered disk reads: 252 MB in 3.02 seconds = 83.56 MB/sec Timing cached reads: 3076 MB in 2.00 seconds = 1538.03 MB/sec Timing buffered disk reads: 254 MB in 3.02 seconds = 84.15 MB/sec /dev/sdf: Timing cached reads: 3188 MB in 2.00 seconds = 1595.06 MB/sec Timing buffered disk reads: 254 MB in 3.02 seconds = 84.19 MB/sec Timing cached reads: 3232 MB in 2.00 seconds = 1617.20 MB/sec Timing buffered disk reads: 254 MB in 3.00 seconds = 84.65 MB/sec /dev/sdg: Timing cached reads: 2504 MB in 2.00 seconds = 1251.51 MB/sec Timing buffered disk reads: 254 MB in 3.01 seconds = 84.38 MB/sec Timing cached reads: 3148 MB in 2.00 seconds = 1574.65 MB/sec Timing buffered disk reads: 254 MB in 3.02 seconds = 84.13 MB/sec /dev/sdh: Timing cached reads: 2724 MB in 2.00 seconds = 1362.59 MB/sec Timing buffered disk reads: 204 MB in 3.00 seconds = 67.92 MB/sec Timing cached reads: 3100 MB in 2.00 seconds = 1550.07 MB/sec Timing buffered disk reads: 206 MB in 3.00 seconds = 68.58 MB/sec /dev/md0: Timing cached reads: 3352 MB in 2.00 seconds = 1675.62 MB/sec Timing buffered disk reads: 244 MB in 3.01 seconds = 81.12 MB/sec Timing cached reads: 2916 MB in 2.00 seconds = 1458.11 MB/sec Timing buffered disk reads: 240 MB in 3.02 seconds = 79.56 MB/sec bonnie++ -n 16 -s 4g -u 500:100 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost.locald 4G 204 99 89235 51 41047 30 679 97 77746 30 849.1 35 Latency 71233us 860ms 321ms 39031us 67873us 1798ms Version 1.93c ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 15673 76 +++++ +++ 24399 87 19356 82 +++++ +++ 18622 84 Latency 5981us 510us 687us 13832us 10us 5286us bonnie++ -n 16 -s 4g -u 500:100 -c 10 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 10 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost.locald 4G 215 99 93297 52 41075 30 623 98 77658 30 828.1 35 Latency 58284us 910ms 488ms 24164us 79160us 1501ms Version 1.93c ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 15239 72 +++++ +++ 20420 81 17146 83 +++++ +++ 22467 81 Latency 3783us 575us 689us 9930us 369us 378us mdadm --stop /dev/md0 mdadm /dev/md1 --create --auto=yes -l 5 -n 6 -z 10000000 /dev/sd[bcdefg] mkfs.ext3 /dev/md1 mount -t ext3 /dev/md1 /t ; chmod a+rwx /t hdparm -tT /dev/md1 /dev/md1: Timing cached reads: 3008 MB in 2.00 seconds = 1504.87 MB/sec Timing buffered disk reads: 378 MB in 3.01 seconds = 125.70 MB/sec Timing cached reads: 2936 MB in 2.00 seconds = 1467.94 MB/sec Timing buffered disk reads: 374 MB in 3.00 seconds = 124.66 MB/sec bonnie++ -n 16 -s 4g -u 500:100 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost.locald 4G 196 98 107678 59 51732 36 799 97 130636 55 894.6 37 Latency 84838us 841ms 595ms 23809us 49033us 1387ms Version 1.93c ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 14401 80 +++++ +++ 23920 82 15402 86 +++++ +++ 23847 78 Latency 7696us 490us 679us 11521us 38946us 4219us bonnie++ -n 16 -s 4g -u 500:100 -c 10 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 10 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost.locald 4G 191 97 110754 60 52446 37 729 94 129467 54 939.4 37 Latency 95648us 382ms 327ms 62412us 141ms 1465ms Version 1.93c ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 13949 76 +++++ +++ 18207 88 15536 86 +++++ +++ 20108 81 Latency 6989us 703us 671us 10915us 116us 530us Try one more (not very fast) disk in the array, raid6. This is also using the full size of the component disks. mdadm --create /dev/md0 --auto=yes -l 6 -n 7 /dev/sd[^a] [root@localhost ~]# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid6 sdh[6] sdg[5] sdf[4] sde[3] sdd[2] sdc[1] sdb[0] 716873280 blocks level 6, 64k chunk, algorithm 2 [7/7] [UUUUUUU] mkfs.ext3 /dev/md0 mount -t ext3 /dev/md0 /t ; chmod a+rwx /t hdparm -tT /dev/md0 /dev/md0: Timing cached reads: 3064 MB in 2.00 seconds = 1531.75 MB/sec Timing buffered disk reads: 296 MB in 3.01 seconds = 98.19 MB/sec bonnie++ -n 16 -s 4g -u 500:100 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost.locald 4G 271 98 100680 54 46675 35 636 96 98022 41 832.0 38 Latency 109ms 731ms 376ms 57257us 58033us 1105ms Version 1.93c ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 13960 73 +++++ +++ 18453 81 20007 77 +++++ +++ 22908 81 Latency 8956us 549us 737us 10518us 106us 743us bonnie++ -n 16 -s 4g -u 500:100 -c 10 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 10 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost.locald 4G 182 99 98632 51 46390 35 969 97 99093 41 805.7 36 Latency 62050us 1244ms 555ms 26144us 61489us 2314ms Version 1.93c ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 14183 71 +++++ +++ 18986 83 23824 74 +++++ +++ 18757 79 Latency 6041us 540us 1955us 12914us 204us 523us And again, raid5, on 7 disks. mdadm --create /dev/md0 --auto=yes -l 5 -n 7 /dev/sd[^a] [root@localhost ~]# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdh[6] sdg[5] sdf[4] sde[3] sdd[2] sdc[1] sdb[0] 860247936 blocks level 5, 64k chunk, algorithm 2 [7/7] [UUUUUUU] hdparm -tT /dev/md0 /dev/md0: Timing cached reads: 3468 MB in 2.00 seconds = 1734.08 MB/sec Timing buffered disk reads: 462 MB in 3.00 seconds = 153.88 MB/sec bonnie++ -n 16 -s 4g -u 500:100 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost.locald 4G 316 99 120625 59 55935 39 607 98 137852 59 977.4 45 Latency 29211us 389ms 165ms 24155us 72309us 1460ms Version 1.93c ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 14893 75 +++++ +++ 19266 85 16173 81 +++++ +++ 19925 79 Latency 10502us 499us 751us 5600us 106us 325us bonnie++ -n 16 -s 4g -u 500:100 -c 10 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 10 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost.locald 4G 272 97 121548 62 55338 40 1036 97 136812 57 935.7 39 Latency 119ms 540ms 567ms 16458us 43814us 1196ms Version 1.93c ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 14659 75 +++++ +++ 19552 89 16259 85 +++++ +++ 18811 84 Latency 7184us 545us 675us 7194us 107us 267us Now keep the 7 disks in raid5 under rhel4 md-raid, but vary SCSI settings for all devices on the bus. This is of interest if lower speed or disabling of fancy things (TCQ, e.g.) is thought to be of help. Change speed to U160 instead of U320: change nothing else. hdparm -tT /dev/md0 /dev/md0: Timing cached reads: 3480 MB in 2.00 seconds = 1740.52 MB/sec Timing buffered disk reads: 404 MB in 3.01 seconds = 134.17 MB/sec bonnie++ -n 16 -s 4g -u 500:100 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost.locald 4G 294 97 96475 56 50128 34 538 99 126111 46 983.8 44 Latency 124ms 550ms 312ms 19694us 389ms 986ms Version 1.93c ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 15936 76 +++++ +++ 18050 87 15435 84 +++++ +++ 18480 85 Latency 5177us 435us 687us 11596us 86us 531us bonnie++ -n 16 -s 4g -u 500:100 -c 10 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 10 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost.locald 4G 139 99 94323 57 50024 34 523 99 127897 45 959.3 44 Latency 63923us 533ms 160ms 27151us 48685us 150ms Version 1.93c ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 14068 78 +++++ +++ 18689 91 28134 77 +++++ +++ 27099 79 Latency 12209us 19832us 653us 8209us 67us 182us Still at U160: now `No' instead of `Yes' to "Disconnect"(CQ?) and "QAS". hdparm -tT /dev/md0 /dev/md0: Timing cached reads: 2960 MB in 2.00 seconds = 1481.47 MB/sec Timing buffered disk reads: 394 MB in 3.01 seconds = 130.93 MB/sec bonnie++ -n 16 -s 4g -u 500:100 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost.locald 4G 125 99 93561 57 49619 34 579 97 125771 45 944.3 37 Latency 74947us 700ms 489ms 28421us 59061us 1407ms Version 1.93c ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 11684 73 +++++ +++ 18244 80 14254 81 +++++ +++ 18131 82 Latency 13530us 519us 9730us 9016us 132us 4051us bonnie++ -n 16 -s 4g -u 500:100 -c 10 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 10 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost.locald 4G 277 99 95408 54 50200 32 737 95 124978 46 945.9 40 Latency 57617us 516ms 573ms 118ms 65644us 1296ms Version 1.93c ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 13928 75 +++++ +++ 26207 83 15682 83 +++++ +++ 18837 79 Latency 7905us 562us 4899us 20565us 9662us 9793us Still at U160: now `No' to the above Disconnect and QAS and also to "Packetized" hdparm -tT /dev/md0 /dev/md0: Timing cached reads: 2636 MB in 2.00 seconds = 1319.28 MB/sec Timing buffered disk reads: 410 MB in 3.01 seconds = 136.43 MB/sec bonnie++ -n 16 -s 4g -u 500:100 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost.locald 4G 298 99 43105 33 29301 19 784 96 124006 45 477.4 22 Latency 57407us 1468ms 467ms 20648us 104ms 2360ms bonnie++ -n 16 -s 4g -u 500:100 -c 10 -d /t/ Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 10 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP localhost.locald 4G 207 98 41601 35 28810 19 832 97 125594 48 467.2 21 Latency 63235us 1369ms 593ms 22957us 98100us 2332ms Version 1.93c ------Sequential Create------ --------Random Create-------- localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 12450 53 +++++ +++ 22495 74 13903 69 +++++ +++ 20178 75 Latency 9486us 535us 679us 51125us 73us 317us Now reinstate the LSI Megaraid 320-1 card, reinstall the same OS (RHEL5) and try the raid5 from the card. Took hours (>8) doing build of 7 in Raid5, even with 100% build-speed. Got just 8MBps during this time. sda is the 73GB Hitachi, used directly (raid0). sdb is raid5 on the other 7 146GB disks. /dev/sda: Timing cached reads: 3304 MB in 2.00 seconds = 1652.79 MB/sec Timing buffered disk reads: 252 MB in 3.02 seconds = 83.48 MB/sec Timing cached reads: 3152 MB in 2.00 seconds = 1576.29 MB/sec Timing buffered disk reads: 252 MB in 3.01 seconds = 83.58 MB/sec /dev/sdb: Timing cached reads: 2972 MB in 2.00 seconds = 1485.66 MB/sec Timing buffered disk reads: 128 MB in 3.02 seconds = 42.36 MB/sec Timing cached reads: 3192 MB in 2.00 seconds = 1597.15 MB/sec Timing buffered disk reads: 126 MB in 3.03 seconds = 41.58 MB/sec Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP rhel5 4G 285 99 51511 43 27708 16 776 98 80507 19 1070 36 Latency 56988us 346ms 176ms 79572us 32495us 134ms Version 1.93c ------Sequential Create------ --------Random Create-------- rhel5 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 15942 78 +++++ +++ 23994 85 16762 83 +++++ +++ 19873 88 Latency 9977us 638us 691us 7708us 87us 132us Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 10 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP rhel5 4G 215 99 51246 43 27441 17 782 96 79915 20 1048 37 Latency 61333us 401ms 172ms 113ms 18198us 90551us Version 1.93c ------Sequential Create------ --------Random Create-------- rhel5 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 14619 80 +++++ +++ 21393 87 15610 83 +++++ +++ 19400 88 Latency 11380us 526us 649us 10318us 31us 102us Settings (from megamgr utility) --- sda: RAID = 0 Size = 70135MB StripeSize = 64KB Write Policy = WRBACK Read Policy = ADAPTIVE Policy = DirectIO #Stripes = 1 State = OPTIMAL --- sdb: RAID = 5 Size = 840078MB StripeSize = 64KB Write Policy = WRBACK Read Policy = ADAPTIVE Cache Policy = DirectIO #Stripes = 7 State = OPTIMAL Now change (dealing with just sdb, the raid5), Cache Policy: Cache Policy = CachedIO. This was done from the utility while running; I'm assuming the policies can be changed on the fly. /dev/sdb: Timing cached reads: 2712 MB in 2.00 seconds = 1356.46 MB/sec Timing buffered disk reads: 176 MB in 3.01 seconds = 58.41 MB/sec Timing cached reads: 2792 MB in 2.00 seconds = 1397.10 MB/sec Timing buffered disk reads: 178 MB in 3.01 seconds = 59.13 MB/sec Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP rhel5 4G 237 99 51676 45 25649 14 944 99 63857 15 1043 37 Latency 65412us 454ms 228ms 29284us 331ms 113ms Version 1.93c ------Sequential Create------ --------Random Create-------- rhel5 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 16647 79 +++++ +++ 20764 88 15626 83 +++++ +++ 19153 89 Latency 6229us 566us 743us 13945us 13us 353us Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 10 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP rhel5 4G 258 99 52427 45 25454 14 587 99 62743 15 1028 39 Latency 64038us 459ms 256ms 36185us 331ms 124ms Version 1.93c ------Sequential Create------ --------Random Create-------- rhel5 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 15479 79 +++++ +++ 19334 88 19379 82 +++++ +++ 21721 87 Latency 10123us 586us 711us 8111us 72us 319us Change Read Policy to Normal (no read-ahead) /dev/sdb: Timing cached reads: 3152 MB in 2.00 seconds = 1576.15 MB/sec Timing buffered disk reads: 150 MB in 3.02 seconds = 49.74 MB/sec Timing cached reads: 2756 MB in 2.00 seconds = 1379.06 MB/sec Timing buffered disk reads: 150 MB in 3.02 seconds = 49.69 MB/sec Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP rhel5 4G 173 99 51848 46 24948 14 637 98 58957 14 1027 37 Latency 67531us 442ms 263ms 35542us 269ms 221ms Version 1.93c ------Sequential Create------ --------Random Create-------- rhel5 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 18526 75 +++++ +++ 20936 88 16646 83 +++++ +++ 24025 86 Latency 14102us 716us 661us 8816us 87us 533us Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 10 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP rhel5 4G 168 99 51762 45 24631 14 882 99 59010 14 1029 38 Latency 72614us 361ms 161ms 21789us 229ms 116ms Version 1.93c ------Sequential Create------ --------Random Create-------- rhel5 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 16041 79 +++++ +++ 20055 88 16562 83 +++++ +++ 23040 85 Latency 8625us 433us 648us 9218us 88us 279us Change Read Policy to ReadAhead (always) (original setting was "Adaptive": read-ahead if there were two succesive adjacent reads) /dev/sdb: Timing cached reads: 2632 MB in 2.00 seconds = 1316.67 MB/sec Timing buffered disk reads: 170 MB in 3.03 seconds = 56.03 MB/sec /dev/sdb: Timing cached reads: 2664 MB in 2.00 seconds = 1331.82 MB/sec Timing buffered disk reads: 174 MB in 3.03 seconds = 57.48 MB/sec Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP rhel5 4G 264 99 52007 46 25664 14 872 99 64195 15 984.4 36 Latency 58933us 486ms 335ms 24600us 269ms 110ms Version 1.93c ------Sequential Create------ --------Random Create-------- rhel5 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 15856 79 +++++ +++ 23960 85 19705 79 +++++ +++ 24475 85 Latency 11749us 499us 698us 12396us 250us 450us Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 10 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP rhel5 4G 246 99 52328 47 25332 14 826 91 64522 15 997.7 32 Latency 64938us 402ms 297ms 20280us 20534us 121ms Version 1.93c ------Sequential Create------ --------Random Create-------- rhel5 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 15710 79 +++++ +++ 19675 87 29403 72 +++++ +++ 25408 86 Latency 7922us 504us 655us 6145us 15us 545us Change Read Policy back to .... and Write Policy to WriteThrough instead of WriteBack. /dev/sdb: Timing cached reads: 2688 MB in 2.00 seconds = 1345.46 MB/sec Timing buffered disk reads: 148 MB in 3.01 seconds = 49.15 MB/sec /dev/sdb: Timing cached reads: 3188 MB in 2.00 seconds = 1594.56 MB/sec Timing buffered disk reads: 148 MB in 3.04 seconds = 48.73 MB/sec Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP rhel5 4G 228 97 10343 9 9167 5 602 98 59175 14 956.1 36 Latency 220ms 1630ms 1133ms 41008us 457ms 112ms Version 1.93c ------Sequential Create------ --------Random Create-------- rhel5 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 10508 52 +++++ +++ 12787 60 11921 61 +++++ +++ 13108 61 Latency 9140us 504us 650us 6546us 10us 340us Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 10 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP rhel5 4G 159 97 10326 9 9156 5 870 98 59393 14 989.0 32 Latency 216ms 1715ms 759ms 49196us 24911us 198ms Version 1.93c ------Sequential Create------ --------Random Create-------- rhel5 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 11331 57 +++++ +++ 13437 54 10644 58 +++++ +++ 13278 62 Latency 9847us 505us 746us 12361us 89us 355us Development: another look at the logs of the initial (Sept 2, 2006) problem shows the three Maxtor Atlas 10k5 147sca disks all not responding to resets. At least for some earlier models this seems to be known, e.g. on Dell's site. There is a firmware update for the earlier models. Assume it is those disks that are the problem: see if firmware update can be done from CD obtained from Dell; try out, if not very satisfied, simply replace those disks. Aiming now for md raid5 on 5 or 6 disks, one hot-spare. [Finally (Sept 2007), couldn't get contact with supplier, instead just replaced the damnable disks and did md raid5 on 6 disks, with the older maxtor remaining as hot-spare.]