* Further tests of speed, 2006-07-29 - 2006-07-31 * Nathaniel * * Use of different kernels with the adaptec raid * card (details in report from the earlit test) * and of the more thorough "bonnie++" tests. * * There are two points to this test as opposed to * the previous: * --- the particular kernel/system seems to have * made a very big difference in the read speeds * (hdparm), which was unexpected -- the * previous tests assumed the card/kernel raid * and the disks chosen for the array were far * more significant * --- other measures, e.g. write or file-creation * speeds, may be of interest * * The physical disks used are the same as in the * previous tests: b,c,d,e,f are 147GB 10krpm U320 * SCSI disks. Note that when the raid card is present, * the letters b,c,d refer to arrays produced by the card. ----------------------------------------------------------------------------- 1) no raid card (direct access to disks), compare kernel md raid drivers of redhat and gentoo kernels and systems ----------------------------------------------------------------------------- Using redhat system (2.6.9-34.ELsmp no updating from u3) all discs (bcdef) now have two primary 5GB partitions at the start raid5 on sd[bcdef] --> 75 MBps raid5 on sd[bcd]1 --> 110 MBps raid0 on sd[ef]1 --> 117 MBps raid5 on sd[bcdef]1 --> 73 MBps raid0 on sd[bcdef]1 --> 130 MBps Using gentoo system ( kernel 2.6.16-gentoo-r13, mdadm - v2.5.2 - 27 June 2006, hdparm v6.3) raid5 on sd[bcdef] --> 145 MBps raid5 on sd[bcd]1 --> 156 MBps raid0 on sd[ef]1 --> 180 MBps raid5 on sd[bcdef]1 --> 143 MBps raid0 on sd[bcdef]1 --> 175 MBps ----------------------------------------------------------------------------- 2) no raid card, test difference in speed of first and last parts of disks ----------------------------------------------------------------------------- Out of interest, also on gentoo system, tried a partition at the end of the disk compared to one at the beginning (at the end was about 7.1GB, beginning was 4.9GB). Big difference: consistently 85 MBps from beginning of disk, 50 MBps from end. Using raid0 on the three fastest disks' end partitions, got 147 MBps. ----------------------------------------------------------------------------- 3) no raid card, running gentoo (judging by speeds -- badly documented results), put arrays into further arrays ----------------------------------------------------------------------------- The partitions 1 and 2 are small (1? 5? GB) at start of disk. # mdadm --create /dev/md0 --auto=yes -l 5 -n 5 /dev/sd[bcdef]1 # mdadm --create /dev/md1 --auto=yes -l 5 -n 5 /dev/sd[bcdef]2 # mdadm --create /dev/md10 --auto=yes -l 0 -n 2 /dev/md[01] # hdparm -tT /dev/md0: Timing buffered disk reads: 452 MB in 3.01 seconds = 150.23 MB/sec /dev/md1: Timing buffered disk reads: 450 MB in 3.01 seconds = 149.66 MB/sec /dev/md10: Timing buffered disk reads: 126 MB in 3.05 seconds = 41.29 MB/sec ----------------------------------------------------------------------------- 4) raid card in place, compare redhat and gentoo by hdparm ----------------------------------------------------------------------------- Now have raidcard back. Physical disk 0 (still with non-RAID system on it) is not used at all. The redhat system is on a volume (sda) on the last physcial disk, 5. The gentoo system is from a livecd. Only hdparm is compared, as running bonnie++ on the livecd without libstdc++ was not a straightforward filecopy. There are three 5GB raid5 arrays, all with 64KB stripe size (which is as with linux md [if that _was_ the same meaning of the 64KB], but less than the 256KB default of the raidcard as used in the first raidcard tests). sdb raid5 on all five disks sdc raid5 on the three similar, fastest disks, write-caching sdd raid5 on the three similar, fastest disks, no write-caching rhel4 2.6.9-34.ELsmp hdparm -tT (thrice, very similar results for sdc and sdd) sdb 93 MBps first time then 112 MBps sdc 70 MBps sdd 71 MBps gentoo livecd 2.6.12-gentoo-r10 #1 SMP hdparm -tT (five times, very similar results for sdc and sdd) sdb 85 MBps first time then 100 MBps sdc 67 MBps sdd 68 MBps So, interesting: speeds are reversed as compared to software raid cases. ----------------------------------------------------------------------------- 5) comparisons with bonnie++ (read,write,create,delete) ----------------------------------------------------------------------------- Here, bonnie++ was used instead of hdparm, to test some other aspects of `speed'. The systems 'redhat' or 'gentoo' are as described in the earlier tests. In all cases, an ext3 filesystem was used. In the linux md raid cases all of the five disks were used in their entirety. In the adaptec raid case a 5GB raid array was used at the beginning of the disk. In the first column of the table, below, "seq" or "rnd" is sequential or random, respectively, r,w,cr,rm are read,write,create,delete. Numbers are MBps speeds and ms latencies (not sure about what these actually are in this case -- some strange figures occur), or files/second for create/delete. (# bonnie++ -n 16 -s 4g -u nt:users -d MOUNTPOINT) task redhat gentoo adaptec(under redhat) ---------------------------------------------------------------------------- seq_r MBps|ms 102 | 66 123 | 54 94 | 20 seq_w MBps|ms 83 | 833 87 | 257 28*| 8336(!?) (seq|rnd)_cr /s 2169 | 2268 2402 | 2498 1233 | 1975 (seq|rnd)_rm /s 18600 | 7693 23539 | 7807 23837 | 7167 * this 28 MBps without write caching on the card could be expected to be more like 90 with write caching, if the raidcard tests with 3 disks in the array can be taken as showing the same ratio of speeds with and without write caching as would happen in the 5-disk case So, there isn't so much redhat/gentoo difference here for all five disks in raid5 as was suggested by the hdparm tests. A small difference in write, create, delete, and about 20% in read. The raid card was generally similar to the redhat md performance, but particularly slow on creating files and very slow on writing if not caching. The difficulty of making the card's management software work therefore decides the matter against it! ----------------------------------------------------------------------------- 6) comparison of ext3 and reiserfs, with bonnie++ ----------------------------------------------------------------------------- Also, from bonnie++ (only with gentoo linux md raid, as redhat makes use of reiserfs hard) note the difference between ext3 and reiserfs for create/delete! reiserfs was used with the -notail option, as is usually done unless one favours small disk usage over speed. Read and write speeds were not far different (write, ext3:87, reiser:90 MBps; read ext3:123, reiser:109) but for file creation|deletion they gave: ext3: 2402 | 2498 files/s reiser: 13289 | 16681 files/s ================================================================================ ================================================================================ THE RAW RESULTS: from bonnie++, including details for some other computers too ================================================================================ == penguinnew (ext3, redhat [2.6.9-34.ELsmp], adaptec 5GB raid5 on all five 10krpm disks) Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP penguin.ets.kth. 4G 284 99 28366 20 21070 11 908 99 94225 21 618.5 25 Latency 54316us 8336ms 3249ms 22102us 20316us 688ms Version 1.93c ------Sequential Create------ --------Random Create-------- penguin.ets.kth.se -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 1233 87 +++++ +++ 23837 45 1975 89 +++++ +++ 7167 86 Latency 39286us 307us 69us 50867us 20us 728us == penguinnew (ext3, redhat linux [2.6.9-34.ELsmp] md raid5 on 5x147GB U320 10krpm SCSI) Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP penguin.ets.kth. 4G 252 98 83495 48 40526 28 714 97 102270 35 875.0 41 Latency 102ms 833ms 283ms 30997us 66051us 1857ms Version 1.93c ------Sequential Create------ --------Random Create-------- penguin.ets.kth.se -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 2169 95 +++++ +++ 18599 80 2268 95 +++++ +++ 7693 91 Latency 52299us 51us 320us 71505us 15us 416us == penguinnew (ext3, gentoo linux [2.6.16-gentoo-r13] md raid5 on 5x147GB U320 10krpm SCSI) Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP penguin 4G 290 98 87458 52 44372 27 1278 99 123697 41 709.4 18 Latency 89367us 257ms 166ms 17943us 54009us 1056ms Version 1.93c ------Sequential Create------ --------Random Create-------- penguin -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 2402 95 +++++ +++ 23539 67 2498 95 +++++ +++ 7807 89 Latency 9205us 62us 1346us 67618us 20us 869us == penguinnew (reiserfs -notail, gentoo linux md raid5 on 5x147GB U320 10krpm SCSI) Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP penguin 4G 335 99 90531 36 42913 23 1262 99 108567 44 674.5 22 Latency 46238us 273ms 75234us 16340us 46555us 88309us Version 1.93c ------Sequential Create------ --------Random Create-------- penguin -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 13289 78 +++++ +++ 15675 91 16681 74 +++++ +++ 14132 93 Latency 164us 2766us 2868us 7689us 21us 1412us == penguinnew (ext3, gentoo linux md raid0 on 5x147GB U320 10krpm SCSI) Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP penguin 4G 284 99 163894 75 74471 23 1058 99 183627 27 897.9 17 Latency 32802us 1322ms 719ms 13677us 15768us 260ms Version 1.93c ------Sequential Create------ --------Random Create-------- penguin -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 2112 97 +++++ +++ +++++ +++ 2161 98 +++++ +++ 8557 95 Latency 24398us 42us 195us 32198us 163us 317us == penguinnew (ext3, gentoo linux md raid1 on 5x147GB U320 10krpm SCSI) Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP penguin 4G 285 98 34558 19 29768 11 992 98 81877 14 1122 23 Latency 122ms 22360ms 1351ms 22684us 44477us 20872us Version 1.93c ------Sequential Create------ --------Random Create-------- penguin -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 1914 95 +++++ +++ 32364 63 2094 96 +++++ +++ 7828 90 Latency 231ms 87us 72us 89565us 50us 375us == penguinnew (ext3, redhat, adaptec 5GB raid5 on three 10krpm disks with writecache) Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP penguin.ets.kth. 4G 281 99 80812 54 34430 17 795 98 67683 15 914.6 37 Latency 55309us 1042ms 1526ms 82648us 63997us 162ms Version 1.93c ------Sequential Create------ --------Random Create-------- penguin.ets.kth.se -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 1521 99 +++++ +++ 26553 46 1737 99 +++++ +++ 9042 98 Latency 17261us 25us 54us 10416us 21us 508us == penguinnew (ext3, redhat, adaptec 5GB raid5 on three 10krpm disks, no writecache) Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP penguin.ets.kth. 4G 239 99 23199 18 17871 9 841 99 69502 16 867.1 34 Latency 69021us 16445ms 1556ms 20721us 39633us 433ms Version 1.93c ------Sequential Create------ --------Random Create-------- penguin.ets.kth.se -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 1535 96 +++++ +++ +++++ +++ 1654 97 +++++ +++ 8358 92 Latency 56145us 27us 71us 79632us 20us 453us ************ ======== other computers ======== ************* == serv[a simple PATA|SATA server with gentoo system] (ext3 on 2*40GB PATA in linux raid1) Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP serv 4G 278 99 30790 15 18436 6 972 94 37701 5 152.7 2 Latency 61117us 2950ms 3507ms 162ms 113ms 497ms Version 1.93c ------Sequential Create------ --------Random Create-------- serv -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 1914 94 +++++ +++ 28344 47 2062 97 +++++ +++ 5809 92 Latency 43278us 113us 92us 69143us 73us 571us == box[nfs client, accessing serv 2*40GB PATA linux raid1 by NFSv3 1Gbps ethernet] Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP box 4G 4 13 13188 12 8025 13 808 92 35876 11 147.9 5 Latency 2015ms 237ms 678ms 82400us 145ms 579ms Version 1.93c ------Sequential Create------ --------Random Create-------- box -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 653 7 3201 18 1349 9 637 7 3293 16 1065 8 Latency 39562us 15691us 282ms 118ms 35862us 73389us == serv[as above] (ext3 on 300GB SATA) Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP serv 4G 282 99 44449 18 22930 6 975 97 49749 6 116.5 2 Latency 50574us 2819ms 520ms 47534us 56284us 373ms Version 1.93c ------Sequential Create------ --------Random Create-------- serv -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 1995 98 +++++ +++ +++++ +++ 2065 98 +++++ +++ 5774 97 Latency 19491us 77us 108us 23869us 25us 644us == gnu (reiser on PATA 7.2krpm) Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP gnu 4G 243 99 43754 27 19637 9 1062 98 36619 9 130.7 4 Latency 104ms 1824ms 4130ms 37321us 86158us 423ms Version 1.93c ------Sequential Create------ --------Random Create-------- gnu -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 11269 48 +++++ +++ 18956 99 17749 95 +++++ +++ 18702 99 Latency 10258us 1181us 1302us 1005us 66us 684us == magsim (ext3 single 73GB U320 10krpm SCSI) Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP magsim 4G 324 99 49771 19 27276 7 1013 97 61088 6 360.2 5 Latency 40396us 2339ms 902ms 18523us 64440us 289ms Version 1.93c ------Sequential Create------ --------Random Create-------- magsim -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 2719 93 +++++ +++ +++++ +++ 2812 95 +++++ +++ 8757 91 Latency 44464us 61us 85us 59288us 38us 297us =================================================================================