UWMLSC > Beowulf Systems > Nemo

Analysis of Performance of ZFS on our SunFire X4500's

For our 3 machines that will house user home directories, archive data, and certain LIGO data, we are going to use the 3 vdev config for the zpool, as this gets us the largest usable space, with an acceptable level of performance; this was decided by Patrick, Scott, and Paul.

The table below summarizes the performance testing we did with these systems. We performed a mix of local and remote tests over nfs. The speeds are in MB/s, and are listed as write/read, example 413w/427r means 413MB/s write / 427MB/s read. The complete results are linked to at the bottom of this page.

  • The local bonnie tests were run using with "bonnie++ -s 32568:64k -x 3 -d /export/parmor/local -u root" (which runs 3 tests, using 32GB files, and 64KB chunk-size)
  • The local dd testing results show the average of 3 passes of dd-ing from /dev/zero to a 100GB file using a 1MB block size, and reading from that same file to /dev/null.
  • The "56 Clients" results show the sum of 56 clients (or 57, in some cases) performing bonnie testing using "bonnie++ -s 4096:64K -x 1 -d /x4500-$i/$name -u root" (one test, with a 4GB file and 64KB chunk-size).
  • The "1 Client bonnie++" results use the same command line as used in the 56 client test.
  • The "1 Client dd" results show the speeds dd-ing to and from a 10GB file, using a 1MB block size.

Number of vdevs Size local bonnie++ local dd 56 Clients, 16 NFSd 56 Clients, 300 NFSd 1 Client bonnie++ 1 Client dd
3 17TB / 18606163968K 454.5w/526.0r 413w/427r 89.7w/149.3r 103.0w/218.0r 78.6w/113.9r 74.7w/76.6r
4 16TB / 17641193472K 447.0w/551.9r 445w/475r 92.4w/161.3r 102.4/255.0r 78.7w/85.2r 74.6w/76.0r
5 16TB / 1664321600K 438w 93.7w/187.8r        
6 15TB / 15713574912K 488w 92.6w/170.5r        

Check this page for dead links, sloppy HTML, or a bad style sheet; or strip it for printing.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.