[devel] Fw: Fwd: XFS vs. ext3

[devel] Fw: Fwd: XFS vs. ext3

[Alexander Bokovoy]

----- Forwarded message from Andrew Klaassen <ak@dkp.com> -----

Date: Tue, 25 Feb 2003 23:43:30 -0500
From: Andrew Klaassen <ak@dkp.com>
To: linux-list@ssc.com, ext3-users@redhat.com, linux-xfs@oss.sgi.com
Subject: XFS vs. ext3

(Sorry for cross-posting; I'm not on either ext3-users or
linux-xfs, but I thought both lists might find this interesting. 
CC me with any replies or questions.  Thanks.)

(The last four paragraphs contain the interesting bits. 
Basically, XFS hath kick-ed the *ss of ext3 under conditions
that are, for our company, critical.)

Some listees might be interested in some testing I did the other
day, XFS vs. ext3.

In our last IMAX film project, right at crunch time, we were
getting a whole bunch of dropped frames during compositing.  Our
compositing program would report "invalid file" partway through
writing, and move on to the next frame.  A year earlier we had
done a very similar project, and stressed the system in very
similar ways, but not seen the same problem at all.  Difference? 
Last year we were using XFS, this year we were using ext3. 
Otherwise, as far as I could tell, the setups were identical.

As far as I could tell; film projects differ in subtle ways, and
that can have a big impact on how hard the filesystems are
stressed.

This Sunday I decided to test my hunch.  We had to know for
sure; if frames were also dropped with XFS under test, or if the
test didn't show any dropped frames with either filesystem - in
other words, if the problem was a Murphy's Law problem, refusing
to show itself until the worst possible moment - we figured we'd
be forced to spend a couple of hundred thousand dollars on
servers and fabric for our next big IMAX project.  The time
needed to check for dropped frames and re-render frame-by-frame
when a deadline is rushing up - to babysit - is simply too
expensive.

The test setup:  4 Shake compositing stations running on Win2k,
communicating, via Samba, with a 1/2TB Linux server with an IDE
software RAID5 setup, over GigE.  Shake's job was simply to pump
through 48MB Cineon frames from local drives to the server as
fast as possible.  I ran tests continuously for about 12 hours;
I had to be able to guarantee my results.

The results were clear and dramatic.  Anywhere from 2 to 44
dropped frames out of 200 with ext3.  (The worst ext3 numbers
came while overwriting already existing files.) Zero dropped
frames with XFS.  Nadda.  None.  After the first few clear XFS
tests I put extra load on the machine while the tests were
running to see if that would make XFS hiccough - copying large
files around internally, spawning CPU-eating programs.  It
didn't.

Well... not until I threw a fork bomb at it, anyway.  <smirk>
But even then, it kept on chuggin' till the load average was
somewhere over 900.

Conclusion, clear as a bell: XFS for high-bandwidth data
transfer over Samba... when running IDE software RAID5, anyway.

Oh yeah - another interesting note:  There were also dropped
frames under ext*2*, which I tried just as a comparison case. 
XFS truly does, in the patois of the time, r0x0r...

Andrew Klaassen


----- End forwarded message -----

-- 
/ Alexander Bokovoy
---
The nation that controls magnetism controls the universe.
		-- Chester Gould/Dick Tracy