My recipe for ZFS at home
After spending a lot of time Googling around, reading forums, blogs and bug reports, etc. I was able to come up with the most amount of storage in the quietest case I could find, that should be compatible with Solaris/OpenSolaris. I've found a recipe that works and I'd like to share it.
My main goal was something with as many drive bays as possible and as quiet as possible. That was it. I think I've been able to do that pretty successfully with this configuration. So much so I just finished building my second one.
Of course, you can piece this together almost 100% from Newegg (which I what I did originally) for a little bit more money, but possibly save on shipping - and of course you'd also be ordering from the best online computer retailer.
Anyway I've gone through and found what I believe to be are the cheapest places to find these parts. Together it totals $1,390.27 and gives you 16 drive bays for data in a pretty quiet configuration. The loudest components are the fans on the 5-in-3 SATA Chassis units - if you can swap those out with something else, you'll have less drives available but could make it almost silent then.
- Case: 1x $319.99 LIAN LI PC-V2110B Black Aluminum ATX Full Tower Computer Case - Retail
- Motherboard: 1x $219.09 SUPERMICRO MBD-PDSME+-O LGA 775 Intel 3010 ATX Intel Xeon/Core 2/Pentium D/Pentium EE/Pentium 4/Celeron D Server Motherboard - Retail
- CPU: 1x $178.90 Intel Core 2 Quad Q6600 Quad-Core Processor 2.4GHz, 1066FSB, LGA775, 8MB cache Retail
- CPU Cooler: 1x $17.00 ARCTIC COOLING Freezer 7 Pro 92mm CPU Cooler - Retail
- RAM: 2x $24.19 Kingston 2GB 240-Pin DDR2 SDRAM ECC Unbuffered DDR2 667 (PC2 5300) Intel Certified Server Memory Model KVR667D2E5/2GI - Retail
- Power Supply: 1x $104.99 PC Power & Cooling S75QB 750W ATX12V / EPS12V SLI Certified CrossFire Ready 80 PLUS Certified Active PFC Power Supply compatible ... - Retail
- SATA Controller: 2x $92.98 SUPERMICRO AOC-SAT2-MV8 64-bit PCI-X133MHz SATA Controller Card - Retail
- SATA Chassis: 2x $119.99 AMS DS-3151SSBK Aluminum 3.5" Black SATA SATA Backplane Module - Retail
- Mirrored Boot Drives: 2x $37.99 Seagate Barracuda 7200.11 ST3160813AS 160GB 7200 RPM SATA 3.0Gb/s 3.5" Internal Hard Drive - OEM
Then all you need is to buy the drives... I used 1.5TB Seagate drives. Their price point is excellent right now, typically running around $105 at Fry's consistently.
I set them up in two 8 disk RAIDZ2 pools, for double redundancy in both vdevs, and combined them into one large pool.
Performance seems to be good, have no complaints yet. It's been reliable as hell too. All off a single power supply!
Compatibility - I am running Solaris 10u7 on one, because I don't need any of the OpenSolaris/SXCE/etc. features, I'd rather have something a bit more "stable" - one of the reasons is mentioned below.
The other one is running snv_104 (SXCE) - I wish I would have tried Solaris 10 first on that one. Oh well. I did try upgrading it to snv_110 or thereabouts at one point and it got stuck at boot time. I am not sure if that bug has been fixed yet or not. So be aware, regression issues can break compatibility. Although it shouldn't! *cough*
Hope this helps some people out!
So how much usable space do you end up with?
If I loan you a Kill-o-watt, can you tell us how much the power bill is for this monster?
Sounds awesome!
(PS: I am hoping that ZFS is usable in FreeBSD 8.0 and I might try this rig).
Sure, I'd check it out idle and under load. Would be interesting to see. I'd probably hook that up to all kinds of gadgets. Even lamps.
Also, you might not need this recipe for FreeBSD. Especially if SATA Port Multipler support is an option. Then all you'd need is some SATA PMP capable cards and you could run one cable to a 5 drive chassis which is pretty quiet and continuously add on top of that. That's what my original goal was. But OpenSolaris *just* started adding support in for that. I think FreeBSD is ahead of them there.
if you don't need hot-swap capability, there are a number of less expensive ways of mounting groups of 3.5" drives in 5.25" bays (eg $10 4:1 http://www.svc.com/scy-hdsx4.html). Lian-Li also makes a ~$25 4:1, part# EX-34NB.
alternatively, the $320 case+2x$120 5:3backplane = $560; which is about the low end of prices for a rackmount case that includes the backplane and hot swap trays.
A used system pull Dell PERC 5/i or 6/i (a rebranded LSI product) might be sourced inexpensively (fyi, the PERC5 does not support SATA NCQ). There are RAID and NON-RAID PERC adapter cards; usually the non-raid versions sell quite inexpensively.
if someone is looking for an alternate means of getting there; the latest version of FreeNAS supports ZFS (yay!)
I didn't care about hot-swap capability. I cared about 5:3's, of which I believe this was the cheapest one I could find with a decent amount of people who have actually used it and it isn't some totally funky unit only used by a few people in Japan or something. It also had a fan, I think some don't. I thought I could swap the fans out but a quick attempt did not yield results and I've been too lazy to try again.