I’ve got a couple of projects I’d like to do that require maybe 100TB of storage: some scientometrics against the sci-hub collection, as well as building a bajillion scala projects from github. I don’t really care about data redundancy.

The cheapest way I’ve been able to figure out to do this is just buy a case with 15 HDD bays, eg the Anidees AI crystal case, and just get a ridiculously beefy processor + 256gb RAM and do all the computation on a single box.

Does this sound right? All of the purchasable NAS cases all seem more expensive, but I’m out of my element.

I expect I’m going to want to figure out ZFS to make a single logical drive.

Does this seem right? Building eg a backblaze pod is outside of my budget, and my eyes glaze over whenever I try to read about NAS controllers.

If you plan on using ZFS, make sure to take into account for the overhead and parity drives. For example, I have 12x 16TB (192TB raw) drives in raidzfs2, including overhead and parity I only have ~125TiB usable.

That's my setup for nearly 1/3 PB.

I previously had a HP N54L with FreeNAS, modded with extra two extra drives to get 6x4TB and that setup was very simple to do.

Now I moved over to a custom one which is a Fractal Node 804 case, i3 8100T, C246M-WU4 mobo and 32GB ECC DDR4 RAM. It has a SF450W Platinum PSU, and it pulls less than 40W in use - could get it lower with some underclocking. This runs Debian Buster with ZFS on Linux, and it's been rock stable for over half a year now - moving the drives over from my N54L was surprisingly simple. Plex transcoding works fine as well. Definitely recommend it.

This is why I picked Synology as my nas. The Synology Hybrid Raid system let me just shove all my extra drives in the 4 bay NAS and still have parity protection.

Then I just slowly upgraded all drives to the current money/capacity optimum one or two at a time.

If I was made of money and could afford to upgrade all the drives in a pool in one go, I'd definitely go for ZFS too.

I personally prefer the QNAP options as those either re-branded or integrated from chipsets that are widely supported in Linux and BSD.

There are rackable ATX cases like the Rosewill RSV-L4500 which don't have the enterprise-y niceties but are perfectly service-able and fairly quiet for a dozen disks.

I found this setup a lot cheaper than the dedicated NAS boxes you can buy and more flexible.

There's really no need to add bays for magnetic drives anymore (if you want magnetic drives, you probably want them in a NAS)

Use a MB with ECC Ram (Asus + AMD seems to be the most cost effective option here). Also, you should max out the ram on the board, especially if you're aiming for a raid z2 array of more than 10GB (I did max my ram, but didn't go ECC).

Source the same drive from as many places as possible to try to get different mfg runs. I had really bad luck here, about 3/4 of the seagate drives I'd purchased died in under 2 years... The drive model itself had a really poor quality and many people had issues with these drives dying a short death.

Ensure your "hot spare" drives work... I'd configured for 2 hot spare drives, however, there was a bug in the version of FreeNAS I was using that caused the hot spares to not go into usage automatically.

Make sure that anything important on your NAS is backed up... if it's only on the NAS, that is not a backup. Hardware can and will fail. I tend to not consider something backed up unless it is in at least 3 copies and at least 2 different locations.

Overall, it was just a piss-poor experience, in the end I went back to Synology for my NAS usage... I spent a lot of money in hopes of having a close to "final" NAS solution, with a lot of defective hardware, a lot of failures, and in general, my "backup" NAS was more reliable. There's something to be said for buying a solution vs. DIY.

I might consider it again in a few years, but am more likely to just get a 5-8 drive product over DIY again. Drives have gotten a lot larger... at the time I went with 3TB, and 4TB were just starting to come out.

If you're going to host this at home you'll probably want/need a firewall, I hear OpenBSD is great in that area.

I was going to get some of the WD HGST HC530 14 TB self-encrypting drives until they came out to ~$600 each. (Ouch.) The regular ones are <$400, similar to the ones BackBlaze uses.

Loving it!

Now, my storage needs are fairly modest in comparison to the author. I'm running with RAID-Z2 and at 44% capacity. I have this unit in my work workstation, at an external office. It is quiet and cool.

I used to have a big house with a room I could put a bunch of computers in. I moved to a smaller house, but more importantly I was just tired of managing a business-class infrastructure at home (VLANs, multiple APs, UPSs, batteries, patch panel, servers, etc).

So I copied a backup of my storage server from an off-site box, to S3 with Glacier, copied the primary to this ZFS array on my workstation, removed junk I was just holding on to, and now my home infrastructure consists of a Cable Modem and Google WiFi mesh. Huge improvement in maintenance!

Here's a blog post I wrote about one of the previous incarnations: https://www.tummy.com/articles/ultimatestorage2008/