Okay, so if this is a professional gig, then off site backups are a must, and raid on site is also. But for a pro photographer I'd have expected disk costs to be a necessary expense. In Australia we have 10TB disks for AUD 450. Double up for redundancy and add on chassis, this isn't trivial $s, but equally it's presumably 'worth it'.
For my non-pro and far more modest collection of around 300GB of images, I keep copies on three local machines, and one remote (family member) with sync changes being able to be done over home grade ADSL. With your volumes you could do off site sync via usb disk easily enough for new large ingests, and propagate smaller changes over the wire. Having a friend or family in the same city is very convenient compared to trying to hunt down the best all you can eat deal du jour, with no need to handle the regular t&c changes those services suffer. Good reciprocal opportunities too, of course.
I've been looking into doing something similar for an offsite backup for my extensive photo collection- how much storage do you have and how much did it cost you?
For local redundancy, use a RAID 1 or RAID 5 setup, either in one of your home systems, or you can buy a simple NAS solution like a Drobo, and set it up for mirroring to multiple drives. The NAS unit ($350-500) + 3-4 4TB HDD's (~$120/each), or $700-1000 which will last years.
To do offsite, in case of fire, you can clone backups onto another harddrive or every few weeks or months onto blurays and store them at a relatives, safety deposit box, or somewhere else you consider safe and accessible. Or you are willing, for offsite matters, go a cloud provided like Google, Box, DropBox, which will charge you $10-20/mo for a TB of storage on their servers, you can use that.
I personally, don't use much offsite storage, for most of my data. I make occasional Blu-Ray or thumbdrive clones of important files and keep them in my bedroom, and accessible if I have to run. And while traveling I'll bring a set of thumbdrives with me.
I was a professional photographer for a couple of years (as in, paid most of my rent with it until web dev became a more lucrative venture) and assisted some VERY big and experienced photographers.
Everyone I know backed up on-site, using manual methods as and when they remembered. I did actually set up a good automated backup solution for one of them, but there's definitely a limit to their interest. Remember they're making pretty much all their money off commissions, and are only keeping backups out of some sense of pride/responsibility.
Another blow is that, at least here in the UK, we're only starting to get the kind of upstream bandwidth necessary to make this level of data transfer feasible.
So, it's a brilliant idea and, as long as confidence in cloud storage grows, as time goes on the market will grow and grow. I'm just not sure it's quite there yet.
Online/offsite backup is a different use case. They are paying $60/year so that, if their house burns down, gets flooded, disk gets fried by lightning, they still have their family pictures.
Local backup is cheap and fast, and you should do it too. But it doesn't provide geographic redundancy.
Yeah, I'd expect the market for this is more niche but could be high revenue-per customer. As a wedding photography studio we'd pay 100$/mo in a heartbeat if this was done well.
[Edit]
Also note that OP wasn't really just pitching "backup". He wants you to reduce the footprint his images take up locally. I think the messaging to casual and hobbyist users probably ought to play on this. Note Loom's taglines "more room to play". Backup is probably boring for people not under the pressure of liability like pros are.
It's for long term offsite backups/archival. If you need to retrieve lots of data from it on a regular basis (e.g. because of a single dead drive) you're doing something wrong.
If our office burns down, and I had our offsite backups in Glacier, the retrieval costs are peanuts compared to cost of losing the data (going out of business). If my home burns down, and I have my offsite backups in Glacier, a few hundred dollars to retrieve data would be nothing compared to the emotional loss of years worth of photos etc.
But I have triple copies at home - minimum - of almost all of my data (mirrored drives + regular snapshots on a third drive; and a lot of the stuff is also synced to/from one or more other computers), and similar setups at work: An offsite backup is last resort.
The economics are certainly moving, but I think it will be a long time before on-site storage is obsolete.
A reliable fast connection is still very expensive, and there are several other considerations.
When we looked at offsite backup a major consideration was restore times. Even if you can do differential backups, in a disaster you might need to get everything back ASAP and even the best internet connection can't get near the throughput of onsite disks/tapes.
Doing exactly what you are suggesting, I will tell you why it's a much bigger cost than just the cost of the drive.
- The Seagate drive you mention has horrible failure rates, IIRC.
- Unless you know how to set up and monitor a ZFS pool, don't bother. Your data will not survive without this.
- Are you confident enough in your backup solution? Are your backups offsite? Are they offsite on another ZFS pool, or similar mechanism? Do you check your backups for integrity? What is your strategy for when backups (or original data) is corrupt?
- Does your home grown solution provide a secure sync capability between all your devices? Alternatively, do you have access to the photos from all your devices?
- Does your home grown solution allow your friends and family to view/download a subset of the photos from any of their devices?
- Is your solutions online a reasonable percentage of the time?
- Is your home grown solution as fast as AWS? As in, if you are traveling and want access to your data, how fast will it download/upload?
Basically, unless you plan on spending quite a bit of time setting this up, and know what you are doing, it is much much cheaper to pay from Prime, or similar.
Edit: BTW, the box you put these drives into must have ECC RAM. Without it, expect corruption. Same goes for your other box, the backup you host offsite.
Upfront costs, with networking, rack and stacked, and wired, were far under $100/TB raw, around $40-$60, but this was quite a while ago and I don't know how it looks in the era of 10+TB drives. Also remember that once you are off S3 you are in the situation of doing your own backup, and the use case dictates the required availability when things fail... we didn't need anything online, but mirrored to a second site. With erasure coding, you can get by with 1.5x copies at each site or so, with a performance hit. So properly backed up with a full double, it's about 3x raw...
Opex will be power, data center rent, and internet access are hugely hugely variable. And of course, the personnel will be at least 1 full time person who's extremely competent.
And with services like Backblaze offering cheap offsite backup, it's a no-brainer.
My home internet connection is 10:1 upload, but that's still 38Mbps up which is more than enough to backup several TB - sure, the initial backup will take a while, but it then keeps up with incremental changes without any issues.
Well data protection is expensive, nobody said the contrary.
Backup what you value the most, ignore what you don't and apply tiers depending on what needs to be kept but you can deal with transferring it back home slowly and what you need immediately in case of a failure.
My rules of thumb are:
- always invest 3x the price of your hot live NAS storage in backups. If you can't afford buying 40TB of storage, you can't afford having 10TB of live storage. Period. Goal is to have at least one copy locally and one externally and have more space to on the backup storages to account for retentions, changes and help with migrations.
- if you can't afford 3 redundant storages(RAID), favor having 3 times non redundant storage (no RAID) over having less copies of redundant one.
Additional tip to reduce cost and avoid expensive cloud offering is to find a reliable and trustable relative or friend that can host your external copy of your backup. Nebula or Tailscale now makes it very easy without having to configure routers and stuff. In exchange you can offer that person to host his/her backup storage.
Also digitalizing material stuff is nice, but printing digital photos is also a great way to preserve copies. I'd rather save the photos I cherish the most than having 3 backup copies of 10TB of blurry or non outstanding photos. After years of having them all digitally, I am inveting back in printing photos and making albums. You can also print photobook multiple times and have some stored at a relative's place.
I had a conversation the other day with a bunch of wedding photographers and it’s amazing how much money they spend on a yearly basis on backup. They all shoot raw and they end up with something like 50-60GB of data per event. And that’s only for photographers, videographers are on a completely different scale.
Now, their number one problem isn’t storage per se, since cost per TB is constantly dropping, but uploading data to the cloud. It takes them anything from days to a full week to upload data.
So here’s my $64.000 question. Why hasn’t anyone tried to disrupt this industry? How about setting up a cloud backup company that sends you a portable disk where you write your data and then send it back. From my understanding money isn’t an issue, they would gladly pay tens of dollars on a monthly fee if someone could handle all their backup issues for them.
> If your nas gets destroyed eg by fire or water or stolen eg by burglar or even the police your data is gone.
Isn't this why offsite backups are important?
Synology has Hyper Backup, and as you mentioned, there's also rclone.
I personally think the biggest benefit to keeping photos as simple files (and maintaining my own offsite backups) is that I'm never at the mercy of one of the SaaS's microservices being down, features being nerfed, or a company being restructured or acquired.
I do see the "peace of mind" that an automatic photo hosting/backup service gives. :) Well, one could always use them all (maintaining file backups and paying for a service like Ente) for real user-side redundancy and convenience.
As a small film production we have approx. 120 TB to backup. Thats a lot of money per month, if you check AWS it is a fortune (half the value of a car) per month.
Well, my online versioned storage via Google is unlimited for $12/mo which syncs my files locally and then I have another backup provider that does the same thing via CrashPlan Small Business which is also unlimited for $10/mo. So that is $22/mo for unlimited continuous backup and versioning. I don''t see how tapes would offer much value, and storage location of those tapes especially for a home use case is likely going to be a single point of failure.
In this case, the majority of my income depends on the edit server being online. I have a full backup plus an offsite copy, but they are over 2.5 Gbps on spinning disk, or take 6+ hours to restore from offsite, so they're emergency use only.
I'm also trying to get my homelab set up to be a bit more robust / automated / reproducible, and trying out a bunch of different ideas. I'll likely settle on something else in a year's time, but the main motivation for now was to get all the storage into my rack on my 10 Gbps network.
You are ignoring the Electricity/BW costs spent by your friend which would raise your costs
I use backuplizard for personal data/photos which works out more like cost of a one 2 TB disk per year to me and to me Its easy to pay it instead of owning disks and worry about them breaking,etc
For my non-pro and far more modest collection of around 300GB of images, I keep copies on three local machines, and one remote (family member) with sync changes being able to be done over home grade ADSL. With your volumes you could do off site sync via usb disk easily enough for new large ingests, and propagate smaller changes over the wire. Having a friend or family in the same city is very convenient compared to trying to hunt down the best all you can eat deal du jour, with no need to handle the regular t&c changes those services suffer. Good reciprocal opportunities too, of course.
reply