Great article...lots of comments here about how you shouldn't keep that much data. I agree with that for hobbyists, but this is pretty relevant for professionals where the option to cull doesn't extend that far and there are business reasons for retaining terabytes of photos for an extended period. Additionally, next-gen cameras with 40-50MP RAWs are right around the corner.
I use a similar combination of home-built network storage (no RAID - just manual multiple backups) and glacier for offsite redundancy. Dealing with images as a business, I typically only work on a couple shoots at a time so syncing across devices is not a big concern, but long-term archiving and redundancy is.
Heh. As a photographer I'm not sure what utility this would have for someone like me. The last thing I want to do is smash my glorious high-bit-depth RAW files with LR adjustments into 8 bit low quality JPEGs. And if I did, ol' Photoshop can do that with image macros, and Lightroom can do it with an export. Storage costs are so cheap now and continue to fall, even a guy like me with 5TB+ of images isn't panicking about how to archive this stuff. A 9TB RAID5 is cheap as dirt compared to the rest of the costs of photography, a few rotating externals to back up is also cheap, and cloud backup is mostly only limited by network speeds. Glacier will archive stuff for $4 per month per terabyte (and falling), there is zero need in most circumstances to archive at such a terribly lossy quality level, at least in my industry. Maybe there are other applications out there, I dunno.
I'm not the poster, but I have local and remote backups of photographs.
When I shoot a model I will take 400-600 images in an hour, each about 25Mb in size. The shoot I had last weekend resulted in approximately 18Gb of RAW files, and output JPG files.
In total I have just under 3Tb of RAW, JPG, and other media files. (Sometimes I film shoots, or do some video-work at the same time.)
That kind of volume is not huge, but still painful to upload remotely. Its also at the cusp of the kind of data you can backup to a cheap SAN-box locally. I currently have two toy NAS devices each with 2x4Tb drives. If I want to bump my local capacity to 8Tb, or similar, it'll get quite expensive.
I'm a pro-photographer and work in IT as well. I have run into this storage problem on more occassions than I can poke a stick at. I've seen first hand accounts like the one linked, where people acquire hard drive after hard drive with no end in sight. But the problem isn't limited to just photographers, it's anyone who acquires data of any type.
However that being said, I use a combination of NAS of which I have about 40TB of pure photos (yes a lot of images). I've been dabbling with LTO to backup images, but you need to know what you are doing with tape. It's not a simple drop type opperation as a hard drive is. You can get a lot of shoeshining if you don't do things properly.
So for long term easy access, I recommend to others to look into 4-Bay NAS units (at a minimum) for future flexibility. QNAP or Synology to aid the config and use a blob storage service to store the data (if upload speed permits). I used to use Backblaze but even with basic B2 storage, I found it more expensive than Google Cloud storage (coldline).
But with anything your use case, preparedness to learn and financial situation will dictate what you end up doing.
As a post semi professional photographer, I want to go on a divergent path here for a moment.
As far as backing up photos, most of you have too many photos. To make your life easier before downloading all these photos go through them with the critical eye, and ask yourself if people would be interested in seeing this photo 50 years from now.
Second, never ever ever use any cloud based back up unless you are just using it for a back up of your multiple hard drive backups. In essence, if she only be a back up of your back ups. And never a back up of your daily workflow.
In my opinion, we are heading for a time of fiscal turmoil that will either make these cloud backups too expensive or obsolete.
Yep, I’m a photographer and I’ve got 4x8TB drives in a NAS and it takes some work to make sure it doesn’t fill up. I like to keep all my wedding raws for a year, and I keep all the raw files I haven’t culled long term.
I'm selfhosting Immich (Google photos clone) on my NAS which has a 70TB's+ of storage. Every night a backup to Glacier runs (which is pennies to store, expensive to retrieve). Every so often I hook up an external hdd and just copy the backup over.
I've had some pretty severe data loss a few years back where I had photos on a cloud service that shut down and I was too careless to properly back them up. Once I got kids I stepped it up a notch and vowed to never let that happen again (the Glacier backup is for when my house burns down or something). Still need to write some proper instructions for my family to retrieve the data if I am unable but they're basically files on a hdd.
As for curating them, I'll leave that to the next person. I think I'd have loved to comb through pics my parents took of us. I do select pictures that are super great and add them to an album. Throwing away isn't worth the effort imo.
I'd be interested in seeing a service like this but for professional photographers. A friend of a friend is a wedding photographer. The work flow for something like this is to have multiple shooters each taking ~25GB worth of photos for one shoot. Then they process the worthy photos, making 3 large files per photo (RAW, .psd, and a final .png). During the turnaround time between wedding and delivery, it would be ideal to keep a copy of each of these files somewhere, but at least keeping the RAW somewhere safe is a minimum requirement. We were trying to figure out a feasible backup solution for the volume of data that such a shoot would require. The time to upload this data is longer than the delivery time for the photos, given the current available upstream bandwidth.
We came to the conclusion that putting files on BluRay disks or hard drives and keeping them physically separate was the best solution for the cost. The problem is that disks take a long time to burn, and external hard drives aren't the best archival medium (not to mention they don't always travel well).
Has anyone here solved the problem of data backups where bandwidth limits make pure online/cloud storage infeasible?
I'm the SO of a professional newborn photographer who generated somewhere in the neighborhood of 20-30TB of RAW photos, backups, LR xmp sidecars, etc in the last year. JPEGs don't even register. High end DSLR RAWs are gigantic!
I'm the technical support for overall workflow, storage & backup engineer, etc, etc. I wear many hats.
Right now I've got a backup on import to a FreeNAS (BSD) filer running on a HP Micro Server. Lightroom backups go to the same filer and are synced as well. My local server rsyncs the delta every night to an identical machine at a friends' house.
This isn't perfect. The one place that isn't 100% backed up is current projects in post-production. That's an acceptable trade off for now.
The most important thing to note is that there are two kinds of photographers: Those who never lost data, and those who care about backups. As mentioned elsewhere in this thread, converting members of Group A into Group B is a tough sell. Video testimonials might be a good option.
A storage service for photographers MUST be invisible to their workflow. Make it as easy to use as an egg timer. At the MOST, an addition of a single screen with sensible defaults (just click next) on photo import.
Ignoring error messages that get in their way is to be expected. Care more about their backups than they do. Send me (tech support guy) an email to notify if there haven't been backups added in X days, along with weekly reports of data usage per day -- how many photos, how large, what root folders, usage trends, etc.
An ideal installation would be automatic detection of what program they're using, what places files are stored (hint: It's not just on a local drive). Detect new file locations and back those up too.
An inexpensive (<$25) option of "only backup current projects I'm working on and anything I shot in the last month", that actually works... I'd sign up in a heartbeat.
That's why I, and every other enthusiast photographer I know, keep my archive locally, meaning on my laptops hardrive and an external one as backup. You cannot trust any cloud provider to properly maintain your data, and external storage is becoming cheaper every year.
Great post. I appreciate the focus on minimalism and hobbyist photographers. I think it'd be an interesting project to take on. The technical side doesn't sound too difficult, plug into Amazon, lightroom/aperture.
While the post mentions avoiding externals I think a handy feature would be to push to both an external drive, glacier, and then remove the RAWs locally. That way you have two copies, so the data is actually backed up. Trusting all of your photography to amazon not losing data doesn't sound ideal.
I'm a similarly semi professional photographer, and I have to agree. Most photos are stored for decades just to never be looked at again. Sure it might be interesting to retrieve a picture of that random piece of furniture or that car one future day, but seriously how many photos of their dog will people retrieve from storage?
I'm taking pictures everywhere. At the supermarket, on the road, in the kitchen, it's more than words. All these photos are super ephemeral, they will literally expire within 30 minutes. I think it is actually correct to store low resolution copies only, as they are just enough to trigger my own memory. I would gain nothing from a RAW copy of that photo. I would lose nothing from deleting them from the archive, but it's a hassle.
My non-professional photo library is at around 50 GByte, excluding astrophotography projects and photogrammetry runs. It's a really handy size for backups, and it indexes quickly with whatever service. I'm using Google Photos as a dumping ground, but PhotoPrism also indexes quickly.
I'll also note that geotags and a correct clock matter much more for archiving than high dynamic range and raw file formats. "Real cameras" suck for easy archiving. I want to zoom into a map of places I've been, and GPS is way too flimsy to be reliable on my Canon gear.
Do pros really want to store their photos "in the cloud"? I'm just an amateur and have ~500GB of photos. Uploading that much is almost impossible over a normal asymmetric home internet connection, and I imagine professional photographers must have multi-terabytes of data easily.
Agreed, this can be a big win, and I did something like this while I learning photojournalism on the side, and generating a lot of images.
Combined with the tech (RAID array, backups, sharing script), it also helped to have a manual practice of culling photos.
I didn't cull as selectively as I might pick photos to cold-submit to a publication. But if I had several almost identical images from the same event in my archive, I'd try to delete all but one of them.
Reducing space requirements to 1/4 has home IT benefits: maybe don't need that NAS or bigger drives yet, backups run 4x faster, backups might fit on a single backup medium or much less expensive one, can afford that second big local drive for a little extra RAID-mirroring protection, etc.
It's also good encouragement to be a little more judicious about pressing the button on the camera that makes more culling work. :)
I only keep the RAW files for the final photos. Usually this is one or two files per “shoot.” So my cloud storage costs aren’t that high. Obviously this doesn’t work for everyone as there are lots of cases where you need to archive everything.
I’m pretty bad at storage discipline. And I admit that most of what I store is generally useless. But there’s diminishing returns on deciding what to store, vs. storing everything.
I have a little hard drive toaster that I keep a revolving 5tb spinning disc drive in. It’s my “storage” drive, and once it fills up I replace it with new ones. I don’t shoot “professionally” as in for clients, but I shoot some fairly large scale images, that result in lots and lots of data. And my working files can get up to 10gb in size. So I vigorously backup my working files, and I’m only halfassedly storing all the captures in case I need to go back and review, or dig up BTS. Lastly I have a pretty cheap and crappy SSD that I use as a sort of scratch disc. If I shoot teatherd, I’ll capture to it, or if I’m working on an image I’ll work off the SSD until the image is complete, and then move it to me of the spinning disc drives.
I currently have 3 filled 5tb drives with captures, and a drobo that I keep my working files on (tiffs, psd’s, psb’s).
If none of what I’m saying adds up, take a look at my images, and hopefully it will make it slightly more clear: http://agroism.com
1) A terabyte holds 250,000+ full-res, jpg photos. To reach that number, you'd have to more than 60 photos a day for 10 years.
2) Most phones come with free (or very cheap) cloud photo backup, so many people won't even have most of their photos on local storage. Pros and other heavy photo takers could just back their phones up to a physical drive that costs very little.
My photography archive, including about 10 years of high-volume commercial photography, is about 14 years old and at about 23TB. At present, it grows about 5GB a day. I have stopped routinely shooting RAW because storing it all has become a mess.
I’ve been shooting for a very long time. And I shoot several sessions per month, and roughly 60gb per session. Sometimes much more, sometimes less. There is no possible way, nor reason, to have all those images in the cloud.
I split up my catalogs aggressively, so I don’t really know how many images I have saved. But it’s in the hundreds of thousands.
New Lightroom might be OK for the casual photographer. But for the pro or even advanced amateur, the new Lightroom is completely useless.
I use a similar combination of home-built network storage (no RAID - just manual multiple backups) and glacier for offsite redundancy. Dealing with images as a business, I typically only work on a couple shoots at a time so syncing across devices is not a big concern, but long-term archiving and redundancy is.
reply