I am glad companies like Facebook is using Blue-Ray. Hopefully this will trigger progress in increasing density of optical discs and/or decreasing cost of high capacity discs and writers for consumer market.
Eager to archive my yearly ever increasing personal media (thanks to having a kid) onto a high capacity disc and just store it in the safety deposit box.
I don't know what the life of blu-rays are but burned optical cds and dvds have an estimated lifespan of only 10 years. Make sure you continue to keep up your backups! How I do it is I update my backups every few years but I also keep everything on functional and "hot" hard drives at all times (so I can see it's all still there).
The common Blu-Ray media (BD-R, not BD-R LTH) discs use a non-organic dye and should be more resilient than DVD-R or CD-R media. According to this source [1] 100-150 years data retention.
Edit: note also, BD-R writers immediately read back and check written data, re-writing damaged sectors to a spare area. This in-hardware defect management should make BD-R more resilient against data corruption than DVD-R or CD-R [2].
CDs and DVDs have horrible shelf life because they all use organic dyes. Some Blu-Rays do too, but you can still buy Blu-Rays with metallic dyes which should work for decades(panasonic BD-Rs are guaranteed for 50 years).
In fact, do you remember the very first, early writable CDs? They all were blue on the bottom - that's because they used a metallic dye as well, I imagine 99% of those discs will be perfectly readable today.
The main downside is that MDiscs have a max capacity is 25GB on the Blu-ray.
I have some myself, but I can't attest to the life span, since I've only had my BDR for about a year now. How does a normal person prove the lifespan claims of an MDiscs other than waiting?
I still find myself burning most backups to 50GB disks, and making redundant copies of only my most important stuff to the 25GB MDiscs.
Where I live, a 3 pack of 25GB MDiscs runs around $20 vs $70 for a spindle of 25 RiData 50GB discs. That's ~6.66/25GB MDisc vs ~2.80/50GB BDR. Not a huge premium for piece of mind, I guess.
A few months ago, Frank Frankovsky left Facebook to start a startup based on this tech! He was previously "VP of Hardware Design & Supply Chain Optimization" and has also been super involved in the Open Compute Project.
I'm pretty excited to see what they produce, and hope they can ship before someone like Amazon tries to acquire them. This level of cold storage is going to be critical as the world generates exponentially more data every year. He also mentioned this is an open source venture, which is doubly awesome. :)
I've gone through over a half dozen Seagate 2.5 drives (spindle and hybrid) and one 3.5 external in the past five years, and I'm lucky if they last 2 years. This includes drives that are mostly offline.
I've switched to WD, but I haven't had them long enough to say that they're only better. The WD Green in my Time Capsule has lasted a few years though.
All these drive failures led me to buy a BD-R burner, which has worked out fairly well, although I would say it's better to buy burning software than using the free software (on Windows at least) if you want to avoid making a >= $3 coaster.
I find the 50GB capacity to be fairly easy to work with, since I'm much less likely to have to span files/folders across discs than the 25GB ones.
I wonder how Facebook is able to implement the EU privacy directives with this technology as their long term storage solution. If a customer requests deletion of their data you're supposed to comply with that, even on back-ups.
Yet I don't think anyone will ever check that. How are you going to find one picture that should've been deleted in a cold storage system with exabytes of data (trillions(?) of photos)?
Facebook allow users to deactivate their accounts, which can then be reactivated. Facebook users can also delete their accounts. Facebook say that this is permanent:
When you delete an account, it is permanently deleted from Facebook.
You should only delete your account if you are certain you never want
to reactivate it.
Permanently deleted still doesn't mean deleted. It just means that you can't recover it. Facebook have never publicly stated what their deletion process is.
Is it logical or physical?
Since Facebook keep anonymous profiles, I wouldn't be surprised if deleted accounts just get migrated to anonymous profiles which they can still profit from. That's what I'd do if I was Facebook.
"Permanently deleted" means exactly what it says. Deactivated means that the account is inactive but the data is still there. When you delete your account the data is gone within 14 days, no options for recovery; it happens in bulk and with a bit of a grace period for 'buyers remorse' so it is somewhere in the tail end of that 14 day period. And yes, Facebook has publicly stated their deletion process on multiple occasions and has explanations of how it works in online help docs and FAQs that you just choose to ignore.
FB isn't exactly good at trust when it comes to its users. The whole business is about extracting value from its user base as an asset. Companies normally don't like throwing away assets if they can help it.
Until I see evidence to the contrary, I will assume that FB doesn't delete user data completely. That is just my preference based on the evidence of FB's previous actions. Others can believe what they will. I don't trust the online help docs or FAQs as far as I can throw them. FB has made a business out of abusing user privacy.
Zuck: Yeah so if you ever need info about anyone at Harvard
Zuck: Just ask
Zuck: I have over 4,000 emails, pictures, addresses, SNS
[Redacted Friend's Name]: What? How'd you manage that one?
Zuck: People just submitted it.
Zuck: I don't know why.
Zuck: They "trust me"
Zuck: Dumb fucks
Finally, as programmers, we all know how hard it is to remove elements such as a user from a graph. A user is key element of FB. It is challenging to remove a user, since they have a multitude of cross-related information, linked to other users and other user content. Either you delete the user logically and then hide their linked content, or you delete the user physically, but first delete all of the linked content, or you replace the user with a 'deleted user profile' and hide their linked content. Any other ways?
FB obviously hadn't done this back in 2010 [1] and maybe they have been trying hard to fix this issue, but I'd welcome an official post from FB tech team that states how they went about implementing this feature. Maybe a bit of transparency would go a long way in helping the more paranoid of us to think twice about FB.
If you encrypt each users data, in every backup, using a unique key, you need only erase the key in a central database to invalidate all possible backups.
Easiest way would be by just using another (slightly more expensive) cold storage medium for EU residents. From the article it sounds like they havent rolled this out to all data (yet)
I used to have a job as one of those robotic arms.
The datacenter were I worked had lots of shelves full of tapes, and the computer would beep and display a tape number and a drive number. You'd have to jump up, find the tape and put it in the drive.
During the day you were running about constantly trying to keep the thing happy. On the night shift you could sit a watch movies and only get up once or twice an hour.
One of the real robots dropped a tape within a swiss data center on December 24th 2010 and subsequent error states stopped all debit card transaction as a result...
I wonder how projects like Archive.org could benefit from something like this. I know it has it's up and downs, but if you have enough backups it shouldn't be too bad. I'm sure there's tons of untouched content that could easily be archived (an archive inside the archive), but other sites might as well, such as wikipedia. Of course they would need a bit of funding for this somewhat new endeavour. Maybe facebook can 'donate' the implementation to such sites.
I really like how open Facebook is about their infrastructure, hardware designs, etc.
I would have thought that they would be trying something with higher capacity, but off the shelf == lower costs.
Interesting that FB needs to handle 900 million photo uploads a day. With higher resolution cameras on smartphones, their storage requirements will keep increasing. My Note 4 takes 4K video and very high resolution photos - FB, Google, etc. can store lower res versions only, but the pressure will be there for higher resolution support, especially with 4K TVs, and other high resolution devices.
Just to help add some clarity, this isn't Blu-Ray in the common consumer sense. I've seen some comments on here about the stability of the dyes and capacity. This technology, as mentioned at the end of the article, is Optical Archive storage [1]. The media are in the form of cartridges with 12 discs in each cartridge, and Sony claims that the media is "50 year rated". Sony currently manufactures cartridges with up to 1.5TB of storage[2]. They also make a 7U, 10-cartridge library [3]. So, in a standard rack, you could fit a maximum of 90TB of Optical Archive storage. That's admittedly not a lot given the storage density of hard drives, so I'm guessing that Facebook's decision to invest in this is partly driven by energy savings or other considerations.
So how does this work with the way Facebook delivers content? I understand the advantages of disc storage for a service like Amazon Glacier as the customer expectations are set for significant time and bandwidth constraints but Facebook is supposed to deliver personal data as fast as it can be served up. They've created a "customer" expectation of instant access that can't be provided by an offline disc.
It makes for a good solution as a 2nd or 3rd level backup but you'd need some form of RAID to prevent users from experience a lengthy delay every single time a disk fails. That's a lot of money to invest in a novel solution that creates a lengthy interruption in service should it ever be needed. Maybe the power savings are worth it when weighed against the rarity of such use and expected level of revenue lost to user frustration but that's an equation made fragile by so many variables.
I could also see it used in conjunction with some amazing use-prediction algorithm to store data that will most likely never be requested. If Facebook truly keeps everything then they have millions of photos, videos, hidden posts, and other digital detritus tied up in dead, inactive, or orphaned repositories. I know half the groups I joined back in high school are now impossible to find but the data they contained is still important for the purpose of linking people and interests together. I also have several friends who have passed away, making their account a kind of digital mausoleum. Their private data will never be accessed again but it must remain in FB's datacenter until someone is given access to the account to delete it. The number of Facebook accounts tied to the deceased is only increasing so these racks of Blu Rays might be considered the first digital cemetery. I don't know how many resources are allocated to this kind of data but this is where cold storage makes the most sense.
reply