Storage.

haroldp

@obajoba

I agree with almost every one of your points.

'Don't trust a RAID array of any kind as a sole method of DR'

I would never trust any single method as a sole method.

Raid (1 or otherwise) on an imbedded device priced for and available to consumers has been unreliable for me because:
- Cheap chip sets are used.
- As you point out, one must often have identical drives.
- The manufacturers do a poor job of supporting backward releases of hardware and firmware, disk arrays written in one device are often unusable in a newer one.

The Raid 1 that I use for my data and photo's is based on OSX disk manager arrays and has been very reliable and well supported. If a drive fails, the options are to rebuild with a new drive, or to delete the array and read the good drive individually. both options work. The disadvantage (compared to imbedded devices) is that it is slightly slower on writes since the OS and IO bus is generating two writes foe track written.

Don't waste your money using RAID1 for an OS disk.

Absolutely, If OS and Application software are on separate disk(s) from any real data, incremental backup (like time machine) is more than good enough to keep up with change, write speed really matters on the system drive since it is the default paging device.

I use an SSD as the system drive for this reason, prior to SSD I used a 3 drive Raid 0 array for write speed, understanding that a failure of any was a failure of all. A periodic clone and TM was good enough system drive recovery for me.

Another reason to separate software from data drives is that for me, computers (and OS ) come and go but my data is persistent. Therefore all of my data are on external drives, local and NAS.

"A basic rule of DR: 3 copies on 3 mediums are minimum for the longest period which you would require a feasible rebuild/recovery of the data: Working copy, onsite (immediate recovery) copy, offsite copy. "

haroldp

I mis-keyed and posted before ready.

"A basic rule of DR: 3 copies on 3 mediums are minimum for the longest period which you would require a feasible rebuild/recovery of the data: Working copy, onsite (immediate recovery) copy, offsite copy."
- Excellent advice

I have every byte of data on 5 drives.
- 2 on Raid 1
- 1 on TM / NAS
- 1 on a local incremental backup
- 1 on a rotating portable device offsite

I have not needed the offsite yet, but each of the others has been useful in different situations.
Recovery is not always from a system / hardware event but may be from ones own error.

OS and Application software is on one system drive, periodic and generational (bootable) clones, and Time Machine. User directories on the system drive have no data, only configurations.

Given the costs of this hobby or profession, and the declining cost of storage, I am happy for a belt and suspenders.

As compared to film of which I have hundreds of thousands of frames, I can actually find, in a reasonable time, every one of my digital photos.

Regards ... Harold

Msmoto

I store everything on my desktop and backup about every month. My Time MAchine has a glitch so I do not have it functioning. LR 4.3 backs up on the main drive. I use two separate hard roves to store images and back these up once a month.
So, hopefully I will not loose everything. Also, by storing on Flickr the large size images, I can access most of the new JPEGs.

obajoba

I store everything on my desktop and backup about every month. My Time MAchine has a glitch so I do not have it functioning.

@MsMoto - Might I suggest the following site? Very useful for quickly resolving issues with Time Machine.
http://pondini.org/TM/FAQ.html

@haroldp - precisely.

So, what I find is that the consumer products push all of this nonsense and make Backup and DR seem intimidating when it should really be A) in your top 3 priorities, Simple (copy a to b and a to c) and, C) should require as little intervention as possible but be tested at least every 6 months if not every quarter. The more complex, the less likely it is to be maintained and even less than that, tested.

I work for a very, very large company but am one of only 2 people that supports all of the SAN, NAS, FC switches, etc. for several of our products, in the multiple PB range of active storage and greater than 10's of TB of new data per day. We have to manage this efficiently with extremely high throughput and redundancy beyond belief. Any decent Enterprise level Storage SysAdmin will tell you something similar.

The reality is that drive corruption, especially on an *nix based filesystem, can make a catalog "forget" about a bunch of images, or it could cause your machine not to boot, or it could cause all of your nefs to get corrupted... All it takes is one unrecoverable inode. So, speculatively (and this is one of a multitude of theories) it's possible that multiple Aperture users have corrupted catalogs and Aperture simply disregards corrupted data so as to not impact what remains and functions properly. I don't know, but I've gone into LR4 before and found hundreds of images unable to be read. So, I hit the LR forums and support and was able to mostly recover from the situation, replace my dead OCZ vertex drive, and carry on.

Disk recovery is always an option with DriveGenius, Data Rescue, etc. and waaaay back in the day when I was a "Desktop Support" guy, I can remember being able to find multiple installs of an OS, multiple users buried by "reimages" of the machine, etc. The bottom line is really that, until you have overwritten every area of the disk (this includes SD and CF cards), the data is likely still recoverable. I've put HDD's in a freezer to aid in recovery, used canned air upside down to spray a drive every 30 seconds to keep it cool and running while we recover important data, you name it and I have probably at least tried it.

And in the end, if you maintain your data properly, you should not have any problems getting back to your most recent recovery point, even with brand new hardware, in < a day or two. If anyone would like, I would be happy to start a thread that walks through the decision making process for backups and DR.

Ironheart

And if all else fails, you can send a bad disk to a service that will scrape 99% of the data off for an exorbitant fee. For the last 1% you can pay even more $$ and they will read the bits with a SEM. Once you go this route you can even read data that has been overwritten several times.

For the truly paranoid you need to physically destroy the media by shredding the platters and then incinerating the shreds.

haroldp

There is software available (usually free) that will take secure erase all the way to the military spec of 35 time s overwrite with psuedo random strings. It is strongly advised not to hold ones breath while this is running.

Deleted, even 'permanently' deleted files are easy to recover as long as they are not overwritten.

A simple 1 X overwrite secure erase will certainly lock out the idle curious, and is good for anything short of real forensic recovery.

Mil spec (35 X random) overwrite is secure against any forensic attack that I know of.

There is always the Romulan disruptor beam.

Regards ... H

[Deleted User]

I use the program - SyncBackPro - I have 3, 2tb drives in my computer and a 256GB SSD with OS and programs. One Main drive where I work on. I put backups from my data on the other 2 drives.

On the SSD I import my photo's and there is also my LR catalog.
After (basic) editing I move the photo's to \Photo\ the main drive.
Every day my photo map on \ is synchronised to the 2 other drives.
My Lightroom catalog (s) are also synchronised by SyncBackPro as all my data, my documents, outlook etc.

Clouds .... brrrr. Some members of our photoclub has a disk in a NAS by another member to keep a full copy outside the house.

I choose to synchronise my photo map, always dangerous, when you delete photo's on the main, by me the next day they are deleted on the other drives too. In SyncBackPro you can choose what you want, of course. There are many backup software programs on the market. I have this one, because I want a 1 on 1 normal file backup as simple as possible. No incremental, encrypted or what more there is you can do.

My photo drive crashed three month ago. The things I did was, panic .. ,then opened the LR catalog from my other drive and went on. Adjusted the line in SyncBackPro, put another 2TB internal drive (€ 89.-) in my computer for the crashed one and that was it.

OK, I have also an old 1TB USB drive, where I copy my photo's on and some other old drives I don't use anymore, so they become still of use. (is this a right sentence in English?), Oh, I hope you know what I mean.

spraynpray

" so they are still of some use" would be correct Ton. Otherwise perfect.

TaoTeJared

PC: (Everyone with their Apple Time Machines - geez)

Import to LR - send a copy of all files to a Nas 2tb drive during import.
Work off of internal drive
Finished work gets moved to an external USB drive -
Copy of "good" files are burned to High-end archive quality DVD or Blu-ray disks.
LR Backup runs once a week - stores to external USB drive
Full back up (Internal drives/USB drives) are backed up once a month to NAS or after important/a ton of work.

I still have not found a back-up software that I actually like.

My thoughts on Raid disks: Raid 0 never seems smart to me. Raid 1 is probably the best. Raid 5 is even better but really only needed if your business is very large. Yes disks fail, and that is the point of Raid systems. Every major business in the world runs on Raid systems - not to trust them is silly.

Cloud storage - Short-term storage to share with people - that is all they are good for. Anything else is a waste of money and they never actually tell you in plain terms that if they go under, get their servers seized, loose their data center, that you will never get your files back. Guarantees mean nothing when a company is bankrupt or out of business. There are many, many examples of this that have happen in the last 5 years. There are even examples how employees have literally taken people's files, sold photos, published written works, sold passwords, etc. Fighting them in court never works. All of the companies are set up to fail, transfer all assets to another name, and continue running thus getting out of their "responsibilities." Cloud storage is in it's infancy and will be for many years to come before all of this gets shaken out. Until then, I will not use them for reliability needs.

spraynpray

Hmm, yes, but don't forget cloud storage is only a third of the backup strategy - still got PC hard drive AND portable hard drive.

sevencrossing

http://www.bbc.co.uk/news/science-environment-21145163

obajoba

@TaoTeJared - As a storage admin in an enterprise environment, I tell you: do not trust RAID arrays. Rely on them? Yes. Trust them? No. RAID is not a replacement for backing up data. I have had to recover 30TB+ Oracle DB's running on RAID6, RAID5, RAID1, RAID10, RAID50, etc. because of multi disk failures. RAID is for your working copy and is one of many solutions used to help attain "Five 9's" of availability, and is typically leveraged more for performance than for redundancy. The type of RAID is chosen carefully, by the storage engineer, based on a long list of requirements. Without knowing the application no RAID type is any better than another.

Everyone is so against cloud storage. Break out the tinfoil hats and prepare for the invasion. X_X Do you run your own SMTP/mail server for your email? If not, It's more vulnerable than cloud storage... Cloud storage of encrypted data is more than safe. It is also only a portion of the DR strategy. Just like you can lose a disk, you can lose your cloud storage, then you find a new cloud storage provider just like you would buy a new disk.

So, back to my basic rule of DR: Minimum 3 Copies on multiple Mediums, minimum of 2 locations.

PaulDi

I have read (where I don't remember) that external hard drives need to be spun every few months (how many seems to be a point of contention) or they might "seize up". I don't know if it is true but it is something to remember if you have a full drive all backed up and not changing any more (which at only 2 or 3 terabyes I'm approaching with just "current" -- last 3 years NEFs and jpgs -- and complicted PSDs). More paranoia.

PB_PM

That must be a myth PaulDi. Some of my drives are 10 years old (5-10GB each), and don't get fired up for years at time, yet they work. I don't keep anything critical on them, but it's nice to look back at some of the stuff I was doing back then.

TaoTeJared

Obajoba - I used to sit on storage/back up compliance committees so you are not the only one who knows what they are speaking too. By reading the other comments, many here have researched it more then enough to know what the ideal is. We also do know the application - user photo storage & photo storage back up. Trying to compare corporate RAID arrays Running something like Oracle DBs with extremely high use on them to amateur and professional photographers is a terrible and false equivalency except for maybe very large art houses (which are not on this site looking for advice.) Attempting people to scare people not to use them is the same as telling people not to trust Memory Cards so they should go back to film. No one has also suggested using only one type of back up either.

Raid 1 has 2 drives that are mirrored - for pure storage for the vast majority of home and very small/individual photographers that is probably the best price/TB solution for fail safe solution as one medium. The odds of both drives failing is very low. Having an off-site back-up copy of that storage, adds an additional layer that should be done, but probably is the one most not done.

Cloud storage for long term is just asking for trouble and adds a whole level of uncontrollable variables that you can't control - mainly with most being ran by start-up companies that have to survive in a new market. If they go under, your storage will be gone as well. You have (just business wise) Economic slowdown, Regulatory changes, Increasing competition, Damage to reputation/brand, Business interruption, Failure to innovate/meet customer needs, Failure to attract or retain top talent, Commodity price risk, Technology failure/system failure, Cash flow/liquidity risk. These are from Aon's Global Risk Management Survey's top ten risks. In the past year alone various government agencies from around the world have taken the physical computers to combat crime. If your data was on those, it's gone and you will get it back when they are done.
The costs are even more compelling not to touch these services and are at this point around $0.10 per GB. One TB of storage is $100. I have now seen 3TB usb drives going for $100.

Correlli

I also use TimeMachine for the regular backups. I have got another, external HD with multiple partitions. One I use for a complete backup using SuperDuper (once a week) and another I use for an Aperture backup. This disk only gets connected when I do the backup. I know that having two backups on one disk is no help against disk failure, but it is more about data file corruption. So in total I got three backups on two physical drives in one location (I store one of the disks at work when we leave for vacation though).

Regarding my old negatives and slides I started a project about two years ago when I moved all negatives and slides to new storage. Basically I was afraid that over time the old containers could degrade over time and damage the film. So I moved everything to PrintFile storage pages and also moved the slides to better frames. I was quite some work and in the process I also threw away a number of old slides that I kept back then. But as Warprints said earlier: still only one copy in one location...

proudgeek

Two redundant 2TB drives backed up weekly. If I were really serious, I'd put one in a fireproof safe and only take it out to write to it.

obajoba

@TaoTeJared - I think my point was overlooked. We could argue about corporate storage, RAID uses, and compliance all day. For home users, unfortunately, you typically have to rely on software RAID and even I have had a RAID1 set on a home machine lose a disk, thrash all of the data and had trouble recovering. So, how does the average user handle that? They spend a ton of money getting data recovered. I would say you're better off having two separate drives, with two separate copies of the original data before contemplating the use of RAID$ at home.

As for "cloud storage", what companies are people using that run this risk? Apple, Google, DropBox and Amazon are highly available, competitively priced, and are not going to "just collapse." If you use a small company that is high-risk then you are asking for any trouble you might have. And once again, the point was still overlooked. Unlike a RAID array, you are creating redundancy by choosing multiple mediums - a local hard drive, a backup copy on another drive (whether it is local or in a safe deposit box, wherever), plus a copy online "in the cloud." What are the chances that all 3 of those copies will be lost at the same time? I would bet dollars to donuts that it is a far smaller chance than having a double disk failure in a RAID1 array... Plus, Apple and Microsoft both make it very easy to encrypt your data as a consumer. So, to me, that's a very sound solution in comparison to trying to keep a physical copy offsite. Another benefit of online storage is, of course, backing up new data while travelling. I hear many people say they bring a second drive with them and dump their cards to that drive. That is great until you are robbed and lose your computer, the second drive, the cards and the camera. for my last trip to ZA, I uploaded all of my photos at night while I was sleeping and if I had been robbed I would still have the photos.

Regardless, everyone has their own ideas and solutions. It really depends on how much your data means to you as to how much work and money you put into securing it. I put a generic version of my solution up early on in the thread and showed how it is easily implemented, not all that expensive, and requires very little intervention. I happen to use Google Drive because I purchased storage with them long ago and still have the original pricing model (which I believe is much cheaper than the new pricing structure.)

Ironheart

RAID was never intended to do anything other than offer the hope of recovery from a single failed drive. Multiple failures will almost always result in data loss, and even single failures can propagate corrupt data in such a way as to make recovery difficult. There are still advantages and reasons to use RAID, but it should be part of an overall strategy rather than your only hope.

As far as cloud providers, if you are using one of the big boys, gone are the days of a government seizure of equipment having any effect on your backups. Today the data is highly replicated and geographically distributed, and only your credentials can actually re-assemble the data in a meaningful way. Do your research on which companies are in it for the long haul (hint, one of them has been around for 30+ years already) and you will be fine. Remember this is also only part of your overall strategy rather than all your eggs in one basket.

I personally prefer 200 year archive quality Blu-Ray DVD (BD-R) as the third leg of my archive stool (Cloud and RAID being the others). Remember CF and SD are good for 10-15 year max archive-wise.