I would be interested in knowing how others cope with storing image files.
I ask because a friend of mine (who is much more computer savvy than I am), has just had about 2,000 of his images wiped from his Aperture account. I don't have fine detail to explain why but he has tried all the tricks he can to get them back, been to Aperture, help pages etc. but without success. He says also that he is not the only one and swears blind that it was not he, but Aperture who deleted them. He is not a professional photographer but is understandably rather peeved! Some are not backed up elsewhere.
My method is to make a CD/DVD of all the camera files before I do anything else, then a back up of same on a portable hard drive. I then import into LR for editing and usually store edited files back with the camera files on the portable HD, sometimes making a CD/DVD as well. I don't wipe the card before I have safe camera file copies. I have considered Cloud storage but have decided that it is all too remote and unknown for me. I don't understand where it is! I want the digital equivalent of a neg. file that I can actually lay my hands on!
My concerns for my method are that CD/DVDs can delaminate after time (never had this myself but I believe it does happen), and hard drives, portable or otherwise, can go wrong. Am I being unnecessarily paranoid? Is there a better system? All this has become more significant to me recently as my file sizes have increased somewhat with the new D800!
Best practice would be for me to keep a backup copy offsite, and if accused of not using best practices, well, I'm guilty as charged. Everything's getting cheaper all the time so I'm coming to the point where I've no excuse.
So, at the end of a days shooting I backup the card/s to a portable storage device - this enables me to wipe the cards and start again the next day.
Once home I import into Lightroom and store the RAW data on my hard disk in a master file dedicated to photos.
I regularly backup the master photo file (incrementially) to a second hard disk which is often stored off-site (ie; in my office). I also backup the Lightroom Catalog file - that way I shouldn't lose my modifications... (Both of these to a large RAID configured disk setup)
I also, and this is nuts, backup the portable storage device to a second hard disk stored off-site.
Of course, with all this backing up going on, I only actually get time to take about three pictures a day... :P
However, when I got Lightroom 4, I downloaded straight from card into LR and got totally freaked out by the way the files seemed to be stored all over the place on my computer without me really knowing where they were! There seemed to be dozens of duplicates and the whole experience made me so nervous that I have never done it again!
Perhaps I did something dumb (likely) but I just make normal files now on the hard drive and import them to LR as I need them. It all takes much longer I suppose but at least I know where the blooming things are!
Lastly, sorry to hear about your friends loss. One of the things I don't like about Apature is that it store all the images in one single library file. If it goes...so does all your goodies.
After importing, I run a backup routine of SuperDuper which makes incremental backups. This goes to my usual backup drive. Twice a month I ran another backup on a separate drive. Then, there's also the possibility to backup the originals in Aperture, that's just one click after importing. But I only do that for recent pictures not for each library.
The weak point of my backup is, the drives are all at home in more or less the same place. I just don't want to take them to the office and bring them back home, because I can't walk into the office 24/7.
I'm not making copies with CD, DVD or blurays. There's just not enough space on them and they are not reliable in terms of longtime archives. I know, verbatim has "archival grade" DVDs, but I can't check the quality after burning and would have the false feeling of security until I want to reread the media. Also, who knows how long DVD drives will survive? Anybody of you with floppy drives or the old cartridges or ZIP drives and whatever was used in the old days? Even if the drive is mechanically still working - how to get drivers for recent operating systems?
three things I would add
from time to time, practice restoring to a new or friends computer
check you also have a back up copy of your LR catalog; unless you tell it to do otherwise , LR will back up to the original hard drive
finally make prints of your precious, very best work, and keep them in a fire proof safe
If you want make your photos available for future generations, ( 100s of years time ) make Black and white prints on archive paper
@DJB: It would be worth your while to learn the way that Lightroom works so you don't worry. You can either do it my way (different catalogues/different image folders) or the way some very prominent Lightroom evangelists do it i.e. use the same catalogue for everything and let Lighroom put the images where it wants. There is - despite early worries about PC speed being affected by large catalogues - apparently no difference in reality.
Hello, my name is Warprints, and I am a paranoid. I run four computers at my house, and 22 at my office. My office computers are linked to my server, which runs a RAID 5 array. I back up daily to an NAS with a RAID 1 array that is located at the opposite side of the building. I back up weekly to a removable disc that is stored in a media fire safe. I do not trust the cloud. Stuff is always getting lost, hacked, etc. in the cloud.
At home, I download photos from my camera directly into LR4. I have LR set to automatically copy all images to an NAS with RAID 1. I also occasionally copy all images to a removable hard drive that is kept in an off site media fire safe.
I like WD Black series drives (the greens and blues tend to be too slow and crash more often). I have been managing computer systems for a few decades, and would guess that out of the about 30 machines I manage, I have one hard drive failure every year - oldest drive to fail was about 5 years old and youngest was about 1 year old.
Kidding aside, like @Golf007sd, I have dual hard drives in my MBP: a 256GB SSD for the OS, Applications, frequently accessed files (LR catalog) and a second drive (a 750GB HDD, /Volumes/datdisk) with three main directories: Photos, Data, Media. On my SDD, using symlinks, I point /Volumes/Google Drive/LR_Data -> ~/$LR_Install and /Volumes/Google Drive/Photos -> /Volumes/datdisk/Photos.
So, my workflow looks something like this:
1) Import to LR, which renames files and copies them to /Volumes/datdisk/Photos/$Year/$Month/$Day/$new_file
2) Google Drive is running in the background all the time and it sees the files pop into their new directory then syncs/uploads them to my Google Drive (it will also sync removal of files, so as I pare them down after import I regain that space)
3) LR Catalog backup once/week within LR
4) TimeMachine Full Backup on Sundays -> 2TB TimeCapsule which backs up my entire machine
5) TimeMachine Incrementals daily (use TimeMachineEditor.app to modify the defaults of hourly, it's too much)
6) Monthly Archive of TimeCapsule -> Synology NAS
I end up with all of my data in "The Cloud" (Google Drive) and 2 local copies of my recent data, including my LR catalog, plus 3 copies of all data at the one month mark. It takes a couple hours at absolute most to set up this way and it's fully automated once implemented.
If you want to go a step further, you can get your hands on Parallels or VMWare and create a virtual machine once/month and restore the entire config into the VM for testing...
Quite right. I will indeed have to give the LR filing another go. Where IS the cloud? Can I go and see it to see if my files are safe?! Only kidding (sort of) but it makes me nervous!
I should have checked the archive really but I probably would have wanted to ask more questions anyway so I am grateful for the responses. I am glad that you recommend the black WD drives as those are the ones I have! So far, they have worked fine as has my Lacie Rugged, although I have heard that they can be troublesome. When your hard drives 'fail' can you retrieve the data, get them running again, repair them or are they totally dead? I don't like the odds you quoted!
Interesting point re CDs/DVDs. The new iMac (I am a iMac user) has no optical drive on it so I think that the writing is on the wall there, although you can always use a remote optical drive instead. Additionally, I have had considerable trouble with Apple optical drives. The problem is, I quite like the physical presence of a DVD or CD! As an aside, I like my large music collection on CD for the same reason. I can browse through it and the quality is very good- but that is a different thing really.
I will try to absorb your advice on work flow and try it- and the recommendations from others- out.
Thanks to all!
Re backing up on CD/DVD .... used to do that years ago, but with the volume and size of RAW files being what they are now ...
I just happened to throw out an old hard drive today that was from 1995 !! An old WD Caviar 2540. That's 540 MB (not GB). Glad I don't need to rely on those drives to store D800 RAW files!!
1. Time machine backup (everything) on external drive
2. Fully Boot-able backup (every two days) on another external
3. All Images backup to an additional media storage drive (another external)
4. Older files are backed-up on DVD
5. All my favorites are also stored (full size jpg) on my website
1. Time machine backup on external drive
2. CrashPlan backup to 'the cloud'
3. I occasionally (~quarterly) update my Aperture Vault on a different external drive.
Hopefully your friend understand's there's a Trash in Aperture, and how that integrates with OSX's trash. It's actually quite hard these days to 'instantly' lose anything. Even a corrupt Aperture library should be openable (rt-click > show package contents) where you'll find a directory of masters and can copy them out.
I have experienced Optical discs that deteriorated and became unreadable, but only in the early days of the technology. These days optical drives are too small and too slow to waste time on.
Use at least two external hard drives, fw800 or faster, usb2 is a painfully slow way to move a terabyte of data.
Don't rely on raid drives - they fail too and are a bigger pain in the ass to recover - it's often shared electronics or power supplies that go, not just one drive that's relativelyveasy to recover
Don't trust time machine on a mac an inch. It has failed repeatedly on me when i've needed to recover an entire system in anger. Make manual copies regularly or use carbon copy cloner. That does work.
I'm anal. I detest the way that windows and os-x default to putting your photo and catalogue files several levels down a directory tree on the same drive as the operating system.
I keep OS and data on separate physical drives (replaced the CD drive on my laptop with a second hard disk), keep the photos, catalogues etc on the data drive and regularly clone it for backup.
I make a clone of the clean drive with OS and apps when freshly installed, then when the OS finally crashes (and it will when you run it ever day for several years) you can just plug in the clone and boot straight into a good system while you fix the problem
All this from many many years hard experience so ignore it at your peril :-)
Good luck !
All of my files are stored in hierarchic directories under OSX and opened in NX2 or CS5 / ACR as needed for processing.
My operating disks are organized as Raid 1 arrays under the control of OSX ( which is BSD Unix under the User interface ) which means that each is stored in parallel on two separate disk drives paired to each other in near real time.
Additionally, a backup is taken by apple's 'time machine' every hour to a NAS disk drive elsewhere on my network, and another incremental backup daily to another local drive.
In the field each image is always on two media, either two flash cards or one card one disk, until uploaded at home, when I re initialize the cards.
Regards ... H
Nikon N90s, F100, F, lots of Leica M digital and film stuff.
I think the consensus is (and as it should be if possible):
- OS and Applications on one disk
- Libraries and photos on a separate disk (whether it be internal, FW, USB, NAS, FC, iSCSI, whatever)
- Backup the "data disk" using more than one methodology (rsync to another disk + TimeMachine to a 3rd, TM to a second disk + clone to a 3rd, etc.)
- Keep a copy offsite (cloud, safe deposit box, wherever)
- Don't believe for a second that a little used HDD or SDD is any more reliable than a CD/DVD/BluRay
If you adhere to this, you can always build a new OS drive or machine, install your application and point it to your data.
A basic rule of DR: 3 copies on 3 mediums are minimum for the longest period which you would require a feasible rebuild/recovery of the data: Working copy, onsite (immediate recovery) copy, offsite copy. In other words, if you promise a client you will have their images available for one year, you better have them for a year. If you don't have clients, then you need to decide which photos you want to keep and for how long you want to guarantee recovery.
@haroldp - since you're a DBA and I'm a Storage Admin I have to give you a hard time, it's required (but in good fun only) :P So, everyone, if you take his advice, you'll end up with triple the storage you need, at least two more copies of all of the logs and data you could possibly need, and it will all run on SSD in RAID10, with a separate RAID group for every directory.
Safe to say, I guess, at this point if you have two copies of the original files then you're living a life of luxury in comparison to the good old days of film.
In my sillier moments I sometimes consider just going back to shooting B&W film, processing it properly, putting the negs, in my beautifully indexed neg. files carefully arranged in my office, cross-referenced in my old-fashioned paper book and card index system with hand-writing on the paper. I can find them easily, always know where they are, can hold them in my hands and know as a matter of fact that they will be there in good order for at least 150 years and probably much longer! Hmmmm.
Unless my house burns down.