Storage.

DJBee49

I would be interested in knowing how others cope with storing image files.

I ask because a friend of mine (who is much more computer savvy than I am), has just had about 2,000 of his images wiped from his Aperture account. I don't have fine detail to explain why but he has tried all the tricks he can to get them back, been to Aperture, help pages etc. but without success. He says also that he is not the only one and swears blind that it was not he, but Aperture who deleted them. He is not a professional photographer but is understandably rather peeved! Some are not backed up elsewhere.

My method is to make a CD/DVD of all the camera files before I do anything else, then a back up of same on a portable hard drive. I then import into LR for editing and usually store edited files back with the camera files on the portable HD, sometimes making a CD/DVD as well. I don't wipe the card before I have safe camera file copies. I have considered Cloud storage but have decided that it is all too remote and unknown for me. I don't understand where it is! I want the digital equivalent of a neg. file that I can actually lay my hands on!

My concerns for my method are that CD/DVDs can delaminate after time (never had this myself but I believe it does happen), and hard drives, portable or otherwise, can go wrong. Am I being unnecessarily paranoid? Is there a better system? All this has become more significant to me recently as my file sizes have increased somewhat with the new D800!

shawnino

Two big, fat, 5400 spinning hard drives.

Best practice would be for me to keep a backup copy offsite, and if accused of not using best practices, well, I'm guilty as charged. Everything's getting cheaper all the time so I'm coming to the point where I've no excuse.

[Deleted User]

I admit to being completely anal about storing/backing-up as I'm terrified that I'll lose my images.

So, at the end of a days shooting I backup the card/s to a portable storage device - this enables me to wipe the cards and start again the next day.

Once home I import into Lightroom and store the RAW data on my hard disk in a master file dedicated to photos.

I regularly backup the master photo file (incrementially) to a second hard disk which is often stored off-site (ie; in my office). I also backup the Lightroom Catalog file - that way I shouldn't lose my modifications... (Both of these to a large RAID configured disk setup)

I also, and this is nuts, backup the portable storage device to a second hard disk stored off-site.

Of course, with all this backing up going on, I only actually get time to take about three pictures a day... :P

DJBee49

Thanks for those answers. Not a million miles from what i am doing then.

However, when I got Lightroom 4, I downloaded straight from card into LR and got totally freaked out by the way the files seemed to be stored all over the place on my computer without me really knowing where they were! There seemed to be dozens of duplicates and the whole experience made me so nervous that I have never done it again!

Perhaps I did something dumb (likely) but I just make normal files now on the hard drive and import them to LR as I need them. It all takes much longer I suppose but at least I know where the blooming things are!

Golf007sd

I backup using Time Machine which is built in to OSX. I also have two hard drives in my laptop ...one or apps and the OS, the other for movies, songs and photo's. I backup once a month to two external drives...both are in the house. If a fire happens, the photo's are the least of my concern.

Lastly, sorry to hear about your friends loss. One of the things I don't like about Apature is that it store all the images in one single library file. If it goes...so does all your goodies.

[Deleted User]

There's another alternative of course - storing stuff in the cloud. Cloud based storage is becoming cheaper and cheaper - and most importantly it's backed-up by the supplier - usually in a 24h cycle, then once a week etc. but sometimes even more frequently. There are simple to use apps that take advantage of our sleep patterns - ie; they do all the backing-up at night while we'redoing more important things - I think this might be the way to go - but given that thus far I have about 500Gb it'll take a while to get it all out there...

JJ_SO

Aperture usually don't eats files. So, in most of the cases the problem sits in front of the screen. However, people who never care to backup files will have to learn their lesson one day, unfortunately in a hard way.

After importing, I run a backup routine of SuperDuper which makes incremental backups. This goes to my usual backup drive. Twice a month I ran another backup on a separate drive. Then, there's also the possibility to backup the originals in Aperture, that's just one click after importing. But I only do that for recent pictures not for each library.

The weak point of my backup is, the drives are all at home in more or less the same place. I just don't want to take them to the office and bring them back home, because I can't walk into the office 24/7.

I'm not making copies with CD, DVD or blurays. There's just not enough space on them and they are not reliable in terms of longtime archives. I know, verbatim has "archival grade" DVDs, but I can't check the quality after burning and would have the false feeling of security until I want to reread the media. Also, who knows how long DVD drives will survive? Anybody of you with floppy drives or the old cartridges or ZIP drives and whatever was used in the old days? Even if the drive is mechanically still working - how to get drivers for recent operating systems?

sevencrossing

I would be interested in knowing how others cope with storing image files.

!

you system seems good
three things I would add
from time to time, practice restoring to a new or friends computer
check you also have a back up copy of your LR catalog; unless you tell it to do otherwise , LR will back up to the original hard drive

finally make prints of your precious, very best work, and keep them in a fire proof safe

If you want make your photos available for future generations, ( 100s of years time ) make Black and white prints on archive paper

spraynpray

I import directly into Lightroom having selected or created the catalogue I am going to use first, and selected or created the folder the actual images are going into. At the import stage I have Lightroom pop a copy of the files onto a portable hard drive and I also (for £30 per year) have unlimited cloud storage so my off-site storage is taken care of. I don't delete the images on my card until all of that is done and confirmed so if I need to shoot more images than I could store on card and complete that routine before deleting, I would buy more cards. I carry an emergency card which I never count so it is truly for emergency use.

@DJB: It would be worth your while to learn the way that Lightroom works so you don't worry. You can either do it my way (different catalogues/different image folders) or the way some very prominent Lightroom evangelists do it i.e. use the same catalogue for everything and let Lighroom put the images where it wants. There is - despite early worries about PC speed being affected by large catalogues - apparently no difference in reality.

PB_PM

One of the things I don't like about Apature is that it store all the images in one single library file. If it goes...so does all your goodies.

The was only in Aperture 1, since version 2 you have been able to choose the library or to manage the files yourself.

warprints

This was discussed in a thread that is now archived.
Hello, my name is Warprints, and I am a paranoid. I run four computers at my house, and 22 at my office. My office computers are linked to my server, which runs a RAID 5 array. I back up daily to an NAS with a RAID 1 array that is located at the opposite side of the building. I back up weekly to a removable disc that is stored in a media fire safe. I do not trust the cloud. Stuff is always getting lost, hacked, etc. in the cloud.
At home, I download photos from my camera directly into LR4. I have LR set to automatically copy all images to an NAS with RAID 1. I also occasionally copy all images to a removable hard drive that is kept in an off site media fire safe.
I like WD Black series drives (the greens and blues tend to be too slow and crash more often). I have been managing computer systems for a few decades, and would guess that out of the about 30 machines I manage, I have one hard drive failure every year - oldest drive to fail was about 5 years old and youngest was about 1 year old.

obajoba

Since designing "Backup and DR strategies" is 50% of my job, my home structure is of course a mess.

Kidding aside, like @Golf007sd, I have dual hard drives in my MBP: a 256GB SSD for the OS, Applications, frequently accessed files (LR catalog) and a second drive (a 750GB HDD, /Volumes/datdisk) with three main directories: Photos, Data, Media. On my SDD, using symlinks, I point /Volumes/Google Drive/LR_Data -> ~/$LR_Install and /Volumes/Google Drive/Photos -> /Volumes/datdisk/Photos.

So, my workflow looks something like this:
1) Import to LR, which renames files and copies them to /Volumes/datdisk/Photos/$Year/$Month/$Day/$new_file
2) Google Drive is running in the background all the time and it sees the files pop into their new directory then syncs/uploads them to my Google Drive (it will also sync removal of files, so as I pare them down after import I regain that space)
3) LR Catalog backup once/week within LR
4) TimeMachine Full Backup on Sundays -> 2TB TimeCapsule which backs up my entire machine
5) TimeMachine Incrementals daily (use TimeMachineEditor.app to modify the defaults of hourly, it's too much)
6) Monthly Archive of TimeCapsule -> Synology NAS

I end up with all of my data in "The Cloud" (Google Drive) and 2 local copies of my recent data, including my LR catalog, plus 3 copies of all data at the one month mark. It takes a couple hours at absolute most to set up this way and it's fully automated once implemented.

If you want to go a step further, you can get your hands on Parallels or VMWare and create a virtual machine once/month and restore the entire config into the VM for testing...

obajoba

I should also mention, to address the concerns of @warprints, that I dumbed down my setup for the layman, but I use encrypted ExFat and/or TrueCrypt to encrypt all of my data that goes into Google Drive. I also use Google OTP/OTA so that my Google account is inaccessible without the password and an OTP code generated by an app on my phone and my data account is not associated with my primary GMail account, rather it is a secondary Google account used only for data and Google Drive.

DJBee49

spraynpray.
Quite right. I will indeed have to give the LR filing another go. Where IS the cloud? Can I go and see it to see if my files are safe?! Only kidding (sort of) but it makes me nervous!

warprints.
I should have checked the archive really but I probably would have wanted to ask more questions anyway so I am grateful for the responses. I am glad that you recommend the black WD drives as those are the ones I have! So far, they have worked fine as has my Lacie Rugged, although I have heard that they can be troublesome. When your hard drives 'fail' can you retrieve the data, get them running again, repair them or are they totally dead? I don't like the odds you quoted!

JJSO.
Interesting point re CDs/DVDs. The new iMac (I am a iMac user) has no optical drive on it so I think that the writing is on the wall there, although you can always use a remote optical drive instead. Additionally, I have had considerable trouble with Apple optical drives. The problem is, I quite like the physical presence of a DVD or CD! As an aside, I like my large music collection on CD for the same reason. I can browse through it and the quality is very good- but that is a different thing really.

obajoba.
I will try to absorb your advice on work flow and try it- and the recommendations from others- out.

Thanks to all!

warprints

warprints.
When your hard drives 'fail' can you retrieve the data, get them running again, repair them or are they totally dead? I don't like the odds you quoted!

Generally, the hard drives have pretty much everything backed up somewhere else (server, NAS) so I don't try to recover, just get a new hard drive and rebuild.

Re backing up on CD/DVD .... used to do that years ago, but with the volume and size of RAW files being what they are now ...

I just happened to throw out an old hard drive today that was from 1995 !! An old WD Caviar 2540. That's 540 MB (not GB). Glad I don't need to rely on those drives to store D800 RAW files!!

PB_PM

I have what I consider a good backup system with my iMac.

1. Time machine backup (everything) on external drive
2. Fully Boot-able backup (every two days) on another external
3. All Images backup to an additional media storage drive (another external)
4. Older files are backed-up on DVD
5. All my favorites are also stored (full size jpg) on my website

KnockKnock

Similar, but different:

1. Time machine backup on external drive
2. CrashPlan backup to 'the cloud'
3. I occasionally (~quarterly) update my Aperture Vault on a different external drive.

Hopefully your friend understand's there's a Trash in Aperture, and how that integrates with OSX's trash. It's actually quite hard these days to 'instantly' lose anything. Even a corrupt Aperture library should be openable (rt-click > show package contents) where you'll find a directory of masters and can copy them out.

obajoba

Hopefully your friend understand's there's a Trash in Aperture, and how that integrates with OSX's trash. It's actually quite hard these days to 'instantly' lose anything. Even a corrupt Aperture library should be openable (rt-click > show package contents) where you'll find a directory of masters and can copy them out.

Not to mention there's half a dozen reliable drive recovery tools/programs he can use provided he isn't running FileVault encryption on the disk.

sideways

Summarising from archived thread so being brief, not curt :-) ,
I have experienced Optical discs that deteriorated and became unreadable, but only in the early days of the technology. These days optical drives are too small and too slow to waste time on.
Use at least two external hard drives, fw800 or faster, usb2 is a painfully slow way to move a terabyte of data.
Don't rely on raid drives - they fail too and are a bigger pain in the ass to recover - it's often shared electronics or power supplies that go, not just one drive that's relativelyveasy to recover
Don't trust time machine on a mac an inch. It has failed repeatedly on me when i've needed to recover an entire system in anger. Make manual copies regularly or use carbon copy cloner. That does work.

I'm anal. I detest the way that windows and os-x default to putting your photo and catalogue files several levels down a directory tree on the same drive as the operating system.
I keep OS and data on separate physical drives (replaced the CD drive on my laptop with a second hard disk), keep the photos, catalogues etc on the data drive and regularly clone it for backup.
I make a clone of the clean drive with OS and apps when freshly installed, then when the OS finally crashes (and it will when you run it ever day for several years) you can just plug in the clone and boot straight into a good system while you fix the problem

All this from many many years hard experience so ignore it at your peril :-)
Good luck !

haroldp

I am an IT professional and a Database Administrator, and completely anal about recovery.

All of my files are stored in hierarchic directories under OSX and opened in NX2 or CS5 / ACR as needed for processing.

My operating disks are organized as Raid 1 arrays under the control of OSX ( which is BSD Unix under the User interface ) which means that each is stored in parallel on two separate disk drives paired to each other in near real time.

Additionally, a backup is taken by apple's 'time machine' every hour to a NAS disk drive elsewhere on my network, and another incremental backup daily to another local drive.

In the field each image is always on two media, either two flash cards or one card one disk, until uploaded at home, when I re initialize the cards.

Regards ... H

obajoba

I'm just going to throw something out there for everyone, and I am not poking/prodding/trying to argue. IME, RAID arrays fail. I manage PETAbytes of data in every RAID combination you can imagine. I have seen double-disk failures in Enterprise Class RAID5 arrays, and triple disk failures in RAID6 arrays. I've had RAID1's lose a disk and been unable to rebuild them (especially when the proprietary RAID software requires an identical disk), and RAID10's trashed by a single disk failure. Don't trust a RAID array of any kind as a sole method of DR. IMHO - Don't waste your money using RAID1 for an OS disk for personal computing. It's senseless and not cost effective in the slightest. If you'd like to discuss IOPS, throughput and number of operations required by disk , interface, or RAID type, I would be happy to join you in an appropriate thread.

I think the consensus is (and as it should be if possible):
- OS and Applications on one disk
- Libraries and photos on a separate disk (whether it be internal, FW, USB, NAS, FC, iSCSI, whatever)
- Backup the "data disk" using more than one methodology (rsync to another disk + TimeMachine to a 3rd, TM to a second disk + clone to a 3rd, etc.)
- Keep a copy offsite (cloud, safe deposit box, wherever)
- Don't believe for a second that a little used HDD or SDD is any more reliable than a CD/DVD/BluRay

If you adhere to this, you can always build a new OS drive or machine, install your application and point it to your data.

A basic rule of DR: 3 copies on 3 mediums are minimum for the longest period which you would require a feasible rebuild/recovery of the data: Working copy, onsite (immediate recovery) copy, offsite copy. In other words, if you promise a client you will have their images available for one year, you better have them for a year. If you don't have clients, then you need to decide which photos you want to keep and for how long you want to guarantee recovery.

@haroldp - since you're a DBA and I'm a Storage Admin I have to give you a hard time, it's required (but in good fun only) :P So, everyone, if you take his advice, you'll end up with triple the storage you need, at least two more copies of all of the logs and data you could possibly need, and it will all run on SSD in RAID10, with a separate RAID group for every directory.

Benji2505

My solution is very plain: I have two drives for my pictures (only), one at home and one in the office. Every Friday one of them is brought up to date.

warprints

And to think that most of us are trusting only one set of negatives/positives with no duplicates as backups. IF my house burns down, I've lost 1000s of pre-digital images. (Yeah, I am slowly scanning everything important.)

PB_PM

That is a scary thought. Almost as scary as sitting in front of my computer scanning the negatives all day.

obajoba

And to think that most of us are trusting only one set of negatives/positives with no duplicates as backups. IF my house burns down, I've lost 1000s of pre-digital images. (Yeah, I am slowly scanning everything important.)

I actually thought about this after my earlier post. I remember my uncle having a (what I called) vault but it was really a gun safe and inside he had all of the fireproof boxes that were labeled by year and that's how he kept his negatives "safe".

Safe to say, I guess, at this point if you have two copies of the original files then you're living a life of luxury in comparison to the good old days of film.

DJBee49

Grief! My head is now spinning with all this amazing information! I will have to sit down and go through it all seriously! I am slightly more paranoid than when I started but with more consideration, I am sure this will pass.

In my sillier moments I sometimes consider just going back to shooting B&W film, processing it properly, putting the negs, in my beautifully indexed neg. files carefully arranged in my office, cross-referenced in my old-fashioned paper book and card index system with hand-writing on the paper. I can find them easily, always know where they are, can hold them in my hands and know as a matter of fact that they will be there in good order for at least 150 years and probably much longer! Hmmmm.

Unless my house burns down.