• Welcome to the Lightroom Queen Forums! We're a friendly bunch, so please feel free to register and join in the conversation. If you're not familiar with forums, you'll find step by step instructions on how to post your first thread under Help at the bottom of the page. You're also welcome to download our free Lightroom Quick Start eBooks and explore our other FAQ resources.
  • Stop struggling with Lightroom! There's no need to spend hours hunting for the answers to your Lightroom Classic questions. All the information you need is in Adobe Lightroom Classic - The Missing FAQ!

    To help you get started, there's a series of easy tutorials to guide you through a simple workflow. As you grow in confidence, the book switches to a conversational FAQ format, so you can quickly find answers to advanced questions. And better still, the eBooks are updated for every release, so it's always up to date.
  • Dark mode now has a single preference for the whole site! It's a simple toggle switch in the bottom right-hand corner of any page. As it uses a cookie to store your preference, you may need to dismiss the cookie banner before you can see it. Any problems, please let us know!

Catalogs Dealing with extremely large catalog file

Status
Not open for further replies.

theoldwizard1

Member
Joined
Oct 18, 2014
Messages
27
Lightroom Experience
Beginner
Lightroom Version
Operating System: Windows 7

Lightroom Version: LR CC
(Please go to Help menu > System Info to double check the exact version number)

Question or Description of Problem:First, I am not the end user, that is my best friend. I am the "computer guy" who maintains his PC. I know nothing about USING LR, but I do have somewhat of an idea on how the data is stored.

He has an extremely LARGE catalog file. It is so large that when shutting LR down and letting it do a backup, it takes in excess of 1 hour ! The catalog file and all of the pictures are stored on a 3 disk (two 2TB and one 4TB disks) RAID 5 set. (J: drive) The backup files are stored on a different disk (no RAID).

The main reason he has kept all of the pictures in on catalog is because he heavily uses the comment (correct term ?) feature so that he can cross reference the images. Overall performance of LR is acceptable, it is just doing the backup.
 
Last edited:
How large is the .lrcat file? and how many photos are in his catalog?
One hr seems long for a backup. Is the backup on an external drive? USB 2?

For reference, I have a .lrcat file that is 2.53 GB in size, and has 128,683 photos in it (as of right now).
While I have not timed a backup and optimize for a while, the full backup takes less than 10 minutes.
I am running a Win 10 system with a SSD that hold both the catalog and the backup. The SSD is backed up every day.
 
How large is the .lrcat file? and how many photos are in his catalog?

For reference, I have a .lrcat file that is 2.53 GB in size, and has 128,683 photos in it (as of right now).
Well, after hie last trip he said he added "8-10,000" new images so I am sure he has at least double (yeah, I should have got this info first before asking the question) Another thing he does that is likely different than most LR users, he spends many hours individually removing "artifacts" in his final images (the photos are used for commercial material or presentation on high definition screens). One image may have over 200 edits.

One hr seems long for a backup. Is the backup on an external drive? USB 2?
No the backup drive is internal.

While I have not timed a backup and optimize for a while, the full backup takes less than 10 minutes.
I am running a Win 10 system with a SSD that hold both the catalog and the backup. The SSD is backed up every day.
Does LR do the optimize automatically when you shut it down or do you have to check a box in the setup to make sure this gets done.

I have been thinking about moving the catalog to a SSD. Sounds like a good idea ! When you say "backup" are you referring to letting the operating system software backup the catalog or having LR do the backup at shutdown ? Are the backups then just kept on a rotating device ?
 
If you are using the O/S or 3rd party tool to back up the catalog make sure that the preview folders are not included. The only file that needs to be backed up is the *.lrcat file.
 
Lightroom's backup is different from an os backup:

It checks the integrity of the Catalogue, to avoid repeatedly backing up a corrupt one.

It compresses the backed up file.

It does not do any versioning - it's up to the use to decide on retention strategy.

Dave
 
Jimmsp ands davidedric have raised important issues that have not been addressed.
First is the size (in MB) of the catalog file. An extremely large file would suggest some corruption or orphaned records in the database that is the LR catalog file. When exiting RL the app does three thing that Lightroom calls a "backup" First it checked the database for referential integrity, second it optimizes the catalog by writing out all of the records into a new compact file. Then last, it renames the new file with the old file name. The end result is a copied file and not a true backup.
I suspect it is the first step that is taking time. The second step can be delayed IF there is not enough free space on the primary drive where working storage (/TEMP) is maintained.
If there are lots of local adjustment to individual pixels (Spot removal and Brush tool) then LR will have recorded every one including the history of these adjustments. These local adjustments can make the *.lrcat file very large.
One solution to reduce the file to the essentials is to create a new catalog from the old one. This is achieved by using the "Export as catalog" function to build a catalog containing all of the images cataloged in the original. Some things like Smart collections will not be transferred, but this is a small price to pay for a more efficient catalog file.
So, to review what I have covered here:
  • The Lightroom "Backup" is a simple file copy with some database tuning thrown in.
  • Lightroom uses working storage and needs lots of free space on the primary drive (100GB free is recommended for Windows in general).
  • The "Export as Catalog" function will create anew catalog from the old leaving the garbage behind. This should result in a much smaller *.lrcat file.
 
Before you do something as drastic as 'Export as catalog', you may want to try to delete the edit history (which you will also lose if you do decide to use 'Export as catalog'). That should also reduce the catalog size considerably.
 
I agree. I wouldn't do Export as Catalog, instead delete the edit history using Develop's menu Develop > Clear History.
 
Before you do something as drastic as 'Export as catalog', you may want to try to delete the edit history (which you will also lose if you do decide to use 'Export as catalog'). That should also reduce the catalog size considerably.
Am I correct in that deleting the edit history removes the users capability to "rewind" modification to the images ?
 
Yes, mostly. The user can still reset to the import state.
 
Am I correct in that deleting the edit history removes the users capability to "rewind" modification to the images ?

Yes, but how useful is it to do so? Short term, yes, but for images that you adjusted 6 months ago? 2 years ago? By that stage you've got quite a build-up of this log data. I had one recent case where a user had 9 years of history in a 700000 image catalogue. Clearing the history halved the 16gb catalogue size and drastically improved performance.
 
The history in Lightroom is much less useful than the history in Photoshop. That's because Lightroom is a parametric editor, not a pixel editor. For a pixel editor, it matters that you can 'undo' edits, because the pixels have changed. Changing the pixels once more, and this time with the opposite settings, isn't the same as not doing the edit in the first place.

A parametric editor stores the edits as a kind of 'to do list'. That means that there is no difference between using 'undo' for a certain edit setting to get it back to zero, and simply setting it to zero again yourself.
 
An extremely large file would suggest some corruption or orphaned records in the database that is the LR catalog file. When exiting LR the app does three thing that Lightroom calls a "backup" First it checked the database for referential integrity, second it optimizes the catalog by writing out all of the records into a new compact file. Then last, it renames the new file with the old file name. The end result is a copied file and not a true backup.
I suspect it is the first step that is taking time. The second step can be delayed IF there is not enough free space on the primary drive where working storage (/TEMP) is maintained.
I think we are closing in on the the root problem ! The .lrcat file is OVER 14GB !! As stated, this is because the user wants to have keywords available for ALL of the images he has captured for the past 7-10 years. This is easily >200,000 images. I don't think LR will associate keywords across different catalog. Also, the user wants to keep the ability to "undo" edits on each individual image (sometimes hundreds of edits on 1 image).

As stated before, the .lrcat file is on a RAID 5 set. The free space on the C: (which is a smallish SSD) where /TEMP is located is <10GB. This will be resolved ! (The plan is to purchase a new, larger (>500GB SSD to replace the current C: drive. Likely the old C: drive will become the destination for backups. Would moving just the .lrcat file from the RAID 5 set to the new SSD C: drive improve things ? Comments please !)

The LR backup file is NOT on the the RAID set or on the C: drive. It is in a separate drive. (Not sure if that helps or not.)

If I recall correctly, ZIPing the backup was a big issue (for us) in the past. Has Adobe done anything about making ZIPing the back up an option ?
 
Keywords aren't a problem, not unless there are truly extreme numbers of them. It's the hundreds of edits that eat space partly because (IIRC) each History step contains a record of all adjustments at that point. Local adjustments are even more data.

Delete the History, and the user retains the ability to change any adjustment. He just loses the ability to step back through all the adjustments - and if you think of other software, the ability to undo x years later is pretty unusual.
 
The .lrcat file is OVER 14GB ...where /TEMP is located is <10GB.
If there is insufficient space for the 14GB file in /TEMP, I am surprised that Windows did not simply give up the process. I think resolving this free space problem will speed up your Lightroom backup on exit issue. Cleaning up the *.lrcat file by deleting the edit history as others have suggested should resolve much of the overly large file size part of the problem. There may still be some orphan records in some database tables that will not go away by deleting the edit history.
Having a CPU w/ 4-6 cores is necessary for performance as is sufficient RAM (16GB is the minimum inspire of what the LR specs suggest)
 
Let your friend try the suggested option (deleting edit history) on one picture to see what it means. i'm sure he can live with it once he sees he can change the image (back or forward) as he wish.
When he has some doubts, there is always the option to go back via one of the backup catalogs
 
At some point Adobe or the underlying Windows functions they used changed some of the backup code. The system no longer just barfs and dies when low on temp space.
Instead the backup process takes forever. If you know you are going to upgrade the hardware. I would do so before you bother with the Lr tuning.
In addition, there are multiple threads on here about backup processes. There are three separate pieces to the backup. I would start with those aspects to get ready for the hardware upgrades. There are also links and information on moving Lr to a new computer. Read up on that aspect also, since replacing the C drive depending on method may mean exactly that.

Tim

Sent from my LG-TP260 using Tapatalk
 
If I recall correctly, ZIPing the backup was a big issue (for us) in the past. Has Adobe done anything about making ZIPing the back up an option ?

The catalog backup used to not be zipped automatically before LR6. I used an extension that would zip them up in the background, at configurable intervals, while continuing to do other things in LR. LR6 now forces us to wait and do nothing (in LR) for the backup to be zipped, with no option to disable that.

As others say, the edit history is not all that useful IMO after some time has passed. You look at all the edits and wonder what the heck you were attempting to do with many/most/all of them. Instead of that, my practice is to use snapshots and name them with much more useful information like date, version, what was changed, what you were attempting to do, or what still needs to be done, etc.

And also as said, each and every single history step is a complete snapshot internally (as far as I can tell)..
So think of how many local adjustments he does, say your stated 200, and multiply that by how many edit steps are in the history: one additional brush stroke, 201 recorded in the step snapshot - another brush stoke, another 202 recorded in the step snapshot, and so on. That can inflate the catalog big-time - for little to no benefit. As stated, LR is a parametric editor, and therefore things can be changed at any time regardless of what was in each history step.
 
Last edited:
If there is insufficient space for the 14GB file in /TEMP, I am surprised that Windows did not simply give up the process.
This is top on the list of things to do (user is in "nose to the grindstone" mode - major public presentation in 2 weeks !) and then out of town.

Cleaning up the *.lrcat file by deleting the edit history as others have suggested should resolve much of the overly large file size part of the problem.
I am confused. What does "edit history" mean if not the ability to individually undo each image change ?

There may still be some orphan records in some database tables that will not go away by deleting the edit history.
Is there a procedure to take care of this ?

Having a CPU w/ 4-6 cores is necessary for performance as is sufficient RAM (16GB is the minimum inspire of what the LR specs suggest)
6 cores, 16GB.
 
I am confused. What does "edit history" mean if not the ability to individually undo each image change ?
It does mean that, but in a non-destructive editor, the ability doesn't buy you much. You have to undo them in reverse order, back to where you want to make your correction, and that potentially means you'd have to re-apply all of the steps you skipped over. The point is that you can make the correction just by moving a slider and treat the history as irrelevant. The history is a record of changes in the order you made them, but in practice it isn't terribly useful.
 
I am confused. What does "edit history" mean if not the ability to individually undo each image change ?

It does mean that, but to understand why it is often not very useful at all, let's compare sharpening in Photoshop and in Lightroom. When you sharpen in Photoshop, you change the pixels. So if you don't have some way to undo the sharpening, you would have to blur the image if you want to undo the sharpening or if you think you sharpened a bit too much. Of course that won't work very well, so an undo option is very useful.

In Lightroom you set the sharpening slider to say 80. If you think that is too much, you can simply change it to 50, even if the initial sharpening was done two months ago. It also doesn't matter if you changed the saturation after you set the sharpening, so there is no reason to step back through history if you want to (partially) undo the sharpening. You can just set the sharpening slider to 50 and leave the saturation setting the way it is. That means that using history is actually more cumbersome, because stepping back through history would first reverse the saturation adjustment, that you would have to re-apply too after making the sharpening adjustment.
 
If you look in the Develop module. You will the individual history steps. Each represents a single instruction to modify the original image. Each change to the crop window results in a new instruction. This may result in 5- 10 crop instructions to get to the final crop window when all that was needed was the original image and the final crop instruction. The same is said for the basic settings as well as local adjustments such as brush strokes. Clear history results is one set set of instruction that combines all of the final settings of the adjustments parameters into one combined instruction.
I know of no other way to eliminate orphaned database records other that to Export the (valid records of the) catalog to new catalog. I do not know to what extent orphaned records exist if any, I do know that I have seen very large catalog files reduced by using the Export as Catalog function. For me it was the only way to prevent my master catalog from crashing LR with some of the earlier versions to LR6/CC2015
 
As stated before, the .lrcat file is on a RAID 5 set. The free space on the C: (which is a smallish SSD) where /TEMP is located is <10GB. This will be resolved ! (The plan is to purchase a new, larger (>500GB) SSD to replace the current C: drive. Likely the old C: drive will become the destination for backups. Would moving just the .lrcat file from the RAID 5 set to the new SSD C: drive improve things ? )
No one commented on this, so I would like to bring it up again.

"Back in my day" when I was professionally admining larger server systems (+10 years ago) spreading IO across different rotating media had a huge positive impact in performance when copying data for one place to another (LR backup is primarily a "copy" operation). But that was with old SCSI drives and on UN*X OSes.

I wonder how Windows 7 deals with IO between 2 different SSD SATA drive (zero latency) and if it will make a major improvement in performance. Comments ?
 
Don't leave out buffer size and speeds. Windows will move data to the buffer quickly until it fills, then there could be some lag waiting on the drive to respond to additional write requests.
The drive (every drive) will always be the slowest I/O component. Data transfer between SSDs might be a little faster, but if there is any bottleneck, Windows will simply wait. I'd guess that with 2 SSDs, any waiting might be transparent to the user.
 
No one commented on this, so I would like to bring it up again.

"Back in my day" when I was professionally admining larger server systems (+10 years ago) spreading IO across different rotating media had a huge positive impact in performance when copying data for one place to another (LR backup is primarily a "copy" operation). But that was with old SCSI drives and on UN*X OSes.

I wonder how Windows 7 deals with IO between 2 different SSD SATA drive (zero latency) and if it will make a major improvement in performance. Comments ?

Since I am building out new HW for myself and pretty much most of the family, I did some testing with the HW I ordered for others before I ordered mine.
A few things I found out using benchmark software on Windows 10:
  • A software mirror SSD over SATA was actually slower than single SATA SSD.
  • A software mirror with SATA HDD was much faster than a single SATA HDD (7,500 RPM drives)
  • M.2 NVMe SSD drives blow the pants of other choices.
This was all done with AMD Ryzen 3 1200 (on what is about to be my daughter's desktop).

Tim
 
Status
Not open for further replies.
Back
Top