FOG Photos - Problem with weird patterns on JPEGS ( in-camera converted )

Ironheart

@Paperman, I can see the concentric circles in the sky as @msmoto describes on the JPGs you posted on dropbox. However, I couldn't see them on my phone, my tablet, or my laptop. I threw them up on my big monitor, and I could see them plain as day, so I went back and stared at my laptop screen again and they are there, but really faint. I should also say that I could only see them on the dropbox preview display, once I downloaded the JPGs to my system, using preview and photoshop the image is clean.

Now for more fun, I pulled the JPGS and NEFs into ViewNX2 and I can say for certain that you have clipped/blown highlights at every one of these exposure values. The one that is 2/3 under is super minimal, but there nonetheless.

My guess is that when dropbox, this forum (e.g. PAD), and about 1000 other web servers serve up a picture, they are doing transformations, such as resizing to fit the screen, and putting watermarks, or other types of manipulations, this is all being done in an 8-bit space, rather than what LR, PS, NX*, do which is to convert to 16-bit first and then do any transforms. This will result in posterization which will have a greater effect on clipped values. Here's my favorite link on the subject; Laura does an awesome job 'splaining what's actually going on:
http://laurashoe.com/2011/08/09/8-versus-16-bit-what-does-it-really-mean/

In any event is is supper subtle, unless some computer server does a crappy transform along the way.

As to why you aren't seeing the blown highlights in the on-camera histogram, that's a bit strange since the JPGs clearly show blown highlights. I guess I can load your JPGs onto a card and view them in-camera :-/

Paperman

Thanks Ironheart . I am relieved in a way as your response is just proof that I haven't wasted people's time here just due to a faulty screen ( on the other hand, it is not so relieving as it brings us back to the original problem ! ).

What I will do now if ( I can ) is to add the LCD screen shots of the in-camera histogram - that was my plan anyway if anyone had said they had seen the patterns. As I have been saying since my first post, the highlights did not blink & histogram did not touch the right for another 2/3 stops which is too large a margin of error if you are shooting for max DR.

Before I proceed to the next step, could you please tell me by looking at the above 3 posted JPEG photos if the histogram touches the right in any of them in your software . ( JPEGS from camera, not JPEGs you have converted from RAW ) . I see it touching in none, tiny bit at f6.3 , fully at f5.6

Ironheart

The histogram doesn't touch the right, but it touches the top. That's what I call clipping. If I drop the EV comp in the first raw, I can see that the detail is actually in the RAW, but at EV comp=0, you are clearly flat-topped on the highlights.

spraynpray

You sure you are reading the Histogram right Ironheart? My understanding of histograms is the vertical axis merely represents the quantity of light of various colours but the x axis represents the EV range hence banging the ends is over or under exposure.

Paperman

Touching top does not mean anything Ironheart as Spraynpray has indicated. Y-axis is the number of pixels at that light value. X-axis is what matters . Since most of the photo is that dull bright gray, that is where no of pixels max out ( going above a preset limit or an average estimate ). The right hand side is the clipping point.

Ironheart

But my point is that you are still over-exposed because there is data in that flat region that is being lost. If you look at your two other less exposed shots, or pull the EV down on the first one, you will see that flat top turn into the usual mountain-scape that a histogram should have.

Thanks, I understand that to clip highlights or shadows you need to fall off the left or the right of the graph, I mis-spoke when I said highlight clipping, what I should have said is you are getting tonal clipping, because your range of exposure data, the dynamic range of this photo, won't fit into an 8-bit space. One way to solve this is via tone-mapping, another would be to use HDR-like techniques. Regardless, the flat-top on the histogram is presenting itself as tonal posterization in certain circumstances.

Paperman

But my point is that you are still over-exposed because there is data in that flat region that is being lost......Regardless, the flat-top on the histogram is presenting itself as tonal posterization in certain circumstances.

Not at all, Ironheart.

The histogram is an x/y axis graph. We have discussed what the x-axis is; left side is blacks/darkest tones ( shade 0 ), right side is whites/lightest tones ( shade 255 ) .

Y-axis shows no of pixels, not light/exposure value. However, having a full scale from 0 to a maximum no of pixels is pointless as one will never have a pixel count of a shade reaching that high. One can still set a limit that high but then the figures will be so small we will be looking at a histogram very low in height revealing minimal detail in such a small LCD screen.

So what you have as maximum in y-axis in a histogram is an average figure - maximums one would normally expect to see in an image. ( take a 36 Mp camera / 256 shades . Will give 140k pixels or so per tone. So it is reasonable to set y-axis between limits 0 to maybe 500. Or maybe 0 to 1000 but it would totally ruin it to set it from 0 to 36,000 .)

When the graph touches top and gets cut, it just means there are more pixels of that shade or color than the graph shows . Does not mean at all any detail is lost .

Take NYSE stock exchange graph for example. If the scale is set from 0 to 20,000, one will never see the daily movements. If it is scaled it between 10,000 and 11,000, one can read the data easier.

spraynpray

In summary, if you showed the full number for the y axis on that small screen, you could have significant highlight or shadow clipping that you could not even see thereby rendering the histogram feature useless.

Msmoto

@Paperman

This is he way I understand it. The histogram is useful for determining if one is using the the high or low end of the sensor sensitivity, and where most of the pixels are being utilized. It is especially good for showing us when we make an error, like leaving our exposure compensation set at = or - 1.7 stops...LOL

Paperman

So I guess we are back to square one. Will simply have to accept that the histogram/highlight warning may not show overexposure ( for maybe another 2/3 stops ) though there is enough visible digital detoriation to suggest there is.

heartyfisher

I used to set all my photos to -0.7 to make sure as much as possible highlights are retained. However, nowadays I have set it in camera to -0.7 and -0.5 and -0.3 in the various metering modes .

Paperman

I easily get frustrated dealing with exposure compensation ....I find Manual to be the most practical if shooting same landscape but different positions/framing/angles. I will get the first 2-3 wrong finding the clipping point but then the exposure is good enough till I walk to another scene. ( except that my confidence has been a bit dented with the mentioned issue in this topic ) ) ) ) ) )

With auto modes & exposure compensation, the camera tends to read a different exposure with each framing or view change ( unless shooting spot ) . When you think of it, the highlights /clipping point in a landscape actually rarely changes - be it a cloud, snowy peak, a white boat, etc., no matter where you point the camera or whatever the angle/focal length is.

Of course, not talking about action/sports/wildlife or any other photography where the scene changes all the time ...

spraynpray

Just hold your breathe a while to see what the new firmware update raw histogram is like. I'm really excited about it and my Canon owning mate is rabidly jealous!

Funny how our experience varies - I find the histogram that works on the embedded jpeg actually over-estimates the over exposure. In fact I routinely shoot up to a little clipping as I know I can recover the apparently blown highlights. The trick is to know how much to push it but I have never had your problem shooting raw.

Ironheart

Guys, I'm back again, like a dog with a bone here :-)

So I've been digging around on the histograms, and have found a couple of surprises. My first observation is that the histogram displays in many programs (and likely the camera body too) are using the imbedded JPEG to produce the histogram. The only program so far that accurately displays the histogram of the RAW image is RawTherapee.

What it shows, for your first "properly exposed" image (the f/7 one) the RAW histogram is equally distributed between highlights and shadows without much midtone (two roughly equal sized peaks and a valley). What happens during the conversion to JPEG is that the highlights are all pushed into a very narrow region, and the shadows are redistributed across the shadow and midtone range. The "clipping" or "flat top" region I was describing before is accurately depicted in the RawTherapee histogram and shows a narrow highlight region that is 2-3 times taller than the shadow and mid region. Apparently the folks there put more thought into how to properly display histograms :-)

My interpretation of this, hopefully more accurate, histogram is that all of your highlight or bright tones (i.e. the foggy sky) have been compressed into a narrow region. This means 1) there are 2-3 times less tones used to represent the sky, therefore the bars are 2-3 times taller and 2) This is why the posterizaiton is occurring, due to the smaller number of actual tones used to represent this large area of the photograph.

What I haven't determined yet is if this is an artifact of how the RAW to JPEG conversion is done, or if this is a limitation of the sRGB color gamut, or something else entirely. I had a ton of fog in my backyard this morning, so I should have some of my very own RAW files to mess with as well. What is true for certain is that the RAW file is using a much wider range of tones to represent the sky, and the data is all there. It is the process of conversion that compresses the tones of this region into a very narrow range, and therefore a limited number of tones, hence the subtle, but present posterizing of the sky.

Paperman

I checked my old fog photos ; don't have many but I did not see any of the artifacts ( those contours/concentric circles ) in those under similar conditions.

So what I experienced last week is, as Ironheart has expressed, digital deterioration due to data loss during conversion /compression of RAWS. In camera JPEGS have the artifacts where as the RAWS don't. JPEGS converted from RAWS in PC is another story - different levels of degradation ( or posterization if you want to call it that ) depending on a) what software I convert with b) what program I view with :-S .

Add to that the factor that some monitors show these artifacts and some don't, I am in far far away dark place ( deep space nine 8-> )