Tech discussion: Data throughput on today's high MPix cameras vs. SD UHS-ii & CFast memory cards.

Badger

Hi all,
I am ´new´ to the forum, although I have been an avid reader / follower (/lurker?) of NR & NRF for about a year now. I have been out of photography for quite a few years now, relying on my camera phone and a Point&Shot for snapshots, ever since I put my old (analog SLR) camera on the shelf… and ´forgot´ about the joy of taking proper photographs soon after stopping using my old trusty SLR camera.
I’m looking to get back into proper photography again as a hobby, as soon as I’m able to foot the bill for new gear without receiving an angry phone-call from my bank manager.

Anyhow, with the ´introduction´ and formalities out of the way; let’s get down to the real intent of this post (my first on NRF).

Start reading here for the tech stuff:
I’ve been following several discussions related to Nikon, and other manufacturers, camera frame-rate( fps), buffer size (or lack of appropriate buffer size, as it may be), and also discussions about different storage media (SD card, CF card, XQD card, etc. in all their variations & iterations).
To my mind; the number and speed of continuous shots is dependent on the system speed (throughput from camera sensor to storage media), and I see little-or-no discussions about the new standards that are available – like CFast and SDHC/SDXC UHS-ii (ver.4.00/4.10).

Now, in my (admittedly) limited knowledge & understanding of the data transfer flow from point A (sensor) to point B (storage medium) it appears that bottleneck is the system write speed of the storage cards in use today; where storage cards capable of higher sustained write speeds would solve (to some extent) the limitations in frames per second and number of frames in a continuous burst of pictures.
Looking at the specs of readily available storage technology; like SD UHS-ii which has a theoretical maximum throughput of 156/312MB pr. second (single/dual bus) and CFast which has a theoretical throughput of up to 600MB pr. second, I can’t help but wonder why camera manufacturers, especially those that have products with very high megapixel sensors like Nikon d800 & Sony A7R, haven’t embraced the latest technology and built this in to their products?

I would be interested in hearing the views on this from the people here on NRF, I know there is a lot of technically inclined people on here and the combined knowledge of NRF should make this an interesting thread IMHO.
So, what are your thoughts on (a) Nikon (and other manufacturers) incorporating the new, faster technology in their camera range(s), and (b) would improved write speeds to the storage medium be sufficient or would the current processing power of today’s cameras need to be improved to keep up with the data throughput. Also, (c) are there other bottlenecks in the system that (might) limit the speed of data throughput if we got cameras that supported storage medium capable of 300MB/sec. or faster?
I would love to hear your insights and opinions.

Oh,… and you’ll have to excuse any spelling mistakes and poor syntax because English isn’t my native language.

Cheers
Badger

shawnino

Your English is better than mine and I am a native speaker, so you'll get no quarter from me

When Nikon introduced the D4 they made one of the card slots for XQD, a Sony card which is faster than what had typically been offered up for still cameras in the past. If this blog or my friends who bought the D4 are any indication, Nikon took a fair amount of grief for that as the cards were deemed to be very expensive. I never quite understood the hullabaloo (you're spending $6K on a body, so hey... and it still takes CF) but it was amplified when the XQD cards didn't play nicely with certain CF cards (which seemed much more reasonable to me BTW).

I can't imagine why any camera company would rush to be an early adopter after that.

Badger

Your point about early adopters of new technology is valid, and a point I've thought about - in conjunction with the camera industry basicly being the last consumer products still holding on to the CF card standard.

Personally I think SDxx cards will become the standard for the non-commercial market in the long run, due to SDxx cards being produced in bigger quantities (more consumer products using SDxx compared to CF).
Now, I can certainly understand why some favour CF cards - especially those that aren't sensitive to the (somewhat steep) price of CF cards), but I secretly wish for the successor to the d7100 and d800 t have dual SDxx card slots that are compatible with dual bus UHS-ii SD cards for up to 300MB/sec sustained write speeds.

Now, for Nikon and other camera manufacturers to adopt the UHS-ii SD card standard I realise that the major SD card manufacturers like Sandisk and Lexar need to start producing these cards on a large scale (Sandisk, are you listening?)... which sort of brings us full circle to your "early adopter" argument - nobody want's to implement new technology before they're fairly certain it will be/become the "industry standard".

WestEndBoy

Cards are very expensive, especially considering that you need multiple cards to set yourself up. I don't believe that card write speed, while it is important, is the true bottleneck on cameras. It is the buffer. I only get 17 shots on raw from my D800 until the camera goes from 4 shots per second to less than one. I am using the fastest CF and SC card that money can buy. Doubling the speed of the card would not double those shots. I might get an extra 4 or 5 and maybe 1 shot per second after the buffer filled up. But a double fast card would be at least another $500 times 2 (two cards in my D800) plus how many do I need for a trip.

But a double sized buffer would give me about 35 shots in the buffer and I bet it would be less than $500, and only once. One way that the D4 is superior to its Canon counterpart (and my D800) is the bigger buffer.

I would trade a bigger buffer for a slower card any day.

Badger

Thank you jshickele, a valuable insight there.

Ade

Using XQD instead of CF on the D4 can help extend the buffer maybe 20%, which is not insignificant when you need all-out performance. A faster card also means the buffer is cleared more quickly, so you don't have to wait as long to shoot the next full burst.

Here's the obvious caveat: when using a dual-slot camera in backup mode, then the camera's performance can only be as good as the slowest card in the two slots.

So when using a D4, buying the latest 180 MB/s XQD card will not make the camera any more faster if the second slot has a 90 MB/s CF card. The buffer can only be cleared at 90 MB/s.

Similarly when shooting the D800, it doesn't make sense (speed-wise) to get the fastest CF card possible only to pair it with a slow SD card leftover from an old point-and-shoot.

Personally, I think having two UHS-II SD card slots is a fine idea.

There is still however an unfortunate sentiment in the market that SD cards are "for amateurs" while "professional" equipment use CF (or XQD). This feeling is perhaps driven more by marketing (and profits) than by any technical merit.

DenverShooter

As someone who has a D800E, D4 & D7000 and a raft of cards for all of them let me toss in my two cents worth.

The XQD format rocks. Fast, fast and faster when it comes to shooting and really fast when it comes to ingesting. I use the Sony QDA-EX1 adapter which slides right into my MacBook Pro and it is lightning fast. I typically run 64G sizes on both the XQD and CF which when are full is a lot to ingest.

CF on the other hand requires an adapter to USB to be ingested and runs about half the speed. CF also has the bad habit of bending the slot pins if you are less than precise when inserting it. And Nikon charges a pretty penny to straighten them out..

Large, slot fragile and antiquated, CF is really yesterdays' technology...

I will take SD and XQD anytime.

Denver Shooter

Photobug

+1 regarding CF cards and the slot pins. DenverShooter nicely said.

I have always been careful when inserting CF cards in my D300 and card reader. Bought a Wolverine backup drive for traveling and after four months damaged the pins and had to send it back for repair. Was in Yellowstone NP when that happened. At least I did not lose any images.

Never used XQD but do use SD cards and have the CFs laying on the shelf incase I buy a full frame DSLR that uses them.

Badger

Thanks for sharing your $0.02 on this.

If I could spec my own camera it would have two identical card slots. I can not see any benefit to having two different card slots on a camera.
I do (as previously mentioned) favour the SD format. CF cards are wildly expensive where I live, and the extortionist prices of XQD cards would force my bank manager to up his medication again.

And I have picked up on the ´stigma´ of CF cards being the "professional choice", tho in my line of work; the "professional choice" for connecting auxillary equipment to Process Automation computers is via PS/2 ports.
So, as have been mentioned; the "professional choice" can often be antiquated remnants from yesteryear.

I'm just pussled there haven't been an outcry from the photo enthusiasts for the new(est) technology; like UHS-II and CFast, to be incorporated in the next release of their camera model(s).
I love reading the "needs & wants" of the enthusiasts and gear heads out there, and their expectations for the next camera / lens release, because that's entertainment in itself (especially for me, seeing as I'm a gear head myself).

Thanks for all the real world insights, much appreciated!

PB_PM

Having mixed cards is a pain, but survivable. Would I be happier if my D800 had CF or just SDxx yes. All my larger (16GB) cards are SD, simply because SD with reasonable performance, and size is cheaper.

Golf007sd

@Badger: To begin with: Welcome to NRF. We look forward to your contribution. As to your first topic and introduction, well done.

From my personal experience, I agree with everything DenverShooter has stated; given that I currently own the D4 & D7000. Like yourself, I too prefer having two slots of the same type...be it XQD or SD. I'm not a fan of CF cards due to issues that have been mentioned above by other members. As to why Nikon and other manufactures do spilt the type of slots they offer on their newer bodies vs just one type (i.e D4 or D800); I believe it is to keep those that have invested in their current memory cards the ability to utilize them, instead of just collecting dust in a drawer.

Now to the meat of your questions: a) I personally welcome new tech in the photography world...storage among the top. With that said, for new storage technology, the end user may require some patience as the interface filters through the high-tech digital world. Case in point, Apple's Thunderbolt. b) Nikon's current processor, namely the Expeed3 & Expeed4, can fully handle the processing of the image itself. I'm sure as newer models arrive to market, this area too will see improvements. Thus, I do not believe it is the processor that is slowing down the image-to-card. What I and many others hope for is: higher buffer capacity...specially on the higher-end fast FPS bodies (Ds's and D4's). We shall see what the up and coming D4s has.
Yet, IMHO the vast majority of the DSLR owners do not need a body that offers high frame rate (>5 FPS). Those that do require it, need to work on mastering their technique to get that "one shot" vs just holding down the shutter button. Only on a few occasions have I ever filled the buffer on my D4 while taking an actions shot, hence, bad technique on my part.
c) The bottleneck within "the system" is: time, adoption and cost. The modern digital world is all about speed; moreover, it hungers for it. Take the USB interface for example, USB->USB 2.0->USB 3.0. Over time the interface has gotten faster and faster. The the adoption process was very much global, for all tech manufactures due to cost and the implementation process. For consumer it was very seamless, due mainly to its downward compatibility and ease of use; hence the connector shape itself never changed. One cannot say the same about the XQD...on all fronts. Yet, upon its introduction (not to mention its future capability) the XQD is clearly a more robust than anything being offered or used. So why haven't other camera manufactures utilized it? Food for thought: HDD vs SSD? Do you see what I mean at it relates to the "bottleneck?"

TaoTeJared

@Badger - Correct me if I wrong, but it seems that you are trying to find a solution to camera's having slower FPS and think it could be the cards? In reality the card a camera uses is about 1/10th of all the variables.

As jshickele said above it mostly comes down to the size of the buffer. But it also is in direct correlation to the size of the files. i.e. D4 & D800 have the same buffer/chip size but the D800's files are much larger. Shooting Jpeg on my D800 I does seem to pick up a bit of speed (not much though and probably barely measurable) vs RAW files. Add to that the "speed" of the card used. If you use the newest, fastest cards, then the buffer can clear faster.

SD cards are the slowest, CF cards are faster and the XQD cards are the fasted to date.
Lets be really clear when you speak to "maximum throughput"; That is a theoretical number and is never achieved in devices. Even tests I have seen with connecting them to computers, those numbers can get close with the best of everything, but still isn't achieved. That said, it doesn't mean that the card can make a difference, just that don't think the "published" throughput is something to pay attention to.

My card set up is a really fast smaller GB card for High speed shots (CF), and slower big cards for the normal stuff.

As for the Dual different slots, I have grown to like it a bit on my D800 for one reason only - my ipad has a "camera" connection kit that only takes SD cards Shame it can only handle basic jpeg files though.

Badger

@Golf007sd:
First of all, thank you for the warm welcome.

I do agree with your point regarding those who have already invested a lot in current memory cards, and can certainly see a ´business-case´ for catering to those customers that do not need or want to invest in a completely new, and different; memory cards.

As jshickele stated, and others have touched upon; today’s cameras are relying on a buffer to intermediately store the processed data from the sensor while waiting for the image data to be written to the memory card – and hence; I feel that I am/was partly right in my assumption that the memory card write speed is a limiting factor in today’s DSLRs.
Seeing as the camera processor is able to read the camera sensor, process the sensor/image data and write to the buffer at a faster rate than the current memory cards are able to ingest the data that has been stored in the buffer intermediately I think the memory card write speed to be the weak link in the system (or at least a substantial part of the ´weak link´ in the system).
Tho, jshickele (and others) made a valid point, so I’ve modified my thoughts about this to something along the lines of: ”A cameras capability to throughput large amounts of image data (as a function of high resolution sensor, little-or-no data compression, fast frame rate or a combination of the mentioned) is largely dependant on the size of the buffer and the write speed of the memory card(s).

Thank you for a well formulated and informative reply, much appreciated.


@TaoTeJared:
I’m more trying to gain knowledge for myself (/my own benefit), and to settle my curiosity as to why there is little-or-no discussion about the SD UHS-II and CFast standards, than trying to find a ”solution” to an actual issue or problem.

I might have fumbled my meaning when I have typed up my posts above (blame it on my lack of ´grammacular skillarity´), but I try to think in terms of amount of data transferred from point A to point B as how the amount came about (i.e. from high res sensor, little data compression, RAW vs. Jpeg and/or fast frame rate) is largely irrelevant.
A camera’s sustained ability to capture and store image data is in effect limited by the memory card’s speed, seeing as a buffer of any size will eventually fill up. In practical, real world terms, I think any buffer that can keep the camera going full tilt for 30 seconds at max frame rate, full resolution shooting RAW at no (or lossless) compression should be sufficient for 99.99% of us. (Those that actually need more than that should/could potentially look into shooting video?)

As you (broadly speaking) correctly state; SD cards are the slowest. This is why I am eagerly awaiting the UHS-II cards to be incorporated into DSLR cameras capable of generating large amounts of image data (see my note on this above), and subsequently led me to write my initial post on NRF. Tho, as far as I know; the only manufacturer of UHS-II SD cards today is Toshiba – so we probably won’t see them implemented in consumer products until more manufacturers start producing them on a large scale.
And I fully realize that we will never achieve the published figures from the manufacturers, but as stated above; the rest of the system handling the data flow from the camera sensor is more than up to the task – hence the camera buffer that allows you to circumvent the maximum image data transfer rate imposed by the memory card… for a limited time (where the time is dependent on the buffer size).


I now have a slightly new perspective on the real world importance of the in-camera buffer, after thinking about the comment(s) first sparked by jsickele.

@all:
Thank you all,
I very much appreciate all the knowledge out there, and the fact that the members here take the time to share their knowledge on NRF.
I have learned quite a bit from this thread, and a lot from NRF in general.

PB_PM

...today’s cameras are relying on a buffer to intermediately store the processed data from the sensor while waiting for the image data to be written to the memory card...

While this is somewhat true it's not the whole story. The files are also in the buffer while the camera is processing the images. If there was no buffer space then the camera would have to try and process the files while they were on the memory card, and that would be far worse. The buffer is more like RAM on a computer, while the memory card is the hard drive. Active files are in the buffer, processed files on the memory card. It's true that some files are kept in the buffer when a large number of shots are fired at once. Even if the cards could read 300+ MB/s they would still have a hard time handling 40+ MB files, if they were coming in at 10+FPS (400+MB/s).

The fact of the matter is that faster memory, in a small package (SD/CF/XQD), while still providing 8+GB of storage space, would be very expensive. We all know that there are faster memory types available, but using them in cameras at this point would be cost prohibitive for the end user.

Badger

@PB_PM:
You are correct.
Part of what this discussion accomplished was to have me pull my head out of a purely theoretical train of thoughts, and implement some real world practicality into said train of thoughts.
Size, cost, availability, backwards compability, whether it is a world wide standard, proprietary or just an oddball format, etc. are all part of the equation.

~ OK, so what's the speed of dark then? ~

Ade

@Badger

You're not "wrong" really, we are mostly photographers here, but what've been driving all the XQD and CFast activities is really video.

For stills it's possible to have a relatively large buffer (a la the D4) to absorb the fast data rate while slowly writing the contents to flash.

For video, the flash speed cannot be slower than the data rate.

As the market demand moves to things like RAW video, high fps footage, 4K and even 8K resolutions, the need for next-generation memory cards becomes clear. Video not only demands very high data rates, but some guaranteed performance profiles as well.

What I question is why the industry seems to be heading into a divided-standard world, XQD vs. CFast. I don't know the politics behind it, but XQD was originally backed by SanDisk, which has since made a 180 degree turn and backed a new CFast standard instead.

The cynic in me thinks the manufacturers will be happy with fragmentation. Nikon cameras get XQD, Canon cameras get CFast. Sony camcorders get XQD, ARRI camcorders get CFast. Just another form of vendor lock in? I hope not.

Badger

@Ade:
It's not about being right or wrong,... it's about reading other peoples views and comparing notes.

And, as you mention; the politics behind a (new) format's way to global standardisation isn't neccesarily logical or understandable to the lay-man.
VHS beat Betamax (before my time, mind you), CD's took over the market from vinyl records, Blueray got to be the standard while HD-DVD and LaserDisk (sp?) went all but extinct.
My impression (be that right or wrong) is that the comercial industry had quite a lot of involvement in what turned out to be the global standard and what was left to be forgotten.

You are on to something when you say that the video market is driving the development of fast memory.
There's a better-than-average chance that the ´mainstream´ fast memory adopted by the video crowd will trickle down into the next generations of camera bodies, as long as there is some degree of backwards compability to existing systems... or bar that; we will continue to get cameras with mixed cards.

~ Reality is nothing more than a collective hunch. ~

Ironheart

One other thing to consider is the relatively long development cycles for new camera technology. While it may seem that the D3xxx and D5xxx churn quickly, in reality the base platform is roughly the same across these lines. The pro models have even longer cycles. Compare this to the computer industry which can bring new technologies to market much faster due to the greater modularity of these platforms. Cameras need reliability more than speed, it seems some computer manufacturers take the opposite approach.

TaoTeJared

...curiosity as to why there is little-or-no discussion about the SD UHS-II and CFast standards,

That one is really easy. 1) few really understand the backside data, 2) There isn't really options at this time.

I do think you have miss-understood when we are speaking about the buffer. It is the limiter and needs to be larger. A larger buffer would do more than any change in cards right now.

The other limiter is the Parallel ATA bus of all camera systems. It is limited to only 166mb/s. So there is a need for a broader system change. Serial ATA allow the 600mb/s that would be a possible next step.

Consider my D800 raw files max out at about 40mb each. With the fastest performing card, the most the system could clear with the 166mb/s limit (of the system bus not the card) is 4 shots per sec. A 30 sec burst is competently unrealistic and most would be max around 5 sec, and that is still very long. The system needs about 3sec of buffer just to clear 1sec buffered shots out. And that 3 sec is about when my D800 blocks up and falls to about 1fps. That would equal about 480mb. With that logic I'm guessing that Nikon uses a 512mb buffer. Double that and you get 6 seconds or in real world terms about 5 seconds with with a sustained write around 135mb/s (which is realistic.)

That covers just the throughput side of the files. The next hang-up is the processors that process the image. I'm convinced it is this real hang-up that limits the D800 since Raw or JPEG files still shoot at 3-4 FPS. Jpegs wouldn't bog the system down. That means a much faster Expeed processor.

With so many companies sticking with 16mp-ish cameras I'm guessing it really comes down to the processing at this point. Sony's A7r (36mp) is limited to 1.5 fps. This is really shocking if you follow Sony's camera systems. Generally they are the one's who try to have the really high FPS and larger buffers.

So Faster cards will help, but systems would have to be changed over to Serial ATA so they work. That is not a small change. The easiest change is to double the Buffer but even with that, the processor still has to keep up as well.

Badger

@TaoTeJared:
I think the reason we put different emphasis on the buffer in this discussion is that I’m thinking/speaking about the system’s sustained data transfer capability, while you appear to discuss this from a more ’real world application’ perspective (which I admit is probably a more sensible approach) where bursts of short duration (i.e. up to a couple of seconds) are common.

I am in full agreement with you on the parallel vs. serial ATA bus topic.
The brunt of the digital world is using/adopting serial bus for data transfer, which is one of the reasons why I am favoring SD cards.
The fact that CFast is (or will be, however you want to look at it) using serial bus, and I think this will be a very, very good memory card standard (tho, I’m worried that the CFast cards will be prohibitively expensive).

Thank you for your valuable insights, they are most welcome.

Cheers
Badger

~ A fight to the death with a vampire has a few inherent problems. ~

Ade

Modern cameras do not use PATA or SATA as an internal bus. Instead, the camera is architected as System on a Chip and direct interfaces are provided to peripherals.

Both PATA and SATA are now considered "dead-end" legacy technologies. PATA is very slow, and we've reached the scalability limit of SATA.

Instead, newer designs are based on direct PCI Express (PCIe) connectivity. And this approach is already used in the Nikon D4: the XQD card interface is PCI Express 3.0 1x (1 GB/s).

CFast is actually an older standard based on SATA. The 600 MB/s limitation comes from SATA-3's max speed. I haven't read the new CFast 2.0 spec yet since I don't feel like paying $100 for it!

There are no plans to increase SATA beyond 600 MB/s. The SATA 3.2 specification defines a new faster SATA Express standard, but that will be achieved by adding PCI Express lanes to a backward-compatible SATA connector. So the SATA portion itself will be capped at 600 MB/s, and any speed improvements will come from PCIe.

Basically the industry is not going to put any more money into developing SATA, and we'll see more and more direct-PCIe peripherals out there (including XQD and various SSD drives).

TaoTeJared

Sorry I should have added "based on Parallel ATA bus standards".

Last I checked CFast still has a different connection than current CF. So a change would have to happen there.
I looked up the Toshiba EXCERIA™ UHS-II and it's max write is only 120mb/s which doesn't add much.

The brunt of the digital world is using/adopting serial bus for data transfer, which is one of the reasons why I am favoring SD cards.
The fact that CFast is (or will be, however you want to look at it) using serial bus, and I think this will be a very, very good memory card standard (tho, I’m worried that the CFast cards will be prohibitively expensive.

Actually Serial bus is old and everyone is leaving it behind. Ade is correct about PCI Express (PCIe), that is where everything has been going for a while. I'm not entirely sure why XQD hasn't caught on (camera world as well as other devices) and I guessing they are charging a pretty penny for licencing or companies are moving to SSDs instead of cards in various devices that keeps the tech still high cost since low supply/inventories. (Realize that most CF & SD cards sold are for industrial/small computer device-type use and not just cameras.)

Everything is expensive until it becomes commonplace. I remember paying $200 for a sandisk extreme 16gb card that was something like 30mb/s. Before my last trip, I bought a off brand 128gb 120mb/s CF card for $89.

Ade

Yea basically CF is an extension of PATA, while CFast is based on SATA. Hence they have different connectors and will never really be compatible with each other.

The last version of PATA (ATA-7) before CF took over the spec actually provided a bridge mode to SATA 1.0, but really no one bothered to use it.

Toshiba has two different UHS-II lines:

- The regular Exceria UHS-II is 120 MB/s write, 260 MB/s read.

- The Exceria Pro is 240 MB/s write, 260 MB/s read.

However for the moment the Pro is only offered at half capacity (32GB max) and is only available in Japan.

Rumors say the upcoming Fuji X-T1 will support UHS-II. The theoretical limit for UHS-II is 312 MB/s.

Badger

See, SanDisk was listening. =P

250 MB/s sustained write speed:
http://www.sandisk.com/about-sandisk/press-room/press-releases/2014/sandisk-announces-world’s-fastest-sd-card/

Now, where's the Nikon D7100 successor that supports the UHS-II SD cards?
...and while you're at it Nikon; please add a dedicated AF-ON back focus button, together with the new EXPEED 4 processor, to the D7100 successor - and you've pretty much got my dream camera.

Thank you all for your contributions.

Golf007sd

@Badger: have a look at this topic in how to configure you AF/AL to work as an AF-ON button.

Badger

Thank you for the link Golf007sd.
Tho, I already know you can configure the AF-L/AE-L button to function as a back focus button.
Comparing the D7100 and the D800, at a local shop that let me 'fondle' both cameras, I find the dedicated AF-ON button on the D800 to be placed more ergonomically correct (for my hands/fingers at-least) - which is why I would be happy if Nikon added a dedicated AF-ON button, placed on the right side of the AF/AE button, on the D7100 successor.