They address that in the paper. They make use of the rolling shutter common in CMOS imaging chips to effectively increase the sampling rate to well above the frame rate.
They article says they were able to use the rolling shutter on standard cameras to extract data at a much higher frequency than the nominal framerate would allow.
Did you read the article? There are tricks you can do that leverage how a rolling shutter works so you can get an effective high framerate across a small area of the image.
Correct, a CMOS sensor with a rolling shutter looks and handles a lot like a piece of DRAM. Pixels are arranged in rows and colums; capturing a frame works just like a sequential read through a DRAM array. The nominal frame-rate at a given resolution is set by the read-out speed of the array, so reducing resolution increases speed almost linearly.
I think they make use of the fact that individual lines are updated on the camera by the light and can subsequently be read out. If it were a camera with global shutter, chances are it could not be configured to update lines as fast indivually. A global shutter design copies each pixelsite or line into an indiviual buffer when the "shutter" signal is triggered. This happens in parallell.
I'd say these high framerates on Raspi cameras definitely is an artefact of the rolling shutter design in these cameras.
Maybe one could get even higher framerates by hooking the analog part (or very close thereafter) of the rolling shutter camera up to something else, but I don't see how.
I love the creative use of the rolling shutter, instead of seeing it as a downside, they turned the line-by-line nature of the sensor into sample rate multiplier.
> Finally, I should point out that rolling shutter, standard on most mobile cameras, is causing all sorts of problems for traditional image and video analysis algorithms, which often make the assumption, sometimes implicitly, that the entire frame was captured at a single instance in time. This is not true anymore, and can lead to gross errors in many methods. Hence the many recent papers on correcting for, and in some cases exploiting, rolling shutter effects.
This is not just true of mobile phones, but of any current CMOS-sensored imaging device (most of them on the market). Compact cameras and SLR's included.
So it's like if 960-row video at 60fps were actually a 57600 rows-per-second video, right? Which they can extract info from because having more rows in a still frame doesn't mean having more information (at least not linearly), i.e. in still frames with no rolling shutter, rows contain redundant vibration already extracted from previous rows.
So having a rolling shutter is good for this specific application because it trades off resolution (most of which is redundant or insignificant information) for sampling rate.
You still have an adjustable exposure time between the pixel reset and read, which you can make pretty small (<1ms)
But the real problem comes from the time taken to read an entire frame. On a cheap camera this is about the frame time because the electronics is slow, but on a high end camera it is still often close to the frame time because you have a lot of pixels and there is a limit to how fast you can read while still having low noise. Some scientific CMOS cameras get round this by having massively parallel outputs.
The problem gets worse at 48fps - if it takes close to 1/48s to read the chip then a moving object will have stretched across the entire frame from top to bottom. In the worst case a vertical post in a fast pan will be at 45deg. A 24fps camera run at the same pixel clock only has half the effect.
The cameras do have software to try and correct this - basically they look for vertical edges and de-skew them, but this puts in artifacts that you don't want in a Hollywood movie. The other secret is to not fast pan at 48fps.
They show in the last demo usage of this technique with rolling shutter. Considering image height for most cameras is now in thousands of pixels, this somewhat overcomes Nyquist limit problem
They actually got Nyquist to budge to some extent. Check out the last portion of the video.
They took advantage of the fact that digital cameras tend to scan line by line over a short period of time, so there's actually more timing information available in an image than the base 60fps.
Not from a YouTube video, due to the video compression obliterating the small details needed to get the frequency. But the article notes that a rolling shutter allows even a 60fps camera to capture higher frequencies. Each scan line of the captured video is at a sub-frame time slice. If the object occupies a hundred scan lines, then you have 100 slightly shifted 60 hz samples that you can combine to reconstruct higher frequencies.
There's some exciting work being done in our lab with a "coded" rolling shutter, where you change the readout times on individual rows to give you better pictures or videos -- high dynamic range, super-fast capture, etc.
There's no project page up yet, but this video explains most of the key parts:
I wonder if there are interesting applications of being able to capture images at this kind of speed at high resolution. For example, using image analysis between frames to get some kind of depth map, based on subtle differences between each frame due to slight hand movements.
Let's say that it would read the entire image in 1/120 second, then it is waiting and does nothing another 1/120 second before it starts reading next frame.
The real number would be significantly smaller. Therefore they can not bump the sample rate more then five or six times. And I imagine they are using some intelligent algorithm to evenly space out the captured samples already.
> it displays the entire area for the entire time and switches frame content almost immediately.
I've heard this called the sample-and-hold effect. It looks a bit like a fast slide show, and really stands out in high-contrast, steady motion scenes.
I don't quite understand how your original point refutes the headline. They are saying that it captures frames at a rate that, measured in seconds, would be ten trillion frames per second.
reply