Google Research on Experimental Nighttime Photography
This article was published by
On a full moon night last year I carried a professional DSLR camera, a heavy lens and a tripod up to a hilltop in the Marin Headlands just north of San Francisco to take a picture of the Golden Gate Bridge and the lights of the city behind it.
I thought the photo of the moonlit landscape came out well so I showed it to my (then) teammates in Gcam, a Google Research team that focuses on computational photography
- developing algorithms that assist in taking pictures, usually with
smartphones and similar small cameras. Seeing my nighttime photo, one of
the Gcam team members challenged me to re-take it, but with a phone
camera instead of a DSLR. Even though cameras on cellphones have come a
long way, I wasn’t sure whether it would be possible to come close to
the DSLR shot.
Probably the most successful Gcam project to date is the image processing pipeline that enables the HDR+ mode
in the camera app on Nexus and Pixel phones. HDR+ allows you to take
photos at low-light levels by rapidly shooting a burst of up to ten
short exposures and averaging them them into a single image, reducing
blur due to camera shake while collecting enough total light to yield surprisingly good pictures.
Of course there are limits to what HDR+ can do. Once it gets dark
enough the camera just cannot gather enough light and challenging shots
like nighttime landscapes are still beyond reach.
The Challenges
To learn what was possible with a cellphone camera in extremely low-light conditions, I looked to the experimental SeeInTheDark app, written by Marc Levoy and presented at the ICCV 2015 Extreme Imaging Workshop,
which can produce pictures with even less light than HDR+. It does this
by accumulating more exposures, and merging them under the assumption
that the scene is static and any differences between successive
exposures must be due to camera motion or sensor noise. The app reduces
noise further by dropping image resolution to about 1 MPixel. With
SeeInTheDark it is just possible to take pictures, albeit fairly grainy
ones, by the light of the full moon.
However, in order to keep motion blur due to camera shake and moving
objects in the scene at acceptable levels, both HDR+ and SeeInTheDark
must keep the exposure times for individual frames below roughly one
tenth of a second. Since the user can’t hold the camera perfectly still
for extended periods, it doesn’t make sense to attempt to merge a large
number of frames into a single picture. Therefore, HDR+ merges at most
ten frames, while SeeInTheDark progressively discounts older frames as
new ones are captured. This limits how much light the camera can gather
and thus affects the quality of the final pictures at very low light
levels.
Of course, if we want to take high-quality pictures of low-light scenes
(such as a landscape illuminated only by the moon), increasing the
exposure time to more than one second and mounting the phone on a tripod
or placing it on some other solid support makes the task a lot easier.
Google’s Nexus 6P and Pixel phones support exposure times of 4 and 2
seconds respectively. As long as the scene is static, we should be able
to record and merge dozens of frames to produce a single final image,
even if shooting those frames takes several minutes.
Even with the use of a tripod, a sharp picture requires the camera’s
lens to be focused on the subject, and this can be tricky in scenes with
very low light levels. The two autofocus mechanisms employed by
cellphone cameras — contrast detection and phase detection
— fail when it’s dark enough that the camera's image sensor returns
mostly noise. Fortunately, the interesting parts of outdoor scenes tend
to be far enough away that simply setting the focus distance to infinity
produces sharp images.
Experiments & Results
Taking all this into account, I wrote a simple Android camera app with
manual control over exposure time, ISO and focus distance. When the
shutter button is pressed the app waits a few seconds and then records
up to 64 frames with the selected settings. The app saves the raw frames
captured from the sensor as DNG files, which can later be downloaded onto a PC for processing.
To test my app, I visited the Point Reyes lighthouse
on the California coast some thirty miles northwest of San Francisco on
a full moon night. I pointed a Nexus 6P phone at the building and shot a
burst of 32 four-second frames at ISO 1600. After covering the camera
lens with opaque adhesive tape I shot an additional 32 black frames.
Back at the office I loaded the raw files into Photoshop. The individual
frames were very grainy, as one would expect given the tiny sensor in a
cellphone camera, but computing the mean of all 32 frames cleaned up
most of the grain, and subtracting the mean of the 32 black frames
removed faint grid-like patterns caused by local variations in the
sensor's black level. The resulting image, shown below, looks
surprisingly good.
The lantern in the lighthouse is overexposed, but the
rest of the scene is sharp, not too grainy, and has pleasing, natural
looking colors. For comparison, a hand-held HDR+ shot of the same scene
looks like this:
Satisfied with these results, I wanted to see if I could capture a nighttime landscape as well as the stars in the clear sky above it, all in one picture. When I took the photo of the lighthouse a thin layer of clouds conspired with the bright moonlight to make the stars nearly invisible, but on a clear night a two or four second exposure can easily capture the brighter stars. The stars are not stationary, though; they appear to rotate around the celestial poles, completing a full turn every 24 hours. The motion is slow enough to be invisible in exposures of only a few seconds, but over the minutes it takes to record a few dozen frames the stars move enough to turn into streaks when the frames are merged. Here is an example:
Seeing streaks instead of pinpoint stars in the sky can
be avoided by shifting and rotating the original frames such that the
stars align. Merging the aligned frames produces an image with a
clean-looking sky, and many faint stars that were hidden by noise in the
individual frames become visible. Of course, the ground is now
motion-blurred as if the camera had followed the rotation of the sky.
We now have two images; one where the ground is sharp, and one where the sky is sharp, and we can combine them into a single picture that is sharp everywhere. In Photoshop the easiest way to do that is with a hand-painted layer mask. After adjusting brightness and colors to taste, slight cropping, and removing an ugly "No Trespassing" sign we get a presentable picture:
Using Even Less Light The pictures I've shown so far were shot on nights with a full moon, when it was bright enough that one could easily walk outside without a lantern or a flashlight. I wanted to find out if it was possible to take cellphone photos in even less light. Using a Pixel phone, I tried a scene illuminated by a three-quarter moon low in the sky, and another one with no moon at all. Anticipating more noise in the individual exposures, I shot 64-frame bursts. The processed final images still look fine:
In the second image the distant lights of the cities around the San Francisco Bay caused the sky near the horizon to glow, but without moonlight the night was still dark enough to make the Milky Way visible. The picture looks noticeably grainier than my earlier moonlight shots, but it's not too bad. Pushing the Limits How far can we go? Can we take a cellphone photo with only starlight - no moon, no artificial light sources nearby, and no background glow from a distant city? To test this I drove to a point on the California coast a little north of the mouth of the Russian River, where nights can get really dark, and pointed my Pixel phone at the summer sky above the ocean. Combining 64 two-second exposures taken at ISO 12800, and 64 corresponding black black frames did produce a recognizable image of the Milky Way. The constellations Scorpius and Sagittarius are clearly visible, and squinting hard enough one can just barely make out the horizon and one or two rocks in the ocean, but overall, this is not a picture you'd want to print out and frame. Still, this may be the lowest-light cellphone photo ever taken.
Here we are approaching the limits of what the Pixel
camera can do. The camera cannot handle exposure times longer than two
seconds. If this restriction was removed we could expose individual
frames for eight to ten seconds, and the stars still would not show
noticeable motion blur. With longer exposures we could lower the ISO
setting, which would significantly reduce noise in the individual
frames, and we would get a correspondingly cleaner and more detailed
final picture.
Getting back to the original challenge - using a cellphone to reproduce a
night-time DSLR shot of the Golden Gate - I did that. Here is what I
got:
At 9 to 10 MPixels the resolution of these pictures is not as high as what a DSLR camera might produce, but otherwise image quality is surprisingly good: the photos are sharp all the way into the corners, there is not much visible noise, the captured dynamic range is sufficient to avoid saturating all but the brightest highlights, and the colors are pleasing. Trying to find out if phone cameras might be suitable for outdoor nighttime photography was a fun experiment, and clearly the result is yes, they are. However, arriving at the final images required a lot of careful post-processing on a desktop computer, and the procedure is too cumbersome for all but the most dedicated cellphone photographers. However, with the right software a phone should be able to process the images internally, and if steps such as painting layer masks by hand can be eliminated, it might be possible to do point-and-shoot photography in very low light conditions. Almost - the cellphone would still have to rest on the ground or be mounted on a tripod.
Here’s a Google Photos album with more examples of photos that were created with the technique described above.
Via GoogleResearch Blog
On a full moon night last year I carried a professional DSLR camera, a heavy lens and a tripod up to a hilltop in the Marin Headlands just north of San Francisco to take a picture of the Golden Gate Bridge and the lights of the city behind it.
 |
| A view of the Golden Gate Bridge from the Marin Headlands, taken with a DSLR camera (Canon 1DX, Zeiss Otus 28mm f/1.4 ZE). Click here for the full resolution image. |
 |
| Point Reyes lighthouse at night, photographed with Google Nexus 6P (full resolution image here). |
 |
| Point Reyes Lighthouse at night, hand-held HDR+ shot (full resolution image here). The inset rectangle has been brightened in Photoshop to roughly match the previous picture. |
Satisfied with these results, I wanted to see if I could capture a nighttime landscape as well as the stars in the clear sky above it, all in one picture. When I took the photo of the lighthouse a thin layer of clouds conspired with the bright moonlight to make the stars nearly invisible, but on a clear night a two or four second exposure can easily capture the brighter stars. The stars are not stationary, though; they appear to rotate around the celestial poles, completing a full turn every 24 hours. The motion is slow enough to be invisible in exposures of only a few seconds, but over the minutes it takes to record a few dozen frames the stars move enough to turn into streaks when the frames are merged. Here is an example:
 |
| The North Star above Mount Burdell, single 2-second exposure. (full resolution image here). |
 |
| Mean of 32 2-second exposures (full resolution image here). |
 |
| Mean of 32 2-second exposures, stars aligned (full resolution image here). |
We now have two images; one where the ground is sharp, and one where the sky is sharp, and we can combine them into a single picture that is sharp everywhere. In Photoshop the easiest way to do that is with a hand-painted layer mask. After adjusting brightness and colors to taste, slight cropping, and removing an ugly "No Trespassing" sign we get a presentable picture:
 |
| The North Star above Mount Burdell, shot with Google Pixel, final image (full resolution image here). |
Using Even Less Light The pictures I've shown so far were shot on nights with a full moon, when it was bright enough that one could easily walk outside without a lantern or a flashlight. I wanted to find out if it was possible to take cellphone photos in even less light. Using a Pixel phone, I tried a scene illuminated by a three-quarter moon low in the sky, and another one with no moon at all. Anticipating more noise in the individual exposures, I shot 64-frame bursts. The processed final images still look fine:
 |
| Wrecked fishing boat in Inverness and the Big Dipper, 64 2-second exposures, shot with Google Pixel (full resolution image here). |
 |
| Stars above Pierce Point Ranch, 64 2-second exposures, shot with Google Pixel (full resolution image here). |
In the second image the distant lights of the cities around the San Francisco Bay caused the sky near the horizon to glow, but without moonlight the night was still dark enough to make the Milky Way visible. The picture looks noticeably grainier than my earlier moonlight shots, but it's not too bad. Pushing the Limits How far can we go? Can we take a cellphone photo with only starlight - no moon, no artificial light sources nearby, and no background glow from a distant city? To test this I drove to a point on the California coast a little north of the mouth of the Russian River, where nights can get really dark, and pointed my Pixel phone at the summer sky above the ocean. Combining 64 two-second exposures taken at ISO 12800, and 64 corresponding black black frames did produce a recognizable image of the Milky Way. The constellations Scorpius and Sagittarius are clearly visible, and squinting hard enough one can just barely make out the horizon and one or two rocks in the ocean, but overall, this is not a picture you'd want to print out and frame. Still, this may be the lowest-light cellphone photo ever taken.
 |
| Only starlight, shot with Google Pixel (full resolution image here). |
 |
| Golden Gate Bridge at night, shot with Google Nexus 6P (full resolution image here). |
 |
| The Moon above San Francisco, shot with Google Nexus 6P (full resolution image here). |
At 9 to 10 MPixels the resolution of these pictures is not as high as what a DSLR camera might produce, but otherwise image quality is surprisingly good: the photos are sharp all the way into the corners, there is not much visible noise, the captured dynamic range is sufficient to avoid saturating all but the brightest highlights, and the colors are pleasing. Trying to find out if phone cameras might be suitable for outdoor nighttime photography was a fun experiment, and clearly the result is yes, they are. However, arriving at the final images required a lot of careful post-processing on a desktop computer, and the procedure is too cumbersome for all but the most dedicated cellphone photographers. However, with the right software a phone should be able to process the images internally, and if steps such as painting layer masks by hand can be eliminated, it might be possible to do point-and-shoot photography in very low light conditions. Almost - the cellphone would still have to rest on the ground or be mounted on a tripod.
Here’s a Google Photos album with more examples of photos that were created with the technique described above.
Via GoogleResearch Blog
Post a Comment