Capturing very detailed normal maps using Photometric Stereo (R&D)

Photometric stereo.

Edit: Since releasing this article, I’ve spent some time to create easy to use processing tool for non-technical artists that would like to try this technique – https://dabarti.com/capture .

A while ago I did small research project with the aim of developing best way to acquire accurate normal bump maps from photos. In this post I’m going to describe how I did it.

Initial idea was to use Agisoft and photogrammetry, this seemed as the most straight forward approach, but while it produced some promising results I found that detail of the textures was not nearly good enough. After going though some papers on that subject I struck gold by stumbling upon technique called Photometric Stereo.

Great explanation of what it is and how it works is found on Wikipedia.  What I love about this idea the most is that I could potentially take photos from single point of view and generate normal map with the quality limited only by the lens and camera sensor. Think about this like “reverse rendering”. If you know the color of the given part of the object and direction of the light, you can basically find out the surface normal.

With first try I converted this:

Into that:

Before I managed to do that, it took me 2-3 days of trial and error. At first I couldn’t find any software package that could process my photos using photometric stereo. Luckily I found some implementations written Python on Github that I could use. I fixed up the code a little to make it work on my end and processed first batch of photos.

The first test made me super happy, so after testing how the leaf would hold up in animation with  V-Ray 2sided shader I got even more excited.

Workflow.

At the moment I’m sure that this is the best way to go for leaves, cloth, concrete, bricks, anything that you need only one side “scanned”.  I planned my workflow like this:

1. Take as many photos as possible with different light direction. Intervalometer was very helpful. Super bright portable bike light would work great as spotlight.
2. If there is additional ambient lighting: Take one photo without spotlight, so you can substract it from photos with the spotlight… This would leave you something like “Light Select” pass. ( video ).
3. Automate calibration process for the light vector with the help of mirror sphere and Python. Light vector can be calculated based on the XY coordinates of the light reflection in relation to center of the sphere.
4. Connect it all together with already working Python code and start having fun.

This was the next test:

What now?

I plan to do little more research and testing until the flora wakes up after winter to capture some high resolution leaves. So far I figured that since the camera is fixed, it’s easy to capture additional maps like translucency by taking one additional photo with the leafs back lit with even lighting.

In the meantime you can download textures used for above image. Photometric stereo sample textures

I will revisit this technique very soon.