cloudLight - Relighting using cloudlet inside Nuke

cloudLight - Relighting using cloudlet inside Nuke

| February 27, 2012 | 9:46 pm or 10:46 pm GMT

| 0 Comments

Hassan Uriostegui has created a plug-in that would let you use the XYZ and color pass to relight your scene inside Nuke.

1. Introduction

Enabling 3D relighting in compositing stage, allow a great flexibility, to accomplish fine tuning results. In this field several researches was done base in ren -dering “normals” or “point” passes in order to export in ( floating point raw data ) , the world position cor -responding for each pixel to 2D compositing pack -ages.

2. Background

On 2005 “sexy light” shake macro from Jose Fer-nandez de Castro implements expression to simu -late 2.5D lighting using normal passes. ost recently Shervin Shogian from “Image Engine” show us a plugin developed around “District 9” Feature Film, where pixels where plotted in 3D point couds, allow-ing volumetric relighting with built-in Nuke lights and scanline render, based in a color and point pass -es. After that several plugins and techniques were developed enabling point cloud visualization as CG_POINTS, lacking any posibility to render using standar surface rendering.

3. Goal.

This research is an effort to create an open source NDK C++ plugin to bring full 3D surface relighting on
Nuke to VFX community.

4. Technical Approach. Cloudlets.

In order to accomplish this, designing a pipeline where the “point cloud” could be represented by light weight geometry that responds to usal 3D lights and shading. So instead points the system uses cloud -lets.

Cloudlet is a light holder class wich represents a ge-ometry in the world space of a given color . In ab -stract way cloudlets are the atomic elements of a geometry cloud.

Thus, the solutions consists in 2 basic plugins. The first is the actual 3D geometry cloud generator wich takes the “point” and “color” passes, and using a uni-form sampling* driven by a “resolution” UI slider, creates cubes in the corresponding point pass posi -tion and maps in the corresponding vertex colors the color coming from the “color” pass. Additionally if the user is working with 2D footage can select to use the average luma as the depth channel, and adjust magnitud based on corresponding knob.*

float scale = 1.0 / resolution;

for( int x=0; x< (columns*resolution); x++){
for( int y=0; y<(rows*resolution); y++){

colorMap->sample(x*scale,y*scale,1,1,colorPixel);
pointMap->sample(x*scale,y*scale,1,1,pointPixel);

if(colorPixel[Chan_Alpha] > 0.5 ){
//only create if its solid
cloudlet CL;

CL.r= colorPixel[Chan_Red];
CL.g= colorPixel[Chan_Green];
CL.b= colorPixel[Chan_Blue];

CL.x= pointPixel[Chan_Red];
CL.y= pointPixel[Chan_Green];
CL.z= pointPixel[Chan_Blue];

if(useLuma){
float lum = (CL.x+CL.y+CL.z)/3.0f;
CL.x= x;
CL.y= y ;
CL.z= lum * ((columns +rows)/10.0f);
}

As an optimization only does pixels with alpha value greater than 0.0 are considered for drawing.

Aditionally in order to allow plugin running on non high-end computers, users can enable wich faces of the cloudlets wants to be draw..

int cube_faces = 0;
if(useFront) cube_faces++;
if(useBack) cube_faces++;
if(useTop) cube_faces++;
if(useBottom) cube_faces++;
if(useLeft) cube_faces++;
if(useRight) cube_faces++;

unsigned cube_points = cube_faces * (2 * 3); //2 triangles per face
unsigned num_points = cube_points*clouds.size();

Then color is assigned as a point attibute.

for (unsigned c = 0; c < clouds.size(); c++){

cloudlet cloud=clouds[c];

float r=cloud.r;
float g=cloud.g;
float b=cloud.b;

for(int i=0; i int p = (cube*cube_points)+i;

cf->vector4(p).set( r,g,b, 1.0 ); } }

The second is a modification of The Foundry Phong shader wich takes the vertex color and multiplies by the standar shading algorithm

surface[channel[z]] = vtx.r()* (emission_.x * mapEmission[Chan_Red]+ surface[channel[0]] + Cd.x * color_.x *
mapDiffuse[Chan_Red] + Ck.x * mapSpecular[Chan_Red] + vtx.ambient.x);

5.Development.

The plugin was compiled through xCode IDE using external makefile template. Also a bash script ( de -pending on test script) was writen to allow quick de -ployment to .nuke folder, current software instance killing, and script relaunching to start Nuke with the latest compiled version.

#!/bin/bash
echo “Deploying and lauching on run.nk”
cp -f *.dylib ~/.nuke
pid=$(ps -fe | grep ‘[N]uke6’ | awk ‘{print $2}’)
if [[ -n $pid ]]; then
kill $pid
else
echo “Nuke is dead”
fi
cd ..
cd ..
sleep sleep 10
open run.nk

6. Applications.

Current vesion is realesed with an example script showing how to connect cloudLight ( geometry cloud generator ) and cloudPhong ( surface shader * Phong ) to allow surface relighting, as well as using the color luma to depth feature.

But also the plugin can be used in other passes than the “color” to create new lighting stages.

7. Future development.

This plugin needs to be tested in High-End comput -ers using a very dense geometry cloud in order to prove its accuracy. Several improvements could be done on the sampling algorithm in order to optimize the perceptual results. Also further shaders that take the vertex color information could benefit to the cur -rent system.

8. Links.

Images from Adam Wierzchowski
http://www.pxfactory.eu/plug/PointCloud/pointCloud.html
https://github.com/randominteractive/cloudLight
https://github.com/hassanvfx/cloudLight
Demo: http://www.youtube.com/watch?v=LJ39zm1Ehzo

Follow the LINK to get all the files.