Note

This tutorial is generated from a Jupyter notebook that can be downloaded and run interactively.

📓 Download notebook

Color Map Optimization#

Consider color mapping to the geometry reconstructed from depth cameras. As color and depth frames are not perfectly aligned, the texture mapping using color images is subject to results in blurred color map. CloudViewer provides color map optimization method proposed by [Zhou2014]. The following script shows an example of color map optimization.

Input#

This code below reads color and depth image pairs and makes rgbd_image. Note that convert_rgb_to_intensity flag is False. This is to preserve 8-bit color channels instead of using single channel float type image.

It is always good practice to visualize the RGBD image before applying it to the color map optimization. The debug_mode switch can be set to True to visualize the RGBD image.

[2]:

def load_fountain_dataset():
    rgbd_images = []
    fountain_rgbd_dataset = cv3d.data.SampleFountainRGBDImages()
    for i in range(len(fountain_rgbd_dataset.depth_paths)):
        depth = cv3d.io.read_image(fountain_rgbd_dataset.depth_paths[i])
        color = cv3d.io.read_image(fountain_rgbd_dataset.color_paths[i])
        rgbd_image = cv3d.geometry.RGBDImage.create_from_color_and_depth(
            color, depth, convert_rgb_to_intensity=False)
        rgbd_images.append(rgbd_image)

    camera_trajectory = cv3d.io.read_pinhole_camera_trajectory(
        fountain_rgbd_dataset.keyframe_poses_log_path)
    mesh = cv3d.io.read_triangle_mesh(fountain_rgbd_dataset.reconstruction_path)

    return mesh, rgbd_images, camera_trajectory

The code below reads a camera trajectory and a mesh.

[3]:

# Load dataset
mesh, rgbd_images, camera_trajectory = load_fountain_dataset()

[CloudViewer INFO] Downloading https://github.com/isl-org/open3d_downloads/releases/download/20220201-data/SampleFountainRGBDImages.zip
[CloudViewer INFO] Downloaded to /root/cloudViewer_data/download/SampleFountainRGBDImages/SampleFountainRGBDImages.zip
[CloudViewer INFO] Created directory /root/cloudViewer_data/extract/SampleFountainRGBDImages.
[CloudViewer INFO] Extracting /root/cloudViewer_data/download/SampleFountainRGBDImages/SampleFountainRGBDImages.zip.
[CloudViewer INFO] Extracted to /root/cloudViewer_data/extract/SampleFountainRGBDImages.

To visualize how the camera poses are not good for color mapping, this code intentionally sets the iteration number to 0, which means no optimization. color_map_optimization paints a mesh using corresponding RGBD images and camera poses. Without optimization, the texture map is blurred.

[4]:

# Before full optimization, let's visualize texture map
# with given geometry, RGBD images, and camera poses.
mesh, camera_trajectory = cv3d.pipelines.color_map.run_rigid_optimizer(
    mesh, rgbd_images, camera_trajectory,
    cv3d.pipelines.color_map.RigidOptimizerOption(maximum_iteration=0))
cv3d.visualization.draw_geometries([mesh],
                                   zoom=0.5399,
                                   front=[0.0665, -0.1107, -0.9916],
                                   lookat=[0.7353, 0.6537, 1.0521],
                                   up=[0.0136, -0.9936, 0.1118])

[CloudViewer WARNING] GLFW Error: X11: The DISPLAY environment variable is missing
[CloudViewer WARNING] GLFW initialized for headless rendering.
../../_images/tutorial_pipelines_color_map_optimization_7_1.png

Rigid Optimization#

The next step is to optimize camera poses to get a sharp color map.

The code below sets maximum_iteration = 300 for actual iterations.

[5]:

# Optimize texture and save the mesh as texture_mapped.ply
# This is implementation of following paper
# Q.-Y. Zhou and V. Koltun,
# Color Map Optimization for 3D Reconstruction with Consumer Depth Cameras,
# SIGGRAPH 2014

# Run rigid optimization.
maximum_iteration = 100 if is_ci else 300
with cv3d.utility.VerbosityContextManager(
        cv3d.utility.VerbosityLevel.Debug) as cm:
    mesh, camera_trajectory = cv3d.pipelines.color_map.run_rigid_optimizer(
        mesh, rgbd_images, camera_trajectory,
        cv3d.pipelines.color_map.RigidOptimizerOption(
            maximum_iteration=maximum_iteration))

cv3d.visualization.draw_geometries([mesh],
                                   zoom=0.5399,
                                   front=[0.0665, -0.1107, -0.9916],
                                   lookat=[0.7353, 0.6537, 1.0521],
                                   up=[0.0136, -0.9936, 0.1118])

[CloudViewer DEBUG] [ColorMapOptimization] CreateUtilImagesFromRGBD
[CloudViewer DEBUG] [ColorMapOptimization] CreateDepthBoundaryMasks
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 0/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 1/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 2/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 3/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 4/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 5/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 6/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 7/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 8/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 9/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 10/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 11/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 12/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 13/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 14/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 15/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 16/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 17/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 18/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 19/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 20/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 21/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 22/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 23/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 24/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 25/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 26/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 27/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 28/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 29/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 30/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 31/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 32/33
[CloudViewer DEBUG] [ColorMapOptimization] CreateVertexAndImageVisibility
[CloudViewer DEBUG] [cam 0]: 238774/536872 (44.47503%) vertices are visible
[CloudViewer DEBUG] [cam 1]: 264228/536872 (49.21620%) vertices are visible
[CloudViewer DEBUG] [cam 2]: 32910/536872 (6.12995%) vertices are visible
[CloudViewer DEBUG] [cam 3]: 207076/536872 (38.57083%) vertices are visible
[CloudViewer DEBUG] [cam 4]: 208953/536872 (38.92045%) vertices are visible
[CloudViewer DEBUG] [cam 5]: 121789/536872 (22.68492%) vertices are visible
[CloudViewer DEBUG] [cam 6]: 139892/536872 (26.05686%) vertices are visible
[CloudViewer DEBUG] [cam 7]: 94112/536872 (17.52969%) vertices are visible
[CloudViewer DEBUG] [cam 8]: 136523/536872 (25.42934%) vertices are visible
[CloudViewer DEBUG] [cam 9]: 88157/536872 (16.42049%) vertices are visible
[CloudViewer DEBUG] [cam 10]: 119823/536872 (22.31873%) vertices are visible
[CloudViewer DEBUG] [cam 11]: 183991/536872 (34.27092%) vertices are visible
[CloudViewer DEBUG] [cam 12]: 238849/536872 (44.48900%) vertices are visible
[CloudViewer DEBUG] [cam 13]: 222246/536872 (41.39646%) vertices are visible
[CloudViewer DEBUG] [cam 14]: 220761/536872 (41.11986%) vertices are visible
[CloudViewer DEBUG] [cam 15]: 257217/536872 (47.91030%) vertices are visible
[CloudViewer DEBUG] [cam 16]: 123058/536872 (22.92129%) vertices are visible
[CloudViewer DEBUG] [cam 17]: 155657/536872 (28.99332%) vertices are visible
[CloudViewer DEBUG] [cam 18]: 109616/536872 (20.41753%) vertices are visible
[CloudViewer DEBUG] [cam 19]: 132968/536872 (24.76717%) vertices are visible
[CloudViewer DEBUG] [cam 20]: 125766/536872 (23.42570%) vertices are visible
[CloudViewer DEBUG] [cam 21]: 118639/536872 (22.09819%) vertices are visible
[CloudViewer DEBUG] [cam 22]: 112538/536872 (20.96179%) vertices are visible
[CloudViewer DEBUG] [cam 23]: 76603/536872 (14.26839%) vertices are visible
[CloudViewer DEBUG] [cam 24]: 74932/536872 (13.95714%) vertices are visible
[CloudViewer DEBUG] [cam 25]: 79252/536872 (14.76181%) vertices are visible
[CloudViewer DEBUG] [cam 26]: 107727/536872 (20.06568%) vertices are visible
[CloudViewer DEBUG] [cam 27]: 114058/536872 (21.24491%) vertices are visible
[CloudViewer DEBUG] [cam 28]: 128703/536872 (23.97275%) vertices are visible
[CloudViewer DEBUG] [cam 29]: 149558/536872 (27.85729%) vertices are visible
[CloudViewer DEBUG] [cam 30]: 165234/536872 (30.77717%) vertices are visible
[CloudViewer DEBUG] [cam 31]: 104629/536872 (19.48863%) vertices are visible
[CloudViewer DEBUG] [cam 32]: 50546/536872 (9.41491%) vertices are visible
[CloudViewer DEBUG] [ColorMapOptimization] Rigid Optimization
[CloudViewer DEBUG] [Iteration 0001]
[CloudViewer DEBUG] Residual error : 21627.768621 (avg : 0.004597)
[CloudViewer DEBUG] [Iteration 0002]
[CloudViewer DEBUG] Residual error : 21450.291135 (avg : 0.004559)
[CloudViewer DEBUG] [Iteration 0003]
[CloudViewer DEBUG] Residual error : 21273.138416 (avg : 0.004522)
[CloudViewer DEBUG] [Iteration 0004]
[CloudViewer DEBUG] Residual error : 21095.402935 (avg : 0.004484)
[CloudViewer DEBUG] [Iteration 0005]
[CloudViewer DEBUG] Residual error : 20918.021855 (avg : 0.004446)
[CloudViewer DEBUG] [Iteration 0006]
[CloudViewer DEBUG] Residual error : 20741.426493 (avg : 0.004409)
[CloudViewer DEBUG] [Iteration 0007]
[CloudViewer DEBUG] Residual error : 20565.403267 (avg : 0.004371)
[CloudViewer DEBUG] [Iteration 0008]
[CloudViewer DEBUG] Residual error : 20389.813726 (avg : 0.004334)
[CloudViewer DEBUG] [Iteration 0009]
[CloudViewer DEBUG] Residual error : 20214.920850 (avg : 0.004297)
[CloudViewer DEBUG] [Iteration 0010]
[CloudViewer DEBUG] Residual error : 20041.235823 (avg : 0.004260)
[CloudViewer DEBUG] [Iteration 0011]
[CloudViewer DEBUG] Residual error : 19868.146860 (avg : 0.004223)
[CloudViewer DEBUG] [Iteration 0012]
[CloudViewer DEBUG] Residual error : 19696.485295 (avg : 0.004186)
[CloudViewer DEBUG] [Iteration 0013]
[CloudViewer DEBUG] Residual error : 19526.257939 (avg : 0.004150)
[CloudViewer DEBUG] [Iteration 0014]
[CloudViewer DEBUG] Residual error : 19357.003360 (avg : 0.004114)
[CloudViewer DEBUG] [Iteration 0015]
[CloudViewer DEBUG] Residual error : 19188.390473 (avg : 0.004078)
[CloudViewer DEBUG] [Iteration 0016]
[CloudViewer DEBUG] Residual error : 19020.769133 (avg : 0.004043)
[CloudViewer DEBUG] [Iteration 0017]
[CloudViewer DEBUG] Residual error : 18853.673959 (avg : 0.004007)
[CloudViewer DEBUG] [Iteration 0018]
[CloudViewer DEBUG] Residual error : 18686.442401 (avg : 0.003972)
[CloudViewer DEBUG] [Iteration 0019]
[CloudViewer DEBUG] Residual error : 18520.269019 (avg : 0.003936)
[CloudViewer DEBUG] [Iteration 0020]
[CloudViewer DEBUG] Residual error : 18353.936846 (avg : 0.003901)
[CloudViewer DEBUG] [Iteration 0021]
[CloudViewer DEBUG] Residual error : 18188.177884 (avg : 0.003866)
[CloudViewer DEBUG] [Iteration 0022]
[CloudViewer DEBUG] Residual error : 18022.433052 (avg : 0.003831)
[CloudViewer DEBUG] [Iteration 0023]
[CloudViewer DEBUG] Residual error : 17857.744919 (avg : 0.003796)
[CloudViewer DEBUG] [Iteration 0024]
[CloudViewer DEBUG] Residual error : 17692.343942 (avg : 0.003760)
[CloudViewer DEBUG] [Iteration 0025]
[CloudViewer DEBUG] Residual error : 17527.758880 (avg : 0.003726)
[CloudViewer DEBUG] [Iteration 0026]
[CloudViewer DEBUG] Residual error : 17362.914704 (avg : 0.003690)
[CloudViewer DEBUG] [Iteration 0027]
[CloudViewer DEBUG] Residual error : 17197.453287 (avg : 0.003655)
[CloudViewer DEBUG] [Iteration 0028]
[CloudViewer DEBUG] Residual error : 17032.392931 (avg : 0.003620)
[CloudViewer DEBUG] [Iteration 0029]
[CloudViewer DEBUG] Residual error : 16866.963843 (avg : 0.003585)
[CloudViewer DEBUG] [Iteration 0030]
[CloudViewer DEBUG] Residual error : 16700.219471 (avg : 0.003550)
[CloudViewer DEBUG] [Iteration 0031]
[CloudViewer DEBUG] Residual error : 16533.402242 (avg : 0.003514)
[CloudViewer DEBUG] [Iteration 0032]
[CloudViewer DEBUG] Residual error : 16366.152025 (avg : 0.003479)
[CloudViewer DEBUG] [Iteration 0033]
[CloudViewer DEBUG] Residual error : 16197.984542 (avg : 0.003443)
[CloudViewer DEBUG] [Iteration 0034]
[CloudViewer DEBUG] Residual error : 16028.964073 (avg : 0.003407)
[CloudViewer DEBUG] [Iteration 0035]
[CloudViewer DEBUG] Residual error : 15858.129949 (avg : 0.003371)
[CloudViewer DEBUG] [Iteration 0036]
[CloudViewer DEBUG] Residual error : 15686.642635 (avg : 0.003334)
[CloudViewer DEBUG] [Iteration 0037]
[CloudViewer DEBUG] Residual error : 15513.719628 (avg : 0.003297)
[CloudViewer DEBUG] [Iteration 0038]
[CloudViewer DEBUG] Residual error : 15339.369246 (avg : 0.003260)
[CloudViewer DEBUG] [Iteration 0039]
[CloudViewer DEBUG] Residual error : 15163.439300 (avg : 0.003223)
[CloudViewer DEBUG] [Iteration 0040]
[CloudViewer DEBUG] Residual error : 14985.757900 (avg : 0.003185)
[CloudViewer DEBUG] [Iteration 0041]
[CloudViewer DEBUG] Residual error : 14806.617845 (avg : 0.003147)
[CloudViewer DEBUG] [Iteration 0042]
[CloudViewer DEBUG] Residual error : 14626.059264 (avg : 0.003109)
[CloudViewer DEBUG] [Iteration 0043]
[CloudViewer DEBUG] Residual error : 14443.625413 (avg : 0.003070)
[CloudViewer DEBUG] [Iteration 0044]
[CloudViewer DEBUG] Residual error : 14259.326270 (avg : 0.003031)
[CloudViewer DEBUG] [Iteration 0045]
[CloudViewer DEBUG] Residual error : 14073.449922 (avg : 0.002991)
[CloudViewer DEBUG] [Iteration 0046]
[CloudViewer DEBUG] Residual error : 13886.382375 (avg : 0.002952)
[CloudViewer DEBUG] [Iteration 0047]
[CloudViewer DEBUG] Residual error : 13697.917537 (avg : 0.002911)
[CloudViewer DEBUG] [Iteration 0048]
[CloudViewer DEBUG] Residual error : 13508.312134 (avg : 0.002871)
[CloudViewer DEBUG] [Iteration 0049]
[CloudViewer DEBUG] Residual error : 13317.338242 (avg : 0.002831)
[CloudViewer DEBUG] [Iteration 0050]
[CloudViewer DEBUG] Residual error : 13125.903241 (avg : 0.002790)
[CloudViewer DEBUG] [Iteration 0051]
[CloudViewer DEBUG] Residual error : 12934.127414 (avg : 0.002749)
[CloudViewer DEBUG] [Iteration 0052]
[CloudViewer DEBUG] Residual error : 12742.598063 (avg : 0.002708)
[CloudViewer DEBUG] [Iteration 0053]
[CloudViewer DEBUG] Residual error : 12551.614612 (avg : 0.002668)
[CloudViewer DEBUG] [Iteration 0054]
[CloudViewer DEBUG] Residual error : 12361.992528 (avg : 0.002628)
[CloudViewer DEBUG] [Iteration 0055]
[CloudViewer DEBUG] Residual error : 12174.346045 (avg : 0.002588)
[CloudViewer DEBUG] [Iteration 0056]
[CloudViewer DEBUG] Residual error : 11989.307660 (avg : 0.002548)
[CloudViewer DEBUG] [Iteration 0057]
[CloudViewer DEBUG] Residual error : 11807.571722 (avg : 0.002510)
[CloudViewer DEBUG] [Iteration 0058]
[CloudViewer DEBUG] Residual error : 11630.608171 (avg : 0.002472)
[CloudViewer DEBUG] [Iteration 0059]
[CloudViewer DEBUG] Residual error : 11459.226849 (avg : 0.002436)
[CloudViewer DEBUG] [Iteration 0060]
[CloudViewer DEBUG] Residual error : 11294.101928 (avg : 0.002401)
[CloudViewer DEBUG] [Iteration 0061]
[CloudViewer DEBUG] Residual error : 11136.617621 (avg : 0.002367)
[CloudViewer DEBUG] [Iteration 0062]
[CloudViewer DEBUG] Residual error : 10987.829629 (avg : 0.002335)
[CloudViewer DEBUG] [Iteration 0063]
[CloudViewer DEBUG] Residual error : 10846.494783 (avg : 0.002305)
[CloudViewer DEBUG] [Iteration 0064]
[CloudViewer DEBUG] Residual error : 10715.235101 (avg : 0.002278)
[CloudViewer DEBUG] [Iteration 0065]
[CloudViewer DEBUG] Residual error : 10593.381129 (avg : 0.002252)
[CloudViewer DEBUG] [Iteration 0066]
[CloudViewer DEBUG] Residual error : 10480.699566 (avg : 0.002228)
[CloudViewer DEBUG] [Iteration 0067]
[CloudViewer DEBUG] Residual error : 10378.046761 (avg : 0.002206)
[CloudViewer DEBUG] [Iteration 0068]
[CloudViewer DEBUG] Residual error : 10284.482360 (avg : 0.002186)
[CloudViewer DEBUG] [Iteration 0069]
[CloudViewer DEBUG] Residual error : 10199.318639 (avg : 0.002168)
[CloudViewer DEBUG] [Iteration 0070]
[CloudViewer DEBUG] Residual error : 10122.739402 (avg : 0.002152)
[CloudViewer DEBUG] [Iteration 0071]
[CloudViewer DEBUG] Residual error : 10053.677972 (avg : 0.002137)
[CloudViewer DEBUG] [Iteration 0072]
[CloudViewer DEBUG] Residual error : 9991.653715 (avg : 0.002124)
[CloudViewer DEBUG] [Iteration 0073]
[CloudViewer DEBUG] Residual error : 9935.956504 (avg : 0.002112)
[CloudViewer DEBUG] [Iteration 0074]
[CloudViewer DEBUG] Residual error : 9885.848506 (avg : 0.002101)
[CloudViewer DEBUG] [Iteration 0075]
[CloudViewer DEBUG] Residual error : 9840.681676 (avg : 0.002092)
[CloudViewer DEBUG] [Iteration 0076]
[CloudViewer DEBUG] Residual error : 9800.219243 (avg : 0.002083)
[CloudViewer DEBUG] [Iteration 0077]
[CloudViewer DEBUG] Residual error : 9763.323634 (avg : 0.002075)
[CloudViewer DEBUG] [Iteration 0078]
[CloudViewer DEBUG] Residual error : 9730.056207 (avg : 0.002068)
[CloudViewer DEBUG] [Iteration 0079]
[CloudViewer DEBUG] Residual error : 9699.678138 (avg : 0.002062)
[CloudViewer DEBUG] [Iteration 0080]
[CloudViewer DEBUG] Residual error : 9672.215074 (avg : 0.002056)
[CloudViewer DEBUG] [Iteration 0081]
[CloudViewer DEBUG] Residual error : 9646.627006 (avg : 0.002050)
[CloudViewer DEBUG] [Iteration 0082]
[CloudViewer DEBUG] Residual error : 9623.530113 (avg : 0.002045)
[CloudViewer DEBUG] [Iteration 0083]
[CloudViewer DEBUG] Residual error : 9601.971151 (avg : 0.002041)
[CloudViewer DEBUG] [Iteration 0084]
[CloudViewer DEBUG] Residual error : 9582.367893 (avg : 0.002037)
[CloudViewer DEBUG] [Iteration 0085]
[CloudViewer DEBUG] Residual error : 9564.142260 (avg : 0.002033)
[CloudViewer DEBUG] [Iteration 0086]
[CloudViewer DEBUG] Residual error : 9547.028282 (avg : 0.002029)
[CloudViewer DEBUG] [Iteration 0087]
[CloudViewer DEBUG] Residual error : 9531.077865 (avg : 0.002026)
[CloudViewer DEBUG] [Iteration 0088]
[CloudViewer DEBUG] Residual error : 9516.119413 (avg : 0.002023)
[CloudViewer DEBUG] [Iteration 0089]
[CloudViewer DEBUG] Residual error : 9502.171065 (avg : 0.002020)
[CloudViewer DEBUG] [Iteration 0090]
[CloudViewer DEBUG] Residual error : 9488.918634 (avg : 0.002017)
[CloudViewer DEBUG] [Iteration 0091]
[CloudViewer DEBUG] Residual error : 9476.499042 (avg : 0.002014)
[CloudViewer DEBUG] [Iteration 0092]
[CloudViewer DEBUG] Residual error : 9464.696972 (avg : 0.002012)
[CloudViewer DEBUG] [Iteration 0093]
[CloudViewer DEBUG] Residual error : 9453.515185 (avg : 0.002009)
[CloudViewer DEBUG] [Iteration 0094]
[CloudViewer DEBUG] Residual error : 9442.825386 (avg : 0.002007)
[CloudViewer DEBUG] [Iteration 0095]
[CloudViewer DEBUG] Residual error : 9432.423517 (avg : 0.002005)
[CloudViewer DEBUG] [Iteration 0096]
[CloudViewer DEBUG] Residual error : 9422.587274 (avg : 0.002003)
[CloudViewer DEBUG] [Iteration 0097]
[CloudViewer DEBUG] Residual error : 9413.130207 (avg : 0.002001)
[CloudViewer DEBUG] [Iteration 0098]
[CloudViewer DEBUG] Residual error : 9404.124481 (avg : 0.001999)
[CloudViewer DEBUG] [Iteration 0099]
[CloudViewer DEBUG] Residual error : 9395.318127 (avg : 0.001997)
[CloudViewer DEBUG] [Iteration 0100]
[CloudViewer DEBUG] Residual error : 9386.564204 (avg : 0.001995)
[CloudViewer DEBUG] [ColorMapOptimization] Set Mesh Color
[CloudViewer DEBUG] [RemoveDuplicatedVertices] 0 vertices have been removed.
[CloudViewer DEBUG] [RemoveDuplicatedTriangles] 0 triangles have been removed.
[CloudViewer DEBUG] [RemoveUnreferencedVertices] 734 vertices have been removed.
[CloudViewer DEBUG] [RemoveDegenerateTriangles] 0 triangles have been removed.
[CloudViewer DEBUG] Triangle mesh sampled from 536872 vertices and 1033745 triangles to 504121 vertices and 976218 triangles.
[CloudViewer WARNING] GLFW initialized for headless rendering.
../../_images/tutorial_pipelines_color_map_optimization_9_1.png

The residual error implies inconsistency of image intensities. Lower residual leads to a better color map quality. By default, ColorMapOptimizationOption enables rigid optimization. It optimizes 6-dimensional pose of every cameras.

Non-rigid Optimization#

For better alignment quality, there is an option for non-rigid optimization. To enable this option, simply set option.non_rigid_camera_coordinate to True before calling color_map_optimization. Besides 6-dimensional camera poses, non-rigid optimization even considers local image warping represented by anchor points. This adds even more flexibility and leads to an even higher quality color mapping. The residual error is smaller than the case of rigid optimization.

[6]:

# Run non-rigid optimization.
maximum_iteration = 100 if is_ci else 300
with cv3d.utility.VerbosityContextManager(
        cv3d.utility.VerbosityLevel.Debug) as cm:
    mesh, camera_trajectory = cv3d.pipelines.color_map.run_non_rigid_optimizer(
        mesh, rgbd_images, camera_trajectory,
        cv3d.pipelines.color_map.NonRigidOptimizerOption(
            maximum_iteration=maximum_iteration))

[CloudViewer DEBUG] [ColorMapOptimization] CreateUtilImagesFromRGBD
[CloudViewer DEBUG] [ColorMapOptimization] CreateDepthBoundaryMasks
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 0/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 1/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 2/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 3/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 4/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 5/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 6/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 7/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 8/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 9/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 10/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 11/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 12/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 13/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 14/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 15/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 16/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 17/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 18/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 19/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 20/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 21/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 22/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 23/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 24/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 25/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 26/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 27/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 28/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 29/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 30/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 31/33
[CloudViewer DEBUG] [MakeDepthMasks] geometry::Image 32/33
[CloudViewer DEBUG] [ColorMapOptimization] CreateVertexAndImageVisibility
[CloudViewer DEBUG] [cam 0]: 249227/536872 (46.42205%) vertices are visible
[CloudViewer DEBUG] [cam 1]: 270723/536872 (50.42599%) vertices are visible
[CloudViewer DEBUG] [cam 2]: 230773/536872 (42.98473%) vertices are visible
[CloudViewer DEBUG] [cam 3]: 224391/536872 (41.79600%) vertices are visible
[CloudViewer DEBUG] [cam 4]: 212453/536872 (39.57237%) vertices are visible
[CloudViewer DEBUG] [cam 5]: 176854/536872 (32.94156%) vertices are visible
[CloudViewer DEBUG] [cam 6]: 194256/536872 (36.18293%) vertices are visible
[CloudViewer DEBUG] [cam 7]: 146473/536872 (27.28267%) vertices are visible
[CloudViewer DEBUG] [cam 8]: 149126/536872 (27.77683%) vertices are visible
[CloudViewer DEBUG] [cam 9]: 120544/536872 (22.45302%) vertices are visible
[CloudViewer DEBUG] [cam 10]: 159456/536872 (29.70093%) vertices are visible
[CloudViewer DEBUG] [cam 11]: 182111/536872 (33.92075%) vertices are visible
[CloudViewer DEBUG] [cam 12]: 247864/536872 (46.16817%) vertices are visible
[CloudViewer DEBUG] [cam 13]: 243854/536872 (45.42125%) vertices are visible
[CloudViewer DEBUG] [cam 14]: 221039/536872 (41.17164%) vertices are visible
[CloudViewer DEBUG] [cam 15]: 257706/536872 (48.00139%) vertices are visible
[CloudViewer DEBUG] [cam 16]: 153151/536872 (28.52654%) vertices are visible
[CloudViewer DEBUG] [cam 17]: 159136/536872 (29.64133%) vertices are visible
[CloudViewer DEBUG] [cam 18]: 114113/536872 (21.25516%) vertices are visible
[CloudViewer DEBUG] [cam 19]: 134865/536872 (25.12051%) vertices are visible
[CloudViewer DEBUG] [cam 20]: 129277/536872 (24.07967%) vertices are visible
[CloudViewer DEBUG] [cam 21]: 123877/536872 (23.07384%) vertices are visible
[CloudViewer DEBUG] [cam 22]: 112334/536872 (20.92380%) vertices are visible
[CloudViewer DEBUG] [cam 23]: 106054/536872 (19.75406%) vertices are visible
[CloudViewer DEBUG] [cam 24]: 70273/536872 (13.08934%) vertices are visible
[CloudViewer DEBUG] [cam 25]: 82594/536872 (15.38430%) vertices are visible
[CloudViewer DEBUG] [cam 26]: 128280/536872 (23.89396%) vertices are visible
[CloudViewer DEBUG] [cam 27]: 130563/536872 (24.31920%) vertices are visible
[CloudViewer DEBUG] [cam 28]: 129309/536872 (24.08563%) vertices are visible
[CloudViewer DEBUG] [cam 29]: 158027/536872 (29.43476%) vertices are visible
[CloudViewer DEBUG] [cam 30]: 172541/536872 (32.13820%) vertices are visible
[CloudViewer DEBUG] [cam 31]: 154058/536872 (28.69548%) vertices are visible
[CloudViewer DEBUG] [cam 32]: 99088/536872 (18.45654%) vertices are visible
[CloudViewer DEBUG] [ColorMapOptimization] Non-Rigid Optimization
[CloudViewer DEBUG] [Iteration 0001]
[CloudViewer DEBUG] Residual error : 12219.409463, reg : 0.000000
[CloudViewer DEBUG] [Iteration 0002]
[CloudViewer DEBUG] Residual error : 11738.098182, reg : 13.917949
[CloudViewer DEBUG] [Iteration 0003]
[CloudViewer DEBUG] Residual error : 11305.643607, reg : 42.816618
[CloudViewer DEBUG] [Iteration 0004]
[CloudViewer DEBUG] Residual error : 10918.813332, reg : 78.968639
[CloudViewer DEBUG] [Iteration 0005]
[CloudViewer DEBUG] Residual error : 10573.282008, reg : 119.248857
[CloudViewer DEBUG] [Iteration 0006]
[CloudViewer DEBUG] Residual error : 10266.231727, reg : 161.993518
[CloudViewer DEBUG] [Iteration 0007]
[CloudViewer DEBUG] Residual error : 9994.651602, reg : 206.321346
[CloudViewer DEBUG] [Iteration 0008]
[CloudViewer DEBUG] Residual error : 9755.193775, reg : 251.393934
[CloudViewer DEBUG] [Iteration 0009]
[CloudViewer DEBUG] Residual error : 9543.846376, reg : 296.364797
[CloudViewer DEBUG] [Iteration 0010]
[CloudViewer DEBUG] Residual error : 9357.188969, reg : 340.856491
[CloudViewer DEBUG] [Iteration 0011]
[CloudViewer DEBUG] Residual error : 9192.092685, reg : 384.427386
[CloudViewer DEBUG] [Iteration 0012]
[CloudViewer DEBUG] Residual error : 9044.590116, reg : 427.430174
[CloudViewer DEBUG] [Iteration 0013]
[CloudViewer DEBUG] Residual error : 8912.308474, reg : 470.262477
[CloudViewer DEBUG] [Iteration 0014]
[CloudViewer DEBUG] Residual error : 8793.536352, reg : 512.533792
[CloudViewer DEBUG] [Iteration 0015]
[CloudViewer DEBUG] Residual error : 8685.983981, reg : 553.838998
[CloudViewer DEBUG] [Iteration 0016]
[CloudViewer DEBUG] Residual error : 8587.864179, reg : 594.333689
[CloudViewer DEBUG] [Iteration 0017]
[CloudViewer DEBUG] Residual error : 8497.926911, reg : 634.226733
[CloudViewer DEBUG] [Iteration 0018]
[CloudViewer DEBUG] Residual error : 8415.553582, reg : 673.262629
[CloudViewer DEBUG] [Iteration 0019]
[CloudViewer DEBUG] Residual error : 8339.698617, reg : 710.823110
[CloudViewer DEBUG] [Iteration 0020]
[CloudViewer DEBUG] Residual error : 8269.788296, reg : 746.891344
[CloudViewer DEBUG] [Iteration 0021]
[CloudViewer DEBUG] Residual error : 8205.175962, reg : 781.905671
[CloudViewer DEBUG] [Iteration 0022]
[CloudViewer DEBUG] Residual error : 8145.819129, reg : 815.374447
[CloudViewer DEBUG] [Iteration 0023]
[CloudViewer DEBUG] Residual error : 8090.392274, reg : 847.772555
[CloudViewer DEBUG] [Iteration 0024]
[CloudViewer DEBUG] Residual error : 8039.272935, reg : 878.228357
[CloudViewer DEBUG] [Iteration 0025]
[CloudViewer DEBUG] Residual error : 7991.235360, reg : 907.652260
[CloudViewer DEBUG] [Iteration 0026]
[CloudViewer DEBUG] Residual error : 7946.467818, reg : 935.692518
[CloudViewer DEBUG] [Iteration 0027]
[CloudViewer DEBUG] Residual error : 7904.266849, reg : 963.316839
[CloudViewer DEBUG] [Iteration 0028]
[CloudViewer DEBUG] Residual error : 7865.129914, reg : 989.824742
[CloudViewer DEBUG] [Iteration 0029]
[CloudViewer DEBUG] Residual error : 7827.801464, reg : 1015.936346
[CloudViewer DEBUG] [Iteration 0030]
[CloudViewer DEBUG] Residual error : 7792.384305, reg : 1041.294793
[CloudViewer DEBUG] [Iteration 0031]
[CloudViewer DEBUG] Residual error : 7758.293884, reg : 1066.399713
[CloudViewer DEBUG] [Iteration 0032]
[CloudViewer DEBUG] Residual error : 7725.480592, reg : 1090.541659
[CloudViewer DEBUG] [Iteration 0033]
[CloudViewer DEBUG] Residual error : 7692.987831, reg : 1114.471951
[CloudViewer DEBUG] [Iteration 0034]
[CloudViewer DEBUG] Residual error : 7662.118429, reg : 1137.683806
[CloudViewer DEBUG] [Iteration 0035]
[CloudViewer DEBUG] Residual error : 7632.939895, reg : 1160.772181
[CloudViewer DEBUG] [Iteration 0036]
[CloudViewer DEBUG] Residual error : 7604.799390, reg : 1183.075286
[CloudViewer DEBUG] [Iteration 0037]
[CloudViewer DEBUG] Residual error : 7577.027691, reg : 1205.065050
[CloudViewer DEBUG] [Iteration 0038]
[CloudViewer DEBUG] Residual error : 7550.057836, reg : 1226.295555
[CloudViewer DEBUG] [Iteration 0039]
[CloudViewer DEBUG] Residual error : 7524.371723, reg : 1247.180239
[CloudViewer DEBUG] [Iteration 0040]
[CloudViewer DEBUG] Residual error : 7500.377846, reg : 1267.044466
[CloudViewer DEBUG] [Iteration 0041]
[CloudViewer DEBUG] Residual error : 7477.983037, reg : 1286.393874
[CloudViewer DEBUG] [Iteration 0042]
[CloudViewer DEBUG] Residual error : 7456.448528, reg : 1304.842710
[CloudViewer DEBUG] [Iteration 0043]
[CloudViewer DEBUG] Residual error : 7435.973168, reg : 1323.017066
[CloudViewer DEBUG] [Iteration 0044]
[CloudViewer DEBUG] Residual error : 7416.670574, reg : 1340.582645
[CloudViewer DEBUG] [Iteration 0045]
[CloudViewer DEBUG] Residual error : 7398.311362, reg : 1357.998457
[CloudViewer DEBUG] [Iteration 0046]
[CloudViewer DEBUG] Residual error : 7380.679978, reg : 1374.857620
[CloudViewer DEBUG] [Iteration 0047]
[CloudViewer DEBUG] Residual error : 7363.572714, reg : 1391.570315
[CloudViewer DEBUG] [Iteration 0048]
[CloudViewer DEBUG] Residual error : 7347.553292, reg : 1407.570378
[CloudViewer DEBUG] [Iteration 0049]
[CloudViewer DEBUG] Residual error : 7332.039104, reg : 1423.439561
[CloudViewer DEBUG] [Iteration 0050]
[CloudViewer DEBUG] Residual error : 7317.365274, reg : 1438.657591
[CloudViewer DEBUG] [Iteration 0051]
[CloudViewer DEBUG] Residual error : 7303.051216, reg : 1453.837042
[CloudViewer DEBUG] [Iteration 0052]
[CloudViewer DEBUG] Residual error : 7289.325203, reg : 1468.384926
[CloudViewer DEBUG] [Iteration 0053]
[CloudViewer DEBUG] Residual error : 7276.009296, reg : 1482.989508
[CloudViewer DEBUG] [Iteration 0054]
[CloudViewer DEBUG] Residual error : 7263.114785, reg : 1497.018099
[CloudViewer DEBUG] [Iteration 0055]
[CloudViewer DEBUG] Residual error : 7250.626699, reg : 1511.164147
[CloudViewer DEBUG] [Iteration 0056]
[CloudViewer DEBUG] Residual error : 7238.652913, reg : 1524.843013
[CloudViewer DEBUG] [Iteration 0057]
[CloudViewer DEBUG] Residual error : 7226.846554, reg : 1538.593435
[CloudViewer DEBUG] [Iteration 0058]
[CloudViewer DEBUG] Residual error : 7215.280636, reg : 1551.903145
[CloudViewer DEBUG] [Iteration 0059]
[CloudViewer DEBUG] Residual error : 7203.936045, reg : 1565.520095
[CloudViewer DEBUG] [Iteration 0060]
[CloudViewer DEBUG] Residual error : 7192.995198, reg : 1578.700997
[CloudViewer DEBUG] [Iteration 0061]
[CloudViewer DEBUG] Residual error : 7182.103948, reg : 1591.921342
[CloudViewer DEBUG] [Iteration 0062]
[CloudViewer DEBUG] Residual error : 7171.855879, reg : 1604.630742
[CloudViewer DEBUG] [Iteration 0063]
[CloudViewer DEBUG] Residual error : 7161.562146, reg : 1617.355774
[CloudViewer DEBUG] [Iteration 0064]
[CloudViewer DEBUG] Residual error : 7151.758845, reg : 1629.488044
[CloudViewer DEBUG] [Iteration 0065]
[CloudViewer DEBUG] Residual error : 7142.100305, reg : 1641.625626
[CloudViewer DEBUG] [Iteration 0066]
[CloudViewer DEBUG] Residual error : 7132.826719, reg : 1653.229472
[CloudViewer DEBUG] [Iteration 0067]
[CloudViewer DEBUG] Residual error : 7123.704177, reg : 1664.910146
[CloudViewer DEBUG] [Iteration 0068]
[CloudViewer DEBUG] Residual error : 7114.863577, reg : 1676.148222
[CloudViewer DEBUG] [Iteration 0069]
[CloudViewer DEBUG] Residual error : 7106.177615, reg : 1687.443548
[CloudViewer DEBUG] [Iteration 0070]
[CloudViewer DEBUG] Residual error : 7097.934961, reg : 1698.397956
[CloudViewer DEBUG] [Iteration 0071]
[CloudViewer DEBUG] Residual error : 7089.657147, reg : 1709.383998
[CloudViewer DEBUG] [Iteration 0072]
[CloudViewer DEBUG] Residual error : 7081.596743, reg : 1720.160037
[CloudViewer DEBUG] [Iteration 0073]
[CloudViewer DEBUG] Residual error : 7073.597634, reg : 1730.918839
[CloudViewer DEBUG] [Iteration 0074]
[CloudViewer DEBUG] Residual error : 7065.908579, reg : 1741.536532
[CloudViewer DEBUG] [Iteration 0075]
[CloudViewer DEBUG] Residual error : 7058.337071, reg : 1751.983908
[CloudViewer DEBUG] [Iteration 0076]
[CloudViewer DEBUG] Residual error : 7050.963975, reg : 1762.284984
[CloudViewer DEBUG] [Iteration 0077]
[CloudViewer DEBUG] Residual error : 7043.798168, reg : 1772.432863
[CloudViewer DEBUG] [Iteration 0078]
[CloudViewer DEBUG] Residual error : 7036.825314, reg : 1782.551663
[CloudViewer DEBUG] [Iteration 0079]
[CloudViewer DEBUG] Residual error : 7030.028055, reg : 1792.377245
[CloudViewer DEBUG] [Iteration 0080]
[CloudViewer DEBUG] Residual error : 7023.041359, reg : 1802.308287
[CloudViewer DEBUG] [Iteration 0081]
[CloudViewer DEBUG] Residual error : 7016.342084, reg : 1811.972518
[CloudViewer DEBUG] [Iteration 0082]
[CloudViewer DEBUG] Residual error : 7009.728887, reg : 1821.723036
[CloudViewer DEBUG] [Iteration 0083]
[CloudViewer DEBUG] Residual error : 7003.492377, reg : 1831.188477
[CloudViewer DEBUG] [Iteration 0084]
[CloudViewer DEBUG] Residual error : 6997.146977, reg : 1840.716332
[CloudViewer DEBUG] [Iteration 0085]
[CloudViewer DEBUG] Residual error : 6991.111762, reg : 1850.044389
[CloudViewer DEBUG] [Iteration 0086]
[CloudViewer DEBUG] Residual error : 6985.083067, reg : 1859.512327
[CloudViewer DEBUG] [Iteration 0087]
[CloudViewer DEBUG] Residual error : 6978.925543, reg : 1868.754710
[CloudViewer DEBUG] [Iteration 0088]
[CloudViewer DEBUG] Residual error : 6973.144358, reg : 1877.914578
[CloudViewer DEBUG] [Iteration 0089]
[CloudViewer DEBUG] Residual error : 6967.202136, reg : 1887.324650
[CloudViewer DEBUG] [Iteration 0090]
[CloudViewer DEBUG] Residual error : 6961.490244, reg : 1896.446263
[CloudViewer DEBUG] [Iteration 0091]
[CloudViewer DEBUG] Residual error : 6955.739066, reg : 1905.621188
[CloudViewer DEBUG] [Iteration 0092]
[CloudViewer DEBUG] Residual error : 6950.256386, reg : 1914.469787
[CloudViewer DEBUG] [Iteration 0093]
[CloudViewer DEBUG] Residual error : 6944.474468, reg : 1922.937838
[CloudViewer DEBUG] [Iteration 0094]
[CloudViewer DEBUG] Residual error : 6939.090890, reg : 1931.558575
[CloudViewer DEBUG] [Iteration 0095]
[CloudViewer DEBUG] Residual error : 6933.843502, reg : 1939.838771
[CloudViewer DEBUG] [Iteration 0096]
[CloudViewer DEBUG] Residual error : 6928.335215, reg : 1948.237033
[CloudViewer DEBUG] [Iteration 0097]
[CloudViewer DEBUG] Residual error : 6923.186785, reg : 1956.282267
[CloudViewer DEBUG] [Iteration 0098]
[CloudViewer DEBUG] Residual error : 6917.679679, reg : 1964.501767
[CloudViewer DEBUG] [Iteration 0099]
[CloudViewer DEBUG] Residual error : 6912.341878, reg : 1972.443931
[CloudViewer DEBUG] [Iteration 0100]
[CloudViewer DEBUG] Residual error : 6907.042253, reg : 1980.497469
[CloudViewer DEBUG] [ColorMapOptimization] Set Mesh Color
[CloudViewer DEBUG] [RemoveDuplicatedVertices] 0 vertices have been removed.
[CloudViewer DEBUG] [RemoveDuplicatedTriangles] 0 triangles have been removed.
[CloudViewer DEBUG] [RemoveUnreferencedVertices] 768 vertices have been removed.
[CloudViewer DEBUG] [RemoveDegenerateTriangles] 0 triangles have been removed.
[CloudViewer DEBUG] Triangle mesh sampled from 536872 vertices and 1033745 triangles to 509770 vertices and 984365 triangles.

[7]:

cv3d.visualization.draw_geometries([mesh],
                                   zoom=0.5399,
                                   front=[0.0665, -0.1107, -0.9916],
                                   lookat=[0.7353, 0.6537, 1.0521],
                                   up=[0.0136, -0.9936, 0.1118])

[CloudViewer WARNING] GLFW initialized for headless rendering.
../../_images/tutorial_pipelines_color_map_optimization_13_1.png