High-Quality Illumination of Virtual Objects Based on an Environment Estimation in Mixed Reality Applications

Description

Visualizations of virtual objects in the real environment is often done by a simplified representation with simple surfaces and without reference to the surrounding environment. The seamless fusion of the virtual and real environment is, however, an essential factor in many areas, which is of particular importance when calculating lighting in mixed realities on mobile devices. Current approaches focus on approximations, which allow the calculation of diffuse lighting, whereby the rendering of glossy reflection properties is often neglected. The aim of this book is to enable the visualization of mirror-like reflective surfaces in mixed reality. In order to achieve this goal, various approaches are explored enabling high-quality visualization of virtual objects in realtime with a focus on the use of common hardware such as cameras, sensors in mobile devices, and partially depth sensors. Complete ambient lighting can be estimated, which enables detailed reflections. The results provide a novel way to embed complex and simple geometric shapes with glossy surfaces in the real world which offers a higher level of detail in the reflections without using additional hardware. Acknowledgment Abstract Kurzfassung Contents Abbreviations List of Algorithms List of Figures List of Tables 1 Introduction 1.1 Motivation 1.2 Contributions 1.3 Thesis Outline 2 Fundamentals & State of the Art 2.1 Light Transport 2.1.1 Photometry & Radiometry 2.1.2 Rendering Equation 2.2 Global Illumination 2.2.1 Ray Tracing 2.2.2 Rasterization 2.3 Mixed Reality 2.3.1 Object Detection 2.3.2 Light Estimation 2.4 Reconstruction 2.4.1 Geometry 2.4.2 Material 2.4.3 Light 2.5 Neural Networks & Deep Learning 3 Illumination in Mixed Reality 3.1 Scene Reconstruction 3.1.1 Spatial Information 3.1.2 Materials 3.1.3 Local Light Sources 3.2 Environment Light 3.2.1 Single Image Estimation 3.2.2 Lookup Texture 3.2.3 Stitching 3.2.4 Estimation with Neural Network 3.3 Reflections 4 Realization 4.1 Architecture 4.2 Physically Based Rendering 4.3 Light & Shadows 4.4 Lookup Texture 4.5 Stitching 4.6 Neural Network 4.7 Filtering of Light Probes 4.8 Rendering in Mixed Reality 4.9 Post-Processing 4.9.1 Ambient Occlusion 4.9.2 Screen Space Reflections 4.9.3 Anti-Aliasing 5 Results 5.1 Scenarios 5.1.1 Punctual Lights 5.1.2 Directional Light 5.1.3 Area Lights 5.1.4 Environment 5.1.5 Manipulation 5.2 Rendering 5.3 Performance 6 Discussion & Limitations 7 Conclusion & Outlook Glossary Bibliography