Virtual Reality and Light Field Immersive Video Technologies for Real-World Applications (Computing and Networks)

Description

Virtual reality (VR) refers to technologies that use headsets to generate realistic images, sounds and other sensations that replicate a real-world environment or create an imaginary setting. VR also simulates a user's physical presence in this environment. In virtual reality, six degrees of freedom allows users to not only look around, but also to move around the virtual world and look from above, below or behind objects. To have a true VR experience, the hardware must provide six degrees of freedom, using both orientation tracking (rotational) and positional tracking (translation). This book is addressed to video experts who want to understand the basics of 3D representations and multi-camera video processing to target new immersive media applications. Unlike single camera video coding, future VR technologies address new challenges that arise beyond compression-only, including the pre- and post-processing (depth acquisition and 3D rendering). This book is inspired by the MPEG-I (immersive media) and JPEG-PLENO (plenoptic media) standardization activities, and offers a glimpse of their underlying technologies. Contents About the authors 1. Immersive video introduction References 2. Virtual reality 2.1 Introduction/history 2.2 The challenge of three to six degrees of freedom 2.3 The challenge of stereoscopic to holographic vision References 3. 3D gaming and VR 3.1 OpenGL in VR 3.2 3D data representations 3.3 OpenGL pipeline References 4. Camera and projection models 4.1 Mathematical preliminaries 4.2 The pinhole camera model 4.3 Intrinsics of the pinhole camera 4.4 Projection matrices References 5. Light equations 5.1 Light contributions 5.2 Physically correct light models 5.3 Light models for transparent materials 5.4 Shadows rendering 5.5 Mesh-based 3D rendering with light equations References 6. Kinematics 6.1 Rigid body animations 6.2 Quaternions 6.3 Deformable body animations 6.4 Collisions in the physics engine References 7. Raytracing 7.1 Raytracing complexity 7.2 Raytracing with analytical objects 7.3 VR challenges References 8. 2D transforms for VR with natural content 8.1 The affine transform 8.2 The homography 8.3 Homography estimation 8.4 Feature points and RANSAC outliers for panoramic stitching 8.5 Homography and affine transform revisited 8.6 Pose estimation for AR References 9. 3DoF VR with natural content 9.1 Stereoscopic viewing 9.2 360 panoramas References 10. VR goggles 10.1 Wide angle lens distortion 10.2 Asynchronous high frame rate rendering 10.3 Stereoscopic time warping 10.4 Advanced HMD rendering References 11. 6DoF navigation 11.1 6DoF with point clouds 11.2 Active depth sensing 11.3 Time of flight 11.4 Point cloud registration and densification 11.5 3D rendering of point clouds References 12. Towards 6DoF with image-based rendering 12.1 Introduction 12.2 Finding relative camera positions 12.3 Graph cut 12.4 MPEG reference depth estimation 12.5 Depth estimation challenges 12.6 6DoF view synthesis with depth image-based rendering 12.7 Use case I: view synthesis in holographic stereograms 12.8 Use case II: view synthesis in integral photography 12.9 Difference between PCC and DIBR References 13. Multi-camera acquisition systems 13.1 Stereo vision 13.2 Multiview vision 13.3 Plenoptic imaging References 14. 3D light field displays 14.1 3D TV 14.2 Eye vision 14.3 Surface light field system 14.4 1D-II 3D display system 14.5 Integral photography 14.6 Real-time free viewpoint television 14.7 SMV256 14.8 Light field video camera system 14.9 Multipoint camera and microphone system 14.10 Walk-through system 14.11 Ray emergent imaging (REI) 14.12 Holografika 14.13 Light field 3D display 14.14 Aktina Vision 14.15 IP by 3D VIVANT 14.16 Projection type IP 14.17 Tensor display 14.18 Multi-, plenoptic-, coded-aperture-, multi-focus-camera to tensor display system 14.19 360 light field display 14.20 360 mirror scan 14.21 Seelinder 14.22 Holo Table 14.23 fVisiOn 14.24 Use cases of virtual reality systems References 15. Visual media compression 15.1 3D video compression 15.2 MPEG standardization and compression with 2D video codecs 15.3 Future challenges in 2D video compression 15.4 MPEG codecs for 3D immersion References 16. Conclusion and future perspectives Index