Virtual and Augmented Reality (VR/AR): Foundations and Methods of Extended Realities (XR)

Description

his comprehensive textbook offers a scientifically sound and at the same time practical introduction to Virtual and Augmented Reality (VR/AR). Readers will gain the theoretical foundation needed to design, implement or enhance VR/AR systems, evaluate and improve user interfaces and applications using VR/AR methods, assess and enrich user experiences, and develop a deeper understanding of how to apply VR/AR techniques. Whether utilizing the book for a principal course of study or reference reading, students of computer science, education, media, natural sciences, engineering and other subject areas can benefit from its in-depth content and vivid explanation. The modular structure allows selective sequencing of topics to the requirements of each teaching unit and provides an easy-to-use format from which to choose specific themes for individual self-study. Instructors are provided with extensive materials for creating courses as well as a foundational text upon which to build their advanced topics. The book enables users from both research and industry to deal with the subject in detail so they can properly assess the extent and benefits of VR/AR deployment and determine required resources. Technology enthusiasts and professionals can learn about the current status quo in the field of VR/AR and interested newcomers can gain insight into this fascinating world. Grounded on a solid scientific foundation, this textbook, addresses topics such as perceptual aspects of VR/AR, input and output devices including tracking, interactions in virtual worlds, real-time aspects of VR/AR systems and the authoring of VR/AR applications in addition to providing a broad collection of case studies. Foreword Preface Contents Chapter 1: Introduction to Virtual and Augmented Reality 1.1 What Is VR/AR About? 1.1.1 The Perfect Virtual Reality 1.1.2 The Simulation of the World 1.1.3 Suspension of Disbelief 1.1.4 Motivation 1.2 What Is VR? 1.2.1 Technology-Centered Characterizations of VR 1.2.2 VR as an Innovative Kind of Human–Computer Interaction 1.2.3 Mental Aspects of the VR Experience 1.3 What Is AR? 1.4 Historical Development of VR and AR 1.5 VR Systems 1.6 AR Systems 1.7 Using the Book 1.7.1 Structure of the Book 1.7.2 Usage Instructions 1.7.3 Target Groups Lecturers in the Field of VR/AR Students Users and Those Who Want to Become Users The Technology-Savvy 1.8 Summary and Questions Recommended Reading References Chapter 2: Perceptual Aspects of VR 2.1 Human Information Processing 2.2 Visual Perception 2.2.1 Stereo Vision 2.2.2 Perception of Space 2.3 Multisensory Perception 2.3.1 Auditory Perception 2.3.2 Haptic Perception 2.3.3 Proprioception and Kinaesthesia 2.3.4 Perception of Movement 2.3.5 Presence and Immersion 2.4 Phenomena, Problems, Solutions 2.4.1 Deviating Observation Parameters 2.4.2 Double Vision 2.4.3 Frame Cancellation 2.4.4 Vergence-Focus Conflict 2.4.5 Discrepancies in the Perception of Space 2.4.6 Discrepancies in the Perception of Movement 2.4.7 Cybersickness 2.4.8 Vertical Parallax Problem 2.5 Use of Perceptual Aspects 2.5.1 Salience 2.5.2 User Guidance 2.6 Summary and Questions Recommended Reading References Chapter 3: Virtual Worlds 3.1 Introduction 3.1.1 Requirements on 3D Object Representations for Virtual Worlds 3.1.2 Creation of 3D Models 3.1.3 Preparation of 3D Models for VR/AR 3.1.4 Integration of 3D Models into VR/AR Runtime Environments 3.2 Scene Graphs 3.3 3D Objects 3.3.1 Surface Models Polygonal Representations Polygons Polygon Meshes Triangle Strips 3.3.2 Solid Models Boundary Representations (B-Reps) Primitive Instancing 3.3.3 Appearance Materials Textures Shader 3.3.4 Optimization Techniques for 3D Objects Simplification of Polygon Meshes Level-of-Detail Techniques Texture Baking Billboards 3.4 Animation and Object Behavior 3.4.1 Keyframe Animation 3.4.2 Physics-Based Animation of Rigid Bodies 3.4.3 Object Behavior 3.4.4 Behavior and Animation in Scene Graphs 3.5 Light, Sound, Background 3.5.1 Light Sources 3.5.2 Sound 3.5.3 Backgrounds 3.6 Special Purpose Systems 3.6.1 Virtual Humans 3.6.2 Particle Systems 3.6.3 Terrain 3.6.4 Vegetation 3.7 Summary and Questions Recommended Reading References Chapter 4: VR/AR Input Devices and Tracking 4.1 Fundamentals of Input Devices 4.2 Tracking Techniques 4.2.1 Acoustic Tracking 4.2.2 Magnetic Field-Based Tracking 4.2.3 Inertial Tracking 4.2.4 Laser-Based Tracking 4.2.5 Outdoor Position Tracking 4.3 Camera-Based Tracking 4.3.1 Marker-Based Methods 4.3.2 Tracking Using Black and White Markers Use of Marker Tracking Basic Operation Intrinsic and Extrinsic Camera Parameters 4.3.3 Feature-Based Tracking Techniques Geometry-Based Tracking Other Feature-Based Tracking Techniques 4.3.4 Visual SLAM 4.3.5 Hybrid Tracking Techniques Cloud-Based Tracking Microsoft Hololens Tracking 4.4 Finger Tracking 4.5 Eye Tracking 4.5.1 Eye Movements 4.5.2 Methods 4.5.3 Functionality of an Eye Tracker 4.5.4 Calibration 4.5.5 Eye Tracking in Head-Mounted Displays 4.5.6 Remote Eye Tracker 4.6 Further Input Devices 4.6.1 3D Mouse 4.6.2 Mechanical Input Devices 4.6.3 Treadmills for Virtual Reality 4.7 Summary and Questions Recommended Readings References Chapter 5: VR/AR Output Devices 5.1 Introduction 5.2 Basics of Visual Output 5.3 Head-Mounted Displays (HMDs) 5.3.1 VR Glasses 5.3.2 AR Glasses 5.3.3 General Characteristics and Properties of HMDs 5.3.4 Special HMDs 5.4 Stationary VR Systems 5.4.1 Single-Sided Displays 5.4.2 Multi-Sided Displays 5.4.3 Tiled Displays 5.4.4 Stereo Output Methods 5.5 Audio Output Devices 5.6 Haptic Output Devices 5.7 Summary and Questions Recommended Reading References Chapter 6: Interaction in Virtual Worlds 6.1 Fundamentals of Human–Computer Interaction 6.2 System Control 6.3 Selection 6.3.1 Pointing in Virtual Worlds 6.3.2 Interaction Design 6.3.3 Examples of Selection Techniques 6.4 Manipulation of Objects 6.5 Navigation 6.5.1 Control Techniques for Traveling 6.5.2 Walking Technique for Natural Movement Control 6.5.3 Leaning Interfaces for Movement Control 6.5.4 Teleportation for Movement Control 6.5.5 Route Plan, Goal-Based and Guided Movement Techniques 6.5.6 Criteria for Navigation Techniques 6.6 Processes for the Design and Implementation of Interaction 6.6.1 Characteristics of VR/AR User Interfaces 6.6.2 Human-Centered Design of VR/AR Interactions Analysis of the Context of Use Specification of Requirements Concept, Design and Implementation Evaluation (Especially User Tests) 6.7 User Tests 6.8 Ethical, Social and Legal Aspects of VR/AR 6.9 Summary and Questions Recommended Reading References Chapter 7: Real-Time Aspects of VR Systems 7.1 Latency in VR Systems 7.1.1 What Are the Requirements on Latency? 7.1.2 Where Do Latencies Actually Arise? 7.1.3 Is Latency in a VR System Constant? 7.1.4 What Are the Approaches to Determining Latency? Latency Estimation from Datasheets Measuring the Latency of Tracking Systems Measuring End-to-End Latency 7.1.5 Summary of Latency 7.2 Efficient Collision Detection in Virtual Worlds 7.2.1 Bounding Volumes Axis-Aligned Bounding Box (AABB) Bounding Spheres Oriented Bounding Boxes (OBBs) Discrete-Oriented Polytopes (k-DOPs) 7.2.2 Bounding Volume Hierarchies and Space Partitioning Techniques Bounding Volume Hierarchies (BVHs) Space Partitioning Techniques 7.2.3 Collision Detection in Large Environments Broad Phase Collision Detection Narrow Phase Collision Detection 7.2.4 Summary and Advanced Techniques 7.3 Real-Time Rendering of Virtual Worlds 7.3.1 Algorithmic Strategies View Volume Culling Hierarchical View Volume Culling Occlusion Culling Backface Culling Small Feature Culling Portal Culling Level of Detail (LOD) 7.3.2 Hardware-Related Strategies Object Size Indexing Caching Stripping (Triangle and Quadrilateral Strips) Minimizing State Changes 7.3.3 Software Systems for Virtual Worlds Scene Graph Systems Game Engines 7.4 Summary and Questions Recommended Reading References Chapter 8: Augmented Reality 8.1 Introduction 8.1.1 Getting Started 8.1.2 AR – An Overview 8.2 Registration 8.2.1 Geometric Registration 8.2.2 Photometric Registration 8.3 Visual Output 8.3.1 Handheld Devices 8.3.2 Projection-Based Output 8.3.3 Further Types of Spatial AR 8.3.4 AR Mirrors 8.4 Special AR Techniques 8.4.1 Head-Up Content 8.4.2 Occlusions and Phantom Objects 8.4.3 Crossfading Markers 8.4.4 Virtual Holes 8.4.5 X-Ray Vision 8.5 Special AR Interaction Techniques 8.5.1 Interaction by Navigation 8.5.2 Gaze-Based Interaction 8.5.3 Tangible User Interfaces 8.6 AR Applications 8.7 Diminished and Mediated Reality 8.7.1 Diminished Reality 8.7.2 Mediated Reality 8.8 Summary and Questions Recommended Reading References Chapter 9: VR/AR Case Studies 9.1 Introduction and Overview 9.2 Using Virtual Reality for Design Processes in the Automotive Industry 9.3 AR/VR Revolutionizes Your In-Car Experience 9.4 VR-Based Service Training in the Life Sciences and Diagnostics Industry 9.5 Utilizing Augmented Reality for Visualizing Infrastructure 9.6 Enhancing the Spatial Design Process with CADwalk 9.7 The aixCAVE at RWTH Aachen University 9.8 Augmented Reflection Technology: Stroke Rehabilitation with XR 9.9 Collaborative Virtual Trainers in VR Applications 9.10 Virtual Patients: A Case Study from Research to Real-World Impact 9.11 Embodied Social XR for Teaching, Learning and Therapy 9.12 Virtual Reality for Teaching Literacy to Prisoners References References for Sect. 9.1 References for Sect. 9.2 References for Sect. 9.3 References for Sect. 9.5 References for Sect. 9.6 References for Sect. 9.7 References for Sect. 9.8 References for Sect. 9.9 References for Sect. 9.10 References for Sect. 9.11 References for Sect. 9.12 Chapter 10: Authoring of VR/AR Applications 10.1 Supporting Authors 10.2 Foundations of Authoring Software 10.2.1 Unity How It Works VR/AR Development with Unity Summary 10.2.2 Unreal Engine How It Works Developing VR/AR Applications Summary 10.2.3 AR Frameworks: ARCore and ARKit Availability Tracking and Mapping Reconstruction Estimation of Environment Light Summary 10.3 Examples of the Creation of VR/AR Applications 10.3.1 Making of: Immersive VR Presentation of CAD Files with the Vive Cosmos in Unity 10.3.2 Making of: Interaction in VR Using the Vive Cosmos and Unreal Engine 10.3.3 Making of: An Application for the Microsoft HoloLens 2 with Unity 10.3.4 Making of: Basics for the Development of a Native ARCore Application for Android 10.4 Summary and Questions Recommended Reading References Software, Online Documentation and Tutorials Chapter 11: Mathematical Foundations of VR/AR 11.1 Vector Spaces 11.2 Geometry and Vector Spaces 11.3 Points and Affine Spaces 11.4 Euclidean Space 11.5 Analytical Geometry in ℝ3 11.6 Matrices 11.7 Affine Transformations 11.8 Determination of Transformation Matrices About the Authors Index