Creating Augmented and Virtual Realities: Theory and Practice for Next-Generation Spatial Computing

Description

Despite popular forays into augmented and virtual reality in recent years, spatial computing still sits on the cusp of mainstream use. Developers, artists, and designers looking to enter this field today have few places to turn for expert guidance. In this book, Erin Pangilinan, Steve Lukas, and Vasanth Mohan examine the AR and VR development pipeline and provide hands-on practice to help you hone your skills. Through step-by-step tutorials, you'll learn how to build practical applications and experiences grounded in theory and backed by industry use cases. In each section of the book, industry specialists, including Timoni West, Victor Prisacariu, and Nicolas Meuleau, join the authors to explain the technology behind spatial computing. In three parts, this book covers: Art and design: Explore spatial computing and design interactions, human-centered interaction and sensory design, and content creation tools for digital art Technical development: Examine differences between ARKit, ARCore, and spatial mapping-based systems; learn approaches to cross-platform development on head-mounted displays Use cases: Learn how data and machine learning visualization and AI work in spatial computing, training, sports, health, and other enterprise applications Cover Copyright Table of Contents Foreword Preface A Note on Terminology Why We Wrote This Book What We Didn’t Cover in This Book How This Book Is Organized Conventions Used in This Book Using Code Examples O’Reilly Online Learning How to Contact Us Acknowledgments Part I. Design and Art Across Digital Realities Chapter 1. How Humans Interact with Computers Common Term Definition Introduction Modalities Through the Ages: Pre-Twentieth Century Modalities Through the Ages: Through World War II Modalities Through the Ages: Post-World War II Modalities Through the Ages: The Rise of Personal Computing Modalities Through the Ages: Computer Miniaturization Why Did We Just Go Over All of This? Types of Common HCI Modalities New Modalities The Current State of Modalities for Spatial Computing Devices Current Controllers for Immersive Computing Systems Body Tracking Technologies A Note on Hand Tracking and Hand Pose Recognition Voice, Hands, and Hardware Inputs over the Next Generation Chapter 2. Designing for Our Senses, Not Our Devices Envisioning a Future Sensory Technology Explained So, Who Are We Building This Future For? The Role of Women in AI Sensory Design An Introduction Five Sensory Principles 1. Intuitive Experiences Are Multisensory 2. 3D Will Be Normcore 3. Designs Become Physical Nature 4. Design for the Uncontrollable 5. Unlock the Power of Spatial Collaboration Adobe’s AR Story Conclusion Part II. How eXtended Reality Is Changing Digital Art Chapter 3. Virtual Reality for Art A More Natural Way of Making 3D Art VR for Animation Chapter 4. 3D Art Optimization Introduction Options to Consider Ideal Solution Topology Baking Draw Calls Using VR Tools for Creating 3D Art Acquiring 3D Models Versus Making Them from Scratch Summary Part III. Hardware, SLAM, Tracking Chapter 5. How the Computer Vision That Makes Augmented Reality Possible Works Who Are We? A Brief History of AR How and Why to Select an AR Platform I’m a Developer, What Platform Should I Use and Why? Performance Is Statistics Integrating Hardware and Software Optical Calibration Inertial Calibration The Future of Tracking The Future of AR Computer Vision Mapping How Does Multiplayer AR Work? What’s the Difficult Part? How Does Relocalization Work? What’s the State of the Art in Research (and Coming Soon to Consumer)? Can the Relocalization Problem Really Be Solved for Consumers? Can’t Google or Apple Just Do This? Relocalization != Multiplayer; It’s Also Critical for… How Is Relocalization Really Being Done Today in Apps? Platforms Apple’s ARKit Some Mysteries Explained Isn’t ARCore Just Tango-Lite? So, Should I Build on ARCore Now? What About Tango, Hololens, Vuforia, and Others? Other Development Considerations Lighting Multiplayer AR—Why It’s Quite Difficult How Do People Connect Through AR? How Do AR Apps Connect to the World and Know Where They Really Are? How Do AR Apps Understand and Connect to Things in the Real World? The AR Cloud What I Envision When I Think About the AR Cloud How Big a Deal Is the AR Cloud? So, Just What Is This AR Cloud? Why All the “Meh” from the Press for ARKit and ARCore? What’s Missing to Make a Great AR App? Is Today’s Mobile Cloud up to the Job? Is ARKit (or ARCore) Useless Without the AR Cloud? The Dawn of the AR Cloud The Bigger Picture—Privacy and AR Cloud Data Glossary Part IV. Creating Cross-Platform Augmented Reality and Virtual Reality Chapter 6. Virtual Reality and Augmented Reality: Cross-Platform Theory Why Cross-Platform? The Role of Game Engines Understanding 3D Graphics The Virtual Camera Degrees of Freedom Portability Lessons from Video Game Design Simplifying the Controller Input Development Step 1: Designing the Base Interface Development Step 2: Platform Integration Summary Chapter 7. Virtual Reality Toolkit: Open Source Framework for the Community What Is VRTK and Why People Use It? The History of VRTK Welcome to the SteamVR Unity Toolkit VRTK v4 The Future of VRTK The Success of VRTK Getting Started with VRTK 4 Chapter 8. Three Virtual Reality and Augmented Reality Development Best Practices Developing for Virtual Reality and Augmented Reality Is Difficult Handling Locomotion Locomotion in VR Locomotion in AR Effective Use of Audio Audio in VR Audio in AR Common Interactions Paradigms Inventory for VR Augmented Reality Raycasts Conclusion Part V. Enhancing Data Representation: Data Visualization and Artificial Intelligence in Spatial Computing Chapter 9. Data and Machine Learning Visualization Design and Development in Spatial Computing Introduction Understanding Data Visualization Principles for Data and Machine Learning Visualization in Spatial Computing Why Data and Machine Learning Visualization Works in Spatial Computing The Evolution of Data Visualization Design with the Emergence of XR 2D and 3D Data Represented in XR 2D Data Visualizations versus 3D Data Visualization in Spatial Computing Interactivity in Data Visualizations in Spatial Computing Animation Failures in Data Visualization Design Good Data Visualization Design Optimize 3D Spaces “Savings in Time Feel Like Simplicity” Data Representations, Infographics, and Interactions What Qualifies as Data Visualization? Defining Distinctions in Data Visualization and Big Data or Machine Learning Visualizations How to Create Data Visualization: Data Visualization Creation Pipeline WebXR: Building Data Visualizations for the Web Data Visualization Challenges in XR Data Visualization Industry Use Case Examples of Data Visualizations 3D Reconstruction and Direct Manipulation of Real-World Data: Anatomical Structures in XR A Closer Look at Glass Brain TVA Surg Medical Imaging VR Module Data Visualization Is for Everyone: Open Source–Based Data Visualization in XR Protein Data Visualization Hands-On Tutorials: How to Create Data Visualization in Spatial Computing How to Create Data Visualization: Resources Conclusion References Resources Data Visualization Tools Machine Learning Visualization Tools Data Journalism Visualizations Glossary Chapter 10. Character AI and Behaviors Introduction Behaviors Current Practice: Reactive AI Adaptability Complexity and Universality Feasibility More Intelligence in the System: Deliberative AI Machine Learning Reinforcement Learning Deep Reinforcement Learning Imitation Learning Combining Automated Planning and Machine Learning Applications Conclusion References Part VI. Use Cases in Embodied Reality Chapter 11. The Virtual and Augmented Reality Health Technology Ecosystem VR/AR Health Technology Application Design Standard UX Isn’t Intuitive Pick a Calm Environment Convenience Tutorial: Insight Parkinson’s Experiment What Insight Does How It Was Built Companies Planning and Guidance Experiences Designed for Medical Education Experiences Designed for Use by Patients Proactive Health Case Studies from Leading Academic Institutions Chapter 12. The Fan Experience: SportsXR Introduction Part 1: Five Key Principles of AR and VR for Sports Nothing Is Live Part 2: The Next Evolution of Sports Experiences Part 3: Making the Future Ownership Final Thought Conclusion Chapter 13. Virtual Reality Enterprise Training Use Cases Introduction: The Importance of Enterprise Training Does VR Training Work? Use Case: Flood House Training What Is VR Training Good for? R.I.D.E. What Makes Good VR Training? Spherical Video The Benefits of Spherical Video The Challenges of Spherical Video Interactions with Spherical Video Use Case: Factory Floor Training The Role of Narrative Use Case: Store Robbery Training The Future of XR Training: Beyond Spherical Video Computer Graphics Use Case: Soft Skills Training The Future: Photogrammetry The Future: Light Fields The Future: AR Training The Future: Voice Recognition The Future: The Ideal Training Scenario References Afterword Index About the Authors Colophon