Knowledge-Based Explorable Extended Reality Environments

Description

This book presents explorable XR environments―their rationale, concept, architectures as well as methods and tools for spatial-temporal composition based on domain knowledge, including geometrical, presentational, structural and behavioral elements. Explorable XR environments enable monitoring, analyzing, comprehending, examining and controlling users’ and objects’ behavior and features as well as users’ skills, experience, interests and preferences. The E-XR approach proposed in this book relies on two main pillars. The first is knowledge representation technologies, such as logic programming, description logics and the semantic web, which permit automated reasoning and queries. The second is imperative programming languages, which are a prevalent solution for building XR environments. Potential applications of E-XR are in a variety of domains, e.g., education, training, medicine, design, tourism, marketing, merchandising, engineering and entertainment. The book’s readers will understand the emerging domain of explorable XR environments with their possible applications. Special attention is given to an in-depth discussion of the field with taxonomy and classification of the available related solutions. Examples and design patterns of knowledge-based composition and exploration of XR behavior are provided, and an extensive evaluation and analysis of the proposed approach is included. This book helps researchers in XR systems, 3D modeling tools and game engines as well as lecturers and students who search for clearly presented information supported by use cases. For XR and game programmers as well as graphic designers, the book is a valuable source of information and examples in XR development. Professional software and web developers may find the book interesting as the proposed ideas are illustrated by rich examples demonstrating design patterns and guidelines in object-oriented, procedural and declarative programming. Preface Acknowledgments Contents 1 Introduction 2 Extended Reality Environments 2.1 Relationships Between Users, Content, and Interfaces 2.2 Creation of XR Environments 2.2.1 Programming XR Environments Procedural and Object-Oriented Programming Aspect-Oriented Programming 2.2.2 3D Content Formats 2.2.3 Visual Modeling of XR Environments 2.3 Human–System Interfaces for Extended Reality 2.3.1 Motion Tracking Optical Motion Tracking Mechanical Motion Tracking Inertial Motion Tracking Magnetic Motion Tracking 2.3.2 Eye Tracking Scleral Contact Lens/Search Coil Electrooculography Photo- and Videooculography Video-Based Pupil and Corneal Reflection Applications and Relationships to Motion Capture 2.3.3 Controllers 2.3.4 Displays Depth Perception Head-Mounted Displays Touchscreens 3 Knowledge Representation with Logical Systems 3.1 Foundations of Logical Systems 3.1.1 Statements and Knowledge Bases Sentences Satisfiability and Validity Horn Clauses 3.1.2 Reasoning in Logical Systems Logical Entailment Rules of Inference Soundness and Completeness Inference Algorithms Monotonicity 3.1.3 Computational Properties of Logical Systems Computability Decidability Computational Complexity 3.2 First-Order Logic 3.2.1 Syntax and Semantics of First-Order Logic Expressivity Terms Predicates Quantifiers Axioms 3.2.2 Queries and Reasoning on Knowledge Bases 3.2.3 Knowledge Engineering 3.2.4 Representation of Spatial–Temporal Entities 3.2.5 Logic Programming Separation of Knowledge Bases and Inference Procedures Prolog Parallel Reasoning 3.3 Description Logics 3.3.1 Semantics of Description Logics Types of Entities Interpretation Open and Closed World Assumptions 3.3.2 Types of Statements Terminological Box Relational Box Assertional Box Knowledge Bases in Description Logics 3.3.3 Constructors of Concepts and Roles Concept Constructors Role Restrictions Nominals Role Constructors 3.3.4 Reasoning Tasks in Description Logics 3.3.5 Families of Description Logics 3.3.6 Spatial and Temporal Description Logics 3.4 Semantic Web 3.4.1 Standards for Building Ontologies Resource Description Framework RDF Schema and Web Ontology Language 3.4.2 Representation of Temporal Entities Overview of Available Approaches 4D-Fluents Time Ontology in OWL 4 Ontology-Based Creation of Extended Reality 4.1 Role of Ontologies in Extended Reality 4.2 Ontology-Based 3D Modeling 4.2.1 Concept 4.2.2 Preliminary Steps 4.2.3 Activities of Ontology-Based 3D Modeling Semantic Reflection Semantic Selection Semantic Configuration Semantic Transformation Summary of Modeling Activities 4.2.4 Taxonomy of Ontology-Based Approaches 4.3 Ontologies of 3D Content 4.3.1 Specificity Levels and Represented Features Concrete Representation Conceptual Representation Hybrid Representation 4.3.2 Knowledge Representation Technologies Used 4.3.3 Location of Semantic Annotations 4.4 Methods and Tools for Ontology-Based 3D Modeling 4.4.1 Supported Modeling Activities 4.4.2 3D Content Adaptation 4.4.3 Reasoning in 3D Content Creation 4.4.4 Separation of Concerns in 3D Content Creation 4.5 Open Challenges 5 E-XR: Explorable Extended Reality Environments 5.1 Problem Statement 5.1.1 Domain Terminology in XR Environments 5.1.2 Declarative Specification of XR Behavior 5.1.3 On-Demand Composition of XR Environments 5.1.4 Representation of Past, Current, and Future XR Behavior 5.1.5 Knowledge Exploration in XR Environments 5.2 Requirements 5.2.1 Functional Requirements Functional Requirement 1: Knowledge-Based Representation of XR Behavior Oriented to Domain Semantics Functional Requirement 2: Declarative Specification of Behavior in the XR Development Process Functional Requirement 3: On-Demand Composition of XR Behavior Functional Requirement 4: Representation of the Past, the Current, and the Potential Future XR Behavior Functional Requirement 5: Knowledge-Based Exploration of XR Behavior Functional Requirement 6: Development of New as well as Transformation of Existing XR Environments 5.2.2 Non-functional Requirements Non-functional Requirement 1: Compatibility with Prevalent Tools and Languages Non-functional Requirement 2: Low Delay in Logging XR Behavior While Using XR Environments Non-functional Requirement 3: Maintaining Performance of the Original XR Environments Non-functional Requirement 4: Efficient Reasoning and Query Processing 5.3 Concept and Scope 5.4 Architecture 6 E-XR Visual Knowledge-Based Behavior Model 6.1 Ontology-Based Component 6.1.1 Domain Ontologies 6.1.2 States and Events States Events 6.1.3 Fluent Ontology 6.1.4 Visual Semantic Behavior Logs 6.2 Rule-Based Component 6.2.1 Time Time Domain Temporal Reasoning 6.2.2 Relations Between States and Events 6.2.3 Behavior Occurrence of States Over Time Transitions Between States 6.2.4 Fluent Rule Set 7 E-XR Semantic Link Model 7.1 Representation of Activities and Features 7.1.1 Classes, Methods, and Variables 7.1.2 Join Points 7.1.3 Structure of Activities Instant Activities Interval Activities 7.1.4 Structure of Features 7.2 Workflow Representation 7.2.1 Events and States for Methods 7.2.2 Linking Method States and Events with Time 7.2.3 Generating Transitions 7.3 Mapping 7.3.1 Mapping Classes Formalism Classification of Application Objects 7.3.2 Mapping Events Formalism Generating Event Mappings 7.4 Code Templates 7.4.1 Processing Events Method Wrappers Data Structures 7.4.2 Processing Variables Generating Unique Variable Values Inferring Variable Values 8 E-XR Development Pipeline of Explorable Environments 8.1 Stage 1: Designing Domain Ontology 8.2 Stage 2: Developing XR Components 8.2.1 Step 2-1: Preparing Shapes and Navigation 8.2.2 Step 2-2: Preparing Animations and Programming Behavior 8.2.3 Step 2-3: Specifying Activities and Features 8.2.4 Step 2-4: Generating Component Transition Sets 8.2.5 Step 2-5: Mapping Classes and Events 8.3 Stage 3: Composing XR Environment with Queries 8.3.1 Step 3-1: Merging Knowledge Basesand Transition Sets 8.3.2 Step 3-2: Processing Queries Specifying Queries Environment Composition Algorithm 8.4 Stage 4: Compiling XR Environment 9 E-XR Exploration Methods 9.1 Simulation and Forward Exploration 9.2 Executing Explorable Environments 9.3 Generating Behavior Logs 9.4 Backward Exploration 10 E-XR Development Tools for Explorable Environments 10.1 Ontology Browser 10.2 Activity Editors 10.2.1 Visual Activity Editor 10.2.2 Textual Activity Editor Attribute Presentation Attribute Completion 10.3 Mapping Editor 10.4 Environment Composer 10.4.1 Architecture 10.4.2 Visual Query Design 10.5 Activity Compiler 10.6 Log Library 10.6.1 Generating Temporal Statements 10.6.2 Generating Visual Descriptors 11 Applications of E-XR 11.1 Explorable Immersive Car Showroom 11.1.1 Architecture 11.1.2 Functionality 11.1.3 Development of Explorable Car Showroom 11.1.4 Example of Behavior Log 11.1.5 Reasoning and Querying on Behavior Logs How much time did the customer spend to watch a particular car outside? Which places does the customer prefer inside cars? 11.2 Explorable Immersive Service Guide for Home Appliances 11.2.1 Architecture 11.2.2 Functionality 12 Evaluation of E-XR 12.1 Configuration of Testing System 12.2 Users' Effort in Environment Composition 12.3 Performance of E-XR Tools 12.3.1 Environment Composition Query Translation Query Processing 12.3.2 Activity Compilation Compilation of Activity Knowledge Bases Transformation of XR Environments 12.3.3 Logging Behavior 12.3.4 FPS in Explorable Environments 12.4 Complexity and Size of E-XR Data Structures 12.4.1 Activity Representation 12.4.2 Code Injected to Explorable Environments Code Injected by Processing Activity Knowledge Bases Code Injected by Processing Attributes 12.4.3 Mapping Knowledge Bases 12.4.4 Behavior Logs 12.5 Discussion of E-XR Approach 12.5.1 Evaluation Results Transforming XR Environments Composing XR Environments Mapping Logging Behavior 12.5.2 Alternative Implementation of E-XR Tools Activities and Features Representing Parts of Methods Using Aspect-Oriented Libraries Using Declarative 3D Formats 12.5.3 Requirements of Technical Knowledge and Skills in Using E-XR 12.5.4 Logical System Properties 12.5.5 Knowledge Exploration with Probabilistic Approaches 13 Conclusions References Index