Building a Data Integration Team: Skills, Requirements, and Solutions for Designing Integrations

Description

Find the right people with the right skills. This book clarifies best practices for creating high-functioning data integration teams, enabling you to understand the skills and requirements, documents, and solutions for planning, designing, and monitoring both one-time migration and daily integration systems. The growth of data is exploding. With multiple sources of information constantly arriving across enterprise systems, combining these systems into a single, cohesive, and documentable unit has become more important than ever. But the approach toward integration is much different than in other software disciplines, requiring the ability to code, collaborate, and disentangle complex business rules into a scalable model. Data migrations and integrations can be complicated. In many cases, project teams save the actual migration for the last weekend of the project, and any issues can lead to missed deadlines or, at worst, corrupted data that needs to be reconciled post-deployment. This book details how to plan strategically to avoid these last-minute risks as well as how to build the right solutions for future integration projects. What You Will Learn • Understand the “language” of integrations and how they relate in terms of priority and ownership • Create valuable documents that lead your team from discovery to deployment • Research the most important integration tools in the market today • Monitor your error logs and see how the output increases the cycle of continuous improvement • Market across the enterprise to provide valuable integration solutions Who This Book Is For The executive and integration team leaders who are building the corresponding practice. It is also for integration architects, developers, and business analysts who need additional familiarity with ETL tools, integration processes, and associated project deliverables. Table of Contents About the Author About the Technical Reviewer Acknowledgments Introduction Chapter 1: Integration Background The Current Landscape: Two Distinct Models Waterfall Vs. Agile Models The Waterfall Model The Agile Model Integration Process Vs. Integration Product Ownership Integration Approaches One-Time Migration Nightly Integration Scheduled Service Web Service Hybrid Approach Chapter 2: Key Terms The Simplicity of Migration Source System Connection Type Source Endpoints Metadata Model Data Profiling Structure Discovery Content Discovery Relationship Discovery ETL Tool Source-to-Target Mapping Design Code Business Rules Exception Handling Version Control Scheduling Logging Target Target Endpoint Data Matching Chapter 3: Team Qualifications Team Personality Types of Teams Basic Team Flow Functional Cross-Functional Virtual Team Self-Directed Team Team Sizes Waterfall Vs. Agile Team Expectations Waterfall Agile Best Approaches The T-Shaped Person Integration Team Roles Project Sponsor Stakeholders Subject Matter Expert (SME) Product Owner Project Manager (PM) Technical Lead Data Integration Architect Software Developers Software Testers User Acceptance Testers Business Analyst Release Manager Recruiting Your Team Determining the Roles The Job Description Finding Candidates Internal Candidates External Candidates Referrals Outsourcing Posting to Job Boards Hiring a Recruiter Networking Conducting Interviews Summary Chapter 4: Finding Your Purpose: Project Deliverables Introduction Project Management Phases Initiation Definition Design Phase Development Implementation Maintenance Integration Milestones Initiation Definition Data Migration Plan The Discovery Process: Tips and Tricks Learn the Architecture Keep Your Session Small Get System Access Early Design Data Profile Analysis Issues Log Source-to-Target Mapping Purpose Design Usage Data Migration Process Flow Design Order of Operations Mapping Logic Deployment Steps Log File Delivery Migration Results Final Signoff ETL Scripts Migration Results Report Go-Live/Deployment Plan Post-deployment: Maintenance and Updates Summary Chapter 5: Choosing an ETL Tool Introduction ETL Vs. ELT ETL As Middleware The Components of ETL: Choosing the Right Tool Extraction Establishing Connectors Storing Temporary Data Validating Inbound Data Transformation Modifying Data Analyzing Data Integrity Validating Result Data Loading Establishing Connectors Determining Load Types Rolling Back Errors Tools on the Market Jitterbit Talend Dell Boomi Pentaho Data Integration (PDI) Microsoft SQL Server Integration Services (SSIS) MuleSoft Informatica Scripting Languages Summary Chapter 6: A Sample ETL Project Introduction Project Management Phases Revisited Initiation Definition Design Data Profiling Source-to-Target Mapping Data Migration Process Flow Design Development and Implementation Getting Ready Clone the GitHub Directory Download and Install PDI (Community Edition) Opening the “create_target_table_buyer” Script Opening the “Prototype” Script Summary Chapter 7: Platform Automation Introduction All About Environments Why It Matters Types of Environments Development Staging Production Environments for One-Time Migration Environments for Repeatable Integrations DevOps Practices: A Sample Deployment Our Architecture Preparing for Automation Proposed Migration and Integration Approach Demonstration The Output Step Rollback Testing the Automation Testing Job Scheduling Summary Chapter 8: Monitoring Results Introduction Day One: Post-deployment Level of Support Common Integration Complaints and Solutions The Daily Data Log Updating Production Week One: Post-deployment Planning Your Next Steps Continuous Improvement Month One: Post-deployment The Data Management Maturity (DMM) Model Year One: Post-deployment MDM Summary Chapter 9: Marketing Your Team Introduction Addressing Leadership Educating Others Internal Education Formal Seminars Workshops Online Training Lunch and Learns External Education Meetups Networking Event Conferences The Data Integration Roadmap Data Governance and the Data Governance Office Summary Appendix A: Data Migration Strategy Plan Overview Introduction Purpose References Assumptions, Constraints, Risks, and Gaps Assumptions Constraints Risks Gaps Architecture Current State Current State Data Sources Current State Record Counts Current State Data Model Current State Integration Points Future State Future State Data Sources Future State Data Model Future State Integration Points Development Tools Environment Usage Data Migration Approach Scope Approach Team Roles and Responsibilities Migration Process Pre-migration Activities Migration Activities Post-Migration Activities Contingency Plan Testing/Validation Testing Methods Migration Integration Appendix B: Data Profiling Template Appendix C: Issues Log Template Appendix D: Source-to-Target Mapping Template Appendix E: Data Migration Process Flow Design Template Order of Operation Mapping Logic Deployment Steps Description of Prerequisites Contact Account Project Signoff/Document Delivery Log File Delivery Individual Delivery Portal Storage Migration Results Contact Inserts Updates Account Final Signoff Appendix F: Migration Results Report Template Index