Description
Senior Data Engineer – Customer Platform & DataPosition OverviewWe are seeking Senior Data Engineers with deep experience in distributed data systems, Spark-based data processing, and production-grade data platform engineering.As a Senior Data Engineer, you will design, build, optimize, and operate large-scale batch and real-time data pipelines supporting critical customer data initiatives, including customer data, identity resolution, bookings, loyalty programs, and AI-powered customer insights.The ideal candidate is a highly autonomous engineer who can own technical solutions end-to-end, from system design and implementation through production operations, reliability, performance, and cost optimization. You will work in a highly automated, engineering-focused environment that embraces AI-assisted software development and spec-to-code methodologies.Key ResponsibilitiesDesign, develop, and optimize large-scale batch and streaming data pipelines using Scala and Apache Spark.Build and support real-time data processing solutions using Kafka, Kafka Streams, Flink, or similar technologies.Develop and maintain Apache Airflow DAGs, production workflows, and backfill processes.Design robust data models, schemas, and source-to-target mappings for large-scale data platforms.Implement data quality, validation, reconciliation, and monitoring controls.Troubleshoot production issues and drive root-cause analysis and resolution.Ensure data platforms meet required standards for reliability, scalability, performance, cost, and data quality.Apply strong software engineering practices, including testing, version control, CI/CD, code review, and automated deployment.Review, validate, and improve AI-generated code, ensuring that generated solutions meet production engineering and quality standards.Take end-to-end ownership of datasets, pipelines, and platform components throughout their lifecycle.Contribute to technical design discussions and make pragmatic architecture and implementation decisions.Work effectively with configuration-heavy code and multiple programming languages, including Scala, Java, and Python.Adopt modern spec-to-code and AI-assisted development practices, including emerging frameworks, agent skills, and specification-led engineering approaches.Required Qualifications5+ years of professional Data Engineering experience.Strong hands-on experience with Scala and Apache Spark in production environments.Proven experience building, deploying, and owning production ETL/ELT pipelines.Strong experience with distributed data processing and large-scale data systems.Practical experience with Kafka, Kafka Streams, Flink, or comparable streaming technologies.Hands-on experience with Apache Airflow, including DAG development, scheduling, backfills, and production troubleshooting.Strong knowledge of data modeling, schema design, and source-to-target mapping.Experience implementing data quality, validation, reconciliation, and monitoring.Strong software engineering fundamentals, including automated testing, CI/CD, version control, and code review.Experience troubleshooting and supporting production data systems.Ability to independently own technical solutions from design through production.Ability to critically review and validate AI-generated code rather than relying on generated output without verification.Strong communication skills and the ability to clearly explain technical decisions and trade-offs.Nice-to-Have QualificationsStrong Apache Flink expertise.Experience with ScyllaDB, Cassandra, DynamoDB, or other NoSQL technologies.Experience with Customer Data Platforms (CDP), identity resolution, loyalty systems, clickstream data, customer data, or booking data.Experience working with SLAs, SLOs, observability, monitoring, and production reliability.Experience with AI-assisted development tools such as GitHub Copilot, Claude, Cursor, or similar platforms.Familiarity with spec-led/spec-to-code development, Spec Kit, agent skills, or comparable engineering methodologies.Experience working with large-scale customer-facing or data-intensive platforms.Success ProfileThe successful candidate will be a senior, highly autonomous engineer who can confidently demonstrate:Experience designing and optimizing large-scale Spark jobs.Strong, hands-on Scala production experience.Practical experience building and operating streaming systems using Kafka Streams, Flink, or similar technologies.Ability to design and maintain Airflow DAGs, production workflows, and backfill strategies.Strong understanding of data quality and pipeline validation.Comfortable working across Java, Scala, Python, and configuration-heavy codebases.Strong production troubleshooting and deployment discipline.Experience reviewing and improving AI-generated code with sound engineering judgment.An ownership mindset covering systems, datasets, reliability, performance, cost, scalability, and quality.Ability to work effectively in a spec-to-code, highly automated engineering environment.Engineering EnvironmentThe Customer Platform & Data organization operates at significant scale, supporting data-intensive capabilities across customer experiences and business operations.Engineers will work with technologies including:Scala | Apache Spark | Kafka | Kafka Streams | Apache Flink | Apache Airflow | Java | Python | NoSQL | CI/CD | AI-Assisted DevelopmentThe environment places strong emphasis on automation, engineering quality, observability, reliability, and responsible use of AI development tools.Why This PositionWork on large-scale customer data platforms supporting critical customer-facing experiences.Solve complex distributed data engineering challenges involving both batch and real-time processing.Own systems end-to-end and have meaningful influence over technical architecture and engineering standards.Work with modern technologies including Spark, Scala, Kafka, Flink, and Airflow.Be part of an engineering culture embracing AI-assisted and spec-to-code development.Help build highly reliable, scalable, and data-quality-focused platforms used across a global organization.Collaborate with experienced engineers on technically challenging initiatives spanning customer data, identity, bookings, loyalty, and AI-powered insights.EducationSpecialist / Professional Certified