✨ AI Summary
The Senior Data Engineer will collaborate with data scientists and analysts to understand data requirements and design solutions. Responsibilities include designing, building, and maintaining scalable data pipelines for data ingestion, processing, and transformation from various sources. The role involves optimizing pipelines for performance and reliability, implementing data validation and quality checks, and performing data modeling and schema design. The engineer will also explore new technologies and build data marts to support analysis.
Key requirements include a Bachelor's degree in a technical field or equivalent experience, at least 4 years of data engineering experience, advanced SQL skills, and proficiency in Python. Experience with data warehouses, data modeling, and cloud platforms (AWS, Azure, GCP) is necessary. Familiarity with data streaming technologies like Kafka, Flink, or Spark Streaming is a plus. Strong problem-solving skills, attention to detail, and a willingness to learn are also important.
Requirements
Bachelor’s degree in Computer Science, Information Systems, or other related technical field or equivalent work.Proven experience as a Data engineer or similar role, with at least 4 years of experience.Advanced SQL knowledge/experience - complex queries and query optimization.Proficiency in programming languages such as Python.Experienced in the development of data warehouses and proficient in data modeling.Familiarity with cloud platforms such as AWS, Azure, or Google Cloud Platform.Experience in building data streams using technologies such as Kafka, Flink, or Spark Streaming is considered a plus.Strong problem-solving skills and attention to detail.The ability and willingness to learn new technologies on the job.Self driven, highly motivated, fast learner, ambitious and creative.
Description
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and design appropriate solutions.Work closely with IT and infrastructure teams to deploy and maintain data storage and processing systems, including databases, data warehouses, and cloud infrastructure.Design, build, and maintain robust, scalable data pipelines to ingest, process, and transform data from various sources, including internal databases and external APIs.Optimize data pipelines for performance, scalability, and reliability, ensuring timely and accurate delivery of data.Implement data validation and quality checks to identify and rectify any issues in data pipelines, ensuring data integrity.Explore and evaluate new technologies, tools, and frameworks to improve data engineering processes and capabilities.Perform data modeling and schema design to support analytical and reporting requirements.Design and build data marts and business layers to facilitate efficient data access and analysis.