✨ AI Summary
The Data Engineer/Test role involves designing, building, and maintaining scalable data pipelines for ingestion, transformation, and loading. The position requires developing and implementing automated testing frameworks for data quality, integrity, and pipeline performance. Collaboration with data scientists and analysts to understand data requirements and translate them into technical solutions is key. The role also focuses on optimizing data warehouse and data lake performance for efficient query execution and data accessibility.
Key requirements include a Bachelor's degree in Computer Science, Engineering, or a related quantitative field, with 3+ years of experience in data engineering and data quality assurance. Proficiency in SQL and experience with relational/NoSQL databases (PostgreSQL, MongoDB) are essential. Strong programming skills in Python or Scala for data manipulation and pipeline development are also necessary.
Design, build, and maintain scalable and robust data pipelines for efficient data ingestion, transformation, and loading.Develop and implement automated testing frameworks for data quality, data integrity, and pipeline performance.Collaborate with data scientists and analysts to understand data requirements and translate them into technical solutions.Optimize data warehouse and data lake performance, ensuring efficient query execution and data accessibility.
Requirements
- Bachelor's degree in Computer Science, Engineering, or a related quantitative field.
- 3+ years of hands-on experience in data engineering and data quality assurance.
- Proficiency in SQL and experience with relational and NoSQL databases (e.g., PostgreSQL, MongoDB).
- Strong programming skills in Python or Scala for data manipulation and pipeline development.
Description
Design, build, and maintain scalable and robust data pipelines for efficient data ingestion, transformation, and loading.Develop and implement automated testing frameworks for data quality, data integrity, and pipeline performance.Collaborate with data scientists and analysts to understand data requirements and translate them into technical solutions.Optimize data warehouse and data lake performance, ensuring efficient query execution and data accessibility.