Senior Data Engineer

Senior Data Engineer

We are looking for an experienced Senior Data Engineer to own and evolve our data engineering architecture, ensuring reliable, scalable, and high-quality data pipelines. The role will focus on big data architecture, workflow orchestration using Kestra, canonical data models and catalog schemas, LLM evaluation pipelines, and technical leadership within the data engineering team.

Key Responsibilities:
- Design, build, and maintain scalable data engineering and big data architectures.
- Own end-to-end data pipelines, ensuring reliability, scalability, performance, and data quality.
- Design and manage Kestra workflows for data ingestion, transformation, validation, and downstream processing.
- Build reusable, fault-tolerant workflows with appropriate scheduling, retries, monitoring, alerting, and failure handling.
- Develop and maintain canonical data models, schemas, metadata, and data catalog structures.
- Establish data standards, governance practices, lineage, and quality controls across data assets.
- Design and maintain LLM evaluation pipelines, including dataset preparation, automated evaluation, benchmarking, and monitoring.
- Integrate data pipelines with AI/ML and LLM-based applications where required.
- Optimize data processing, storage, and compute costs across cloud-based environments.
- Implement automated testing and data quality checks across pipelines and datasets.
- Monitor pipeline performance and proactively troubleshoot production issues.
- Work closely with data scientists, ML engineers, software engineers, and product teams to deliver reliable data solutions.
- Conduct technical reviews and establish engineering best practices.
- Mentor junior and mid-level data engineers and contribute to technical decision-making.
- Drive improvements in data platform architecture, automation, observability, and operational reliability.

Required Skills & Experience:
- 5+ years of experience in data engineering or a closely related field.
- Strong experience designing and implementing large-scale data pipelines and data platforms.
- Strong proficiency in SQL and Python.
- Hands-on experience with Kestra or similar workflow orchestration platforms.
- Strong understanding of ETL/ELT, data modeling, data warehousing, and data lake architectures.
- Experience with cloud platforms such as AWS, Azure, or GCP.
- Experience with distributed data processing technologies such as Spark.
- Strong understanding of data quality, testing, observability, and pipeline reliability.
- Experience with Git, CI/CD, Docker, and infrastructure/cloud environments.
- Experience designing canonical schemas, metadata models, or data catalogs.
- Exposure to LLM/AI evaluation frameworks and pipelines is highly desirable.
- Strong problem-solving, communication, and technical leadership skills.