Lead Data Engineer

Lead Data Engineer

Job Description / Duties and Responsibilities

▪ Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory and related Azure services.
▪ Develop complex and optimized T-SQL queries, stored procedures, views, functions, and database objects in SQL Server.
▪ Perform SQL Server database development and administration activities, including performance tuning, troubleshooting, and data integrity management.
▪ Build and maintain data integration pipelines connecting multiple source and target systems.
▪ Work with Azure data platform services to develop and support cloud-based data solutions.
▪ Analyze source systems and changing data structures and adapt data pipelines accordingly.
▪ Apply light-to-medium data modeling techniques to support analytical and operational data requirements.
▪ Work with large volumes of structured and semi-structured data and ensure efficient data processing.
▪ Troubleshoot data pipeline failures, data quality issues, performance problems, and production incidents.
▪ Monitor data pipelines and proactively identify and resolve issues.
▪ Collaborate with developers, architects, QA teams, business analysts, and other stakeholders to deliver data solutions.
▪ Participate in Agile ceremonies including sprint planning, daily stand-ups, backlog refinement, reviews, and retrospectives.
▪ Create and maintain technical documentation covering data flows, mappings, processes, dependencies, and troubleshooting procedures.
▪ Participate in code reviews and follow established development, security, and deployment standards.
▪ Support production deployments, releases, and operational activities as required.

Job Specification / Skills and Competencies

▪ 7+ years of experience in Data Engineering, ETL, data integration, or related roles.
▪ Strong hands-on experience with Microsoft SQL Server development and administration.
▪Advanced knowledge of T-SQL, including:
o Complex queries
o Stored procedures
o Views and functions
o Query optimization
o Indexing and performance tuning
o Error handling and troubleshooting
▪ Strong experience with Azure Data Factory (ADF) for ETL/ELT and data integration.
▪ Practical experience working with Azure data platform services.
▪ Good understanding of data warehousing and data integration concepts.
▪ Experience with data modelling, including dimensional modelling and logical/physical data models.
▪ Ability to understand and work with evolving or changing source data structures.
▪ Strong troubleshooting and problem-solving skills.
▪ Strong technical documentation and communication skills.
▪ Experience working in an Agile/Scrum development environment.
▪ Ability to work independently while collaborating effectively with distributed/onshore-offshore teams.
▪ Experience with CI/CD and source control tools such as Azure DevOps, Git, or Bitbucket.
▪ To adhere to the Information Security Management policies and procedures.