Job description

Job Description: The Azure Data Engineer will be responsible for designing, developing, and maintaining robust data pipelines, ETL/ELT processes, and other data cleansing and transformation solutions to support data-driven initiatives. This role requires technical expertise in data engineering, database management and cloud technologies, with proven experience in implementing complex data projects in a healthcare or research environment. Responsibilities: • ETL/ELT Development: Build and optimize ETL/ELT processes and work within Cloud technology such as Azure and Databricks to ensure efficient data extraction, transformation, and loading while maintaining data quality, consistency, and integrity. Collaborate with a variety of stakeholders across the organization to understand data requirements and translate them into effective ETL/ELT solutions that meet organizational goals. • Cloud Infrastructure and Technologies: Implement and maintain cloud-based data platforms and services, leveraging technologies such as Azure, or Databricks. Design and deploy data solutions using cloud-native services, such as Azure SQL Database, to support advanced analytics, machine learning, and data visualization initiatives. • Performance Monitoring: Monitor performance of data pipelines ETL/ELT processes, databases and data warehouses, troubleshoot issues, and implement optimizations to improve efficiency, reliability, and scalability. Qualifications: 10+ years of experience in data engineering, database management, or related roles, with significant experience managing work is required. 5+ years of experience in Azure services, including Azure Data Factory (ADF), Databricks, and Synapse. 5+ years of experience with SQL skills for database management and query optimization with additional experience with PySpark, SparkQL also preferred. 3+ years of programming skills in Python. 5+ years’ experience with data modeling, ETL processes, and data warehousing and familiarity with big data technologies and frameworks for 3+ years. Proven track record of leading and executing moderately complex projects with minimal oversight is required.

Requirements