Job description

Job Details: Considering Python as the ETL tool for reading data from the source, handling datatype and data transformations, managing error handling, logging, and loading data into Databricks-managed Delta tables, the candidate should have strong expertise in Python, along with solid knowledge of Databricks and PySpark, enabling them to evaluate and perform data transformations within Databricks. They should be proficient in all aspects of Python, including classes, objects, methods, dictionaries, loops, Wheel files, and packages, and should be able to connect to various data sources, including databases and streaming data, Expertise in Azure services such as Azure SQL Database, Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake is also required. • The ideal candidate has extensive experience in data engineering, working with large-scale data infrastructures, and optimizing ETL processes • Expertise in Azure services such as Azure SQL Database, Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake • At least 6+ years of experience in data engineering, with a focus on building data pipelines • Proven experience with Spark, Scala, and other related technologies • Familiarity with Could Warehouses, data lakes, storage formats. • Strong knowledge of data modeling and ETL processes • Experience leading data engineering projects. • Proven experience with Databricks and Apache Spark • Strong proficiency in Python • Experience in Unix shell scripting and scheduling tools About the Company: Expert In Recruitment Solutions

Requirements