Infocus Technologies

Description : Position Summary : We are seeking a highly skilled Azure Databricks Engineer to design, develop, and optimize large-scale data pipelines and analytical solutions on the Azure Cloud ecosystem.The ideal candidate will possess hands-on experience in Azure Databricks, Azure Data Factory (ADF), and PySpark, with a strong understanding of distributed data processing, ETL development, and data integration best practices.Key Responsibilities : Data Engineering & Pipeline Development : - Design, build, and maintain ETL/ELT pipelines in Azure Databricks using PySpark and Azure Data Factory, ensuring data reliability, accuracy, and performance.Data Integration : - Integrate and transform structured, semi-structured, and unstructured data from multiple data sources into unified, analytics-ready datasets.Performance Optimization : - Implement performance tuning, caching, and partitioning strategies to improve Databricks and Spark job efficiency.Cloud Architecture & Deployment : - Leverage Azure components (Data Lake, Synapse Analytics, Key Vault, Event Hub, etc.) for data ingestion, transformation, and storage.Automation & CI/CD : - Work with DevOps teams to automate deployment and monitoring processes using Azure DevOps pipelines, Git integration, and version control best practices.Collaboration : - Collaborate with data scientists, BI developers, and business analysts to translate analytical requirements into technical solutions.Data Governance & Security : - Ensure data quality, lineage, and compliance with data governance, privacy, and security standards within Azure.Technical Skills & Expertise : - Strong experience in Azure Databricks, Azure Data Factory (ADF), and PySpark.- Proficiency in Python for data engineering and automation tasks.- Familiarity with Azure Data Lake Storage (ADLS), Azure Synapse Analytics, and SQL-based data modeling.- Understanding of Spark architecture, data partitioning, job scheduling, and performance optimization.- Experience with data orchestration, workflow automation, and error handling in ADF pipelines.- Hands-on with CI/CD implementation, Git, and Azure DevOps workflows.- Working knowledge of ETL best practices, data transformation logic, and data quality frameworks.- Familiarity with Power BI or other visualization tools (optional but desirable).Preferred Candidate Profile : - Experience : 3 - 5 years in Azure Cloud, Azure Data Factory, and PySpark.- Educational Qualification : Bachelors degree in Computer Science, Information Technology, or a related discipline.- Strong analytical, problem-solving, and debugging skills.- Excellent communication and collaboration abilities within cross-functional teams.- Ability to deliver quality results under tight deadlines and evolving priorities (ref:hirist.tech)