Azure Databricks

Job Description Key Responsibilities: - Data Pipeline Development: Design, build, and maintain scalable ETL (Extract, Transform, Load) data pipelines using Azure Databricks, Apache Spark, and Python. - Spark Optimization: Develop and optimize Spark jobs for large-scale data processing on Databricks. Ensure that the jobs run efficiently, leveraging the capabilities of distributed computing for optimal performance. - Data Integration: Integrate data from various sources, including structured and unstructured data, into the Azure cloud environment using Databricks and related tools. - Collaboration with Data Scientists & Analysts: Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver robust data solutions that enable advanced analytics, machine learning, and reporting. - Azure Integration: Work closely with Azure services such as Azure Data Lake, Azure SQL Database, Azure Blob Storage, Azure Synapse Analytics, and Azure Data Factory for comprehensive data processing solutions. - Data Transformation: Use Spark SQL, PySpark, and Databricks notebooks to perform data transformations and enable the conversion of raw data into actionable insights. - Automation & Scheduling: Implement automated job scheduling and orchestration for regular data processing tasks, ensuring data is consistently processed and available for downstream consumption. - Performance Tuning & Troubleshooting: Optimize the performance of data workflows and Spark applications on Databricks. Troubleshoot and resolve data-related issues and bottlenecks. - Cloud Security: Ensure that data security and compliance standards are followed for cloud-based solutions, including managing data access, encryption, and auditing within the Azure Databricks environment. - Monitoring & Logging: Implement logging and monitoring practices for the Azure Databricks environment to track job performance, failures, and troubleshooting efforts. - Documentation & Best Practices: Maintain proper documentation for data pipelines, processes, and technical workflows. Follow best practices for coding, version control, and deployment. - Stay Updated with Technology Trends: Keep up to date with the latest developments in Azure Databricks, Apache Spark, and related technologies. Apply new techniques to improve performance and scalability. Required Qualifications & Skills: - 3-5 years of hands-on experience in data engineering and working with Azure Databricks. - Strong proficiency in Apache Spark, particularly in Databricks for building large-scale data pipelines and distributed data processing applications. - Solid experience with Azure cloud services, including Azure Data Lake, Azure SQL Database, Azure Blob Storage, Azure Synapse, and Azure Data Factory. - Proficiency in Python, Scala, or SQL for data engineering tasks, with a focus on PySpark for data processing. - Experience working with structured and unstructured data from a variety of sources, including relational databases, APIs, and flat files. - Familiarity with Databricks notebooks for developing, testing, and sharing data workflows, and using them for collaboration. - In-depth understanding of ETL processes, data pipelines, and data transformation techniques. - Hands-on experience with cloud-based data storage solutions (e.g., Azure Data Lake, Blob Storage) and data warehousing concepts. - Knowledge of data security best practices in a cloud environment (e.g., data encryption, access controls, Azure Active Directory). - Experience with CI/CD pipelines and version control systems like Git. - Familiarity with containerization and deployment practices using Docker and Kubernetes is a plus. - Strong debugging, performance tuning, and problem-solving skills. - Excellent written and verbal communication skills, with the ability to collaborate effectively across teams. - Bachelor's degree in Computer Science, Information Technology, or a related field.