Pyspark Developer

PySpark Developer – Bhopal (Onsite – no hybrid)

Experience: 5+ years

Key Responsibilities

• Develop and maintain PySpark pipelines in Azure Databricks for batch and streaming data.
• Implement ETL/ELT workflows integrating Azure Data Lake, Synapse, and Delta Lake.
• Write efficient Python and SQL code for transformations, joins, aggregations, and validations.
• Optimise Spark jobs with partitioning, caching, and tuning to reduce runtime and cost.
• Integrate pipelines into Azure Data Factory with CI/CD deployment via Git/DevOps.
• Apply monitoring, logging, and documentation standards for reliable production pipelines.

Key Requirements (Technical & Delivery Skills)

• 5+ years' hands-on experience in PySpark development within the Azure ecosystem.
• Mastery of:
• o Azure Databricks (PySpark, Delta Lake, Spark SQL)o Azure Data Factory for orchestrationo Azure Synapse Analytics for data integration/consumption
• Experience with Delta Lake/Parquet for scalable storage and analytics.
• Skilled in debugging and optimising Spark jobs for performance and cost efficiency.
• Familiar with CI/CD using Azure DevOps, Git versioning, and Agile delivery.
• Proven track record delivering production-grade pipelines in enterprise environments.

Job Type: Full-time

Work Location: In person