Pyspark Developer
2 days ago
PySpark Developer – Bhopal (Onsite – no hybrid)
Experience: 5+ years
Key Responsibilities
- Develop and maintain PySpark pipelines in Azure Databricks for batch and streaming data.
- Implement ETL/ELT workflows integrating Azure Data Lake, Synapse, and Delta Lake.
- Write efficient Python and SQL code for transformations, joins, aggregations, and validations.
- Optimise Spark jobs with partitioning, caching, and tuning to reduce runtime and cost.
- Integrate pipelines into Azure Data Factory with CI/CD deployment via Git/DevOps.
- Apply monitoring, logging, and documentation standards for reliable production pipelines.
Key Requirements (Technical & Delivery Skills)
- 5+ years' hands-on experience in PySpark development within the Azure ecosystem.
- Mastery of:
- o Azure Databricks (PySpark, Delta Lake, Spark SQL)o Azure Data Factory for orchestrationo Azure Synapse Analytics for data integration/consumption
- Experience with Delta Lake/Parquet for scalable storage and analytics.
- Skilled in debugging and optimising Spark jobs for performance and cost efficiency.
- Familiar with CI/CD using Azure DevOps, Git versioning, and Agile delivery.
- Proven track record delivering production-grade pipelines in enterprise environments.
Job Type: Full-time
Work Location: In person
-
Lead Azure Databricks Data Engineer
4 days ago
Bhopal, Madhya Pradesh, India SAA Consultancy Full time ₹ 15,00,000 - ₹ 25,00,000 per yearLocation: Bhopal, MP, IndiaExperience: 7+ yearsKey ResponsibilitiesDesign and build ETL pipelines in Azure Databricks (PySpark, Delta Lake) to load, clean, and deliver data across Bronze, Silver, and Gold layers.Implement Data Lakehouse Architecture on Azure Data Lake Gen2 with partitioning, schema management, and performance optimization.Develop data models...
-
Senior Data Integration Engineer
5 days ago
Bhopal, India Whatjobs IN C2 Full timeJob Title - Senior Data Integration Engineer Experience - 6+Years Job Summary We are seeking a highly experienced and strategic Senior Data Integration Engineer to architect, build, and manage the data pipelines that power our customer intelligence ecosystem. In this critical role, you will be the subject matter expert responsible for designing and...