Pyspark Azure Databricks Data Engineer

2 days ago

India Crisil Full time ₹ 6,00,000 - ₹ 18,00,000 per year

Job Summary

Crisil Risk Technology group is seeking a strong Python/Pyspark Developer. He/she will be responsible for designing, developing and maintaining scalable data pipeline and ETL process using Azure cloud platform and Azure Databricks.

SCOPE

A key focus of this role is to build and manage advanced data pipelines, perform large-scale data processing, and support analytics and machine learning initiatives. The person will contribute to the growth and scalability of our cloud infrastructure while also managing complex interface development, resolving technical issues, and providing support during weekend maintenance and production operations.

PRIMARY RESPONSIBILITIES

  • Develop and maintain Databricks notebooks using Python/PySpark and SQL
  • Manage & Monitor Databricks clusters and integrate with version control systems such as GitHub.
  • Enable seamless integration between on-premises databases and Power BI for reporting and analytics.
  • Design and build large-scale data pipelines using Azure native data processing frameworks.
  • Collaborate with architects, engineers, analysts, and business stakeholders to deliver enterprise-grade, data-driven solutions.
  • Provide technical leadership and guidance on cloud architecture and implementation strategies.
  • Coordinate with platform teams, Azure API Management (APIM), GitHub, and support teams to ensure smooth operations.
  • Analyze business requirements and design scalable, secure, and efficient solutions on the Azure cloud platform.
  • Develop, test, and optimize software components to enhance the performance and reliability of data platforms.
  • Integrate and standardize data from diverse sources while ensuring compliance with data quality and accessibility standards.
  • Implement streaming data solutions and reusable design patterns in a big data environment.
  • Apply sound judgment and technical expertise to resolve moderately complex data engineering challenges.
  • Review and provide feedback on core code changes and support production deployments.

CORE TECHNOLOGIES

  • Azure: Azure Databricks, Azure Synapse Analytics, Azure Functions, Azure Data Lake Storage Gen2, Azure Event Grid, Azure Event Hubs, Azure Service Bus, Azure Key Vault, Azure Monitor, Azure Log Analytics, Azure API Management (APIM), Azure DevOps.
  • Scripting: PySpark, Python, SQL, Bash.
  • Databases: SQL Server, Oracle, PostgreSQL, Delta Lake
  • Big Data: Apache Spark
  • Version Control:  GitHub, Git, Azure DevOps
  • Visualization: Power BI & Integration with REST APIs for custom dashboards.
  • Data Integration & Workflow Orchestration: Azure Data Factory, Databricks Workflows

QUALIFICATIONS

  • IT professional experience in Azure Cloud with Minimum 5 years of experience in developing and maintaining data pipelines using Python, Spark, and other Big Data technologies.
  • Solid proficiency in Python, PySpark & SQL
  • Strong understanding of ETL processes and data warehousing concepts.
  • Ability to recreate existing legacy application logic and functionality into Azure Databricks/Data Lake, SQL Database and SQL Datawarehouse environment.
  • Experience with Azure services such as Data Factory, Azure Machine Learning, and Azure DevOps.
  • Excellent interpersonal and communication skills
  • Experience with software configuration management tools such as Git/GitHub
  • Working experience with AI is nice to have

  • ▷ (Urgent Search) Azure Databricks

    2 weeks ago

    Chennai, India Viraaj HR Solutions Private Limited Full time

    Job Description Industry & Sector: HR & IT Staffing firm focused on placing cloud data engineering and analytics talent into enterprise clients across finance, retail, and technology sectors. This recruitment drive targets senior cloud data engineers with hands-on experience in Azure Databricks and modern data platform architectures. Position: Azure...

  • Azure Data Engineer

    22 hours ago

    india Tata Consultancy Services Full time

    Greetings from TCS!Job Title: Azure Data EngineerRequired Skillset: Azure Databricks, Pyspark, SQL, ADF, Data LakeLocation: KolkataExperience Range: 4-6 yearsJob DescriptionMust-Have**Azure Databricks, Pyspark, SQL, ADFGood To Have:Python, SnowflakeResponsibility of / Expectations from the Role:Analyzing and translating business needs into long-term solution...

  • Azure Data Engineer

    15 hours ago

    india Tata Consultancy Services Full time

    Greetings from TCS!Job Title: Azure Data EngineerRequired Skillset: Azure Databricks, Pyspark, SQL, ADF, Data LakeLocation: KolkataExperience Range: 4-6 yearsJob DescriptionMust-Have**Azure Databricks, Pyspark, SQL, ADFGood To Have:Python, SnowflakeResponsibility of / Expectations from the Role:Analyzing and translating business needs into long-term solution...

  • Databricks + Pyspark

    1 week ago

    Bhubaneswar, India Cognizant Full time

    Job Description Job Summary Location: AIA Group - Bhubaneswar We are seeking a highly skilled Sr. Developer with 6 to 9 years of experience to join our dynamic team. The ideal candidate will have expertise in Delta Live Pipelines Python Databricks SQL Databricks Workflows and PySpark. This hybrid role involves working on cutting-edge projects that drive...

  • Azure Data Engineer

    2 weeks ago

    India SystemBender Full time

    *Role: Azure Data Engineer*• Advance SQL Azure Data Factory (ADF) Databricks with PySpark Azure Synapse (added advantage) • Fully conversant with big-data processing approaches and schema-on-read methodologies is a must and knowledge of Azure Data Factory / Azure Databricks (PySpark) / Azure Data Lake Storage (ADLS Gen 2) • Good to have an excellent...

  • Azure Data Engineer

    4 days ago

    India SystemBender Full time

    Job Description *Role: Azure Data Engineer* Advance SQL Azure Data Factory (ADF) Databricks with PySpark Azure Synapse (added advantage) Fully conversant with big-data processing approaches and schema-on-read methodologies is a must and knowledge of Azure Data Factory / Azure Databricks (PySpark) / Azure Data Lake Storage (ADLS Gen 2) Good to have an...

  • Azure Data Engineer

    1 week ago

    India SystemBender Full time

    *Role: Azure Data Engineer* • Advance SQL Azure Data Factory (ADF) Databricks with PySpark Azure Synapse (added advantage) • Fully conversant with big-data processing approaches and schema-on-read methodologies is a must and knowledge of Azure Data Factory / Azure Databricks (PySpark) / Azure Data Lake Storage (ADLS Gen 2) • Good to have an excellent...

  • Azure Data Engineer

    1 week ago

    Chennai, India Colan Infotech Private Limited Full time

    Job Description Skills: Azure Data Factory, Azure Synapse Analytics, Azure Databricks, Azure SQL Database, Azure Blob Storage, Data Lake Storage, Azure Data Engineer, Job Description Role: Azure Data Engineer Experience: 3-5 Years Skill Set Pyspark / Scala Spark, Python, Azure Databricks and Azure Data Factory Roles And Responsibilities Should have...

  • Data Engineer with Azure

    1 week ago

    Bengaluru, India Innova ESI Full time

    Job Description Role: Data Engineer with Azure Location Bangalore / Gurugram Type: Fulltime Experience - 9 yrs + Notice:-Immediate Job Description Data Engineer (Azure, ADF, Databricks, PySpark, SCD, Unity Catalog, SQL) Role Overview: Looking for highly skilled experienced Data Engineer with expertise in Azure Data Factory (ADF), Azure Databricks, Delta...

  • (High Salary) Data Engineer – Databricks

    3 weeks ago

    India Epergne Solutions Full time

    Project Overview We are seeking skilled Data Engineers with strong hands-on experience in Databricks for a high-impact data platform project The project involves building scalable data pipelines integrating with cloud platforms and managing data governance using Unity Catalog Key Requirements Primary Skills Expertise in Databricks working on either AWS or...