Pyspark Azure Databricks Data Engineer

3 days ago

India Crisil Full time ₹ 6,00,000 - ₹ 18,00,000 per year

Job Summary

Crisil Risk Technology group is seeking a strong Python/Pyspark Developer. He/she will be responsible for designing, developing and maintaining scalable data pipeline and ETL process using Azure cloud platform and Azure Databricks.

SCOPE

A key focus of this role is to build and manage advanced data pipelines, perform large-scale data processing, and support analytics and machine learning initiatives. The person will contribute to the growth and scalability of our cloud infrastructure while also managing complex interface development, resolving technical issues, and providing support during weekend maintenance and production operations.

PRIMARY RESPONSIBILITIES

  • Develop and maintain Databricks notebooks using Python/PySpark and SQL
  • Manage & Monitor Databricks clusters and integrate with version control systems such as GitHub.
  • Enable seamless integration between on-premises databases and Power BI for reporting and analytics.
  • Design and build large-scale data pipelines using Azure native data processing frameworks.
  • Collaborate with architects, engineers, analysts, and business stakeholders to deliver enterprise-grade, data-driven solutions.
  • Provide technical leadership and guidance on cloud architecture and implementation strategies.
  • Coordinate with platform teams, Azure API Management (APIM), GitHub, and support teams to ensure smooth operations.
  • Analyze business requirements and design scalable, secure, and efficient solutions on the Azure cloud platform.
  • Develop, test, and optimize software components to enhance the performance and reliability of data platforms.
  • Integrate and standardize data from diverse sources while ensuring compliance with data quality and accessibility standards.
  • Implement streaming data solutions and reusable design patterns in a big data environment.
  • Apply sound judgment and technical expertise to resolve moderately complex data engineering challenges.
  • Review and provide feedback on core code changes and support production deployments.

CORE TECHNOLOGIES

  • Azure: Azure Databricks, Azure Synapse Analytics, Azure Functions, Azure Data Lake Storage Gen2, Azure Event Grid, Azure Event Hubs, Azure Service Bus, Azure Key Vault, Azure Monitor, Azure Log Analytics, Azure API Management (APIM), Azure DevOps.
  • Scripting: PySpark, Python, SQL, Bash.
  • Databases: SQL Server, Oracle, PostgreSQL, Delta Lake
  • Big Data: Apache Spark
  • Version Control:  GitHub, Git, Azure DevOps
  • Visualization: Power BI & Integration with REST APIs for custom dashboards.
  • Data Integration & Workflow Orchestration: Azure Data Factory, Databricks Workflows

QUALIFICATIONS

  • IT professional experience in Azure Cloud with Minimum 5 years of experience in developing and maintaining data pipelines using Python, Spark, and other Big Data technologies.
  • Solid proficiency in Python, PySpark & SQL
  • Strong understanding of ETL processes and data warehousing concepts.
  • Ability to recreate existing legacy application logic and functionality into Azure Databricks/Data Lake, SQL Database and SQL Datawarehouse environment.
  • Experience with Azure services such as Data Factory, Azure Machine Learning, and Azure DevOps.
  • Excellent interpersonal and communication skills
  • Experience with software configuration management tools such as Git/GitHub
  • Working experience with AI is nice to have

  • Azure databricks, PySpark

    4 weeks ago

    Bengaluru, Karnataka, India, Karnataka Tata Consultancy Services Full time

    Role: Azure Data Engineer(ADF, ADB)Required Technical Skill Set: Azure Data Engineer, ADF, Azure Databricks Spark (PySpark or Scala), Python, PL/SQLExp : 4 to 6 YearLocation : Kolkata, Pune, Mumbai, Bangalore, BhubaneshwarMust-Have · Strong experience in Azure Data Factory , ADB( Azure Databricks) Synapse; establishing the cloud connectivity between...

  • Data Engineering Azure databricks

    4 weeks ago

    india, IN EXL Full time

    Data Engineer (DE) Consultant is responsible for designing, developing, and maintaining data assets and data related products by liaising with multiple stakeholders.Responsibilities:Work with stakeholders to understand the data requirements to design, develop, and maintain complex ETL processes.Create the data integration and data diagram documentation.Lead...

  • Data Engineer – Databricks

    3 weeks ago

    Bengaluru, Karnataka, India, Karnataka Capgemini Engineering Full time

    Data Engineer – Databricks & PySparkLocation: BangaloreExperience: 6–9 yearsChoosing Capgemini means choosing a place where you’ll be empowered to shape your career, supported by a collaborative global community, and inspired to reimagine what’s possible. Join us in helping leading Consumer Products and Retail Services (CPRS) organizations unlock the...

  • Azure Data Engineer

    3 weeks ago

    Pune, India Fragma Data Systems Full time

    Job Description Technology Skills - Building and operationalizing large scale enterprise data solutions and applications using one or more of AZURE data and analytics services in combination with custom solutions - Azure Synapse/Azure SQL DWH, Azure Data Lake, Azure Blob Storage, Spark, HDInsights, Databricks, CosmosDB, EventHub/IOTHub. - Experience in...

  • Azure data engineer

    3 weeks ago

    india Tata Consultancy Services Full time

    Dear CandidateGreetings from TATA Consultancy ServicesJob Openings at TCSSkill: Azure data engineer (Pyspark , databricks , adf)Exp range: 7 yrs to 10 yrsLocation: Hyderabad, Kolkata, Bangalore, Pune, Chennai, DelhiNotice period – 45Pls find the Job Description below.- Experience in Pyspark and Azure Databricks. - · Proficiency in integrating,...

  • Azure Databricks Engineer

    4 weeks ago

    Chennai, Tamil Nadu, India, Tamil Nadu Tata Consultancy Services Full time

    Role**Databricks Engineer (Individual Contributor)Required Technical Skill Set**Azure Databricks, PysparkExperience: 4 to 10 YearsLocation: Hyd,Kolkata, Noida, Chennai, BangaloreDesired Competencies (Technical/Behavioral Competency)Must-Have**1. Azure Data Bricks (Python)2. Pyspark3. ADLS4. complex problem-solving skillGood-to-HaveAzure DevopsAzure Data...

  • Azure Databricks Developer

    1 week ago

    India Adastra Corp Full time ₹ 15,00,000 - ₹ 28,00,000 per year

    Role & responsibilitiesAs an Azure Databricks Developer with Terraform, your responsibilities include:Designing and optimizing data pipelines using Azure Databricks (Spark, Delta Lake, notebooks, jobs)Automating infrastructure provisioning on Azure through TerraformCollaborating with data engineers, analysts, and cloud architects to integrate Databricks with...

  • Azure data engineer

    4 weeks ago

    india Tata Consultancy Services Full time

    Dear CandidateGreetings from TATA Consultancy Services Job Openings at TCS Skill : Azure data engineer (Pyspark , databricks , adf) Exp range:7 yrs to 10 yrs Location:Hyderabad, Kolkata, Bangalore, Pune, Chennai, Delhi Notice period– 45Pls find the Job Description below. Experience in Pyspark and Azure Databricks. · Proficiency in integrating,...

  • Azure Data Engineer

    1 week ago

    India Hungry Hr Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    * 4–8 yrs exp as Azure Data Engineer * Strong in Azure Databricks, Data Lake & Data Factory * Experience with Azure Synapse Analytics * Hands-on with PySpark & Python * Build & optimize scalable data pipelines * 6Months Contract & Extendable job

  • Azure Databricks

    3 weeks ago

    Noida, Uttar Pradesh, India, Ghaziabad Tata Consultancy Services Full time

    Greetings From TCS!!!"Opportunities don't happen, you create them"Job Title: Azure DatabricksExp Range: 4-10 YearsJob Location: Noida or Bangalore or Chennai or Hyderabad or KolkataJob Description:Required skill: Azure Databricks, PySpark and Python.Must have: 1.Azure Data Bricks (Python)2.Pyspark3. ADLS4. complex problem solving skillGood to Have: 1.Azure...