Python | Pyspark| Databricks

2 weeks ago

Pune, Maharashtra, India Michael Page Full time
Competetive SalaryPF and Gratuity

About Our Client

Our client is an international professional services brand of firms, operating as partnerships under the brand. It is the second-largest professional services network in the world.

Job Description

Total Exp- 6+yearsKey Responsibilities:

Develop and optimize ETL pipelines using PySpark and Databricks.Work with large-scale datasets and build distributed computing solutions.Design and implement data ingestion, transformation, and processing workflows.Write efficient and scalable Python code for data processing.Collaborate with data engineers, data scientists, and business teams to deliver insights.Optimize performance and cost efficiency for big data solutions.Implement best practices for CI/CD, testing, and automation in a cloud environment.Monitor job performance, troubleshooting failures, and tuning queries.

Required Skills & Qualifications:

6years of experience in Python programming for data engineering.Strong hands-on experience with Apache Spark (PySpark) for data transformation.Experience working with Databricks for data processing and analytics.Good knowledge of SQL and NoSQL databases.Experience with Cloud platforms (Azure, AWS, or GCP) and storage solutions.Familiarity with CI/CD pipelines, DevOps practices, and automation tools.Understanding of data lake, data warehouse, and data modeling concepts.Knowledge of job orchestration tools like Airflow, Azure Data Factory, or AWS Glue.Strong problem-solving and debugging skills in distributed environments.

Good to Have:

Experience with Delta Lake and Lakehouse architecture.Familiarity with Kafka or real-time data processing.Knowledge of MLflow and integrating ML models in Databricks.Experience with Scala for Spark-based applications.

The Successful Applicant

Total Exp- 6+yearsKey Responsibilities:

Develop and optimize ETL pipelines using PySpark and Databricks.Work with large-scale datasets and build distributed computing solutions.Design and implement data ingestion, transformation, and processing workflows.Write efficient and scalable Python code for data processing.Collaborate with data engineers, data scientists, and business teams to deliver insights.Optimize performance and cost efficiency for big data solutions.Implement best practices for CI/CD, testing, and automation in a cloud environment.Monitor job performance, troubleshooting failures, and tuning queries.

Required Skills & Qualifications:

6years of experience in Python programming for data engineering.Strong hands-on experience with Apache Spark (PySpark) for data transformation.Experience working with Databricks for data processing and analytics.Good knowledge of SQL and NoSQL databases.Experience with Cloud platforms (Azure, AWS, or GCP) and storage solutions.Familiarity with CI/CD pipelines, DevOps practices, and automation tools.Understanding of data lake, data warehouse, and data modeling concepts.Knowledge of job orchestration tools like Airflow, Azure Data Factory, or AWS Glue.Strong problem-solving and debugging skills in distributed environments.

Good to Have:

Experience with Delta Lake and Lakehouse architecture.Familiarity with Kafka or real-time data processing.Knowledge of MLflow and integrating ML models in Databricks.Experience with Scala for Spark-based applications.

  • Pyspark Azure Databricks

    4 weeks ago

    Pune, Maharashtra, India Tata Consultancy Services Full time

    Greetings from Tata Consulting ServicesTCS is Hiring for Pyspark DatabricksExperience : 6- 10 Years Location: Pune onlyPlease find the JD belowRequired Technical Skill - Pyspark DatabricksData Engineer with professional experience on Databricks, managing data pipelines and infrastructure within Databricks Lake house Platform.Proven expertise in Databricks,...

  • Databricks Data Engineer

    3 weeks ago

    Pune, Maharashtra, India Ascendion Full time

    Job Title: Senior Data Engineer Experience : 3+ years Location: Gurgaon Skills: PySpark, SQL, Databricks, AWS. Role Summary: We are looking for 3–4 experienced Databricks Developers to support a fast-paced, high-impact data engineering initiative. The ideal candidates should have hands-on expertise in building scalable data pipelines using...

  • Databricks Data Engineer

    3 days ago

    Pune, Maharashtra, India Ascendion Full time

    Job Title: Senior Data EngineerExperience : 3+ yearsLocation: Gurgaon /Pune / BangaloreSkills: PySpark, SQL, Databricks, AWS.Role Summary:We are looking for 3–4 experienced Databricks Developers to support a fast-paced, high-impact data engineering initiative. The ideal candidates should have hands-on expertise in building scalable data pipelines using...

  • Databricks Data Engineer

    5 hours ago

    Pune, Maharashtra, India Ascendion Full time

    Job Title: Senior Data Engineer Experience : 3+ years Location: Gurgaon /Pune / Bangalore Skills: PySpark, SQL, Databricks, AWS. Role Summary: We are looking for 3–4 experienced Databricks Developers to support a fast-paced, high-impact data engineering initiative. The ideal candidates should have hands-on expertise in building scalable data...

  • Azure Databricks

    4 weeks ago

    Pune, Maharashtra, India People Prime Worldwide Full time

    Our client is a global technology company headquartered in Santa Clara, California. it focuses on helping organisations harness the power of data to drive digital transformation, enhance operational efficiency, and achieve sustainability. over 100 years of experience in operational technology (OT) and more than 60 years in IT to unlock the power of data from...

  • Databricks Lead

    2 days ago

    Pune, Maharashtra, India beBeeData Full time ₹ 9,00,000 - ₹ 12,00,000

    Senior Databricks Engineer OpportunityWe're seeking a seasoned Databricks expert to lead our Azure data engineering projects.Develop scalable data pipelines with Databricks, PySpark & Spark SQLImplement & manage Delta Lake & modern data lakehouse architectureConfigure & administer Databricks (Workspace, Jobs, DLT, Unity Catalog)Design CI/CD pipelines using...

  • Databricks Data Engineer

    5 days ago

    Pune, Maharashtra, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title:Data EngineerAbout the Role:We are seeking a highly skilled Data Engineer to join our team. The ideal candidate will have hands-on experience with Databricks, including expertise in Databricks SQL, PySpark, and Spark SQL.Key Responsibilities:Migrate SQL Server Stored Procedures to Databricks Notebooks, leveraging PySpark and Spark SQL for complex...

  • Senior Databricks Engineer

    2 days ago

    Pune, Maharashtra, India Newscape Consulting Full time

    We're Hiring: Senior Databricks Engineer / Azure Data Engineer Location: Baner, Pune Work Mode: Onsite Availability: Immediate Joiners / 15 Days NP Are you passionate about building scalable, high-performance data platforms? We're looking for a Senior Databricks Engineer / Azure Data Engineer to join our growing team and work on cutting-edge Azure Data...

  • Senior Databricks Engineer

    3 days ago

    Pune, Maharashtra, India Newscape Consulting Full time

    We're Hiring: Senior Databricks Engineer / Azure Data Engineer Location: Baner, Pune Work Mode: Onsite Availability: Immediate Joiners / 15 Days NPAre you passionate about building scalable, high-performance data platforms?We're looking for a Senior Databricks Engineer / Azure Data Engineer to join our growing team and work on cutting-edge Azure Data...

  • Databricks Opportunity

    3 weeks ago

    Pune, Maharashtra, India Infinite Computer Solutions Full time

    Job Description- Experience as a Databricks Development.- Hands-on experience in Python/ Java- Good in Pyspark/ Java Spark.- Experience in Advanced SQL Or SQLExperience: - 6-10 YearsLocation: - Pune/ BangaloreNote:- 5 Days working from OfficeOpen positions:- 10About Infinite.Infinite Computer Solutions Ltd. is a US$ 180 million, assessed for SEI CMMi Level...