Data Engineer- Pyspark

1 day ago

Pune, Maharashtra, India Pylon Management Consulting Full time US$ 90,000 - US$ 1,20,000 per year

Data Engineer Job Description:

Job Responsibilities:

  • Design and implement highly scalable, distributed data pipelines using Spark and Delta

Lake.

  • Collaborate with the team to integrate Spark workflows into orchestration tools like

Airflow.

  • Implement best practices for job scheduling, resource allocation, and failure recovery in

Apache Spark.

  • Conduct performance benchmarking and recommend improvements for Apache Spark

tuning based on workload patterns.

  • Monitor cluster health and troubleshoot issues to maintain high availability and stability.
  • Work closely with the data engineering team to define data models and transformations

using DBT Core and Spark SQL.

Required Skills:

  • hands-on experience in building Big Data enterprise grade data platforms.
  • Proven track record of optimizing Apache Spark clusters for large-scale workloads and

complex job patterns.

  • Strong proficiency in Apache Spark Core, Spark SQL, and PySpark.
  • Experience with Delta Lake for managing data lakes and supporting ACID transactions.
  • Familiarity with Debezium with Apache Kafka, Airflow, and DBT Core is a plus.
  • Strong understanding of distributed systems, memory management, and resource

allocation strategies.

  • Solid experience with on-premises infrastructure and ability to collaborate with system

administrators for cluster setup and tuning.

  • Problem-solving mindset and ability to work independently.

Nice to Have:

  • Experience with Open Metadata or similar data governance and discovery tools

  • Pyspark Developer

    23 hours ago

    Pune, Maharashtra, India Tech Mahindra Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Pyspark DeveloperRequirements:Mandatory: Primary skill: Pyspark, Data Engineering, Azure Data BricksGood Experience of Hadoop, Hive, and Cloudera/ Azure/GCP 3+ years of experience in the design and implementation of Big Data systems using PySpark, database migration, transformation and integration solutions for any Data warehousing project.Must have...

  • Data Engineer( Azure Databricks, PySpark/Python )

    24 hours ago

    Pune, Maharashtra, India IDESLABS PRIVATE LIMITED Full time US$ 1,20,000 - US$ 2,00,000 per year

    Responsibilities:designing, developing, and maintaining scalable data pipelines using Databricks, PySpark, Spark SQL, and Delta Live Tables.Collaborate with cross-functional teams to understand data requirements and translate them into efficient data models and pipelines.Implement best practices for data engineering, including data quality, and data...

  • Bigdata -Pyspark /Scala Spark- Data Engineer

    1 day ago

    Pune, Maharashtra, India LTIMindtree Full time US$ 1,50,000 - US$ 2,00,000 per year

    30 days to immediate3+ expJob description:Myrefers Big Data PySpark Scala Spark Java Sparkp3PuneWe are seeking a highly skilled and experienced Big Data Engineer to join our data engineering team The ideal candidate will have a strong background in Hadoop ecosystem tools Apache Spark including SparkSQL and Python programming You will be responsible for...

  • Data Engineer Azure synpase, SQL and pyspark

    6 hours ago

    Pune, Maharashtra, India RM Technologies Full time ₹ 1,04,000 - ₹ 3,30,000 per year

    Experience: 5+ years Location: Hybrid – 4 days/week in office Pune Client: MNC Budget: 20-33 LPA Immediate Joiners Only(Max 30 days)Are you a skilled Data Engineer ready to make an impact in a dynamic work environment? We are seeking a talented individual with expertise in Azure Synapse, Python/PySpark, and SQL to join our team. Your role will involve...

  • Pyspark Lead

    6 hours ago

    Pune, Maharashtra, India Wipro Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Position OverviewWe are seeking a skilled and experienced Senior PySpark Developer with expertise in Apache spark, Spark Batch, and Spark Streaming to join our dynamic team. The ideal candidate will design, develop, and maintain high-performance, scalable applications for processing large-scale data in batch and real-time environments.Required Skills and...

  • Data Engineer

    6 hours ago

    Pune, Maharashtra, India Jash Data Sciences Full time ₹ 8,00,000 - ₹ 12,00,000 per year

    Do you love solving real-world data problems with the latest and best techniques? And having fun while solving them in a team Then come join our high-energy team of passionate data people. Jash Data Sciences is the right place for you.We are a cutting-edge Data Sciences and Data Engineering startup based in Pune, India.We believe in continuous learning and...

  • Data Engineer

    4 weeks ago

    Pune, Maharashtra, India Ascendion Full time

    Job Title: Senior Data EngineerExperience : 6+ years Location: Pune Skills: Python , PySpark, AWS, EKS Required Skill Set: • Strong Data Engineering background • Proficiency in AWS, Python, PySpark • Experience with EKS and distributed data processing Technical Skills - AWS, Python, Microservices, Docker, Kubernetes, PySpark (Must have) - Expertise in...

  • Azure databricks, Pyhton, Pyspark

    24 hours ago

    Pune, Maharashtra, India Avertis Infotech Pvt. Ltd. Full time US$ 90,000 - US$ 1,20,000 per year

    Company DescriptionAvertis Infotech Pvt. Ltd. is an IT services company that began as an application development company specializing in integrating high-tech healthcare machinery. Today, we have expanded our expertise to various industries, adapting to the evolving business landscape. Our goal is to provide innovative solutions and exceptional services to...

  • Data Engineer

    1 week ago

    Pune, Maharashtra, India InfoBeans Full time

    Job Role - Python Data Engineer Experience Required - 8+ Years Mandate Skills - Python, Pyspark and Databricks Location - Indore or Pune (Hybrid) Overview: We are seeking a highly skilled Python and PySpark Data Engineer who is passionate about building robust data solutions from the ground up. The ideal candidate should have a deep understanding of core...

  • Data Engineer

    2 weeks ago

    Pune, Maharashtra, India Mount Talent Consulting Pvt Ltd. Full time

    We are hiring a Data Engineer for Pune/Hyderabad/Bangalore.Experience: 6+ YearsDesignation: Senior Software Engineer/Lead Software Engineer –Data EngineerSkill Tech stack: AWS Data Engineer, Python, PySpark, SQL, Data Pipeline, AWS, AWS Glue, LambdaJD:6+ years of experience in data engineering, specifically in cloud environments like AWS.Proficiency in...