Senior PySpark Engineer

2 days ago

Hyderabad, Telangana, India Rapsys Technologies Pte. Ltd Full time
  • Role: Senior PySpark Engineer
  • Experience Required: Minimum 8+ Years
  • Work Location: Hyderabad (5 Days Work from Office)
  • Job Type: Contract to Hire (1 Year/ Renewable)
  • Notice Period: Immediate to 15 Days max
  • Mode of Interview: Virtual

We are seeking a highly skilled PySpark Data Engineer to design, build, and optimize large-scale data pipelines and distributed systems. Beyond deep expertise in Apache Spark (PySpark) and automation, this role requires the ability to manage stakeholders, ensure timely delivery, and assess requirements. You will play a critical role in bridging business needs with technical execution, ensuring high-quality, scalable, and reliable data solutions. Cloudera PySpark experience is preferred.

KEY RESPONSIBILITIES:

  • Architect and guide the refactoring of legacy PySpark scripts into modular, reusable, and configuration-driven frameworks aligned with enterprise standards.
  • Lead migration efforts to Spark 3.3+ and Python 3.10+, ensuring compatibility, performance, and maintainability across distributed systems.
  • Drive modernization by replacing deprecated APIs (e.g., RDDs, legacy UDFs) with efficient DataFrame operations and Pandas UDFs, promoting best practices.
  • Establish and enforce structured logging, robust error handling, and proactive alerting mechanisms for operational resilience.
  • Oversee performance tuning, including partitioning strategies, broadcast joins, and predicate pushdown, to optimize Spark execution plans.
  • Ensure data integrity through schema enforcement, data type consistency, and accurate implementation of Slowly Changing Dimensions (SCD) logic.
  • Collaborate with DevOps and QA teams to integrate Spark workloads into CI/CD pipelines and automated testing frameworks.
  • Mentor and conduct code reviews, providing technical guidance and resolving complex findings to uphold code quality and team growth.
  • Lead performance benchmarking and regression testing initiatives to validate scalability and reliability of Spark applications.
  • Coordinate deployment planning, runbook creation, and production handover, ensuring smooth transitions and operational readiness.
  • Engage with stakeholders to translate business requirements into scalable data processing solutions and contribute to data platform strategy.

Educational Qualification:

  • Graduate/Masters in software engineering/IT/Computer Science or equivalent.

Technical Skills:

PySpark Development (5-7 Years)

  • Refactoring legacy scripts, using DataFrame APIs, avoiding .collect()or equivalent

Spark Optimization (3-5 Years)

  • Broadcast joins, partitioning strategy, predicate pushdown

Pyspark Migration activity (2 Years)

  • Prior experience with Pyspark migration activity.

Testing Frameworks (1+ Years)

  • Pytest, Great Expectations, Deequ for unit/integration/performance testing

Job Type: Contractual / Temporary

Contract length: 12 months

Pay: ₹600, ₹2,700,000.00 per year

Work Location: In person

  • PySpark Engineer

    2 weeks ago

    Hyderabad, Telangana, India Rapsys Technologies Pte. Ltd Full time ₹ 6,00,000 - ₹ 25,00,000 per year

    Role: PySpark EngineerExperience Required: Minimum 5-10 YearsWork Location: Hyderabad (5 Days Work from Office)Job Type: Contract to Hire (1 Year/ Renewable)Notice Period: Immediate to 30 Days maxRequired skill sets and expertise:5+ years of experience in PySpark, Big Data technologies, Data Warehousing.Strong Python programming experience.Domain: Experience...

  • Pyspark Developer

    2 days ago

    Hyderabad, Telangana, India Risk Resources LLP Full time

    Job Summary:We are seeking an experienced Pyspark Developer to join our team. As a Pyspark Developer, you will be responsible for designing, developing, and implementing data processing and analytics solutions using Pyspark, Big Data, and AWS technologies.Responsibilities:Develop and maintain data processing and analytics solutions using Pyspark, Spark, and...

  • PySpark Developer

    3 days ago

    Hyderabad, Telangana, India Algoleap Technologies Full time

    SUMMARY Job SummaryWe are looking for a Senior PySpark Developer with 3 to 6 years of experience in building and optimizing data pipelines using PySpark on Databricks, within AWS cloud environments. This role focuses on the modernization of legacy domains, involving integration with systems like Kafka and collaboration across cross-functional teams.Key...

  • Databricks + Pyspark

    7 days ago

    Hyderabad, Telangana, India Cognizant Full time

    Skills- Databricks+ PysparkExperience: 4 to 13 yearsLocation: AIA-PuneWe are looking for a highly skilled Data Engineer with expertise in PySpark and Databricks to design, build, and optimize scalable data pipelines for processing massive datasets.Key Responsibilities:Build & Optimize Pipelines: Develop high-throughput ETL workflows using PySpark on...

  • PySpark Developer

    5 days ago

    Hyderabad, Telangana, India Algoleap Technologies Pvt Ltd Full time

    Job SummaryWe are looking for a Senior PySpark Developer with 3 to 6 years of experience in building and optimizing data pipelines using PySpark on Databricks, within AWS cloud environments. This role focuses on the modernization of legacy domains, involving integration with systems like Kafka and collaboration across cross-functional teams.Key...

  • EverestDX - Senior Data Engineer - Python/PySpark

    2 days ago

    Hyderabad, Telangana, India EverestDX Inc Full time

    DescriptionJob Summary :We are seeking an experienced Azure Data Engineer / Lead with strong expertise in Azure services, distributed data processing, and team leadership.The ideal candidate will have hands-on experience with Azure Databricks, Azure Data Lake, Azure Data Factory, PySpark/Spark, and SQL, along with the ability to guide a small team on...

  • Lead/ Sr. Pyspark Data Engineer

    2 days ago

    Hyderabad, Telangana, India DATAECONOMY Full time

    Job Title: PySpark Data EngineerExperience: 6+ YearsLocation: Hyderabad/ PuneEmployment Type: Full-TimeJob Summary:We are looking for a skilled and experienced PySpark Data Engineer to join our growing data engineering team. The ideal candidate will have 6+ years of experience in designing and implementing data pipelines using PySpark, AWS Glue,...

  • Senior Data Engineer

    1 week ago

    Hyderabad, Telangana, India Valzo Soft Solutions Full time

    Job Description– Senior Data EngineerExperience:5–7 YearsLocation:Hyderabad, Telangana, IndiaEmployment Type:Full-TimeJoining:Immediate HiringPosition OverviewWe are hiring a Senior Data Engineer to lead the development and optimization of scalable ETL/ELT pipelines using Azure Data Factory, Databricks, and PySpark. The role involves managing Data Lake...

  • ETL,Pyspark/Senior Software Engineer

    2 weeks ago

    Hyderabad, Telangana, India HSBC Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Some careers shine brighter than others.If you're looking for a career that will help you stand out, join HSBC and fulfil your potential. Whether you want a career that could take you to the top, or simply take you in an exciting new direction, HSBC offers opportunities, support and rewards that will take you further.HSBC is one of the largest banking and...

  • ETL,Pyspark/Senior Software Engineer

    3 days ago

    Hyderabad, Telangana, India HSBC Full time

    Some careers shine brighter than others.If you're looking for a career that will help you stand out, join HSBC and fulfil your potential. Whether you want a career that could take you to the top, or simply take you in an exciting new direction, HSBC offers opportunities, support and rewards that will take you further.HSBC is one of the largest banking and...