Senior PySpark Engineer

1 day ago

Hyderabad, Telangana, India Rapsys Technologies Pte. Ltd Full time ₹ 6,00,000 - ₹ 27,00,000 per year
  • Role: Senior PySpark Engineer
  • Experience Required: Minimum 8+ Years
  • Work Location: Hyderabad (5 Days Work from Office)
  • Job Type: Contract to Hire (1 Year/ Renewable)
  • Notice Period: Immediate to 15 Days max
  • Mode of Interview: Virtual

We are seeking a highly skilled PySpark Data Engineer to design, build, and optimize large-scale data pipelines and distributed systems. Beyond deep expertise in Apache Spark (PySpark) and automation, this role requires the ability to manage stakeholders, ensure timely delivery, and assess requirements. You will play a critical role in bridging business needs with technical execution, ensuring high-quality, scalable, and reliable data solutions. Cloudera PySpark experience is preferred.

KEY RESPONSIBILITIES:

  • Architect and guide the refactoring of legacy PySpark scripts into modular, reusable, and configuration-driven frameworks aligned with enterprise standards.
  • Lead migration efforts to Spark 3.3+ and Python 3.10+, ensuring compatibility, performance, and maintainability across distributed systems.
  • Drive modernization by replacing deprecated APIs (e.g., RDDs, legacy UDFs) with efficient DataFrame operations and Pandas UDFs, promoting best practices.
  • Establish and enforce structured logging, robust error handling, and proactive alerting mechanisms for operational resilience.
  • Oversee performance tuning, including partitioning strategies, broadcast joins, and predicate pushdown, to optimize Spark execution plans.
  • Ensure data integrity through schema enforcement, data type consistency, and accurate implementation of Slowly Changing Dimensions (SCD) logic.
  • Collaborate with DevOps and QA teams to integrate Spark workloads into CI/CD pipelines and automated testing frameworks.
  • Mentor and conduct code reviews, providing technical guidance and resolving complex findings to uphold code quality and team growth.
  • Lead performance benchmarking and regression testing initiatives to validate scalability and reliability of Spark applications.
  • Coordinate deployment planning, runbook creation, and production handover, ensuring smooth transitions and operational readiness.
  • Engage with stakeholders to translate business requirements into scalable data processing solutions and contribute to data platform strategy.

Educational Qualification:

  • Graduate/Masters in software engineering/IT/Computer Science or equivalent.

Technical Skills:

PySpark Development (5-7 Years)

  • Refactoring legacy scripts, using DataFrame APIs, avoiding .collect()or equivalent

Spark Optimization (3-5 Years)

  • Broadcast joins, partitioning strategy, predicate pushdown

Pyspark Migration activity (2 Years)

  • Prior experience with Pyspark migration activity.

Testing Frameworks (1+ Years)

  • Pytest, Great Expectations, Deequ for unit/integration/performance testing

Job Type: Contractual / Temporary

Contract length: 12 months

Pay: ₹600, ₹2,700,000.00 per year

Work Location: In person

  • Senior Pyspark Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India DATAECONOMY Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    About UsAbout DATAECONOMY: We are a fast-growing data & analytics company headquartered in Dublin with offices inDublin, OH, Providence, RI, and an advanced technology center in Hyderabad,India. We are clearly differentiated in the data & analytics space via our suite of solutions, accelerators, frameworks, and thought leadership.Job DescriptionJob...

  • Senior Pyspark Data Engineer

    3 days ago

    Hyderabad, Telangana, India DataEconomy Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Job InformationDate Opened10/13/2025Job TypeFull timeIndustryIT ServicesCityHyderabadState/ProvinceTelanganaCountryIndiaZip/Postal Code500081About UsAbout DATAECONOMY: We are a fast-growing data & analytics company headquartered in Dublin with offices inDublin, OH, Providence, RI, and an advanced technology center in Hyderabad,India. We are clearly...

  • PySpark Engineer

    1 week ago

    Hyderabad, Telangana, India Rapsys Technologies Pte. Ltd Full time ₹ 6,00,000 - ₹ 25,00,000 per year

    Role: PySpark EngineerExperience Required: Minimum 5-10 YearsWork Location: Hyderabad (5 Days Work from Office)Job Type: Contract to Hire (1 Year/ Renewable)Notice Period: Immediate to 30 Days maxRequired skill sets and expertise:5+ years of experience in PySpark, Big Data technologies, Data Warehousing.Strong Python programming experience.Domain: Experience...

  • Pyspark Developer

    2 weeks ago

    Hyderabad, Telangana, India NTT DATA Business Solutions Full time ₹ 8,00,000 - ₹ 12,00,000 per year

    Job SummaryWe are looking for a Senior PySpark Developer with 3 to 6 years of experience in building and optimizing data pipelines using PySpark on Databricks, within AWS cloud environments. This role focuses on the modernization of legacy domains, involving integration with systems like Kafka and collaboration across cross-functional teams.Key...

  • Databricks + Pyspark

    6 days ago

    Hyderabad, Telangana, India Cognizant Full time ₹ 1,00,00,000 - ₹ 3,00,00,000 per year

    Skills- Databricks+ PysparkExperience: 4 to 13 yearsLocation: AIA-PuneWe are looking for a highly skilled Data Engineer with expertise in PySpark and Databricks to design, build, and optimize scalable data pipelines for processing massive datasets.Key Responsibilities:Build & Optimize Pipelines: Develop high-throughput ETL workflows using PySpark on...

  • Lead Pyspark Data Engineer

    1 day ago

    Hyderabad, Telangana, India DATAECONOMY Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job Title: PySpark Data EngineerExperience: 10+ YearsLocation: Hyderabad/ PuneEmployment Type: Full-TimeJob Summary:We are looking for a skilled and experienced PySpark Data Engineer to join our growing data engineering team. The ideal candidate will have 10+ years of experience in designing and implementing data pipelines using PySpark, AWS Glue,...

  • Senior PySpark/Python Developer

    1 week ago

    Hyderabad, Telangana, India Zorba Consulting Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    About The Role :We are seeking a highly skilled and experienced Senior PySpark/Python Developer to play a critical role in building a robust and reliable system for managing and disseminating customer notifications regarding PG&E's Planned Power Outages (PPOs). This is an exciting opportunity to tackle complex data challenges within a dynamic environment and...

  • Pyspark + Databricks

    6 days ago

    Hyderabad, Telangana, India Cognizant Technology Solutions Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job SummaryWe are seeking a highly skilled Sr. Developer with 6 to 10 years of experience to join our team. The ideal candidate will have expertise in Databricks SQL Databricks Workflows and PySpark. Experience in the Cards & Payments domain is a plus. This is a hybrid work model with day shifts and no travel required.ResponsibilitiesDevelop and maintain...

  • Senior Consultant-Pyspark

    2 weeks ago

    Hyderabad, Telangana, India Deloitte Full time ₹ 8,00,000 - ₹ 12,00,000 per year

    SummaryPosition SummaryArtificial Intelligence & EngineeringAI & Engineering leverages cutting-edge engineering capabilities to help build, deploy, and operate integrated/verticalized sector solutions in software, data, AI, network, and hybrid cloud infrastructure. These solutions insights are powered by engineering for business advantage, helping...

  • Python Pyspark Developer

    2 weeks ago

    Hyderabad, Telangana, India MatchPoint Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Type: PermanentLocation: Hyderabad (5 Days Onsite)Experience RequiredJob Description:We are seeking a skilledData Engineerwith strong expertise inPySpark Quries, Python programming, and Data Structures & Algorithms (DSA). The ideal candidate will design and optimize large-scale data pipelines, write efficient and scalable code, and solve complex data...