Python Spark Developer

Job Summary

Synechron is seeking a skilled Python Spark Developer to design and optimize large-scale data pipelines and processing systems. The successful candidate will leverage expertise in Python and Apache Spark to build scalable, high-performance data workflows, supporting enterprise analytics, fraud detection, and real-time data applications. This role is instrumental in driving data architecture advancements, operational excellence, and delivering solutions aligned with business and technical standards.

Software Requirements

Required Skills:

• 3+ years of professional experience in Python development with a focus on data engineering and Big Data processing

• Hands-on expertise with Apache Spark (preferably Spark 2.x or 3.x) in batch and streaming environments

• Strong SQL skills with experience working with relational and distributed data systems (e.g., Hive, Snowflake, NoSQL databases)

• Experience with data pipeline orchestration and management tools (e.g., Airflow, Jenkins, Git)

• Solid understanding of software engineering principles, clean code practices, and design patterns

• Familiarity with system design for scalable, data-intensive applications

Preferred Skills:

• Exposure to cloud data platforms such as Snowflake, Databricks, AWS Glue, or GCP DataProc

• Experience working with Kafka, Redis, or similar messaging systems

• Knowledge of observability tools like OpenTelemetry, Grafana, Loki, Tempo

• Understanding of containerization using Docker, orchestration with Kubernetes, and GitOps workflows

Overall Responsibilities

• Design, develop, and optimize scalable data pipelines and workflows utilizing Python and Apache Spark

• Build high-performance data processing applications emphasizing pushdown optimization, partitioning, clustering, and streaming

• Integrate modern data platforms and tools into existing enterprise architectures for improved data accessibility and security

• Engineer feature pipelines to support real-time fraud detection and other critical analytics systems

• Define data models and processing strategies aligned with distributed architecture principles to ensure scalability and consistency

• Develop solutions that are production-ready, maintainable, and feature observability and operational monitoring capabilities

• Adhere to clean code standards, SOLID principles, and architecture best practices to enable extensibility and robustness

• Participate in code reviews, testing, deployment, and performance tuning activities

• Contribute to architectural governance, innovation initiatives, and continuous improvement efforts

Technical Skills (By Category)

Programming Languages:

• Essential: Python (version 3.7+)

• Preferred: Scala, Java for integration purposes

Frameworks & Libraries:

• Essential: Apache Spark, Spark Streaming, Spark SQL, PySpark

• Preferred: Kafka clients, Flink, or other streaming frameworks

Data & Databases:

• Essential: SQL (PostgreSQL, MySQL), Spark dataframes, Hive, or similar distributed storage

• Preferred: NoSQL databases (MongoDB, Cassandra), Data Lake architectures

Cloud & Infrastructure:

• Preferred: Cloud platforms such as Snowflake, Databricks, AWS, or GCP

• Experience with containerization: Docker, Kubernetes, Helm

• Infrastructure automation: Terraform, CloudFormation (desirable)

DevOps & Monitoring:

• Essential: CI/CD (Jenkins, GitHub Actions), observability tools (OpenTelemetry, Prometheus, Grafana)

• Preferred: Log aggregation tools like Loki, Tempo; metrics collection

Experience Requirements

• 3+ years of hands-on experience developing data pipelines in Python with Apache Spark

• Proven experience designing scalable, reliable ETL/ELT workflows in enterprise environments

• Demonstrated ability to optimize Spark jobs for performance in batch and streaming scenarios

• Experience working in distributed system architectures with a focus on data security and compliance

• Background in financial, fraud detection, or data-intensive environments is preferred; relevant industry experience is desirable

• Proven ability to collaborate across cross-functional teams and influence technical decision-making

Day-to-Day Activities

• Develop and maintain large-scale data pipelines supporting enterprise analytics and real-time applications

• Optimize Spark jobs and workflows for throughput, latency, and resource utilization

• Implement pushdown optimizations, partitioning strategies, and clustering techniques to improve data processing efficiency

• Collaborate with data architects, platform teams, and stakeholders to evaluate new tools and platforms for data solutions

• Troubleshoot technical issues, resolve data pipeline failures, and improve system observability

• Conduct code reviews and participate in agile planning, deployment, and operational activities

• Document architecture, processes, and best practices to facilitate knowledge sharing and operational excellence

• Stay current with industry trends, emerging tools, and best practices in big data engineering

Qualifications

• Bachelor's or Master's degree in Computer Science, Software Engineering, Data Science, or related field

• Additional certifications in Big Data, Spark, or cloud data services are a plus

• Extensive hands-on experience developing large-scale data pipelines and processing solutions with Python and Apache Spark

Professional Competencies

• Strong analytical and problem-solving skills for complex data workflows

• Excellent collaboration and communication skills with technical and non-technical stakeholders

• Ability to lead initiatives, influence best practices, and mentor junior engineers

• Adaptability to evolving technologies and organizational needs

• Focus on operational excellence, observability, and sustained performance

• Commitment to continuous learning and process improvement

S​YNECHRON'S DIVERSITY & INCLUSION STATEMENT
 

Diversity & Inclusion are fundamental to our culture, and Synechron is proud to be an equal opportunity workplace and is an affirmative action employer. Our Diversity, Equity, and Inclusion (DEI) initiative 'Same Difference' is committed to fostering an inclusive culture – promoting equality, diversity and an environment that is respectful to all. We strongly believe that a diverse workforce helps build stronger, successful businesses as a global company. We encourage applicants from across diverse backgrounds, race, ethnicities, religion, age, marital status, gender, sexual orientations, or disabilities to apply. We empower our global workforce by offering flexible workplace arrangements, mentoring, internal mobility, learning and development programs, and more.

All employment decisions at Synechron are based on business needs, job requirements and individual qualifications, without regard to the applicant's gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law.

Candidate Application Notice