Pyspark Developer
13 hours ago
Primary Job Title: PySpark Developer
Role & Responsibilities- Design, develop and maintain scalable PySpark ETL pipelines for batch and streaming workloads across data lakes and warehouses.
- Ingest and transform data from diverse sources (Kafka, S3, HDFS, RDBMS) into analytics-ready formats using PySpark and Spark SQL.
- Optimize Spark jobs for performance and cost—tune partitioning, memory, and compute configurations; implement job-level and cluster-level optimizations.
- Integrate pipelines with orchestration and CI/CD systems (Apache Airflow, GitLab/GitHub CI), and implement unit/integration tests for data workflows.
- Implement data quality checks, monitoring, logging and alerting to ensure pipeline reliability and SLA adherence.
- Collaborate with data scientists, analysts and platform teams to define data schemas, metadata, and best-practice engineering playbooks.
Must-Have
- PySpark
- Apache Spark
- Python
- Spark SQL
- Apache Hive
- Apache Airflow
Preferred
- Delta Lake
- Apache Kafka
- AWS
-
Pyspark Developer
4 days ago
Chennai, Tamil Nadu, India Citi Full time ₹ 12,00,000 - ₹ 36,00,000 per yearDiscover your future at CitiWorking at Citi is far more than just a job. A career with us means joining a team of more than 230,000 dedicated people from around the globe. At Citi, you'll have the opportunity to grow your career, give back to your community and make a real impact.Job OverviewAt Citi we're not just building technology, we're building the...
-
Pyspark
1 week ago
Chennai, Tamil Nadu, India Cognizant Technology Solutions Full time ₹ 12,00,000 - ₹ 36,00,000 per yearJob SummaryAs a Sr. Developer you will play a pivotal role in optimizing and enhancing our data processing capabilities using PySpark and AWS technologies. Your expertise in Python and domain knowledge in Cards & Payments will drive impactful solutions. This hybrid role offers the flexibility of day shifts ensuring a balanced work-life...
-
AWS + Pyspark
1 week ago
Chennai, Tamil Nadu, India Cognizant Full time ₹ 20,00,000 - ₹ 25,00,000 per yearSkills- AWS +PysparkExperience: 4 to 9 yearsLocation: AIA-PuneAs an AWS Data Engineer, you will design, build, and manage robust data pipelines on the AWS cloud platform. You will leverage your expertise in AWS data services and programming languages to ingest, process, transform, and store data efficiently, enabling data-driven insights and...
-
databricks/pyspark
2 weeks ago
Chennai, Tamil Nadu, India NRM Analytix Full time ₹ 4,00,000 - ₹ 12,00,000 per yearResponsibilities:* Design, develop & maintain PySpark solutions using Databricks platform* Optimize performance through efficient data processing techniques* Collaborate with cross-functional teams on project deliveryAccessible workspace
-
Pyspark, AWS
1 week ago
Chennai, Tamil Nadu, India Cognizant Technology Solutions Full time ₹ 15,00,000 - ₹ 25,00,000 per yearJob SummaryWe are seeking a skilled Developer with 4 to 6 years of experience to join our team. The ideal candidate will have expertise in Amazon S3 and PySpark and a strong background in the Cards & Payments domain. This hybrid role offers the opportunity to work on cutting-edge projects in a dynamic environment contributing to the companys growth and...
-
Chennai, Tamil Nadu, India Tata Consultancy Servicess Full time ₹ 20,00,000 - ₹ 25,00,000 per yearSkill : PySpark DeveloperExperience Range: 4 to 8Notice : (Immediate to 60 Days)Job Description:Technical Skills: Spark ,PySpark, Python, Hive, Scala, MapReduce, Unix shell scriptingob Description:4+ Years of experience with Developing/Fine tuning and implementing programs/applicationsUsing Python/PySpark/Scala on Big Data/Hadoop Platform.Roles and...
-
Pyspark - Machine Learning
5 hours ago
Chennai, Tamil Nadu, India Virtusa Full time ₹ 20,00,000 - ₹ 25,00,000 per year7+ years of experience in Big Data with strong expertise in Spark and ScalaMandatory Skills: Big Data Primarily Spark and ScalaStrong Knowledge in HDFS, Hive, Impala with knowledge on Unix , Oracle, Autosys,Good to Have : Agile Methodology and Banking ExpertiseStrong Communication SkillsNot limited to Spark batch, need Spark streaming experienceNo SQL DB...
-
PySpark developer_CBS
7 hours ago
Chennai, Tamil Nadu, India Coders Brain Technology Full time ₹ 6,00,000 - ₹ 18,00,000 per yearLocation : Chennai/Hyderabad/Bangalore Details JD: 1. Minimum 6+ years of experience in creating SPARK Jobs using Java/Scala 2. Should have very good experience in developing data loading and transformation tasks using external sources, merge data, perform data enrichment and load in to target data destinations 3. Must have good knowledge on Big data...
-
Chennai, Tamil Nadu, India, Tamil Nadu Tata Consultancy Servicess Full timeSkill : PySpark DeveloperExperience Range: 4 to 8Notice : (Immediate to 60 Days)Job Description:Technical Skills: Spark ,PySpark, Python, Hive, Scala, MapReduce, Unix shell scriptingob Description:4+ Years of experience with Developing/Fine tuning and implementing programs/applicationsUsing Python/PySpark/Scala on Big Data/Hadoop Platform.Roles and...
-
Pyspark Developer
4 weeks ago
Chennai, Tamil Nadu, India, Tamil Nadu Tata Consultancy Services Full timeTCS is Hiring !!!Location :Bangalore, Chennai, Kolkata, Hyderabad, PuneExp : 4-8 Yrs Functional Skills: Experience in Credit Risk/Regulatory risk domainTechnical Skills: Spark ,PySpark, Python, Hive, Scala, MapReduce, Unix shell scripting Good to Have Skills: Exposure to Machine Learning TechniquesJob Description:4+ Years of experience with Developing/Fine...