PySpark Developer

5 days ago

Kolkata, West Bengal, India Apex One Full time ₹ 1,04,000 - ₹ 1,30,878 per year

Type: Contract-to-Hire (C2H)Job SummaryWe are looking for a skilled PySpark Developer with hands-on experience in building scalable data pipelines and processing large datasets. The ideal candidate will have deep expertise in Apache Spark, Python, and working with modern data engineering tools in cloud environments such as AWS.

Key Skills & Responsibilities

  • Strong expertise in PySpark and Apache Spark for batch and real-time data processing.
  • Experience in designing and implementing ETL pipelines, including data ingestion, transformation, and validation.
  • Proficiency in Python for scripting, automation, and building reusable components.
  • Hands-on experience with scheduling tools like Airflow or Control-M to orchestrate workflows.
  • Familiarity with AWS ecosystem, especially S3 and related file system operations.
  • Strong understanding of Unix/Linux environments and Shell scripting.
  • Experience with Hadoop, Hive, and platforms like Cloudera or Hortonworks.
  • Ability to handle CDC (Change Data Capture) operations on large datasets.
  • Experience in performance tuning, optimizing Spark jobs, and troubleshooting.
  • Strong knowledge of data modeling, data validation, and writing unit test cases.
  • Exposure to real-time and batch integration with downstream/upstream systems.
  • Working knowledge of Jupyter Notebook, Zeppelin, or PyCharm for development and debugging.
  • Understanding of Agile methodologies, with experience in CI/CD tools (e.g., Jenkins, Git).

Preferred Skills

  • Experience in building or integrating APIs for data provisioning.
  • Exposure to ETL or reporting tools such as Informatica, Tableau, Jasper, or QlikView.
  • Familiarity with AI/ML model development using PySpark in cloud environments

Skills: ci/cd,zeppelin,pycharm,pyspark,etl tools,control-m,unit test cases,tableau,performance tuning,jenkins,qlikview,informatica,jupyter notebook,api integration,unix/linux,git,aws s3,hive,cloudera,jasper,airflow,cdc,pyspark, apache spark, python, aws s3, airflow/control-m, sql, unix/linux, hive, hadoop, data modeling, and performance tuning,agile methodologies,aws,s3,data modeling,data validation,ai/ml model development,batch integration,apache spark,python,etl pipelines,shell scripting,hortonworks,real-time integration,hadoop

Mandatory Key Skills

ci/cd,zeppelin,pycharm,etl,control-m,performance tuning,jenkins,qlikview,informatica,jupyter notebook,api integration,unix,PySpark*

  • Pyspark

    5 days ago

    Kolkata, West Bengal, India Cognizant Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Skill: PysparkExperience: 6 to 9 yearsLocation: KolkataJob description: ResponsibilitiesDevelop and maintain scalable data pipelines using Python and PySpark.Collaborate with data engineers and data scientists to understand and fulfill data processing needs.Optimize and troubleshoot existing PySpark applications for performance improvements.Write clean,...

  • Pyspark Architect

    7 days ago

    Kolkata, West Bengal, India Talent Worx Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job Description: PySpark ArchitectLocation: Bangalore,Pune,Kolkata Experience Required: 7 to 13 YearsRole Overview:We are seeking a highly skilled and motivated PySpark Manager to lead our dataengineering initiatives. The ideal candidate will have deep expertise in PySpark,Hadoop, AWS, Teradata, Scala, and data visualization tools. You will be responsible...

  • AWS Databricks Developer

    5 days ago

    Kolkata, West Bengal, India Tata Consultancy Services Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Role & responsibilities• Develop and maintain scalable data pipelines using Apache Spark on Databricks.• Build end-to-end ETL/ELT pipelines on AWS using services like S3, Glue, Lambda, EMR, and Step Functions.• Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality data solutions.• Design and implement data...

  • AWS Databricks Developer

    2 weeks ago

    Kolkata, West Bengal, India Integrated Personnel Services Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Job role - DeveloperExperience - 6 to 10 yearsLocation - KolkataAWS Databricks Developer Experience More than 3 years in data integration, pipeline development, and data warehousing, with a strong focus on AWS Databricks.Technical Skills Proficiency in Databricks platform, management, and optimization. Strong experience in AWS Cloud, particularly in data...

  • AWS Glue

    5 days ago

    Kolkata, West Bengal, India Cognizant Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Skill: AWS GlueExperience: 6 to 9 yearsLocation: KolkataJob descriptionTechnical Skills :AWS Glue: 3+ years of hands-on experience in AWS Glue ETL developmentPython/PySpark: Strong programming skills in Python and PySpark for data transformationAWS Services: Proficiency in S3, Redshift, Athena, Lambda, and EMRData Formats: Experience with Parquet, Avro,...

  • AWS Data Engineer_CBS

    3 days ago

    Kolkata, West Bengal, India Coders Brain Technology Full time ₹ 5,00,000 - ₹ 25,00,000 per year

    Location: Pune/kolkata Skillset we are looking for: 6+ experience with Cloud technologies, especially AWS services (S3, Athena, Glue, DynamoDB, Step Functions, etc). Strong understanding of Redshift, EMR, Athena, Aurora, Kinesis, Lambda, EC2, Python etc. Must have Technical strength in SQL, Cloud Technologies, ETL development and data warehousing.. Ensure...

  • Aws Data Engineer

    3 days ago

    Kolkata, West Bengal, India Exavalu Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Role & responsibilities :Design, Develop and Test data ingestion/ETL pipelines with S3 Storage, Postgres, Athena, Redshift.Strong experience in Python, Pyspark, SQL, Glue, Lambda and orchestration techniquesDevelops stored procedures, database triggers and SQL queries with an understanding of data warehousing objects (type-2 dimensions, CDC, aggregations,...

  • Databricks Data Engineer

    2 weeks ago

    Kolkata, West Bengal, India TuTeck Technologies Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job Title: Databricks Data EngineerLocation:Kolkata, India (On-site)Experience:3–8 YearsNotice Period:Immediate Joiners PreferredJob Summary:We are looking for a skilledDatabricks Data Engineerwith hands-on experience inPySparkandSQLto join our data engineering team in Kolkata. The ideal candidate will design, develop, and maintain scalable data pipelines...

  • Databricks Senior Developer

    3 weeks ago

    Kolkata, West Bengal, India, West Bengal Tata Consultancy Services Full time

    Role: Databricks Senior DeveloperLocation: KolkataExperience: 8 Plus YearsRole**Databricks Senior DeveloperRequired Technical Skill Set**Databricks, Spark and Data Migration experienceDesired Competencies (Technical/Behavioral Competency)Must-Have**1. 8+ years with strong Databricks, Spark and Data Migration experience2. Should have experience of end-to-end...

  • Data Engineer with Data Operations

    22 hours ago

    Kolkata, West Bengal, India Adv Techminds Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job Title: Data Engineer with Data OperationsExperience: 5+ Years Employment Type: Contract Duration: 6 Months (Extendable) Location: KolkataKey Responsibilities:Design, develop, and maintain robust data pipelines to support data integration from Oracle Flexcube and other data sources into the Cloudera Data Platform.Utilize Hadoop for efficient data storage...