PySpark Developer

4 days ago

Kolkata, West Bengal, India Apex One Full time ₹ 1,04,000 - ₹ 1,30,878 per year

Type: Contract-to-Hire (C2H)Job SummaryWe are looking for a skilled PySpark Developer with hands-on experience in building scalable data pipelines and processing large datasets. The ideal candidate will have deep expertise in Apache Spark, Python, and working with modern data engineering tools in cloud environments such as AWS.

Key Skills & Responsibilities

  • Strong expertise in PySpark and Apache Spark for batch and real-time data processing.
  • Experience in designing and implementing ETL pipelines, including data ingestion, transformation, and validation.
  • Proficiency in Python for scripting, automation, and building reusable components.
  • Hands-on experience with scheduling tools like Airflow or Control-M to orchestrate workflows.
  • Familiarity with AWS ecosystem, especially S3 and related file system operations.
  • Strong understanding of Unix/Linux environments and Shell scripting.
  • Experience with Hadoop, Hive, and platforms like Cloudera or Hortonworks.
  • Ability to handle CDC (Change Data Capture) operations on large datasets.
  • Experience in performance tuning, optimizing Spark jobs, and troubleshooting.
  • Strong knowledge of data modeling, data validation, and writing unit test cases.
  • Exposure to real-time and batch integration with downstream/upstream systems.
  • Working knowledge of Jupyter Notebook, Zeppelin, or PyCharm for development and debugging.
  • Understanding of Agile methodologies, with experience in CI/CD tools (e.g., Jenkins, Git).

Preferred Skills

  • Experience in building or integrating APIs for data provisioning.
  • Exposure to ETL or reporting tools such as Informatica, Tableau, Jasper, or QlikView.
  • Familiarity with AI/ML model development using PySpark in cloud environments

Skills: ci/cd,zeppelin,pycharm,pyspark,etl tools,control-m,unit test cases,tableau,performance tuning,jenkins,qlikview,informatica,jupyter notebook,api integration,unix/linux,git,aws s3,hive,cloudera,jasper,airflow,cdc,pyspark, apache spark, python, aws s3, airflow/control-m, sql, unix/linux, hive, hadoop, data modeling, and performance tuning,agile methodologies,aws,s3,data modeling,data validation,ai/ml model development,batch integration,apache spark,python,etl pipelines,shell scripting,hortonworks,real-time integration,hadoop

Mandatory Key Skills

ci/cd,zeppelin,pycharm,etl,control-m,performance tuning,jenkins,qlikview,informatica,jupyter notebook,api integration,unix,PySpark*

  • Pyspark

    4 days ago

    Kolkata, West Bengal, India Cognizant Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Skill: PysparkExperience: 6 to 9 yearsLocation: KolkataJob description: ResponsibilitiesDevelop and maintain scalable data pipelines using Python and PySpark.Collaborate with data engineers and data scientists to understand and fulfill data processing needs.Optimize and troubleshoot existing PySpark applications for performance improvements.Write clean,...

  • AWS Databricks Developer

    4 days ago

    Kolkata, West Bengal, India Tata Consultancy Services Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Role & responsibilities• Develop and maintain scalable data pipelines using Apache Spark on Databricks.• Build end-to-end ETL/ELT pipelines on AWS using services like S3, Glue, Lambda, EMR, and Step Functions.• Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality data solutions.• Design and implement data...

  • AWS Databricks Developer

    2 weeks ago

    Kolkata, West Bengal, India Integrated Personnel Services Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Job role - DeveloperExperience - 6 to 10 yearsLocation - KolkataAWS Databricks Developer Experience More than 3 years in data integration, pipeline development, and data warehousing, with a strong focus on AWS Databricks.Technical Skills Proficiency in Databricks platform, management, and optimization. Strong experience in AWS Cloud, particularly in data...

  • AWS Glue

    4 days ago

    Kolkata, West Bengal, India Cognizant Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Skill: AWS GlueExperience: 6 to 9 yearsLocation: KolkataJob descriptionTechnical Skills :AWS Glue: 3+ years of hands-on experience in AWS Glue ETL developmentPython/PySpark: Strong programming skills in Python and PySpark for data transformationAWS Services: Proficiency in S3, Redshift, Athena, Lambda, and EMRData Formats: Experience with Parquet, Avro,...

  • Snowflake Data Engineer

    2 days ago

    Kolkata, West Bengal, India Vidpro Consultancy Services Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Exp: YrsWork Mode: HybridLocation: Bangalore, Chennai, Kolkata, Pune and GurgaonPrimary Skills: Python, pyspark, Azure Data Factory, snowflake, snowpipe, snowsql, Snowsight, Snowpark, ETL, SQL, and Architect Designing.Snowpro certified is plusPrimary Roles and Responsibilities:Developing Modern Data Warehouse solutions using Snowflake, Databricks and...

  • Aws Data Engineer

    2 days ago

    Kolkata, West Bengal, India Exavalu Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Role & responsibilities :Design, Develop and Test data ingestion/ETL pipelines with S3 Storage, Postgres, Athena, Redshift.Strong experience in Python, Pyspark, SQL, Glue, Lambda and orchestration techniquesDevelops stored procedures, database triggers and SQL queries with an understanding of data warehousing objects (type-2 dimensions, CDC, aggregations,...

  • Databricks Senior Developer

    1 day ago

    Kolkata, West Bengal, India, West Bengal Tata Consultancy Services Full time

    Role: Databricks Senior DeveloperLocation: KolkataExperience: 8 Plus YearsRole**Databricks Senior DeveloperRequired Technical Skill Set**Databricks, Spark and Data Migration experienceDesired Competencies (Technical/Behavioral Competency)Must-Have**1. 8+ years with strong Databricks, Spark and Data Migration experience2. Should have experience of end-to-end...

  • Azure Data Engineer

    4 days ago

    Kolkata, West Bengal, India Sys Edge Micro Informatics Full time ₹ 6,00,000 - ₹ 12,00,000 per year

    Role & responsibilitiesAssist in data collection, cleansing, and preparation from various sources.Support basic data modeling and documentation efforts.Develop and maintain simple data pipelines using tools like Azure Data Factory (ADF).Write and optimize queries using SQL.Use Python or PySpark scripts to perform data transformations.Perform data validation...

  • Technology Specialist

    6 days ago

    Kolkata, West Bengal, India Cognizant Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Role: MLOps EngineerLocation: PAN IndiaKey words -SkillsetAWS SageMaker, Azure ML Studio, GCP Vertex AIPySpark, Azure DatabricksMLFlow, KubeFlow, AirFlow, Github Actions, AWS CodePipelineKubernetes, AKS, Terraform, Fast APIResponsibilitiesModel Deployment, Model Monitoring, Model RetrainingDeployment pipeline, Inference pipeline, Monitoring pipeline,...

  • Data Engineer

    1 day ago

    Kolkata, West Bengal, India, West Bengal Tata Consultancy Services Full time

    TCS has been a great pioneer in feeding the fire of young techies like you. We are global leaders in the technology arena and there's nothing that can stop us from growing together. TCS Hiring for skill "Data Engineer". Role: Databricks Senior Developer Required Technical Skill Set: Databricks, Spark and Data Migration experience Experience: 8+yearsWork...