AWS Glue PySpark Developer

1 week ago

India CIGNEX Full time ₹ 15,00,000 - ₹ 25,00,000 per year

We are looking for an experienced AWS Glue PySpark Developer to design, develop, and optimize ETL pipelines and data processing solutions on AWS. The ideal candidate will have deep expertise in PySpark, AWS Glue, and data engineering best practices, along with hands-on experience in building scalable, high-performance data solutions in the cloud.

Key Responsibilities:

  • Design, build, and maintain scalable ETL pipelines using AWS Glue and PySpark.
  • Work with stakeholders to gather and analyse data requirements and translate them into technical solutions.
  • Develop efficient and reusable PySpark scripts to process large-scale structured and unstructured datasets.
  • Optimize ETL jobs for performance, scalability, and cost-effectiveness in AWS environments.
  • Integrate AWS Glue with other AWS services such as S3, Redshift, RDS, Lambda, Step Functions, and Athena.
  • Implement data quality checks, validation frameworks, and error-handling mechanisms within ETL pipelines.
  • Collaborate with data engineers, analysts, and business teams to ensure data accuracy and consistency.
  • Monitor, debug, and resolve production issues related to Glue jobs and data workflows.
  • Ensure compliance with security, governance, and regulatory requirements for data pipelines.
  • Stay current with AWS and big data ecosystem advancements to continuously improve solutions.

Required Skills:

  • 5-6 years of experience in data engineering/ETL development, with at least 3 years in AWS Glue & PySpark.
  • Strong proficiency in PySpark, Spark SQL, and distributed data processing.
  • Hands-on experience with AWS services: S3, Glue Catalog, Redshift, RDS, Lambda, Step Functions, CloudWatch.
  • Expertise in designing data models, partitioning strategies, and optimizing large datasets.
  • Proficiency in SQL and working with relational as well as NoSQL databases.
  • Experience with version control (Git), CI/CD pipelines, and Agile methodologies.
  • Strong problem-solving skills and ability to debug complex data issues.
  • Excellent communication and collaboration skills.

  • TCS Hiring for AWS Senior Data Engineer with Pyspark, AWS, Glue_Hyderabad

    2 weeks ago

    Hyderabad, Telangana, India, Telangana Tata Consultancy Services Full time

    Job Title: AWS Senior Data Engineer with Pyspark, AWS, GlueLocation: HyderabadExperience: 6 to 10 YearsNotice Period: 30-45 daysJob Description:Must: PySpark, AWS[ETL Concepts, S3, Glue, EMR, Redshift, DMS, AppFlow] ,Qlik Replicate, Data TestingNice To Have: Hadoop, Teradata Background, IaC[Cloud Formation / Terraform], GitKind Regards,Priyankha M

  • TCS Hiring for AWS Senior Data Engineer with Pyspark, AWS, Glue_kochi

    2 weeks ago

    Kochi, Kerala, India, Ernakulam Tata Consultancy Services Full time

    Job Title: AWS Senior Data Engineer with Pyspark, AWS, GlueLocation: KochiExperience: 6 to 10 YearsNotice Period: 30-45 daysJob Description:Must: PySpark, AWS[ETL Concepts, S3, Glue, EMR, Redshift, DMS, AppFlow] ,Qlik Replicate, Data TestingNice To Have: Hadoop, Teradata Background, IaC[Cloud Formation / Terraform], GitKind Regards,Priyankha M

  • TCS Hiring for AWS Senior Data Engineer with Pyspark, AWS, Glue_Bangalore

    2 weeks ago

    Bengaluru, Karnataka, India, Karnataka Tata Consultancy Services Full time

    Job Title: AWS Senior Data Engineer with Pyspark, AWS, GlueLocation: BangaloreExperience: 6 to 10 YearsNotice Period: 30-45 daysJob Description:Must: PySpark, AWS[ETL Concepts, S3, Glue, EMR, Redshift, DMS, AppFlow] ,Qlik Replicate, Data TestingNice To Have: Hadoop, Teradata Background, IaC[Cloud Formation / Terraform], GitKind Regards,Priyankha M

  • TCS Hiring for AWS Senior Data Engineer with Pyspark, AWS, Glue_Bhubaneshawar

    2 weeks ago

    Bhubaneswar, Odisha, India, IN Tata Consultancy Services Full time

    Job Title: AWS Senior Data Engineer with Pyspark, AWS, GlueLocation: BhubaneshawarExperience: 6 to 10 YearsNotice Period: 30-45 daysJob Description:Must: PySpark, AWS[ETL Concepts, S3, Glue, EMR, Redshift, DMS, AppFlow] ,Qlik Replicate, Data TestingNice To Have: Hadoop, Teradata Background, IaC[Cloud Formation / Terraform], GitKind Regards,Priyankha M

  • TCS Hiring for AWS Senior Data Engineer with Pyspark, AWS, Glue_Kolkata

    2 weeks ago

    Kolkata, West Bengal, India, West Bengal Tata Consultancy Services Full time

    Job Title: AWS Senior Data Engineer with Pyspark, AWS, GlueLocation: KolkataExperience: 6 to 10 YearsNotice Period: 30-45 daysJob Description:Must: PySpark, AWS[ETL Concepts, S3, Glue, EMR, Redshift, DMS, AppFlow] ,Qlik Replicate, Data TestingNice To Have: Hadoop, Teradata Background, IaC[Cloud Formation / Terraform], GitKind Regards,Priyankha M

  • TCS Hiring for AWS Senior Data Engineer with Pyspark, AWS, Glue_Chennai

    2 weeks ago

    Chennai, Tamil Nadu, India, Tamil Nadu Tata Consultancy Services Full time

    Job Title: AWS Senior Data Engineer with Pyspark, AWS, GlueLocation: ChennaiExperience: 6 to 10 YearsNotice Period: 30-45 daysJob Description:Must: PySpark, AWS[ETL Concepts, S3, Glue, EMR, Redshift, DMS, AppFlow] ,Qlik Replicate, Data TestingNice To Have: Hadoop, Teradata Background, IaC[Cloud Formation / Terraform], GitKind Regards,Priyankha M

  • AWS Data Engineer

    2 weeks ago

    Bhubaneswar, Odisha, India, IN Tata Consultancy Services Full time

    Experience: 5-10 YrsLocation - Bangalore,Chennai,Hyderabad,Pune,Kochi,Bhubaneshawar,KolkataKey SkillsAWS Lambda, Python, Boto3 ,Pyspark, GlueMust have SkillsStrong experience in Python to package, deploy and monitor data science appsKnowledge in Python based automationKnowledge of Boto3 and related Python packagesWorking experience in AWS and AWS LambdaGood...

  • AWS Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India, Karnataka Tata Consultancy Services Full time

    Experience: 5-10 Yrs Location - Bangalore,Chennai,Hyderabad,Pune,Kochi,Bhubaneshawar,KolkataKey Skills AWS Lambda, Python, Boto3 ,Pyspark, GlueMust have Skills Strong experience in Python to package, deploy and monitor data science apps Knowledge in Python based automation Knowledge of Boto3 and related Python packages Working experience in AWS and AWS...

  • Python AWS Data Engineer

    4 weeks ago

    India Digitrix Software LLP Full time

    Experience: 5 to 8 years Job description: Python AWS Data Engineer - Python, AWS Python (core language skill) -- Backend, Pandas, PySpark (DataFrame API), interacting with AWS (e.g., boto3 for S3, Glue, Lambda) - Data Processing: Spark (PySpark), Glue, EMR AWS Core Services: S3, Glue, Athena, Lambda, Step Functions, EMR - Containerization: Docker -...

  • TCS Hiring for AWS Senior Data Engineer with Pyspark, AWS, Glue_Pune

    2 weeks ago

    Pune, Maharashtra, India, Maharashtra Tata Consultancy Services Full time

    Job Title: AWS Senior Data Engineer with Pyspark, AWS, Glue_PuneLocation: PuneExperience: 6 to 10 YearsNotice Period: 30-45 daysJob Description:Must: PySpark, AWS[ETL Concepts, S3, Glue, EMR, Redshift, DMS, AppFlow] ,Qlik Replicate, Data TestingNice To Have: Hadoop, Teradata Background, IaC[Cloud Formation / Terraform], GitKind Regards,Priyankha M