Pyspark, AWS Engineer Lead

4 weeks ago


Hyderabad, India Virtusa Full time
Pyspark, AWS Engineer Lead - CREQ187058 Description 6-8 years of relevant work experience showing growth as a Data Engineer.Hands On programming experienceImplementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation.Experience of performance optimization in Batch and Real time processing applicationsExpertise in Data Governance and Data Security ImplementationWorked on Scheduling tools like AirflowGood hands-on design and programming skills building reusable tools and productsExperience developing in AWS or similar cloud platforms. Preferred:, ECS, EKS, S3, EMR, DynamoDB, Aurora, Redshift, QuickSight or similar.Good knowledge of Python. Good to have Java/Scala.Familiarity with systems with very high volume of transactions, micro service design, or data processing pipelines (Spark).Experience in efficient data processing using open table format Delta, Iceberg.Knowledge and hands-on experience with server-less technologies such as Lambda, MSK, MWAA, Kinesis Analytics a plus.Expertise in practices like Agile, Peer reviews, Continuous IntegrationPrimary Location Hyderabad, Andhra Pradesh, India Job Type Experienced Primary Skills AWS Lake Formation, AWS Glue, Java, Spark, Kafka, Python, Amazon Redshift Years of Experience 8 Qualification

6-8years of relevant work experience showing growth as a Data Engineer.Hands On programming experienceImplementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation.Experience of performance optimization in Batch and Real time processing applicationsExpertise in Data Governance and Data Security ImplementationWorked on Scheduling tools like AirflowGood hands-on design and programming skills building reusable tools and productsExperience developing in AWS or similar cloud platforms. Preferred:, ECS, EKS, S3, EMR, DynamoDB, Aurora, Redshift, QuickSight or similar.Good knowledge of Python. Good to have Java/Scala.Familiarity with systems with very high volume of transactions, micro service design, or data processing pipelines (Spark).Experience in efficient data processing using open table format Delta, Iceberg.Knowledge and hands-on experience with server-less technologies such as Lambda, MSK, MWAA, Kinesis Analytics a plus.Expertise in practices like Agile, Peer reviews, Continuous IntegrationTravel No

  • Hyderabad, India Virtusa Full time

    Pyspark, AWS Engineer Lead - CREQ187058 Description 6-8 years of relevant work experience showing growth as a Data Engineer. Hands On programming experience Implementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation. Experience of performance optimization in Batch and Real time processing applications Expertise in Data Governance and...


  • hyderabad, India Virtusa Full time

    Pyspark, AWS Engineer Lead - CREQ187058 Description 6-8 years of relevant work experience showing growth as a Data Engineer.Hands On programming experienceImplementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation.Experience of performance optimization in Batch and Real time processing applicationsExpertise in Data Governance and Data...


  • Hyderabad, India Virtusa Full time

    Pyspark, AWS Engineer Lead - CREQ187058 Description 6-8 years of relevant work experience showing growth as a Data Engineer. Hands On programming experience Implementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation. Experience of performance optimization in Batch and Real time processing applications Expertise in Data Governance and...


  • Hyderabad, Telangana, India Virtusa Full time

    Pyspark, AWS Engineer Lead CREQ187058 Description 6 8 years of relevant work experience showing growth as a Data Engineer.Hands On programming experienceImplementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation.Experience of performance optimization in Batch and Real time processing applicationsExpertise in Data Governance and Data...


  • Hyderabad, Telangana, India Virtusa Full time

    Pyspark, AWS Engineer Lead - CREQ187058 Description 6-8 years of relevant work experience showing growth as a Data Engineer.Hands On programming experienceImplementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation.Experience of performance optimization in Batch and Real time processing applicationsExpertise in Data Governance and Data...


  • hyderabad, India Virtusa Full time

    Pyspark, AWS Engineer Lead - CREQ187058 Description 6-8 years of relevant work experience showing growth as a Data Engineer.Hands On programming experienceImplementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation.Experience of performance optimization in Batch and Real time processing applicationsExpertise in Data Governance and Data...


  • Hyderabad, India Virtusa Full time

    Python Pyspark with AWS - CREQ188741 Description Responsibilities:1.Analyze various data sources present with the client, including structured and unstructured data formatsentities across different data sources to identify relationships and data inconsistencies3.Analyze existing reports created with Tableau and other tools to understand data usage and...

  • AWS Data Engineer

    2 weeks ago


    Pune/Chennai/Mumbai/Bangalore/Delhi NCR/Hyderabad, India Change Leaders Full time

    Job Description : AWS Glue + Python PysparkJob Overview :Candidate should be responsible for the configuration, administration, and optimization of the AWS Glue + Python Pyspark platform. This role involves ensuring the smooth operation of AWS Glue + Python Pyspark for data integration, transformation, and loading tasks in alignment with the...

  • AWS Data Engineer

    2 months ago


    Pune/Chennai/Mumbai/Bangalore/Delhi NCR/Hyderabad, India Change Leaders Full time

    Job Description : AWS Glue + Python PysparkJob Overview :Candidate should be responsible for the configuration, administration, and optimization of the AWS Glue + Python Pyspark platform. This role involves ensuring the smooth operation of AWS Glue + Python Pyspark for data integration, transformation, and loading tasks in alignment with the...

  • Pyspark Lead

    4 days ago


    Hyderabad, Telangana, India CIEL HR Services Full time

    We are hiring for our client-An International IT service provider Role- Pyspark Lead Mandatory skill set- Python+Spark+Scala(experience in all 3 skill sets should be strong & mandatory) ,Data Engineering+ AWS Exp. Level- 5-12 yrs Location- Bangalore & Hyderabad Notice Period-Immediate to 15days Job Description Design and implement scalable and...

  • AWS (Pyspark) Engg.

    5 hours ago


    Hyderabad, India Zensar Technologies Full time

    4+ years of overall IT experience, which includes hands on experience in Big Data technologies.Mandatory - Hands on experience in Python and PySpark.Build pySpark applications using Spark Dataframes in Python using Jupyter notebook and PyCharm(IDE).Worked on optimizing spark jobs that processes huge volumes of data.Hands on experience in version control...


  • Hyderabad, Telangana, India Virtusa Full time

    Python Pyspark with AWS - CREQ188741 Description Responsibilities:1.Analyze various data sources present with the client, including structured and unstructured data formatsentities across different data sources to identify relationships and data inconsistencies3.Analyze existing reports created with Tableau and other tools to understand data usage and...


  • hyderabad, India Virtusa Full time

    Python Pyspark with AWS - CREQ188741 Description Responsibilities:1.Analyze various data sources present with the client, including structured and unstructured data formats entities across different data sources to identify relationships and data inconsistencies3.Analyze existing reports created with Tableau and other tools to understand data usage and...


  • Hyderabad/Bangalore/Chennai/Vishakhapatnam/Vizag, IN Mouri Tech Ltd Full time

    Senior Cloud Data Engineer (AWS, Python, Pyspark) Job Title : Senior AWS Cloud Data EngineerWork Location : Hyderabad/Kolhapur/Bangalore/Chennai Experience : 6 to 9 yearsRequired Skills : Python, Pyspark, AWS services such as DMS, SAM, Glue, Redshift, Lamda S3 with Apache Airflow, CI/CD, :- Minimum of 6-9 Experience with the core AWS Services and setting up...


  • Hyderabad/Bangalore/Chennai/Vishakhapatnam/Vizag, India Mouri Tech (P) Ltd Full time

    Senior Cloud Data Engineer (AWS, Python, Pyspark) Job Title : Senior AWS Cloud Data EngineerWork Location : Hyderabad/Kolhapur/Bangalore/Chennai Experience : 6 to 9 yearsRequired Skills : Python, Pyspark, AWS services such as DMS, SAM, Glue, Redshift, Lamda S3 with Apache Airflow, CI/CD, : - Minimum of 6-9 Experience with the core AWS Services and setting...


  • Hyderabad,Bangalore,Chennai,Vishakhapatnam,Vizag, India Mouri Tech (P) Ltd Full time

    Senior Cloud Data Engineer (AWS, Python, Pyspark) Job Title : Senior AWS Cloud Data EngineerWork Location : Hyderabad/Kolhapur/Bangalore/Chennai Experience : 6 to 9 yearsRequired Skills : Python, Pyspark, AWS services such as DMS, SAM, Glue, Redshift, Lamda S3 with Apache Airflow, CI/CD, : - Minimum of 6-9 Experience with the core AWS Services and setting...


  • Hyderabad/Bangalore/Chennai/Vishakhapatnam/Vizag, IN Mouri Tech Ltd Full time

    Senior Cloud Data Engineer (AWS, Python, Pyspark) Job Title : Senior AWS Cloud Data EngineerWork Location : Hyderabad/Kolhapur/Bangalore/Chennai Experience : 6 to 9 yearsRequired Skills : Python, Pyspark, AWS services such as DMS, SAM, Glue, Redshift, Lamda S3 with Apache Airflow, CI/CD, :- Minimum of 6-9 Experience with the core AWS Services and setting up...

  • Lead Data Engineer

    3 weeks ago


    Hyderabad/Bangalore/Pune, IN SKANDHA IT SERVICES PRIVATE LIMITED Full time

    Lead Data Engineer Locations : Hyderabad, Bengaluru, Pune. Skills :- Programming : Python , Pyspark/Scala, Flask.- API Microservices , ECS Fargate.- Bigdata AWS : EMR, Glue, Athena, Redshift, Lambda.- Analytics : AWS Quicksight.- Database : Postgres RDS,Mongo (SQL/NoSQL).- Orchestration : preferably Airflow.About the Role : We are searching for a talented...

  • PySpark QA

    3 days ago


    Hyderabad, India Virtusa Full time

    PySpark QA - CREQ191292 DescriptionJOB DESCRIPTIONSkill: PySpark QARole / Tier: Lead Software Engineer/Tier 2Experience: 6 – 9 yearsJob Description:Primary SkillsBIG Data technology mentioned below Hadoop / Big Data (HDFS, PYTHON, SPARK-SQL, MapReduce) with PYSpark.build CI/CD pipelinesSpark APIs to cleanse, explore, aggregate, transform, store &...


  • Hyderabad, India Virtusa Full time

    Python Pyspark with AWS - CREQ188741 Description Responsibilities: 1.Analyze various data sources present with the client, including structured and unstructured data formats entities across different data sources to identify relationships and data inconsistencies 3.Analyze existing reports created with Tableau and other tools to understand data usage and...