Senior AWS Data Engineer

1 month ago

ChennaiBangaloreHyderabadPune, India ProgressX Full time

Role : AWS Data : India


Type : Full-time


Experience : 5-9 years

Functions : Consulting, Finance, Information Technology, Data Engineering & Analytics

Industries : Capital Markets, Investment Banking, Alternative Investments, Financial Services, Management Consulting, Information Technology and Services, Business Travel Healthcare

Job Description :

We are looking for an AWS Data Engineer with primary skills on Python & PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources. There is no requirement for Machine Learning skills.

This is a high visibility, fast-paced key initiative will integrate data across internal and external sources, provide analytical insights, and integrate with the customer's critical systems.

Key Responsibilities :

- Design, build and unit test applications on Spark framework on Python.

- Build Python and PySpark based applications based on data in both Relational databases (e.g. Oracle), NoSQL databases (e.g. DynamoDB, MongoDB) and filesystems (e.g. S3, HDFS)

- Build AWS Lambda functions on Python runtime leveraging pandas, json, boto3, requests, avro libraries

- Build PySpark based data pipeline jobs on AWS Glue ETL requiring in-depth knowledge on AWS Glue Dynamic Frames and Options

- Build Python based event-driven integration with Kafka Topics, leveraging Confluent Kafka libraries

- Design and Build Generic, Reusable utility applications in Python

- Build the Python programs across Glue ETL jobs and Lambda functions

- Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec respectively.

- Design & Build S3 buckets, tiers, lifecycle policies, as strategic storage layer for "Data Lake"

- Optimize performance of Spark applications in Hadoop using configurations around Spark Context, Spark SQL, Data Frame, and Pair RDD's

- Setup the Glue crawlers in order to catalog OracleDB tables, MongoDB collections and S3 objects

- Configure Athena tables and SQL views based on Glue Cataloged datasets

- Ability to monitor, troubleshoot and debug failures using AWS CloudWatch and Datadog

- Ability to solve complex data-driven scenarios and triage towards defects and production issues

- Participate in code release and production deployment.

- Create documentation for user adoption, deployments, runbook, and support client users for enablement or for any issues encountered.

- Perform code reviews with the team and enable them to develop code for complex scenarios

- Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings

- Work collaboratively with onsite and offshore team.

- Voice the opinions to multiple teams and thus driving the entire initiative with strong leadership

Qualifications :

- Bachelor's Degree or equivalent in computer science or related and minimum 5+ years of experience

- Certified on one of - Solution Architect, Data Engineer or Data Analytics Specialty by AWS

- Require 3+ hand-on experience on Python and PySpark programming

- Require 2+ hands-on experience on AWS S3, Glue ETL & Catalog, Lamba Functions, Athena & Kafka

- Require 1+ hands-on experience on Confluent Kafka integration

- Require hands-on experience working on different file formats i.e. avro, parquet, orc, json, xml

- Require hands-on experience on Python pandas, requests, boto3 module

- Require hands-on experience in writing complex SQL queries

- Require hands-on experience using REST API

(ref:hirist.tech)

  • AWS Data Engineer

    3 weeks ago

    Chennai/Bangalore/Hyderabad/Pune, India ProgressX Full time

    Job Title: Senior AWS Data EngineerWe are seeking a highly skilled Senior AWS Data Engineer to join our team at ProgressX. As a key member of our data engineering team, you will be responsible for designing and building scalable data pipelines on AWS Cloud platform.Key Responsibilities:Design and develop data pipelines using AWS Glue ETL and Spark framework...

  • AWS Data Engineer

    2 weeks ago

    Pune, Maharashtra, India NTT DATA Services Full time

    Job Title: AWS Data EngineerAbout the Role:We are seeking a highly skilled AWS Data Engineer to join our team at NTT DATA Services. As an AWS Data Engineer, you will be responsible for designing, developing, and deploying data pipelines on Amazon Web Services (AWS).Key Responsibilities:Design and develop data pipelines using AWS services such as Glue,...

  • Senior Data Engineer

    3 weeks ago

    chennai, India Wavicle Data Solutions Full time

    We are seeking an experienced Senior AWS Data Engineer to join our team. The ideal candidate will have a strong background in designing, building, and maintaining scalable data pipelines and solutions on AWS. You will work closely with data scientists, analysts, and cross-functional teams to optimize and manage our data infrastructure. This role requires a...

  • Senior Data Engineer

    5 days ago

    Chennai, India Wavicle Data Solutions Full time

    We are seeking an experienced Senior AWS Data Engineer to join our team. The ideal candidate will have a strong background in designing, building, and maintaining scalable data pipelines and solutions on AWS. You will work closely with data scientists, analysts, and cross-functional teams to optimize and manage our data infrastructure. This role requires a...

  • AWS Data Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India Wavicle Data Solutions Full time

    We are seeking an experienced Senior AWS Data Engineer to lead our data infrastructure team. The ideal candidate will have a strong background in designing, building, and maintaining scalable data pipelines and solutions on AWS. You will work closely with data scientists, analysts, and cross-functional teams to optimize and manage our data infrastructure....

  • Data Architect with AWS Expertise

    1 week ago

    Chennai, Tamil Nadu, India Wavicle Data Solutions Full time

    Job SummaryWe are seeking an experienced Senior AWS Data Engineer to lead our data engineering team at Wavicle Data Solutions. The ideal candidate will have a strong background in designing, building, and maintaining scalable data pipelines and solutions on AWS.Key ResponsibilitiesDesign and develop robust ETL pipelines on AWS using services like Glue,...

  • AWS Senior Data Engineer

    2 weeks ago

    Hyderabad, India I square soft Full time

    Job Title: AWS Senior Data EngineerAbout the Role:I Square Soft is seeking an experienced AWS Senior Data Engineer to lead the design, development, and deployment of large-scale data pipelines and architectures. The successful candidate will have a strong background in data engineering principles, data warehousing, and data modeling, with expertise in AWS...

  • Senior Data Engineer

    3 weeks ago

    Chennai, India Wavicle Data Solutions Full time

    We are seeking an experienced Senior AWS Data Engineer to join our team. The ideal candidate will have a strong background in designing, building, and maintaining scalable data pipelines and solutions on AWS. You will work closely with data scientists, analysts, and cross-functional teams to optimize and manage our data infrastructure. This role requires a...

  • Senior Data Engineer

    3 weeks ago

    chennai, India Wavicle Data Solutions Full time

    We are seeking an experienced Senior AWS Data Engineer to join our team. The ideal candidate will have a strong background in designing, building, and maintaining scalable data pipelines and solutions on AWS. You will work closely with data scientists, analysts, and cross-functional teams to optimize and manage our data infrastructure. This role requires a...

  • Senior Data Engineer

    3 weeks ago

    Chennai, India Wavicle Data Solutions Full time

    We are seeking an experienced Senior AWS Data Engineer to join our team. The ideal candidate will have a strong background in designing, building, and maintaining scalable data pipelines and solutions on AWS. You will work closely with data scientists, analysts, and cross-functional teams to optimize and manage our data infrastructure. This role requires a...

  • Senior Data Engineer

    3 weeks ago

    Chennai, India Wavicle Data Solutions Full time

    We are seeking an experienced Senior AWS Data Engineer to join our team. The ideal candidate will have a strong background in designing, building, and maintaining scalable data pipelines and solutions on AWS. You will work closely with data scientists, analysts, and cross-functional teams to optimize and manage our data infrastructure. This role requires a...

  • Senior Data Engineer

    3 weeks ago

    Chennai, India Wavicle Data Solutions Full time

    We are seeking an experienced Senior AWS Data Engineer to join our team. The ideal candidate will have a strong background in designing, building, and maintaining scalable data pipelines and solutions on AWS. You will work closely with data scientists, analysts, and cross-functional teams to optimize and manage our data infrastructure. This role requires a...

  • Senior AWS Data Engineer

    2 weeks ago

    Chennai/Bangalore/Hyderabad/Pune, India ProgressX Full time

    Job Title: AWS Data EngineerJob Summary:We are seeking a highly skilled AWS Data Engineer to join our team at ProgressX. As a key member of our data engineering team, you will be responsible for designing and building scalable, secure, and efficient data pipelines on AWS.Key Responsibilities:Design, build, and unit test applications on Spark framework on...

  • Senior Data Engineer AWS

    2 weeks ago

    Chennai, Tamil Nadu, India Fulcrum Digital Full time

    About the RoleFulcrum Digital is a cutting-edge digital transformation company providing innovative technology services to various industries. We are seeking a skilled Senior Data Engineer with expertise in AWS cloud technologies and data engineering.The ideal candidate will be proficient in a wide range of AWS services and tools, including Databricks, MWAA,...

  • Senior Data Engineer

    3 weeks ago

    Pune, Maharashtra, India Data Axle Full time

    About Data AxleData Axle Inc. is a leading provider of data, marketing solutions, sales, and research. With a 50-year history in the US, the company has established a strategic global centre of excellence in Pune, delivering mission-critical data services to its global customers powered by its proprietary cloud-based technology platform and leveraging...

  • Senior AWS Data Engineer

    2 weeks ago

    Hyderabad, India Manuh Technologies Full time

    About the RoleThe Senior AWS Data Engineer will lead the design and implementation of scalable, secure, and efficient data architectures on the AWS platform. This role requires a strong background in data engineering, cloud computing, and AWS services. The successful candidate will work closely with the development team to build, deploy, and maintain data...

  • Senior Data Engineer

    5 months ago

    Pune, India Data Axle Full time

    Job Description About Data Axle: Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the US. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology...

  • Senior Data Engineer

    2 weeks ago

    Pune, Maharashtra, India Data Axle Full time

    About Data AxleData Axle Inc. has been a leading provider of data, marketing solutions, sales, and research for over five decades in the US. Our global centre of excellence in Pune delivers mission-critical data services to our customers worldwide, leveraging our proprietary cloud-based technology platform and business & consumer databases.Job SummaryWe are...

  • AWS Data Engineer

    2 weeks ago

    Pune, Maharashtra, India NTT DATA Services Full time

    Job SummaryWe are seeking a highly skilled AWS Data Engineer to join our team in Pune, Madhya Pradesh, India. The successful candidate will contribute to our client and have the following responsibilities:Key ResponsibilitiesWork with technical development team and team lead to understand desired application capabilities.Develop code using Python, such as...

  • Senior Data Engineer

    3 weeks ago

    Pune, Maharashtra, India Data Axle Full time

    About Data Axle:Data Axle Inc. has been a leading provider of data, marketing solutions, sales, and research for 50 years in the US. With a strategic global centre of excellence in Pune, Data Axle delivers mission-critical data services to its global customers powered by its proprietary cloud-based technology platform and leveraging proprietary business &...