AWS Data Engineer

4 weeks ago


India ITI Data Full time

Job Description

We are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources. There is no requirement for Machine Learning skills. This is a high visibility, fast-paced key initiative will integrate data across internal and external sources, provide analytical insights, and integrate with the customer’s critical systems.

Key Responsibilities

• Design, build and unit test applications on Spark framework on Python.

• Build Python and PySpark based applications based on data in both Relational databases (e.g. Oracle), NoSQL databases (e.g. DynamoDB, MongoDB) and filesystems (e.g. S3, HDFS)

• Build AWS Lambda functions on Python runtime leveraging pandas, json, boto3, requests, avro libraries

• Build PySpark based data pipeline jobs on AWS Glue ETL requiring in-depth knowledge on AWS Glue Dynamic Frames and Options

• Build Python based event-driven integration with Kafka Topics, leveraging Confluent Kafka libraries

• Design and Build Generic, Reusable utility applications in Python

• Build the Python programs across Glue ETL jobs and Lambda functions

• Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec respectively.

• Design & Build S3 buckets, tiers, lifecycle policies, as strategic storage layer for "Data Lake"

• Optimize performance of Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's

• Setup the Glue crawlers in order to catalog OracleDB tables, MongoDB collections and S3 objects

• Configure Athena tables and SQL views based on Glue Cataloged datasets

• Ability to monitor, troubleshoot and debug failures using AWS CloudWatch and Datadog

• Ability to solve complex data-driven scenarios and triage towards defects and production issues

• Participate in code release and production deployment.


  • AWS Data Engineer

    6 days ago


    India ITI Data Full time

    Job Description We are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources....

  • AWS Data Engineer

    3 weeks ago


    India ITI Data Full time

    Job Description We are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources....

  • AWS Data Engineer

    18 hours ago


    India FLYONIT Full time

    Job Summary: We are looking for a highly skilled AWS Data Engineer with 5-6 years of experience to join our dynamic team. The ideal candidate will have hands-on experience with AWS services, data engineering best practices, and a strong understanding of building, deploying, and maintaining data pipelines in the cloud. You will be responsible for designing...

  • AWS Data Engineer

    18 hours ago


    India Mactores Full time

    Mactores is a trusted leader among businesses in providing modern data platform solutions. Since 2008, Mactores has been enabling businesses to accelerate their value through automation by providing automated, agile, and secure end-to-end data solutions. We collaborate with customers to strategize, navigate, and accelerate an ideal path forward with digital...

  • Lead Data Engineer

    3 months ago


    india TIU Consulting Full time

    Job Description : - Design, build and maintain highly scalable and available big data solutions on AWS cloud - Develop and manage AWS Glue data catalog, crawlers, and ETL jobs- Design and implement data synchronization solutions using AWS Datasync- Manage and optimize S3 data storage and access patterns- Develop and maintain Infrastructure as Code (IaC)...


  • India ThreatXIntel Full time

    Company Description ThreatXIntel is a startup cyber security company that specializes in protecting businesses and organizations from cyber threats. Our services include cloud security, web and mobile security testing, cloud security assessment, and DevSecOps. We provide customized and affordable solutions to meet the specific needs of our clients,...


  • India ThreatXIntel Full time

    Company DescriptionThreatXIntel is a startup cyber security company that specializes in protecting businesses and organizations from cyber threats. Our services include cloud security, web and mobile security testing, cloud security assessment, and DevSecOps. We provide customized and affordable solutions to meet the specific needs of our clients, regardless...


  • India UsefulBI Corporation Full time

    Overview:UsefulBI is looking for highly skilled candidates, with expertise in generating power business insights from very large datasets, in this space where the primary aim would be to enable needle moving business impacts through cutting edge statistical analysis. We are looking for passionate data engineers who can envision the design and development of...


  • India UsefulBI Corporation Full time

    Overview: UsefulBI is looking for highly skilled candidates, with expertise in generating power business insights from very large datasets, in this space where the primary aim would be to enable needle moving business impacts through cutting edge statistical analysis. We are looking for passionate data engineers who can envision the design and development...


  • India People Tech Group Inc Full time

    Hello Linkedin connections We are Hiring '' Lead /Data Architect Aws Devops Engineer '' and below are the requirements. This will be through ''People Tech Group Company''. Position : Aws Lead Devops Engineer/Data Architect Company : People Tech Group ( Mnc) Mode of Work: WFO Experience: 6+yrs Location : Bangalore Only ...

  • AWS DevOps Engineer

    5 days ago


    India Space Multimedia Full time

    Space is an innovative and dynamic company that specializes in leading businesses through transformative journeys in the digital era. They are pioneers in driving innovation and helping organizations transition into digitally mature entities. With a wide range of cutting-edge services, including web enterprise application development, AI & ML development,...

  • Data Engineer

    3 weeks ago


    India Insight Global Full time

    Job Title: Senior Data Engineer Location: Remote (India) – Required to work from 1 PM IST to 12 AM IST Pay Rate: Total compensation ranges from 22 to 35 lakhs per annum, based on experience. Duration: This is a contract position. Only apply if you are comfortable with contract work. The position is approved for full-time work up to a year and is likely...

  • Data Engineer

    4 weeks ago


    India Insight Global Full time

    Job Title: Senior Data EngineerLocation: Remote (India) – Required to work from 1 PM IST to 12 AM ISTPay Rate: Total compensation ranges from 22 to 35 lakhs per annum, based on experience.Duration: This is a contract position. Only apply if you are comfortable with contract work. The position is approved for full-time work up to a year and is likely to...

  • AWS Network Engineer

    2 weeks ago


    India System Soft Technologies Full time

    Job Summary:The client is looking for an AWS Network Engineer:We are seeking an experienced AWS Network Engineer with a strong background in networking and hands-on experience with Amazon Web Services (AWS) infrastructure and services. As an AWS Network Engineer, he/she will be responsible for implementing and maintaining our AWS network infrastructure to...


  • India System Soft Technologies Full time

    Job Summary: The client is looking for an AWS Network Engineer: We are seeking an experienced AWS Network Engineer with a strong background in networking and hands-on experience with Amazon Web Services (AWS) infrastructure and services. As an AWS Network Engineer, he/she will be responsible for implementing and maintaining our AWS network infrastructure...

  • Data Engineer

    2 weeks ago


    India Techiebutler Full time

    Job Title: Data EngineerLocation: AhmedabadEmployment Type: Full-TimeExperience Required: 4+ YearsJob DescriptionTechiebutler is looking for an experienced Data Engineer to develop and maintain scalable, secure data solutions. You will collaborate closely with data science, business analytics, and product development teams, deploying cutting-edge...

  • Data Engineer

    1 week ago


    India Techiebutler Full time

    Job Title: Data Engineer Location: Ahmedabad Employment Type: Full-Time Experience Required: 4+ Years Job Description Techiebutler is looking for an experienced Data Engineer to develop and maintain scalable, secure data solutions. You will collaborate closely with data science, business analytics, and product development teams, deploying cutting-edge...

  • Data Engineer

    2 weeks ago


    India Perficient India Full time

    Snowflake Data EngineerLocation - RemoteRole - FulltimeYears of Experience - 5+Implement and manage data transformation in Snowflake and Aurora with DBT.Build and maintain data pipelines to extract, transform and load data effectively.Work with architect and data analysts to understand data models and requirements.Create technical documents and unit test...

  • Data Engineer

    1 week ago


    India Perficient India Full time

    Snowflake Data Engineer Location - Remote Role - Fulltime Years of Experience - 5+ Implement and manage data transformation in Snowflake and Aurora with DBT. Build and maintain data pipelines to extract, transform and load data effectively. Work with architect and data analysts to understand data models and requirements. Create technical documents and unit...

  • Data Engineer

    3 months ago


    india Suitable.AI Full time

    Role Objective : Data Engineer will be responsible for expanding and optimizing our data and database architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building. The Data Engineer will support our...