Big Data- AWS

6 months ago


Greater Noida, India Coforge Full time

Job Description

Job description Role and responsibilities : - Creating Project Technical Documentation - Designing Solution architecture, and work on Data Ingestion, Preparation and Transformation. Debugging the production failures and identifying the solution. - Developing efficient frameworks for development and testing using (AWS Dynamo DB, EKS, Kafka) to enable seamless data ingestion process on to the Hadoop platform. - Enabling Data Governance and Data Discovery on Hadoop Platform - Building data processing framework using Spark, HQL - Exposure of Security Framework with Kerberos, Ranger, Atlas - Exposure of Data Pipeline Automation using DevOps tools - Exposure of Job Monitoring framework along validations automation - Exposure of handling structured, Un Structured and Streaming data Technical skills requirements : - Solid hands-on and Solution Architecting experience in Big-Data Technologies (AWS preferred) - Hands on experience in: AWS Dynamo DB, EKS, Kafka, Kinesis, Glue, EMR - Hands-on experience of programming language like Scala with Spark. - Good command and working experience on Hadoop Map Reduce, HDFS, Hive, HBase, and/or No-SQL Databases - Hands on working experience on any of the data engineering analytics platform (Hortonworks Cloudera MapR AWS), AWS preferred - Hands-on experience on Data Ingestion Apache Nifi, Apache Airflow, Sqoop, and Oozie - Hands on working experience of data processing at scale with event driven systems, message queues (Kafka FlinkSpark Streaming) - Hands on working Experience with AWS Services like EMR, Kinesis, S3, CloudFormation, Glue, API Gateway, Lake Foundation - Hands on working Experience with AWS Athena - Data Warehouse exposure on Apache Nifi, Apache Airflow, Kylo - Operationalization of ML models on AWS (e.g. deployment, scheduling, model monitoring etc.) - Feature Engineering Data Processing to be used for Model development - Experience gathering and processing raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries, etc.) - Experience building data pipelines for structured unstructured, real-time batch, events synchronous asynchronous using MQ, Kafka, Steam processing - Hands-on working experience in analysing source system data and data flows, working with structured and unstructured data - Must be very strong in writing SQL queries - Strengthen the Data engineering team with Big Data solutions - Strong technical, analytical, and problem-solving skills

Posted On

Job description Role and responsibilities : - Creating Project Technical Documentation - Designing Solution architecture, and work on Data Ingestion, Preparation and Transformation. Debugging the production failures and identifying the solution. - Developing efficient frameworks for development and testing using (AWS Dynamo DB, EKS, Kafka) to enable seamless data ingestion process on to the Hadoop platform. - Enabling Data Governance and Data Discovery on Hadoop Platform - Building data processing framework using Spark, HQL - Exposure of Security Framework with Kerberos, Ranger, Atlas - Exposure of Data Pipeline Automation using DevOps tools - Exposure of Job Monitoring framework along validations automation - Exposure of handling structured, Un Structured and Streaming data Technical skills requirements : - Solid hands-on and Solution Architecting experience in Big-Data Technologies (AWS preferred) - Hands on experience in: AWS Dynamo DB, EKS, Kafka, Kinesis, Glue, EMR - Hands-on experience of programming language like Scala with Spark. - Good command and working experience on Hadoop Map Reduce, HDFS, Hive, HBase, and/or No-SQL Databases - Hands on working experience on any of the data engineering analytics platform (Hortonworks Cloudera MapR AWS), AWS preferred - Hands-on experience on Data Ingestion Apache Nifi, Apache Airflow, Sqoop, and Oozie - Hands on working experience of data processing at scale with event driven systems, message queues (Kafka FlinkSpark Streaming) - Hands on working Experience with AWS Services like EMR, Kinesis, S3, CloudFormation, Glue, API Gateway, Lake Foundation - Hands on working Experience with AWS Athena - Data Warehouse exposure on Apache Nifi, Apache Airflow, Kylo - Operationalization of ML models on AWS (e.g. deployment, scheduling, model monitoring etc.) - Feature Engineering Data Processing to be used for Model development - Experience gathering and processing raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries, etc.) - Experience building data pipelines for structured unstructured, real-time batch, events synchronous asynchronous using MQ, Kafka, Steam processing - Hands-on working experience in analysing source system data and data flows, working with structured and unstructured data - Must be very strong in writing SQL queries - Strengthen the Data engineering team with Big Data solutions - Strong technical, analytical, and problem-solving skills

Skills Required

AWS EMR

Posted On AWS EMR Role

y

Posted On

y

Location

Greater Noida

Posted On Greater Noida

  • Greater Noida, Uttar Pradesh, India Coforge Full time

    Coforge is seeking a seasoned Big Data Solutions Architect to lead the design and implementation of large-scale data processing architectures on AWS. This role requires expertise in building efficient frameworks for development and testing, as well as hands-on experience with tools like Apache NiFi, Apache Airflow, Sqoop, and Oozie.Key...


  • Greater Noida, Uttar Pradesh, India Coforge Full time

    As a Senior Big Data Engineer at Coforge, you will play a pivotal role in designing and developing scalable solutions on the Amazon Web Services (AWS) platform. Your primary responsibility will be to architect and implement large-scale data processing systems that cater to our clients' ever-growing data needs.Candidates with hands-on experience in Big-Data...


  • Greater Noida, Uttar Pradesh, India Coforge Full time

    Job Description:Coforge is seeking a highly skilled Big Data Architect to lead our cloud-based data engineering efforts. In this role, you will design and implement scalable data architectures using AWS technologies such as Dynamo DB, EKS, and Kafka.You will work closely with our development teams to ensure seamless data ingestion, processing, and storage on...

  • Big Data Architect

    7 months ago


    Noida, India Skillety Technologies Full time

    **MapReduce.** **experience working with cloud or on-prem Big Data/MPP analytics platform (i.e. AWS Glue/EMR, Google BigQuery,** **Azure Data Warehouse, Databricks, Snowflake, Hortonworks/Cloudera Clusters)** **exp in writing Advanced SQLs** **exp with building dimensional data models to improve accessibility, efficiency and quality of data** **exp in custom...

  • Aws Expert

    6 months ago


    Greater Noida, India Coforge Ltd. Full time

    **Must Have**: 1. 5-6 years of experience in Multi Cloud environment (AWS is must) 2. Understand functional requirements and help Analyst for technical feasibility. 3. Design and implementation of AWS Data Analytics platform (EMR) 4. Should able to capture Technical requirements and design. Help team to implement. 5. Able to build PoC to show sample...

  • Big Data Developer

    3 weeks ago


    Greater Bengaluru Area, India Pixomania Full time

    Job Title: Big Data Developer (Scala + Spark)Location: RemoteExperience: 5+ yearsJob Type: Contractual RemoteAbout Us:We are a forward-thinking technology company committed to delivering innovative solutions in Big Data analytics. Join our talented team as a Big Data Developer with expertise in Scala and Apache Spark to drive impactful projects and...

  • Big Data Developer

    3 weeks ago


    Greater Bengaluru Area, India Pixomania Full time

    Job Title: Big Data Developer (Scala + Spark)Location: RemoteExperience: 5+ yearsJob Type: Contractual RemoteAbout Us:We are a forward-thinking technology company committed to delivering innovative solutions in Big Data analytics. Join our talented team as a Big Data Developer with expertise in Scala and Apache Spark to drive impactful projects and...

  • AWS Data Scientist

    1 month ago


    Bangalore/Pune/Greater Noida, India HyrEzy Talent Solutions Full time

    Job Title : AWS Data Engineer (en)Location : VariousKey Responsibilities :Design, develop, and maintain large-scale data architectures on the AWS cloud platform.Develop and implement efficient data pipelines for ingesting, processing, and transforming massive data volumes from diverse sources.Design and optimize data models for efficient storage, retrieval,...

  • Aws Data Architect

    6 months ago


    Noida, India Skillety Technologies Full time

    **Overall 9+ years of exp in software industry** **5+ years of experience working with cloud or on-prem Big Data/MPP analytics platform (i.e. AWS Glue/EMR, Google** **BigQuery, Azure Data Warehouse, Databricks, Snowflake, Hortonworks/Cloudera Clusters)** **3+ years of exp in writing Advanced SQLs** **3+ years of exp with building dimensional data models to...

  • Big Data Developer

    3 weeks ago


    Greater Bengaluru Area, India Pixomania Full time

    As a Big Data Developer, you will design, develop, and optimize distributed data processing systems using Apache Spark and Hive. You will work on large-scale data platforms and ensure the efficient processing of massive datasets in real-time and batch environments. Your experience with Kafka will be an added advantage in building robust and scalable...

  • Big Data Developer

    3 weeks ago


    Greater Bengaluru Area, India Pixomania Full time

    As a Big Data Developer, you will design, develop, and optimize distributed data processing systems using Apache Spark and Hive. You will work on large-scale data platforms and ensure the efficient processing of massive datasets in real-time and batch environments. Your experience with Kafka will be an added advantage in building robust and scalable...


  • Noida, Uttar Pradesh, India Impetus Full time

    Impetus is a renowned company in the field of data engineering, looking for an experienced Big Data Technology Specialist to join our team.We offer an attractive salary package of $120,000 - $180,000 per annum, depending on experience and qualifications. The ideal candidate will have at least 4-8 years of hands-on experience with Big Data technologies,...


  • Noida, Uttar Pradesh, India Impetus Full time

    Join Impetus, a leading IT solutions company, as a Big Data Developer in Indore, Pune, Noida, or Gurugram. With an estimated salary of 1200000 INR per year, this role offers exciting opportunities for professional growth and development.Company OverviewAbout Impetus: We are a forward-thinking organization that thrives on innovation and collaboration. Our...

  • Big Data Engineer

    5 days ago


    Noida, India Tekshapers Software Solutions (P) Limited Full time

    Job Description :As a Big Data Engineer, you will be responsible for designing, developing, and maintaining data pipelines to extract, transform, and load (ETL) data from various sources into target systems. You will work closely with data analysts and data scientists to ensure data quality, accuracy, and consistency.Key Responsibilities :- Design and...


  • noida, India Big Oh Tech Full time

    DevOps At Big Oh (Noida) Roles and Responsibilities:   1 years of DevOps Engineering and Service Delivery: Proven expertise in CI/CD and containerization with hands-on experience in cloud environments such as AWS, Microsoft Azure, and others.  Automation and Cloud Integration: Skilled in designing and delivering automated CI/CD solutions in cloud...

  • AWS Data Engineer

    2 months ago


    Noida, India Firstcareercentre Full time

    Job Summary :We are seeking a highly skilled and experienced AWS Data Engineer to join our data team. The AWS Data Engineer will be responsible for designing, developing, and optimizing data pipelines, integrating various data sources, and implementing AWS cloud solutions. The ideal candidate should have a strong background in data engineering, cloud...

  • Big data developer

    1 week ago


    Noida, India Impetus Full time

    Lead Software Engineer Locations- Indore, Noida, Hyderabad Job Description We are looking for a Lead Big Data Developer to design and oversee the implementation of large-scale data solutions, lead a talented team, and ensure the efficient processing and analysis of massive datasets. Roles & Responsibilities Key Responsibilities Architect...

  • Big Data Developer

    3 weeks ago


    Noida, India Impetus Full time

    Lead Software EngineerLocations- Indore, Noida, HyderabadJob DescriptionWe are looking for a Lead Big Data Developer to design and oversee the implementation of large-scale data solutions, lead a talented team, and ensure the efficient processing and analysis of massive datasets.Roles & ResponsibilitiesKey ResponsibilitiesArchitect Data Solutions: Design...

  • Big data developer

    3 weeks ago


    Noida, India Impetus Full time

    Lead Software EngineerLocations- Indore, Noida, HyderabadJob DescriptionWe are looking for a Lead Big Data Developer to design and oversee the implementation of large-scale data solutions, lead a talented team, and ensure the efficient processing and analysis of massive datasets.Roles & ResponsibilitiesKey ResponsibilitiesArchitect Data Solutions: Design...

  • Big Data Developer

    3 weeks ago


    Noida, India Impetus Full time

    Lead Software EngineerLocations- Indore, Noida, HyderabadJob DescriptionWe are looking for aLead Big Data Developerto design and oversee the implementation of large-scale data solutions, lead a talented team, and ensure the efficient processing and analysis of massive datasets.Roles & ResponsibilitiesKey ResponsibilitiesArchitect Data Solutions:Design and...