BIG DATA ENGINEER

3 weeks ago

bangalore, India Coforge Full time

Job Description

Job description Role and responsibilities : -

Creating Project Technical Documentation - Designing Solution architecture, and work on Data Ingestion, Preparation and Transformation.

Debugging the production failures and identifying the solution.

- Developing efficient frameworks for development and testing using (AWS Dynamo DB, EKS, Kafka) to enable seamless data ingestion process on to the Hadoop platform.

- Enabling Data Governance and Data Discovery on Hadoop Platform

- Building data processing framework using Spark, HQL

- Exposure of Security Framework with Kerberos, Ranger, Atlas

- Exposure of Data Pipeline Automation using DevOps tools

- Exposure of Job Monitoring framework along validations automation

- Exposure of handling structured, Un Structured and Streaming data Technical skills requirements : - Solid hands-on and Solution Architecting experience in Big-Data Technologies (AWS preferred)

- Hands on experience in: AWS Dynamo DB, EKS, Kafka, Kinesis, Glue, EMR

- Hands-on experience of programming language like Scala with Spark.

- Good command and working experience on Hadoop Map Reduce, HDFS, Hive, HBase, and/or No-SQL Databases
- Hands on working experience on any of the data engineering analytics platform (Hortonworks Cloudera MapR AWS), AWS preferred

- Hands-on experience on Data Ingestion Apache Nifi, Apache Airflow, Sqoop, and Oozie

- Hands on working experience of data processing at scale with event driven systems, message queues (Kafka FlinkSpark Streaming)

- Hands on working Experience with AWS Services like EMR, Kinesis, S3, CloudFormation, Glue, API Gateway, Lake Foundation

- Hands on working Experience with AWS Athena - Data Warehouse exposure on Apache Nifi, Apache Airflow, Kylo - Operationalization of ML models on AWS (e.g. deployment, scheduling, model monitoring etc.)

- Feature Engineering Data Processing to be used for Model development

- Experience gathering and processing raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries, etc.)

- Experience building data pipelines for structured unstructured, real-time batch, events synchronous asynchronous using MQ, Kafka, Steam processing

- Hands-on working experience in analysing source system data and data flows, working with structured and unstructured data

- Must be very strong in writing SQL queries

- Strengthen the Data engineering team with Big Data solutions

- Strong technical, analytical, and problem-solving skills

Posted On

Job description Role and responsibilities : -

Creating Project Technical Documentation - Designing Solution architecture, and work on Data Ingestion, Preparation and Transformation.

Debugging the production failures and identifying the solution.

- Developing efficient frameworks for development and testing using (AWS Dynamo DB, EKS, Kafka) to enable seamless data ingestion process on to the Hadoop platform.

- Enabling Data Governance and Data Discovery on Hadoop Platform

- Building data processing framework using Spark, HQL

- Exposure of Security Framework with Kerberos, Ranger, Atlas

- Exposure of Data Pipeline Automation using DevOps tools

- Exposure of Job Monitoring framework along validations automation

- Exposure of handling structured, Un Structured and Streaming data Technical skills requirements : - Solid hands-on and Solution Architecting experience in Big-Data Technologies (AWS preferred)

- Hands on experience in: AWS Dynamo DB, EKS, Kafka, Kinesis, Glue, EMR

- Hands-on experience of programming language like Scala with Spark.

- Good command and working experience on Hadoop Map Reduce, HDFS, Hive, HBase, and/or No-SQL Databases
- Hands on working experience on any of the data engineering analytics platform (Hortonworks Cloudera MapR AWS), AWS preferred

- Hands-on experience on Data Ingestion Apache Nifi, Apache Airflow, Sqoop, and Oozie

- Hands on working experience of data processing at scale with event driven systems, message queues (Kafka FlinkSpark Streaming)

- Hands on working Experience with AWS Services like EMR, Kinesis, S3, CloudFormation, Glue, API Gateway, Lake Foundation

- Hands on working Experience with AWS Athena - Data Warehouse exposure on Apache Nifi, Apache Airflow, Kylo - Operationalization of ML models on AWS (e.g. deployment, scheduling, model monitoring etc.)

- Feature Engineering Data Processing to be used for Model development

- Experience gathering and processing raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries, etc.)

- Experience building data pipelines for structured unstructured, real-time batch, events synchronous asynchronous using MQ, Kafka, Steam processing

- Hands-on working experience in analysing source system data and data flows, working with structured and unstructured data

- Must be very strong in writing SQL queries

- Strengthen the Data engineering team with Big Data solutions

- Strong technical, analytical, and problem-solving skills

 

Skills Required

AWS EMR

Posted On AWS EMR Location

Bangalore / Greater Noida / Pune / Hyderabad

Posted On Bangalore / Greater Noida / Pune / Hyderabad

  • Big Data Engineer

    4 weeks ago

    bangalore, India Virtusa Full time

    Big Data Engineer - CREQ184231 Description Profile : 5-8 years IT experience with minimum 2 years in Spark and Big Data platform Experience in Big Data technologies Hadoop, HDFS, Hive, Yarn, Jenkins, Ansible Job DescriptionResponsibilitiesSkill: Spark, ScalaLevel: Senior Software EngineerCategory: Special/NicheTenure: 1 YearMission :He (or she) is in charge...

  • Big Data Engineer

    4 weeks ago

    bangalore, India Hexaware Technologies Full time

    Description Data Engineer: Primary : ADF, SQL Seconday : Databricks, Pyspark / Scala, CICD, Data Security, encryption Responsibilities: •Work closely with scrum master, technical leads and provide daily progress updates on the tasks assigned. •Provide support in defining the scope and sizing of the work. •Working closely with...

  • Big Data Developer

    4 weeks ago

    Bangalore Metropolitan Area, India Aptus Data Labs Full time

    Job Title: Big Data Developers Experience: Minimum 8 yearsLocation: Remote / core location - BangaloreMandatory Skills (Hadoop , Hive, HiveQL , Cloudera , ETL, Pyspark/Python, Shell Scripting, Automation )Strong proficiency in Hadoop ecosystem including HDFS, Hive, and HSQLExtensive experience with Python and PySpark for data processing and...

  • Big Data Engineer

    4 weeks ago

    bangalore, India TalentPulse Consultant Full time

    Role : Big Data Engineer Job Description : We are currently seeking a highly skilled Big Data Engineer to join our team on a remote basis. The ideal candidate will have at least 5 years of experience in Big Data technologies and a strong proficiency in Python, Spark, and Scala. This is an active position with a high priority, offering a competitive rate...

  • Big Data Engineer

    3 weeks ago

    Bangalore, Karnataka, India TalentPulse Consultant Full time

    Role : Big Data EngineerJob Description :We are currently seeking a highly skilled Big Data Engineer to join our team on a remote basis.The ideal candidate will have at least 5 years of experience in Big Data technologies and a strong proficiency in Python, Spark, and Scala.This is an active position with a high priority, offering a competitive rate and the...

  • Big Data Engineer

    2 weeks ago

    Bangalore, India TalentPulse Consultant Full time

    Role : Big Data Engineer Job Description : We are currently seeking a highly skilled Big Data Engineer to join our team on a remote basis. The ideal candidate will have at least 5 years of experience in Big Data technologies and a strong proficiency in Python, Spark, and Scala. This is an active position with a high priority, offering a competitive rate...

  • Big Data Engineer

    2 days ago

    bangalore, India Infogain Full time

    Big Data Engineer (Lead) with skills Big Data, Data Warehouse, Azure Datalake for location Any Infogain Base Location (Noida, Gurugram, Bangalore, Mumbai, Pune) Posted on: May 14, Share on Linkedin Share on Twitter Share on Facebook ROLES & RESPONSIBILITIES Responsibilities: · Strong experience of minimum 5+ years in Snowflake &...

  • Aws big data engineer

    4 weeks ago

    bangalore, India Global Pharma Tek Full time

    ob Description Technical skills . Having ETL development experience with overall to years in AWS , Big data engineering related application Minimum end to end ETL project implementation Proficient in PySpark, Python. Proficient in Big Data technologies - Hadoop, Spark, Hive, Databricks. Expert in AWS to develop, deploy and maintain ETL applications...

  • Senior Data Engineer

    3 weeks ago

    Bangalore, Karnataka, India Muoro Full time

    Job Description :About the role :We are looking for a Senior Data Engineer to join our growing engineering team. As a member of the team:- You will work on enterprise data platforms, architect and implement data lakes both on-prem and in the cloud.- You will be responsible for evolving technical architecture, design and implementation of data solutions using...

  • Senior Data Engineer

    4 weeks ago

    Bangalore, India Muoro Full time

    Job Description : About the role :We are looking for a Senior Data Engineer to join our growing engineering team. As a member of the team:- You will work on enterprise data platforms, architect and implement data lakes both on-prem and in the cloud.- You will be responsible for evolving technical architecture, design and implementation of data solutions...

  • Senior Data Engineer

    3 weeks ago

    Bangalore, Karnataka, India Squareroot Consulting Pvt Ltd Full time

    We are seeking a highly skilled and experienced Senior Data Engineer to join our team.Location : Bangalore (Work from Office all 5 days)Experience : 6+ YearsRole : Senior Data EngineerThe ideal candidate will have a strong background in big data technologies and extensive hands-on experience with Cloudera, Spark, Hive, HDFS, Flink, Data Lake, and...

  • Senior Data Engineer

    4 weeks ago

    bangalore, India Squareroot Consulting Pvt Ltd Full time

    We are seeking a highly skilled and experienced Senior Data Engineer to join our team.Location : Bangalore (Work from Office all 5 days)Experience : 6+ YearsRole : Senior Data EngineerThe ideal candidate will have a strong background in big data technologies and extensive hands-on experience with Cloudera, Spark, Hive, HDFS, Flink, Data Lake, and...

  • Big Data Architect

    4 weeks ago

    bangalore, India RiskInsight Consulting Pvt Ltd Full time

    Responsibilities Design and architect Big Data solutions to meet business requirements. Collaborate with stakeholders to understand their data and analytics needs. Create and maintain data models, data flow diagrams, and data integration strategies. Develop and implement data ingestion, processing, and storage solutions using Big Data technologies. ...

  • Senior Data Engineer

    2 weeks ago

    Bangalore, India Squareroot Consulting Pvt Ltd Full time

    We are seeking a highly skilled and experienced Senior Data Engineer to join our team.Location : Bangalore (Work from Office all 5 days)Experience : 6+ YearsRole : Senior Data EngineerThe ideal candidate will have a strong background in big data technologies and extensive hands-on experience with Cloudera, Spark, Hive, HDFS, Flink, Data Lake, and...

  • Big Data Developer

    3 weeks ago

    bangalore, India BigShyft Full time

    JobDescription: We are seeking askilled and experienced Big Data Developer/Data Engineer to joinour clients team in Bengaluru. The ideal candidate should have astrong background in Java Spark Hadoop and Scala with 47 years ofrelevant workexperience.Responsibilities: Designdevelop and maintain largescale data processing systems leveragingHadoop Spark and...

  • Big Data Engineer

    4 days ago

    bangalore, India TekWissen India Full time

    Role: Big Data EngineerLocation: Bangalore-OnsiteWorktype: Contract for 6 monthsPRINCIPAL RESPONSIBILITIES:- Gain core business understanding of Textron Aviation and aircraft design, operation, and support- Query, clean, transform, and stage data (ETL/ELT) across on-prem and cloud environments- Support data analytics and data science activities by...

  • Senior Big Data Engineer

    4 weeks ago

    bangalore, India Recrud Full time

    Senior Big Data Engineer Position. Experience Required: 6 -10 Years. Employment Type: Full-Time. About Our Client:. Our client is a distinguished product engineering consulting firm based in Bangalore, dedicated to providing innovative offshore development solutions. With over half a century of combined expertise in product engineering and consulting...

  • Big data architect

    4 weeks ago

    bangalore, India Virtusa Full time

    Big data architect - CREQ185795 Description Position: Big data architect Location: BLR Primary skills: Big data hadoop spark & aws Experience in building SparkStreaming process. Proficient in understanding distributed computing principles. Experience in managing Hadoop cluster, BigData with all services.Proficiency with Hadoop MapReduce HDFS Pig...

  • Big Data Developer

    4 weeks ago

    bangalore, India Workassist.in Full time

    Our hiring partner is looking for :- This is a full-time on-site role for a Big Data Developer at Big Four Consulting Firm Gurugram.- The Big Data Developer will be responsible for tasks related to Data Engineering, Extract Transform Load (ETL), Big Data, Data Warehousing, and Software Data Engineering, Big Data, Extract Transform Load (ETL), and Data...

  • Big-Data Engineer with Neo4j and Couldera

    2 weeks ago

    bangalore, India HTC Global Services Full time

    Big-Data Engineer with Neo4j and CoulderaExperience: 5+ YearsLocation: Chennai/ Bangalore/ HyderabadNotice period: Immediate to 15 days.Skills:Big-Data Platform (Cloudera+Neo4J), particularly specialized in Spark/Pyspark, Neo4J, HBASE, KUDU, Flink, Kafka, Ontology, APIs (Spring-Boot), Python, Scala, Java, Cypher & GremlinJD:Responsibilities : Mainly involved...