Big Data Engineer

4 weeks ago

Bengaluru, India Coforge Full time
Job Description

Job description Role and responsibilities : -Experience required- 4 to 10 YearsCreating Project Technical Documentation - Designing Solution architecture, and work on Data Ingestion, Preparation and Transformation.Debugging the production failures and identifying the solution.- Developing efficient frameworks for development and testing using (AWS Dynamo DB, EKS, Kafka) to enable seamless data ingestion process on to the Hadoop platform.- Enabling Data Governance and Data Discovery on Hadoop Platform- Building data processing framework using Spark, HQL- Exposure of Security Framework with Kerberos, Ranger, Atlas- Exposure of Data Pipeline Automation using DevOps tools- Exposure of Job Monitoring framework along validations automation- Exposure of handling structured, Un Structured and Streaming data Technical skills requirements : - Solid hands-on and Solution Architecting experience in Big-Data Technologies (AWS preferred)- Hands on experience in: AWS Dynamo DB, EKS, Kafka, Kinesis, Glue, EMR- Hands-on experience of programming language like Scala with Spark.- Good command and working experience on Hadoop Map Reduce, HDFS, Hive, HBase, and/or No-SQL Databases- Hands on working experience on any of the data engineering analytics platform (Hortonworks Cloudera MapR AWS), AWS preferred- Hands-on experience on Data Ingestion Apache Nifi, Apache Airflow, Sqoop, and Oozie- Hands on working experience of data processing at scale with event driven systems, message queues (Kafka FlinkSpark Streaming)- Hands on working Experience with AWS Services like EMR, Kinesis, S3, CloudFormation, Glue, API Gateway, Lake Foundation- Hands on working Experience with AWS Athena - Data Warehouse exposure on Apache Nifi, Apache Airflow, Kylo - Operationalization of ML models on AWS (e.g. deployment, scheduling, model monitoring etc.)- Feature Engineering Data Processing to be used for Model development- Experience gathering and processing raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries, etc.)- Experience building data pipelines for structured unstructured, real-time batch, events synchronous asynchronous using MQ, Kafka, Steam processing- Hands-on working experience in analysing source system data and data flows, working with structured and unstructured data- Must be very strong in writing SQL queries- Strengthen the Data engineering team with Big Data solutions- Strong technical, analytical, and problem-solving skillsPosted On

Job description Role and responsibilities : -Experience required- 4 to 10 YearsCreating Project Technical Documentation - Designing Solution architecture, and work on Data Ingestion, Preparation and Transformation.Debugging the production failures and identifying the solution.- Developing efficient frameworks for development and testing using (AWS Dynamo DB, EKS, Kafka) to enable seamless data ingestion process on to the Hadoop platform.- Enabling Data Governance and Data Discovery on Hadoop Platform- Building data processing framework using Spark, HQL- Exposure of Security Framework with Kerberos, Ranger, Atlas- Exposure of Data Pipeline Automation using DevOps tools- Exposure of Job Monitoring framework along validations automation- Exposure of handling structured, Un Structured and Streaming data Technical skills requirements : - Solid hands-on and Solution Architecting experience in Big-Data Technologies (AWS preferred)- Hands on experience in: AWS Dynamo DB, EKS, Kafka, Kinesis, Glue, EMR- Hands-on experience of programming language like Scala with Spark.- Good command and working experience on Hadoop Map Reduce, HDFS, Hive, HBase, and/or No-SQL Databases- Hands on working experience on any of the data engineering analytics platform (Hortonworks Cloudera MapR AWS), AWS preferred- Hands-on experience on Data Ingestion Apache Nifi, Apache Airflow, Sqoop, and Oozie- Hands on working experience of data processing at scale with event driven systems, message queues (Kafka FlinkSpark Streaming)- Hands on working Experience with AWS Services like EMR, Kinesis, S3, CloudFormation, Glue, API Gateway, Lake Foundation- Hands on working Experience with AWS Athena - Data Warehouse exposure on Apache Nifi, Apache Airflow, Kylo - Operationalization of ML models on AWS (e.g. deployment, scheduling, model monitoring etc.)- Feature Engineering Data Processing to be used for Model development- Experience gathering and processing raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries, etc.)- Experience building data pipelines for structured unstructured, real-time batch, events synchronous asynchronous using MQ, Kafka, Steam processing- Hands-on working experience in analysing source system data and data flows, working with structured and unstructured data- Must be very strong in writing SQL queries- Strengthen the Data engineering team with Big Data solutions- Strong technical, analytical, and problem-solving skills

Skills Required

AWS EMRPosted On AWS EMR Role

yPosted On

yLocation

Bangalore / Greater Noida / Pune / HyderabadPosted On Bangalore / Greater Noida / Pune / Hyderabad

  • Big Data Engineer

    6 hours ago

    Bengaluru, Karnataka, India Virtusa Full time

    Big Data Engineer - CREQ188642 Description Looking for a Senior Software Engineer with strong analytical skills to help build and enhance our Big data application. This individual's day-to-day work will involve using Java, Scala, SQL, Big Data, Linux, git, and maven/gradle to design and build software solutions that are fast, scalable, and...

  • Big Data Engineer

    10 hours ago

    Bengaluru, Karnataka, India TalentPulse Consultant Full time

    Role : Big Data Engineer Job Description : We are currently seeking a highly skilled Big Data Engineer to join our team on a remote basis. The ideal candidate will have at least 5 years of experience in Big Data technologies and a strong proficiency in Python, Spark, and Scala. This is an active position with a high priority, offering a competitive rate...

  • Big Data Engineer

    2 weeks ago

    Bengaluru, India Awign Expert Full time

    This is a remote position.Job Title: Big Data Solutions EngineerReporting to: Director, Product ManagementLocation: India (Remote)Our Ideal CandidateOur company is seeking a talented Big Data Solutions Engineer to join our team.The ideal candidate will be responsible for designing, developing, and implementing our proprietary Bigdata-powered Fin Crime...

  • Big Data Engineer

    1 month ago

    Bengaluru, India Virtusa Full time

    Big Data Engineer - CREQ188642 Description Looking for a Senior Software Engineer with strong analytical skills to help build and enhance our Big data application. This individual's day-to-day work will involve using Java, Scala, SQL, Big Data, Linux, git, and maven/gradle to design and build software solutions that are fast, scalable, and flexible....

  • Big Data Engineer

    1 month ago

    Bengaluru, India Virtusa Full time

    Big Data Engineer - CREQ188642 DescriptionLooking for a Senior Software Engineer with strong analytical skills to help build and enhance our Big data application. This individual's day-to-day work will involve using Java, Scala, SQL, Big Data, Linux, git, and maven/gradle to design and build software solutions that are fast, scalable, and flexible.Bachelor's...

  • Big Data Engineer

    2 weeks ago

    Bengaluru, India Awign Expert Full time

    This is a remote position. Job Title: Big Data Solutions Engineer Reporting to: Director, Product Management Location: India (Remote) Our Ideal Candidate Our company is seeking a talented Big Data Solutions Engineer to join our team. The ideal candidate will be responsible for designing, developing, and implementing our proprietary...

  • Big Data Engineer

    2 weeks ago

    Bengaluru, India Awign Expert Full time

    Job Description This is a remote position. Job Title: Big Data Solutions Engineer Reporting to: Director, Product Management Location: India (Remote) Our Ideal Candidate Our company is seeking a talented Big Data Solutions Engineer to join our team. The ideal candidate will be responsible for designing, developing, and...

  • Big Data Engineer

    2 weeks ago

    Bengaluru, India Awign Expert Full time

    This is a remote position. Job Title: Big Data Solutions Engineer Reporting to: Director, Product Management Location: India (Remote) Our Ideal Candidate Our company is seeking a talented Big Data Solutions Engineer to join our team. The ideal candidate will be responsible for designing, developing, and implementing our proprietary...

  • Big Data Engineer

    8 hours ago

    Bengaluru, Karnataka, India Awign Expert Full time

    This is a remote position. Job Title: Big Data Solutions Engineer Reporting to: Director, Product Management Location: India (Remote) Our Ideal Candidate Our company is seeking a talented Big Data Solutions Engineer to join our team. The ideal candidate will be responsible for designing, developing, and implementing our proprietary Bigdata-powered Fin...

  • Big Data Engineer

    2 weeks ago

    Bengaluru, India Awign Expert Full time

    Job Description This is a remote position. Job Title: Big Data Solutions Engineer Reporting to: Director, Product Management Location: India (Remote) Our Ideal Candidate Our company is seeking a talented Big Data Solutions Engineer to join our team. The ideal candidate will be responsible for designing, developing, and...

  • Big Data Engineer

    2 hours ago

    Bengaluru, Karnataka, India Awign Expert Full time

    This is a remote positionJob Title:Big Data Solutions EngineerReporting to: Director, Product ManagementLocation: India (Remote)Our Ideal CandidateOur company is seeking a talented Big Data Solutions Engineer to join our team.The ideal candidate will be responsible for designing, developing, and implementing our proprietary Bigdata-powered Fin Crime solution...

  • Big Data Engineer

    2 hours ago

    Bengaluru, Karnataka, India Evnek Technologies Pvt Ltd Full time

    Job DescriptionThis is a remote position.Create and maintain data pipelines that transfer data from select locations (maintaining security protocols) into our Big Data financial crime solution.Open source and fintech ka experience MUST.Develop, implement, and standardize ETL processes to transform and cleanse data for pipeline development and...

  • Big Data Engineer

    5 hours ago

    Bengaluru, Karnataka, India Awign Expert Full time

    Job Description This is a remote position. Job Title: Big Data Solutions Engineer Reporting to: Director, Product Management Location: India (Remote) Our Ideal Candidate Our company is seeking a talented Big Data Solutions Engineer to join our team. The ideal candidate will be responsible for designing, developing, and implementing our...

  • Big Data Engineer

    7 hours ago

    Bengaluru, Karnataka, India Evnek Technologies Pvt Ltd Full time

    This is a remote positionDuration:6 MonthLocation:RemoteTimings: Full Time (As per company timings)Notice Period: within 15 days or immediate joinerExperience: 4-6 YearsResponsibilitiesCreate and maintain data pipelines that transfer data from select locations (maintaining security protocols) into our Big Data financial crime solution.Open source and...

  • Big Data Engineer

    9 hours ago

    Bengaluru, Karnataka, India Infogain Full time

    Big Data Engineer (Lead) with skills Big Data, Data Warehouse, Azure Datalake for location Any Infogain Base Location (Noida, Gurugram, Bangalore, Mumbai, Pune)ROLES & RESPONSIBILITIESResponsibilities:Strong experience of minimum 5+ years in Snowflake & Snowpipe having over all 9 to 12 years of experience as a data engineer, database developer, or similar...

  • Big Data Engineer

    5 hours ago

    Bengaluru, Karnataka, India WEN Full time

    Role : Big Data Engineer (Scala)Responsibilities : Develop and maintain scalable data pipelines using Spark and Hive within the Hadoop ecosystem Utilize Scala programming language to implement efficient data processing solutions Collaborate with crossfunctional teams to understand data requirements and deliver optimal solutions Implement best practices for...

  • Big Data Engineer

    11 hours ago

    Bengaluru, Karnataka, India Infogain Full time

    Big Data Engineer (Lead) with skills Big Data, Data Warehouse, Azure Datalake for location Any Infogain Base Location (Noida, Gurugram, Bangalore, Mumbai, Pune)Posted on: June 07, Share on Linkedin Share on Twitter Share on FacebookROLES & RESPONSIBILITIESResponsibilities: Strong experience ofminimum 5+ years in Snowflake & Snowpipehaving over all 9 to 12...

  • Big Data Engineer

    2 months ago

    Bengaluru, India Adobe Full time

    Our CompanyChanging the world through digital experiences is what Adobe’s all about. We give everyone—from emerging artists to global brands—everything they need to design and deliver exceptional digital experiences! We’re passionate about empowering people to create beautiful and powerful images, videos, and apps, and transform how companies...

  • Big Data Engineer

    3 hours ago

    Bengaluru, Karnataka, India Infogain Full time

    Big Data Engineer (Senior) with skills Big Data, Data Warehouse, Azure Datalake for location Any Infogain Base Location (Noida, Gurugram, Bangalore, Mumbai, Pune)ROLES & RESPONSIBILITIESStrong experience of 9 to 12 years as a data engineer, database developer, or similar role, focusing on data warehousing and analytics.Minimum 5 years of experience in...

  • big data engineer

    10 hours ago

    Bengaluru, Karnataka, India Techno Wise Full time

    Apply for BIG DATA ENGINEER, Techno Wise in Bengaluru/ Bangalore ,Mumbai ,Pune for 4 - 8 Year of Experience on