Senior Big Data Engineer

1 week ago

Gurugram, India FxConsulting Full time

Position : Big Data EngineerExperience : 59 Years Job Type : Full-timeJob Summary :We are seeking a highly skilled Big Data Engineer with 59 years of experience to develop, maintain, and optimize robust, scalable big data solutions. This role requires expertise in designing and implementing solutions using Hadoop-based technologies and AWS. The engineer will be crucial in building various complex Big Data platform components, focusing on optimizing data access, handling high data volumes, and ensuring real-time performance using frameworks like Spark, Kafka, and other cloud technologies.Key Responsibilities :Big Data Platform Architecture and Design :- Lead the design of comprehensive data solutions leveraging Hadoop-based technologies in conjunction with AWS cloud infrastructure.- Implement and design various critical Big Data platform components, including Batch Processing, Live Stream Processing, In-Memory Cache, Query Layer (SQL), Rule Engine, and Action Framework.- Design and implement a robust Data Access Layer capable of connecting to diverse data sources.Performance and Scalability :- Implement scalable technical solutions to reliably handle ever-increasing data volumes using modern big data and cloud technologies like Spark, Kafka, and Cloud computing services.- Utilize advanced caching techniques within the Data Access Layer to ensure fast responses to real-time SQL queries against Big Data Technologies.Data Pipeline and Infrastructure :- Set up and operate end-to-end data pipelines and the overall infrastructure required for executing ETL jobs.- Apply proficiency with distributed file systems and computing concepts to optimize data :- Experience: 59 years of hands-on Data engineering experience.- Programming Languages: Good hands-on knowledge and proficiency in one of the primary languages: Java, Scala, or Python.- Data Handling Frameworks: Strong hands-on experience with at least one of the major data handling frameworks: Apache Spark, Apache Beam, or Apache Flink.- Data Store: Good technical understanding of both SQL and NoSQL databases (e.g., MySQL, Mongo).- Data Infrastructure: Familiarity with cloud solutions for data infrastructure, with AWS preferably.- Distributed Concepts: Proficient with distributed file systems and computing concepts (e.g., HDFS, YARN).Preferred Skills :- Familiarity with different types of efficient data formats, including Apache Parquet, Avro, and ORC.- Direct experience integrating streaming technologies like Kafka with processing frameworks like Spark or Flink.- Experience in designing a Rule Engine or Action Framework for real-time data processing.- Certifications in AWS Data Analytics or related Big Data technologies. (ref:hirist.tech)

  • Senior Big Data Software Engineer

    8 hours ago

    Gurugram, India Epam Full time

    Description EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver...

  • Senior Data Engineer

    3 weeks ago

    Gurugram, India DEUTSCHE TELEKOM DIGITAL LABS PRIVATE LIMITED Full time

    Responsibilities :- You will be involved in the design of data solutions using Hadoop based technologies along with Hadoop, AWS.- Responsibilities includes design and implementation of various Big Data platform components like (Batch Processing, Live Stream Processing, In- Memory Cache, Query Layer (SQL), Rule Engine and Action Framework ).- Design and...

  • Senior Engineering Manager: Big Data Diversity

    9 hours ago

    Gurugram, India Questhiring Full time

    As an Engineering Manager, you will ensure consistency and quality by shaping the right strategies. You will keep an eye on all engineering projects and ensure all duties are fulfilled. You will also act as an Architect for this team and will be responsible for design and long-term roadmap. You will analyse other employees tasks and carry on collaborations...

  • Senior Data Engineer

    8 hours ago

    Gurugram, India Invokhr Full time

    Position: Senior Data Engineer / Data Engineer Desired Experience: 3-8 years Salary: Best-in-industry You will act as a key member of the Data consulting team, working directly with the partners and senior stakeholders of the clients designing and implementing big data and analytics solutions. Communication and organisation skills are keys for this position,...

  • Senior Data Engineer

    8 hours ago

    Gurugram, India Invokhr Full time

    Desired Experience: 3-8 years Salary: Best-in-industry Location: Gurgaon ( 5 days onsite) Overview: You will act as a key member of the Data consulting team, working directly with the partners and senior stakeholders of the clients designing and implementing big data and analytics solutions. Communication and organisation skills are keys for this position,...

  • Senior Data Engineer

    2 weeks ago

    Gurugram, India Techmora Full time

    Job Title : Senior Data EngineerLocation : Bangalore / GurgaonEmployment Type : Full-timeExperience : 4 - 9 yearsNotice Period : 30 daysAbout the Role :We are looking for a Senior Data Engineer to design, build, and maintain robust data pipelines and scalable data infrastructure. You will work closely with data scientists, analysts, and other engineering...

  • Azure Data Engineer

    3 weeks ago

    Gurugram, India ACENET CONSULTING PRIVATE LIMITED Full time

    Description :About Us :AceNet Consulting is a fast-growing global business and technology consulting firm specializing in business strategy, digital transformation, technology consulting, product development, start-up advisory and fund-raising services to our global clients across banking & financial services, healthcare, supply chain & logistics, consumer...

  • Big Data Engineer

    6 days ago

    Gurugram, India Source4hire Full time

    Description :- Provides technical leadership in the Big Data space (Hadoop Stack like Spark, M/R, HDFS, Hive, HBase etc.) and contributes to open-source Big Data technologies.Must have : Operating knowledge of cloud computing platforms (GCP, especially Big Query, Dataflow, Dataproc, Storage, VMs, Networking, Pub Sub, Cloud Functions, Composer servics)-...

  • Technical Lead/Senior Technical Lead – Big Data

    8 hours ago

    Gurugram, India Absolutdata Full time

    Summary of Job A revolution is brewing, and Absolutdata IS the epicentre of the revolution called “Big Data” - Amegatrend impacting every facet of business decision making. With our extensive experienceconverting big data into big insights, client are reaping large bottom-line and top-line results. Be itlarge amounts of data (Volume), fast changing or...

  • Big Data Quality Assurance Engineer

    3 weeks ago

    Gurugram, India SK HR Consultants Full time

    Experience : 4 to 6 YearsJob location : Pune, Bangalore and GurgaonWhat You Need :- 4 to 6 years of QA experience with a strong focus on Big Data testing, particularly in Data Lake environments on Azure's cloud platform.- Proficient in Azure Data Factory, Azure Synapse Analytics and Databricks for big data processing and scaled data quality checks.-...