Big Data Engineer

3 weeks ago


Anywhere in IndiaMultiple Locations, IN Talentxo Full time

Job Description :


- Analyze and optimize Hive queries: Identify bottlenecks and opportunities for improvement in Hive scripts. Apply best practices for query optimization, data partitioning, and materialized views.

- Tame the TPC-H & TPC-DS benchmarks: Deeply understand the TPC-H and TPC-DS benchmarks and leverage them to evaluate and compare Presto performance at different scales (1K & 1TB / 10K & 10TB).

- Prestissimo power user: Implement and configure Prestissimo for optimized query execution in our data environment. Fine-tune Prestissimo configurations to maximize performance and resource utilization.

- Hive whisperer: Master the art of sizing and tuning Hive configurations for optimal performance. Manage RAM allocation, adjust Hive settings, and implement best practices for data storage and access.

- Automation Architect: Develop and maintain automation scripts to reproduce your Hive and Presto configurations and optimization techniques. Ensure repeatability and ease of deployment for future changes. (Bonus points for MinIO knowledge)

- Collaborate and communicate: Work closely with data analysts, scientists, and engineers to understand data processing needs and translate them into efficient execution plans. Share your knowledge and findings through clear and concise communication.

Qualifications:

- 10+ years of experience as a Big Data Engineer or related role.

- Strong expertise in HiveQL and PrestoSQL.

- Experience with TPC-H/TPC-DS benchmarks and performance optimization techniques.

- Understanding of distributed file systems and data storage optimizations.

- Scripting skills in Python or similar languages for automation tasks. (Familiarity with MinIO is a plus)

- Excellent analytical and problem-solving skills.

- Strong communication and collaboration skills.

(ref:hirist.tech)

  • Anywhere in India/Multiple Locations, IN Grownex HR Solution Full time

    Role : Lead Big Data EngineerWork Mode : Remote(5 Days working)The Opportunity - Development team is establishing a new offshore product development team for its partner , that is building next-generation Big Data, Cloud-Based Business Operation Support technology for utilities, retail energy suppliers and Community Choice Aggregators (CCA). The candidate...


  • Bangalore/Anywhere in India/Multiple Locations, IN Leadway Resource Full time

    We are seeking a highly skilled and experienced Big Data QA Engineer to join our team. You will play a critical role in ensuring the quality of our big data platform and data solutions. You will leverage your expertise in Big Data testing methodologies, automation frameworks, and scripting languages to design, develop, and execute comprehensive test...

  • Big Data Developer

    3 weeks ago


    Anywhere in India/Multiple Locations, IN Risk Resources LLP Full time

    Responsibilities :- Lead and mentor a team of Big Data engineers in designing, developing, and deploying big data solutions on GCP- Define the technical vision and roadmap for big data architecture within the organization- Architect and implement scalable, high-performance big data pipelines using technologies like MapReduce, Hive, Spark, and BigQuery-...

  • Big Data Engineer

    3 weeks ago


    Bangalore/Anywhere in India/Multiple Locations, IN Connectio IT Pvt Ltd Full time

    Key Skills : Java, Apache Spark, Apache Kafka, Hadoop, AWSJob Description :- Designing, installing, testing, and maintaining scalable data management systems- Ensuring systems meet business requirements and industry standards- Creating custom software components and analytics applications- Research data acquisition opportunities and new uses for existing...

  • PySpark Tech Lead

    3 weeks ago


    Anywhere in India/Multiple Locations, IN TD Newton & Associates Full time

    Technical Skills :- Extensive experience working on software engineering projects and SDLC- Have worked on various development project as developer and/or lead developer.- Capability to understand and comprehend the solution and create a development framework where all the developers will be contributing.- Should be able to create a Test driven development...

  • Sigmoid Analytics

    3 weeks ago


    Anywhere in India/Multiple Locations, IN ASTAR DATA LLP Full time

    This position will be part of a growing team working towards building world-class large scale Big Data Good Development Practices:- Hands-on coder with good experience in programming languages like Python or Scala.- Good Understanding of programming principles and development practices like checkin policy, unit testing, code deployment- Self-starter to be...

  • Sigmoid Analytics

    3 weeks ago


    Anywhere in India/Multiple Locations, IN ASTAR DATA LLP Full time

    Job Description :This position will be part of a growing team working towards building world-class large scale Big Data architectures.Responsibilities : Good Development Practices :- Hands-on coder with good experience in programming languages like Python or Scala.- Good Understanding of programming principles and development practices like checkin policy,...

  • Big Data Architect

    3 weeks ago


    Bangalore/Anywhere in India/Multiple Locations, IN ARR Recruitment Solutions Full time

    About the Role :- We are seeking a highly experienced Big Data Architect to join our team and lead the design, development, and implementation of our big data solutions.- This role is ideal for a candidate with a deep understanding of big data technologies and a proven track record of delivering scalable and robust big data architectures.Responsibilities :-...

  • GCP Data Architect

    3 weeks ago


    Anywhere in India/Multiple Locations, IN Huquo Full time

    Role : GCP Data Architect Description : As a GCP Data Architect, you will be responsible for architecting enterprise data solutions for migration, transformation or modernization on Google Cloud Platform (GCP) integrating native GCP services and other 3rd party data solutions. You will collaborate with cloud partners and our cross-functional teams to enable...

  • Hadoop Administrator

    3 weeks ago


    Metros/Anywhere in India/Multiple Locations, IN Dotflick Solutions Full time

    Designation : Hadoop AdministratorOpen for Pan India(Any Location)Job Description :- Development and deployment of data applications- Design Implementation of infrastructure tooling and work on horizontal frameworks and libraries- Creation of data ingestion pipelines between legacy data warehouses and the big data stack- Automation of application back-end...

  • Big Data Architect

    2 weeks ago


    Metros/Anywhere in India/Multiple Locations, IN Xcel Hire Management Solutions Full time

    Job description : - The decision-making power for data analysis and he/she should also possess the quality of architecting the massive data- Should have skills in big data tools and technologies; it includes technologies like the Hadoop, accumulo, MapReduce, Hive, HBase, panoply and redshift- Overall 10 years of experience as Principal Consultant, Subject...

  • Aptus Data Labs

    2 hours ago


    Anywhere in India/Multiple Locations, IN Aptus Data Labs Full time

    Role - Data Architect/Data Engineer Lead- AWSExp - 8+ YearsLocation - RemoteSkills - Data architecture, Data modelling & design, Data Governance, MDM, Python, SQL, ETL, AWS CloudWe are seeking a highly skilled and experienced Data Architect to join our team. The ideal candidate will have 8+ years of experience in data architecture, with a focus on AWS cloud...

  • Data Engineer

    3 weeks ago


    Pune/Hyderabad/Anywhere in India/Multiple Locations, IN Huquo Full time

    Basic understanding of :- Scheduling and workflow management & working experience in either ADF, Informatica, Airflow, or Similar- Enterprise Data Modelling and Semantic Modelling & working experience in ERwin, ER/Studio, PowerDesigner, or Similar- Logical/Physical model on Big Data sets or modern data warehouse & working experience in ERwin, ER/Studio,...

  • Valiance Solutions

    3 weeks ago


    Anywhere in India/Multiple Locations, IN Valiance Analytics Pvt. Ltd. Full time

    About Us :Valiance is a global AI & Data analytics firm helping clients build cutting-edge technology solutions for digital transformation. We work with some of the marquee brands across India, US and APAC to build transformative solutions for Credit Risk, Fraud, Predictive Maintenance, Quality Inspection, Data lake, IOT analytics etc. Our team comprises...

  • MDM Architect

    3 weeks ago


    Anywhere in India/Multiple Locations, IN TELL Jobs Full time

    Job Description :- Minimum 8+ years of proven experience and Own and aggressively drive forward MDM, Data Governance, Big Data, and Cloud Data Management- Develops data integration processes using the Informatica platform - Creating business solutions which utilize the CDI platform and ensure high quality and scalable solutions- Work with EA (Enterprise...

  • Data Engineer

    7 days ago


    Anywhere in India/Multiple Locations, IN AAR Consultancy Full time

    Technical Skills :1. Google Cloud Platform (GCP) : In-depth knowledge and hands-on experience with GCP services such as BigQuery, Dataflow, Dataproc, Pub/Sub, Data Studio, and Cloud Storage.2. Data Engineering : Strong understanding of data engineering principles, including data ingestion, transformation, processing, and storage.3. Big Data Technologies :...

  • Data Engineer

    3 weeks ago


    Anywhere in India/Multiple Locations, IN Paralleldots Full time

    Job Description :As a Senior Data Engineer at ParallelDots, you will be responsible for overseeing the design and development of advanced data solutions. You will work closely with data scientists, analysts, and other stakeholders to ensure that our data infrastructure meets the highest standards of scalability, reliability, and :- Lead the design,...

  • MDM Architect

    3 weeks ago


    Anywhere in India/Multiple Locations, IN TELL Jobs Full time

    Job Description :- Minimum 8+ years of proven experience and Own and aggressively drive forward MDM, Data Governance, Big Data, and Cloud Data Management- Develops data integration processes using the Informatica platform (eg- Creating business solutions which utilize the CDI platform and ensure high quality and scalable solutions- Work with EA (Enterprise...

  • Data Engineer

    3 weeks ago


    Anywhere in India/Multiple Locations/Bangalore, IN Mobile Programming LLC Full time

    Job Description :Position : Data EngineerExperience : 6+ yearsWork Mode : Work from OfficeLocation : BangaloreExperience in Nifi or Tibco is mandatory.Mandatory Skills : ETL, DevOps platform, Nifi or TibcoPlease note : This position is focused on development rather than migration. We are seeking an experienced Data Engineer to join our team. As a Data...


  • Anywhere in India/Multiple Locations, IN Havells India Limited Full time

    Must have : - 3-7 years of meaningful experience in data engineering working on building ETL pipelines, big data tech like spark- Very good expertise in algorithms and data structures- Excellent in at least one programming language: Python, Go, R- Hands-on expertise big data technologies such as Apache Spark, Hadoop, message queues(Kafka), Apache Airflow,...