Big Data Engineer

4 weeks ago


Bangalore, Karnataka, India Talentxo Full time

Job Description :


Analyze and optimize Hive queries :


- Identify bottlenecks and opportunities for improvement in Hive scripts.


- Apply best practices for query optimization, data partitioning, and materialized views.


Tame the TPC-H & TPC-DS benchmarks :


- Deeply understand the TPC-H and TPC-DS benchmarks and leverage them to evaluate and compare Presto performance at different scales (1K & 1TB / 10K & 10TB).

Prestissimo power user :


- Implement and configure Prestissimo for optimized query execution in our data environment.


- Fine-tune Prestissimo configurations to maximize performance and resource utilization.

Hive whisperer :


- Master the art of sizing and tuning Hive configurations for optimal performance.


- Manage RAM allocation, adjust Hive settings, and implement best practices for data storage and access.

Automation Architect :


- Develop and maintain automation scripts to reproduce your Hive and Presto configurations and optimization techniques.


- Ensure repeatability and ease of deployment for future changes. (Bonus points for MinIO knowledge)

Collaborate and communicate :


- Work closely with data analysts, scientists, and engineers to understand data processing needs and translate them into efficient execution plans.


- Share your knowledge and findings through clear and concise communication.

Qualifications :

- 10+ years of experience as a Big Data Engineer or related role.

- Strong expertise in HiveQL and PrestoSQL.

- Experience with TPC-H/TPC-DS benchmarks and performance optimization techniques.

- Understanding of distributed file systems and data storage optimizations.

- Scripting skills in Python or similar languages for automation tasks. (Familiarity with MinIO is a plus)

- Excellent analytical and problem-solving skills.

- Strong communication and collaboration skills.

(ref:hirist.tech)
  • Big Data Engineer

    4 weeks ago


    Bangalore, Karnataka, India TalentPulse Consultant Full time

    Role : Big Data EngineerJob Description :We are currently seeking a highly skilled Big Data Engineer to join our team on a remote basis.The ideal candidate will have at least 5 years of experience in Big Data technologies and a strong proficiency in Python, Spark, and Scala.This is an active position with a high priority, offering a competitive rate and the...

  • Senior Data Engineer

    4 weeks ago


    Bangalore, Karnataka, India Muoro Full time

    Job Description :About the role :We are looking for a Senior Data Engineer to join our growing engineering team. As a member of the team:- You will work on enterprise data platforms, architect and implement data lakes both on-prem and in the cloud.- You will be responsible for evolving technical architecture, design and implementation of data solutions using...

  • Senior Data Engineer

    4 weeks ago


    Bangalore, Karnataka, India Squareroot Consulting Pvt Ltd Full time

    We are seeking a highly skilled and experienced Senior Data Engineer to join our team.Location : Bangalore (Work from Office all 5 days)Experience : 6+ YearsRole : Senior Data EngineerThe ideal candidate will have a strong background in big data technologies and extensive hands-on experience with Cloudera, Spark, Hive, HDFS, Flink, Data Lake, and...

  • Databricks Architect

    2 weeks ago


    Bangalore, Karnataka, India TecHHire Global Full time

    Job Description :Function : Technical Management - Engineering Management, Software Engineering - Backend Development, Big Data / DWH / ETLResponsibilities :- Design and implement large-scale data processing systems using the Databricks platform.- Work closely with clients to understand their requirements and design solutions that meet their needs.-...

  • Big Data Consultant

    4 weeks ago


    Bangalore, Karnataka, India Nilasu consulting Full time

    Summary :We are seeking a highly skilled consultant with expertise in Spark/Scala and Python to join our team. You will play a key role in developing and maintaining big data applications using Spark and Scala with a focus on functional programming principles. Additionally, your experience in Python and microservices architecture will be valuable in building...

  • Software Engineer

    3 weeks ago


    Bangalore, Karnataka, India symphoni HR Full time

    Skill: Java (Data structures/Algorithims) & Spark - Design, develop and implement, scalable, maintainable, and efficient software solutions using best practices and modern technologies. - Write clean, well designed, and testable code, following coding standards and guidelines. - Conduct code reviews and provide constructive feedback to team members to ensure...

  • Data Engineer

    3 weeks ago


    Bangalore, Karnataka, India LOGIC PLANET IT SERVICES Full time

    About the job :We are hiring for a global technology company, home to more than 224,000 people across 60 countries.Role : Data EngineerExp : 5+ years Mode of Work : HybridMode of Hire : FTEAvailability : Immediate joinee or serving maximum of 30 days notice period.Job Description :- 5+ years of experience operating and deploying solutions using AWS Service...

  • Big Data Engineer

    4 weeks ago


    Bangalore, Karnataka, India randstad Full time

    Primary Responsibilities :- Design, develop, and implement big data solutions using Java, Python, or Scala.- Utilize SQL and Spark Scala for data processing and analysis.- Implement and manage data storage solutions using AWS services like S3, EC2, and Lambda.- Work with various databases such as Postgres, MySQL, Oracle, etc., for data storage and...


  • Bangalore, Karnataka, India People Impact Full time

    Bachelor's degree in Engineering or Computer Science or equivalent OR Master's in Computer Applications or equivalent :- Solid Experience of software development experience and leading teams of engineers and scrum teams.- 5+ years of hands-on experience of working with Map-Reduce, Hive, Spark (core, SQL).- Solid Datawarehousing concepts.- Knowledge...


  • Bangalore, Karnataka, India Zyoin group Full time

    Job Description :We are looking for Principal Data Engineers to help us build and enhance big data platforms to achieve availability, scalability, and operational effectiveness. The right individual will embrace the opportunity to tackle challenging problems and use their influence to drive continual improvement.You will also work on the cutting edge of...

  • Data Engineer

    4 weeks ago


    Bangalore, Karnataka, India ARR Recruitment Solutions Full time

    Job Description : We are looking for a highly skilled Senior Data Engineer with extensive experience in data integration, processing, and application development for handling large-scale data sets measured in Terabytes from multiple sources.- The ideal candidate will have a strong background in AWS services, particularly Lambda, EMR, and Glue, along with...

  • Big Data Architect

    4 weeks ago


    Bangalore, Karnataka, India White Hawk Consultant Full time

    Job Description :- 10+ years of experience in end-to-end software development and system architecture, preferably in a product development organisation.- Hands-on experience in Java/Scala and Big Data tech stack - Apache Pinot/Druid/Hudi, HDFS, Hive, Presto, Flume, Storm, Kafka, Spark, NO-SQL databases is a must.- Hands-on experience in working on...

  • Unison Consulting

    3 weeks ago


    Bangalore, Karnataka, India Unison consulting pte ltd Full time

    About the Role:We are seeking a talented Solution Architect to join our team and play a key role in designing and implementing innovative big data solutions. You will be responsible for translating business needs into scalable, reliable, and high-performance technology architectures, leveraging your expertise in Databricks and other big data technologies....


  • Bangalore, Karnataka, India Adobe Systems India Pvt. Ltd. Full time

    What You'll Do :- Build the underneath data platform and maintain data processing pipelines using best in class technologies. - Special focus on R&D to challenge status-quo and build the next generation big data platform that is efficient and cost effective. - Translate complex technical and functional requirements into detailed designs. - Collaborate...

  • Big Data Engineer

    4 weeks ago


    Bangalore/Hyderabad, Karnataka, India VIDPRO CONSULTANCY SERVICES Full time

    About the job:- This position will be part of a growing team working towards building world class large scale Big Data architectures.- This individual should have a sound understanding of programming principles, experience in programming in Java, Python or similar languages and can expect to spend a majority of their time coding.Location - Bangalore and...

  • Big Data Engineer

    4 weeks ago


    Bangalore, Karnataka, India Techno-Comp Computer Services Pvt Ltd Full time

    Job Description :- Should have hands on experience in Software development for at least 9 years. - Should have hands-on experience in order to handle project work as Technical Lead for at least 2 yr.- Responsible for designing, deploying, and managing high quality data solutions in the AWS cloud ecosystem.- Create and maintain optimal data architecture...

  • Unison Consulting

    3 weeks ago


    Bangalore, Karnataka, India Unison consulting pte ltd Full time

    We are seeking a talented Big Data Engineer to join our team and play a key role in designing, developing, and maintaining our big data infrastructure. You will leverage your expertise in Java, Hadoop, and Kafka to build efficient data pipelines, handle real-time data streams, and ensure high-quality data processing for our data analytics...


  • Bangalore, Karnataka, India Notus Full time

    Job Title : GCP (Google Cloud Platform) SpecialistNumber of Positions : 6 (1 Senior Role, 5 Lead Roles)Shift : 12 : 00 PM to 9 : 00 PM OR 1 : 00 PM to 10 : 00 PMLocation : TechM India locationsNotice Period : Early/Immediate/30 days (Positions need to be filled by May)Budget : Senior Role : 4 to 5/6 years of experience - Depending upon hike % and candidate...

  • Big Data Engineer

    2 weeks ago


    Bangalore/Hyderabad, Karnataka, India TalentXo Full time

    Role & Responsibilities :Good development practices :- Hands on coder with good experience in programming languages like Java or Python.- Hands-on experience on the Big Data stack like PySpark, Hbase, Hadoop, Mapreduce and ElasticSearch.- Good understanding of programming principles and development practices like checkin policy, unit testing, code...


  • Bangalore, Karnataka, India IQVIA RDS India Pvt Ltd Full time

    Job Title / Primary Skill : Big Data DeveloperManagement Level : G130Years of Experience : 2 to 5 yearsJob Location : Bangalore (Hybrid)Must Have Skills : Bigdata Developer, Hadoop, Hive, Scala, and Spark.Educational Qualification : BE/BTech/ MTech/ MCARole & responsibilities : - Minimum 3+ years of experience in Big Data development.- Good understanding of...