Big Data Engineer Lead

6 days ago

Bengaluru, Karnataka, India Brillio Full time ₹ 1,04,000 - ₹ 13,08,780 per year

Key Skills - Hadoop, PySpark, Python, SQL, Apache Iceberg (niche) , Hive

Requirements

  • Solid knowledge of Hadoop ecosystem such as Hive, Iceberg, Spark SQL and Proficiency in Python, Unix ,SQL
  • Experience with Apache Kafka, Apache Flink, and other relevant streaming technologies.
  • Strong hands-on experience with Hive and SQL for querying and data transformation
  • Proficiency in Python for data manipulation and automation
  • Expertise in Apache Spark (batch and streaming)
  • Deep understanding of Hadoop ecosystem (HDFS, YARN, MapReduce)
  • Experience working with Kafka for streaming data pipelines
  • Experience with workflow orchestration tools (Airflow, Oozie, etc.)
  • Knowledge of cloud-based big data platforms (AWS EMR, GCP Dataproc, Azure HDInsight)
  • Fam
  • iliarity with CI/CD pipelines and version control (Git)

Responsibilities

  • Design, develop, and maintain scalable data pipelines and ETL processes.
  • Work with large datasets using Hadoop ecosystem tools (Hive, Spark).
  • Build and optimize real-time and batch data processing solutions using Kafka and Spark Streaming.
  • Write efficient, high-performance SQL queries to extract, transform, and load data.
  • Develop reusable data frameworks and utilities in Python.
  • Collaborate with data scientists, analysts, and product teams to deliver reliable data solutions.
  • Monitor, troubleshoot, and optimize big data workflows for performance and cost efficiency.

  • Leading Big Data Engineer

    1 week ago

    Bengaluru, Karnataka, India beBeeEngineering Full time ₹ 2,00,00,000 - ₹ 3,50,00,000

    Job Opportunity:Big Data Engineering LeadershipWe are seeking a seasoned big data engineer to lead our distributed engineering teams across the globe.Collaborate with cross-functional teams to drive strategic initiatives and foster a culture of innovation.Develop expertise in scalable technologies such as Scala, Python, Apache Spark, HDFS, and Iceberg table...

  • Lead Big Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India beBeeBigData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    We are seeking a seasoned big data professional with expertise in Scala and cloud technologies to lead our data engineering initiatives.">Key Responsibilities:

  • AWS Big Lead Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Coders Brain Technology Private Limited Full time

    About the Role :We are looking for an experienced AWS Big Data Lead Engineer to design, build, and manage scalable data solutions on AWS. The ideal candidate will have extensive hands-on experience in Big Data technologies, cloud-native architecture, and leadership to guide engineering teams in delivering enterprise-grade data pipelines and analytics...

  • Big Data Solutions Engineer

    2 weeks ago

    Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    Big Data Solutions EngineerWe are seeking a highly skilled and experienced Big Data Solutions Engineer to lead and manage our team of data engineers. As a key member of our organization, you will be responsible for designing, building, and maintaining scalable big data platforms and pipelines that meet the needs of our business.Your primary focus will be on...

  • Big Data Architect Lead

    2 weeks ago

    Bengaluru, Karnataka, India beBeeBigData Full time ₹ 1,04,000 - ₹ 1,30,878

    Lead Big Data EngineerYou will be responsible for translating client requirements into design, architecting and implementing big data solutions on Azure Cloud.Provide technical leadership in the areas of data engineering including data ingestion, data access, modeling, data processing, visualization, design and implementation.Lead a team to deliver high...

  • AWS Big Data Lead

    2 weeks ago

    Bengaluru, Karnataka, India CODERS BRAIN TECHNOLOGY PRIVATE LIMITED Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    About the RoleWe are seeking a highly skilled AWS Big Data Lead & Technical Architect with over 10 years of experience in designing, implementing, and leading enterprise-grade Big Data solutions. The ideal candidate will combine deep technical expertise in AWS and Big Data technologies with strong leadership and architectural skills to drive large-scale data...

  • Big Data Engineer

    3 weeks ago

    Bengaluru, Karnataka, India Coders Brain Full time

    Job DescriptionRoles and Responsibilities:- Develop scalable infrastructure and platform to collect, analyze and process large amounts of structured and unstructured data with real-time data interpretation.- Work closely across an array of various teams and organizations in the company (including partners, customers)Basic Qualifications:- Bachelor's degree...

  • Senior Big Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India beBeeBigData Full time ₹ 20,00,000 - ₹ 35,00,000

    We are looking for a seasoned big data professional with expertise in Scala and cloud technologies to lead our data engineering efforts.Job DescriptionThe ideal candidate will have hands-on experience designing, developing, and optimizing data pipelines and ETL workflows using Hadoop, Apache Airflow, Kubernetes, and Containers. They will work closely with...

  • Big Data Lead

    2 weeks ago

    Bengaluru, Karnataka, India IQVIA Full time US$ 90,000 - US$ 1,20,000 per year

    Job Title / Primary Skill: Big Data Developer (Lead/Associate Manager)Management Level: G150Years of Experience: 8 to 13 yearsJob Location: Bangalore (Hybrid)Must Have Skills: Big data, Spark, Scala, SQL, Hadoop Ecosystem.Educational Qualification: BE/BTech/ MTech/ MCA, Bachelor or masters degree in Computer Science,Job OverviewOverall Experience 8+ years in...

  • Big Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Reveille Technologies,Inc Full time ₹ 1,04,000 - ₹ 13,39,200 per year

    We're Hiring: Big Data Engineer – Full-TimeLooking for a skilledBig Data Engineerto join our team inBangaloreRole: Big Data EngineerGrade: G4 |TSRLocation: Bangalore (Onsite/Hybrid – specify if needed)Budget: ₹13,39,200Experience: 4–6 yearsSkills:Azure Databricks (ADB)PySparkSQLSpark (Scala/Python)TerraformAirflow & KubernetesData Warehousing &...