Big Data Engineer

3 hours ago

Hyderabad, Telangana, India Pennant Technologies Private Limited Full time ₹ 12,00,000 - ₹ 36,00,000 per year

Job Description

We are looking for experienced Big Data Engineers to join our growing Data & Analytics team. In this role, you'll design, build, and optimize our modern data platform—enabling scalable pipelines, reliable data lakes, and robust ML operations. You'll work closely with data scientists, analysts, and stakeholders to turn raw data into actionable insights.

Key Responsibilities

  • Data Platform Development & Maintenance
  • Design, deploy, and maintain ETL/ELT pipelines using Apache Airflow and dbt/meltano.
  • Architect and manage data lakes on HDFS and object storage (e.g., S3/HDFS).
  • Implement table formats and governance with Apache Iceberg and Unity Catalog/Apache ranger/Atlas.
  • Big Data Processing
  • Develop distributed data processing jobs in Apache Spark (Scala/Java/Python).
  • Optimize Spark applications for performance, cost, and scalability.
  • Good working knowledge in Databases (Postgres, Mongo, SQL, Sql server etc)
  • Collaboration with Data science teams
  • Integrate MLflow for experiment tracking, model registry, and seamless deployment.
  • Collaborate with data science teams to operationalize models in production.
  • Containerization & Orchestration
  • Containerize data services and workflows using Docker/kubenetes.
  • Orchestrate deployments on Kubernetes clusters (EKS/GKE/AKS).
  • Center of Excellence & Best Practices
  • Contribute to and govern the dbt Center of Excellence: define standards, code reviews, and documentation.
  • Advocate and enforce data engineering best practices: CI/CD, testing, monitoring, and observability.
  • Cross‑Functional Collaboration
  • Partner with Analytics, Product, and Engineering teams to understand requirements and deliver solutions.
  • Mentor junior engineers and share knowledge through code reviews and technical talks.

Required Skills & Experience

  • 5+ years of professional experience in Big Data engineering or similar roles.
  • Airflow:
    Hands‑on experience authoring DAGs, configuring operators/sensors, and managing scheduler/executor backends.
  • Apache Spark:
    Proficient in building and tuning large‑scale ETL jobs (Scala/Python).
  • Apache Iceberg & Unity Catalog:
    Familiarity with table formats, partitioning strategies, and catalog management.
  • Data Lake & HDFS:
    Solid understanding of storage architectures, file formats (Parquet/ORC), and data partitioning.
  • MLflow:
    Working knowledge of experiment tracking, model registry, and deployment workflows.
  • Kubernetes:
    Skilled in deploying, scaling, and debugging containerized applications and stateful workloads.
  • dbt COE:
    Experience with dbt for data modeling, CI/CD integration, and establishing a CoE for standards.
  • Strong programming skills in Python and/or Scala.
  • Experience with version control (Git), CI/CD pipelines (Jenkins/GitHub Actions), and monitoring tools (Prometheus, Grafana).

Nice‑to‑Have

  • Familiarity with cloud platforms (AWS, GCP, or Azure) and managed services (EMR, Dataproc, EKS, GKE).
  • Knowledge of additional data catalogs (e.g., AWS Glue, Hive Metastore).
  • Background in event‑driven architectures (Kafka, Pulsar).
  • Prior involvement in data governance or compliance initiatives.

Educational Qualification

  • Bachelor's or Master's degree in Computer Science, Engineering, or related field.
  • Relevant certifications (e.g., Any cloud certification AWS/GCP/Azure/Databricks Certified Big Data – Specialty) are a plus.

  • Big Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India Sky Systems, Inc. (SkySys) Full time

    Role:Big Data EngineerPosition Type:Full-Time Contract (40hrs/week)Contract Duration:Long TermWork Schedule: 8 hours/day (Mon-Fri)Location: Hybrid - Hyderabad, India (3x days onsite/week)We are looking for a skilledBig Data Engineer (5+ years' experience)to join Experian's dynamic data engineering team. You will design and optimize scalable data pipelines,...

  • Big Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India Harp Technologies Services Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    Job Title: Bigdata + PysparkLocation: HyderabadExperience Level: 5-8 YearsNotice Period: Immediate to 30 DaysInterview Mode: L1 Virtual / L2 F2F (Mandatory)Work Mode: HybridJob SummaryWe are seeking a Specialist with 5 to 8 years of experience in the Big Data Hadoop Ecosystem to design and implement scalable data solutions.Detailed Job DescriptionWork...

  • Big Data Engineer

    7 hours ago

    Hyderabad, Telangana, India Pennant Technologies Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job Title: Senior Big Data EngineerDepartment: Data & AnalyticsEmployment Type: Full‑timeRole OverviewWe are seeking an experienced Big Data Engineer to join our growing Data & Analytics team. In this role, you'll design, build, and optimize our modern data platform—enabling scalable pipelines, reliable data lakes, and robust ML operations. You'll work...

  • Big Data Engineer

    1 week ago

    Hyderabad, Telangana, India Skilviu Soft Solutions Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Role: Big Data EngineerLocation: Hyderabad, India (Hybrid).Experience: 5+ YearsDuration: 1 year+Notice Period: Immediate -15 days.Key Responsibilities:Design, develop, and maintain scalable data processing pipelines using Spark/PySpark and Hadoop on AWS.Implement and optimize data workflows on AWS EMR, EC2, and ECS.Develop and integrate RESTful APIs and AWS...

  • Big Data Operations Engineer

    7 days ago

    Hyderabad, Telangana, India S M Software Solutions Inc Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Job Description – Big Data Operations EngineerLocation :Onsite, Hyderabad SoftilityExperience: 5+ yearsRole OverviewWe are looking for a skilledBig Data Operations Engineerwith hands-on experience in supporting and maintaining large-scale data platforms. The role involves ensuring platform availability, managing incidents, monitoring performance through...

  • Big Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India Hucon Solutions Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    KEY RESPONSIBILITIES:5+ years of experience working in a software product development organization building modern and scalable big data applications• Expertise with functional programming, preferably Scala• Fluency in Hadoop technologies and big data processing technologies, e.g. Spark, YARN, HDFS, Oozie, Hive• Experience implementing RESTful web...

  • Big Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India Techno Facts Solutions Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Experience in developing and delivering scalable big data pipelines using Apache Spark and Databricks on AWS.Position Requirements :Must Have : Build and maintain scalable data pipelines using Databricks and Apache Spark. Develop and optimize ETL/ELT processes for structured and unstructured data. Knowledge of Lakehouse architecture for efficient data...

  • Staff Engineer, Big Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India Nagarro Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    We're Nagarro.We are a Digital Product Engineering company that is scaling in a big way We build products, services, and experiences that inspire, excite, and delight. We work at scale across all devices and digital mediums, and our people exist everywhere in the world experts across 38 countries, to be exact). Our work culture is dynamic and...

  • Big Data Developer

    4 days ago

    Hyderabad, Telangana, India Talent Worx Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    We're hiring for one of the world's leading professional services firms, renowned for its commitment to innovation, excellence, and global impact. With a presence in over 150 countries, this organization provides services across consulting, audit, tax, risk advisory, and financial advisory — helping Fortune 500 companies and governments navigate complex...

  • Big Data Developer

    4 days ago

    Hyderabad, Telangana, India Talent Worx Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    We're hiring for one of the world's leading professional services firms, renowned for its commitment to innovation, excellence, and global impact. With a presence in over 150 countries, this organization provides services across consulting, audit, tax, risk advisory, and financial advisory — helping Fortune 500 companies and governments navigate complex...