Senior Big Data Engineer

6 days ago

New Delhi Noida Thiruvananthapuram, India Iridium Cloud Systems Full time ₹ 15,00,000 - ₹ 25,00,000 per year

Purpose

Design and develop scalable ETL pipelines, APIs, and automations for big data processing in an on-premise Hadoop/Spark/Airflow environment, focusing on real-time log enrichment and exposure via APIs for a visualization app. Take ownership of proposing and implementing solutions for data flow, performance, and integration challenges.

Key Responsibilities

  • Architect and implement ETL/ELT pipelines for streaming (Kafka/Spark) and batch log processing, including data enrichment (e.g., IP geolocation, user mapping) in high-volume scenarios.
  • Build and optimize high-performance APIs (e.g., using FastAPI) to deliver processed logs and automate integrations for internal apps, ensuring low-latency access.
  • Integrate external data sources, optimize queries for concurrent users, and design for scalability (e.g., handling 10x-100x growth in log data).
  • Manage data lifecycle processes: compaction, migration to cold storage (on-prem MinIO equivalents), and cost/efficiency optimizations.
  • Lead performance testing (load, concurrency) and document proposed solutions for ETL designs, API architectures, and automation workflows.
  • Collaborate with infrastructure teams on shared tools like Spark/Kafka, focusing on development-side optimizations without delving into setup/monitoring.

Technical Requirements

  • Core Skills:

  • 5+ years in Python for big data applications (Spark/Hadoop; ELK/Elastic Stack experience is a plus for log handling).

  • 2+ years developing APIs (FastAPI/Flask), streaming pipelines (Kafka/Spark Streaming), and ETL automations in on-premise environments.

  • Proficiency in SQL, pandas, numpy, and on-prem storage solutions (e.g., MinIO).

  • Design Skills: System architecture diagramming (e.g., UML), API integration, and rapid adaptation to tools like Airflow for orchestration.

  • Flexibility: Experience with GCP/AWS tools (e.g., Kinesis, S3) is a plus for conceptual knowledge and adaptability.

Soft Skills

  • Analytical approach to solving performance and scalability issues in big data ETL/APIs.
  • Clear communication for documenting proposed architectures, solutions, and collaborating on cross-role integrations.
  • Results-oriented, with a focus on meeting SLAs/KPIs (e.g., latency, throughput, data accuracy in high-volume logs).

Candidate Profile: Seniors from product or consultancy firms building ETL/APIs for high-volume data (e.g., 1M+ users/logs), with on-premise or hybrid experience.

  • Senior Data Engineer(Big Data)

    2 days ago

    New Delhi, India Andor Tech Full time

    Andor Tech is looking for a highly skilled Senior Big Data Engineer with 8–15 years of experience to join our dynamic team. If you're passionate about big data technologies and thrive in solving complex problems, we want to hear from you!Location: Deployment to a Product-Based Retail MNCImmediate Joiners PreferredKey Skills Required:- Strong expertise in...

  • Senior Big Data Infrastructure Engineer

    6 days ago

    New Delhi, Noida, Thiruvananthapuram, India Iridium Cloud Systems Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    PurposeLead the setup, automation, and maintenance of on-premise infrastructure for high-volume big data environments, including Kafka/Spark clusters for log streaming, ensuring scalability, security, and 24/7 reliability. Provide proactive solutions for infrastructure challenges.Key ResponsibilitiesDesign and automate on-premise deployments/upgrades for...

  • Big Data Developer

    2 days ago

    New Delhi, India Sourcebae Full time

    Job Alert: Senior Big Data Engineer (5-8 Yrs) | Spark, Python, ScalaAre you a seasonedBig Data Engineerready to build and optimize large-scale, distributed data processing applications? We are seeking a senior engineer with5-8 yearsof experience in the core Big Data ecosystem to deliver high-performance solutions. This is a great opportunity to join a...

  • Big Data Engineer

    4 weeks ago

    New Delhi, India HirePower Staffing Solution Full time

    Position Overview:We are seeking a skilled Big Data Developer to join our growing delivery team, with a dual focus on hands-on project support and mentoring junior engineers. This role is ideal for a developer who not only thrives in a technical, fast-paced environment but is also passionate about coaching and developing the next generation of talent.You...

  • Big Data Engineer

    4 days ago

    Thiruvananthapuram, Kerala, India Futuremug Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    Big Data Engineer (Hadoop/Devops) Freelance InterviewerExperience -8 to 12 yearsAbout the companyWe are an HR Tech company based in Trivandrum, offering hiring support to MNCs across India through interview ,assessments and recruitment services. We have a network of 4000+ experienced professionals who take interviews in their available time slots.We're...

  • Big Data Engineer

    12 hours ago

    Thiruvananthapuram, Kerala, India Futuremug Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Big Data Engineer (Hadoop/Devops) Freelance InterviewerExperience -8 to 12 yearsAbout the companyWe are an HR Tech company based in Trivandrum, offering hiring support to MNCs across India through interview ,assessments and recruitment services. We have a network of 4000+ experienced professionals who take interviews in their available time slots.Were...

  • Senior Data Engineer

    2 days ago

    New Delhi, India Eucloid Data Solutions Full time

    Job Description:Eucloid is looking for a Senior Data Engineer to join our Data Platform team supporting various business applications. The ideal candidate will support development of data infrastructure for our clients by participating in activities which may include starting from up- stream and down-stream technology selection to designing and building of...

  • Big Data Engineer

    4 weeks ago

    New Delhi, India Coforge Full time

    Coforge Ltd is Hiring for Big Data Engineer – AWS, Spark & Scala.Must Have Skills:- AWS, Spark & Scala.Experience Required: 3 to 6 YearsJob Locations: Pune, Hyderabad, Greater Noida Only.Send your CV to: Gaurav.2.Kumar@coforge.comFor queries, contact via WhatsApp: 9667427662Key Responsibilities:-• Design, develop, and optimize Big Data architectures...

  • Big Data Engineer

    1 week ago

    Bengaluru, Delhi, Mumbai, NCR, India Suzva Software Technologies Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Big Data Engineer (Remote, Contract 6 Months+) ig Data ecosystem including Snowflake (Snowpark), Spark, MapReduce, Hadoop, and more.We are looking for a Senior Big Data Engineer with deep expertise in large-scale data processing technologies and frameworks. This is a remote, contract-based position suited for a data engineering expert with strong experience...

  • Big Data Developer

    4 weeks ago

    New Delhi, India Coforge Full time

    We are seeking a highly skilled and experienced Senior Big Data Engineer with strong expertise in Streaming Data and Big Data Technologies, AWS & Scala Mandatory. The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks.Key Responsibilities:- Architect and...