Senior Big Data Infrastructure Engineer

6 days ago

New Delhi Noida Thiruvananthapuram, India Iridium Cloud Systems Full time ₹ 20,00,000 - ₹ 25,00,000 per year

Purpose

Lead the setup, automation, and maintenance of on-premise infrastructure for high-volume big data environments, including Kafka/Spark clusters for log streaming, ensuring scalability, security, and 24/7 reliability. Provide proactive solutions for infrastructure challenges.

Key Responsibilities

  • Design and automate on-premise deployments/upgrades for Kafka, Spark, and related big data tools using tools like Kubernetes or Salt, targeting zero-downtime operations.
  • Configure and optimize distributed storage systems (e.g., MinIO or equivalent on-prem solutions) for large-scale log data, including capacity planning, monitoring, and data integrity checks.
  • Implement monitoring and alerting for infrastructure health (e.g., using Prometheus/Grafana), with auto-remediation for issues like CVEs or performance bottlenecks.
  • Enforce on-prem security measures: RBAC, system hardening, SSL/TLS, and compliance auditing to mitigate risks in high-volume data environments.
  • Propose and document infrastructure solutions (e.g., "Scaling on-prem Kafka clusters for 100x log volume without data loss") and collaborate with development teams on integration points like observability.
  • Handle big data volume management, including automation for Spark job scheduling and resource allocation in on-prem clusters.

Technical Requirements

  • Core Skills:

  • 5+ years managing on-premise Linux/Unix systems for mission-critical, high-volume big data operations (e.g., Hadoop/Spark clusters).

  • 3+ years with Kubernetes, CI/CD pipelines, IaC (Salt/Terraform/Ansible), and monitoring tools (Prometheus/Grafana).

  • Deep expertise in Kafka for streaming, MinIO or similar on-prem storage, and security tools (e.g., OpenVAS, auditd).

  • Security Focus: Rapid CVE patching (within 72 hours), anomaly detection, and on-prem compliance standards.

  • Flexibility: Experience with GCP/AWS is a plus for understanding core concepts and tools (e.g., Kinesis or S3 equivalents).

Soft Skills

  • Proactive in identifying and solving infrastructure risks (e.g., scalability, security for big data logs).
  • Strong documentation of configurations, solutions, and rationales for team use.
  • Collaborative mindset to guide developers on infra-related integrations without overlapping into app development.

Prefered Industry Experience

  • Priority Sectors: Banking, Tech, Retail, Healthcare (with on-prem data centers).
  • Candidate Profile: Seniors from enterprises with on-premise or hybrid big data infrastructures handling massive log volumes (e.g., 1M+ events/day).

  • Senior Big Data Engineer

    6 days ago

    New Delhi, Noida, Thiruvananthapuram, India Iridium Cloud Systems Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    PurposeDesign and develop scalable ETL pipelines, APIs, and automations for big data processing in an on-premise Hadoop/Spark/Airflow environment, focusing on real-time log enrichment and exposure via APIs for a visualization app. Take ownership of proposing and implementing solutions for data flow, performance, and integration challenges.Key...

  • Big Data Engineer

    4 days ago

    Thiruvananthapuram, Kerala, India Futuremug Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    Big Data Engineer (Hadoop/Devops) Freelance InterviewerExperience -8 to 12 yearsAbout the companyWe are an HR Tech company based in Trivandrum, offering hiring support to MNCs across India through interview ,assessments and recruitment services. We have a network of 4000+ experienced professionals who take interviews in their available time slots.We're...

  • Big Data Engineer

    11 hours ago

    Thiruvananthapuram, Kerala, India Futuremug Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Big Data Engineer (Hadoop/Devops) Freelance InterviewerExperience -8 to 12 yearsAbout the companyWe are an HR Tech company based in Trivandrum, offering hiring support to MNCs across India through interview ,assessments and recruitment services. We have a network of 4000+ experienced professionals who take interviews in their available time slots.Were...

  • Senior Data Engineer

    1 day ago

    New Delhi, India Eucloid Data Solutions Full time

    Job Description:Eucloid is looking for a Senior Data Engineer to join our Data Platform team supporting various business applications. The ideal candidate will support development of data infrastructure for our clients by participating in activities which may include starting from up- stream and down-stream technology selection to designing and building of...

  • Senior Data Engineer(Big Data)

    2 days ago

    New Delhi, India Andor Tech Full time

    Andor Tech is looking for a highly skilled Senior Big Data Engineer with 8–15 years of experience to join our dynamic team. If you're passionate about big data technologies and thrive in solving complex problems, we want to hear from you!Location: Deployment to a Product-Based Retail MNCImmediate Joiners PreferredKey Skills Required:- Strong expertise in...

  • Big Data Engineer

    1 week ago

    Bengaluru, Delhi, Mumbai, NCR, India Suzva Software Technologies Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Big Data Engineer (Remote, Contract 6 Months+) ig Data ecosystem including Snowflake (Snowpark), Spark, MapReduce, Hadoop, and more.We are looking for a Senior Big Data Engineer with deep expertise in large-scale data processing technologies and frameworks. This is a remote, contract-based position suited for a data engineering expert with strong experience...

  • Senior Data Engineer

    7 days ago

    Delhi, India Eucloid Data Solutions Full time

    Job Description:Eucloid is looking for a Senior Data Engineer to join our Data Platform team supporting various business applications. The ideal candidate will support development of data infrastructure for our clients by participating in activities which may include starting from up- stream and down-stream technology selection to designing and building of...

  • Big Data Developer

    2 days ago

    New Delhi, India Sourcebae Full time

    Job Alert: Senior Big Data Engineer (5-8 Yrs) | Spark, Python, ScalaAre you a seasonedBig Data Engineerready to build and optimize large-scale, distributed data processing applications? We are seeking a senior engineer with5-8 yearsof experience in the core Big Data ecosystem to deliver high-performance solutions. This is a great opportunity to join a...

  • Big Data Engineer

    4 weeks ago

    New Delhi, India HirePower Staffing Solution Full time

    Position Overview:We are seeking a skilled Big Data Developer to join our growing delivery team, with a dual focus on hands-on project support and mentoring junior engineers. This role is ideal for a developer who not only thrives in a technical, fast-paced environment but is also passionate about coaching and developing the next generation of talent.You...

  • Engineer (Devops & Big Data)

    7 days ago

    New Delhi, India AMISEQ Full time

    SRE & Devops (Big Data)Immediate Joiner onlyBengaluru, KAExp- 3-6 YearsYou will be a member of our AI Platform Team, supporting the next generation AI architecture for various research and engineering teams within the organization.● You'll partner with vendors and the infrastructure engineering team for security and service availability● You'll fix...