Big Data

6 days ago

Bengaluru, Karnataka, India Cognizant Technology Solutions Full time ₹ 1,20,00,000 - ₹ 2,40,00,000 per year

Job Summary

Location: AIA-Chennai

We are seeking Senior Data Engineer resources to work on the migration of applications from our legacy Cloudera environment to the new Kubernetes-based data platform. The role requires strong hands-on development skills & Testing in data engineering, with the ability to deliver high-quality pipelines under guidance from internal leads.

Key Responsibilities

  • Develop and optimize data pipelines using Spark 3.5 and Python/Scala.
  • Migrate existing Hive, Spark, and Control-M jobs to Airflow and DBT-based workflows.
  • Integrate data pipelines with messaging systems (Kafka, Solace) and object stores (S3, MinIO).
  • Troubleshoot and optimize distributed jobs running in Kubernetes environments.
  • Collaborate closely with internal leads and architects to implement best practices.
  • Design and implement migration/acceleration framework to automate end to end migration.
  • Continuous enhancements to the frameworks to ensure the stability, scalability and support for diverse use cases and scenarios.
  • Work with various data applications to enable and support the migration process.
  • Deliver assigned migration tasks within agreed timelines.

Required Skills

  • 6–9 years of hands-on data engineering experience.
  • Strong expertise in Apache Spark (batch + streaming) and Hive.
  • Proficiency in Python, Scala, or Java.
  • Knowledge of orchestration tools (Airflow / Control-M) and SQL transformation frameworks (DBT preferred).
  • Experience working with Kafka, Solace, and object stores (S3, MinIO).
  • Exposure to Docker/Kubernetes for deployment.

Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi).

  • Big Data Developer

    1 week ago

    Bengaluru, Karnataka, India Xebia Full time ₹ 5,00,000 - ₹ 25,00,000 per year

    Job Title : Big Data Developer - Spark/ScalaJob location : BengaluruNotice Period : immediate - 15 daysPosition Overview : We are looking for Scala/Spark developers with strong expertise on Python / Hadoop and Azure cloud ..Big Data Orchestration: Airflow, Spark on Kubernetes, Yarn, OozieBig Data Processing: Hadoop, Kafka, Spark & Spark Structured...

  • Big Data Engineer

    1 week ago

    Bengaluru, Karnataka, India Acme Services Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    We are looking for Big Data Engineer who has experience with:- 1) Big Query 2) Pyspark 3) GCP or AWS or Azure Required Candidate profileCandidate should have more than 5 Years of overall experience in Data Engineer. We are not looking for Data Analyst or Snowflake Developer.

  • Big Data Engineer

    6 days ago

    Bengaluru, Karnataka, India Brillio Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    Key skills - Hive, SQL, Python, Spark, Hadoop and kafka , apache airflowJob Title Big Data EngineerRole OverviewWe are seeking a highly skilled Big Data Engineer to join our team. The ideal candidate will have strong experience in building, maintaining, and optimizing large-scale data pipelines and distributed data processing systems. This role involves...

  • Big Data Engineer

    2 days ago

    Bengaluru, Karnataka, India Deqode Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    Profile: Big Data Engineer (System Design)Experience:5+ yearsLocation:BangaloreWork Mode:HybridAbout The RoleWe're looking for an experiencedBig Data Engineerwith system design expertise to architect and build scalable data pipelines and optimize big data solutions.Key ResponsibilitiesDesign, develop, and maintain data pipelines and ETL processes using...

  • Big Data Engineer

    12 hours ago

    Bengaluru, Karnataka, India RiskInsight Consulting Pvt Ltd Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    ResponsibilitiesDesign, develop, and implement robust Big Data solutions using technologies such as Hadoop, Spark, and NoSQL databases. Build and maintain scalable data pipelines for effective data ingestion, transformation, and analysis. Collaborate with data scientists, analysts, and cross-functional teams to understand business requirements and translate...

  • Big Data Developer

    4 days ago

    Bengaluru, Karnataka, India Sourcebae Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job Alert: Senior Big Data Engineer (5-8 Yrs) | Spark, Python, Scala Are you a seasonedBig Data Engineerready to build and optimize large-scale, distributed data processing applications? We are seeking a senior engineer with5-8 yearsof experience in the core Big Data ecosystem to deliver high-performance solutions.This is a great opportunity to join a...

  • Big Data Engineer

    4 days ago

    Bengaluru, Karnataka, India Innova ESI Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Role: Big Data EngineerExperience: 7+ YearsLocation: BangaloreNotice: Immediate Joiners OnlyJob Description:Azure Databricks, Azure Data Factory, Azure Function apps, Apache Spark, Scala, Java, Apache Kafka, event stream and Big DataOptional Skills: Airflow, PythonRoles & ResponsibilitiesOverall, 12 years of experience in IT industry, including 5 years in...

  • Big Data Developer

    2 weeks ago

    Bengaluru, Karnataka, India ITC Infotech Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    Required Skills:Demonstrated ability in designing, building, refactoring and releasing software written in Java, Python.Strong experience with Big Data technologies including Hadoop, Spark, Flink, Hive, HDFS.Ability to handle data integration, bug fixing, performance optimization, and feature enhancements for distributed data pipelines.Having experience with...

  • Big Data Lead

    2 weeks ago

    Bengaluru, Karnataka, India Persistent Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    About PersistentWe are a trusted Digital Engineering and Enterprise Modernization partner, combining deep technical expertise and industry experience to help our clients anticipate what's next. Our offerings and proven solutions create a unique competitive advantage for our clients by giving them the power to see beyond and rise above. We work with many...

  • Senior Big Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Allstate Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    At Allstate, great things happen when our people work together to protect families and their belongings from life's uncertainties. And for more than 90 years our innovative drive has kept us a step ahead of our customers' evolving needs. From advocating for seat belts, air bags and graduated driving laws, to being an industry leader in pricing...