Data Engineer

6 days ago

Bengaluru Karnataka India, Karnataka Mastek Full time

Data Engineer – NiFi / Cloudera / Iceberg / Snowflake / Databricks


Overview


We are seeking a Data Engineer with strong Apache NiFi expertise to design and implement pipelines that move and transform data from Cloudera (HDFS/Hive/Impala) into Apache Iceberg tables, with downstream integration into Snowflake and Databricks. The ideal candidate will have hands-on experience with modern data lakehouse architectures and will play a critical role in enabling scalable, governed, and high-performance data platforms.


Key Responsibilities:


  • Data Ingestion & Pipeline Development
  • Design, configure, and maintain NiFi data flows to extract, transform, and load data from Cloudera into Iceberg tables.
  • Implement streaming and batch ingestion pipelines with NiFi processors and custom scripting where needed.
  • Optimize NiFi workflows for scalability, reliability, and monitoring.
  • Data Lakehouse Enablement
  • Build and manage Apache Iceberg-based datasets for structured, semi-structured, and unstructured data.
  • Ensure schema evolution, partitioning, and metadata management in Iceberg.
  • Develop integration flows from Iceberg to Snowflake and Databricks for analytics, ML, and reporting use cases.
  • Integration & Orchestration
  • Work with Snowflake to ingest curated data from Iceberg for enterprise reporting and commercial insights.
  • Collaborate with Databricks teams to enable advanced analytics and machine learning use cases.
  • Integrate NiFi pipelines with orchestration tools (Airflow, Oozie, or AWS/Azure/GCP schedulers).
  • Performance, Security & Governance
  • Tune NiFi flows and Snowflake/Databricks ingestion for performance and cost optimization.
  • Implement role-based security and ensure compliance (HIPAA, GDPR, SOX if applicable).
  • Work with governance teams to enable lineage, metadata tracking, and auditability.


Qualifications:


  • Bachelor’s degree in Computer Science, Information Systems, or related field.
  • 5+ years of data engineering experience, with at least 2+ years working with Apache NiFi.
  • Strong experience with Cloudera ecosystem (HDFS, Hive, Impala, Spark).
  • Hands-on expertise with Apache Iceberg (schema evolution, time travel, partitioning, compaction).
  • Working knowledge of Snowflake and Databricks integration patterns.
  • Proficiency in SQL and one programming language (Python, Java, or Scala).
  • Understanding of data lakehouse architectures and ETL/ELT best practices.

  • Data Entry Operator

    5 days ago

    Bengaluru, Karnataka, India Design Engineer Full time ₹ 2,64,000 per year

    We are seeking a dedicated and detail-oriented Data Entry Operator (DEO) to support R&D projects. The role involves accurate data entry, documentation, and maintenance of research records in secure systems. The DEO will assist scientists, engineers, and administrative staff by ensuring timely and error-free handling of project information.Key...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack. Provide thought leadership by recommending the most appropriate technologies and solutions for a given use case, covering the entire spectrum from the application layer...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Req ID:321800We are currently seeking a Data Engineer (Talend &Pyspark) to join our team in Bangalore, Karntaka (IN-KA), India (IN)."Job Duties: Key Responsibilities: Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Key Responsibilities:Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack.Provide thought leadership by recommending the most appropriate technologies and solutions for a given use case, covering the entire spectrum from...

  • Technical Lead- Data Engineer

    5 days ago

    Bengaluru, Karnataka, India Astar Data Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Kindly find the Job Description Below.Job Title:Technical Lead- Data EngineerLocation: BangaloreYears of Experience: 8+ years of experienceSigmoidworks with a variety of clients from start-ups to fortune 500 companies. We are looking for a detailed oriented self-starter to assist our engineering and analytics teams in various roles as a Software...

  • Data & AI Engineer Lead

    5 days ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Framework Design & Architecture Architect a metadata-driven, Python/Spark-based framework for automated data validation across high-volume production datasets. Define DQ rule templates for completeness, integrity, conformity, accuracy, and timeliness. Establish data quality thresholds, escalation protocols, and exception workflows. Automation & Integration...

  • Data Engineer

    5 days ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 10,00,000 - ₹ 25,00,000 per year

    Job Duties:Migrate ETL workflows from SAP BODS to AWS Glue/dbt/Talend. Develop and maintain scalable ETL pipelines in AWS. Write PySpark scripts for large-scale data processing. Optimize SQL queries and transformations for AWS PostgreSQL. Work with Cloud Engineers to ensure smooth deployment and performance tuning. Integrate data pipelines with existing Unix...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India, Karnataka AMISEQ Full time

    Data Engineer Bangalore, KA WFO A minimum of 3 years of experience is required. Key ResponsibilitiesDevelop and maintain ETL/ELT pipelines using modern data engineering tools and frameworks7 * 24 On-call support data pipeline health, performance, and SLA complianceDocument data processes, schemas, and best practices SOPImplement data quality checks,...

  • Data Engineer

    4 days ago

    Bengaluru, Karnataka, India, Karnataka Wisemonk Full time

    One of our US Based client who is transforming how people buy and sell secondhand apparel, and data is central to that mission. We’re looking for a Data Engineer to join our Data Engineering team with a focus on analytics engineering and enabling scalable, self-service data across the organization. In this role, you’ll design, build, and maintain...

  • Data Engineer

    6 days ago

    Bengaluru, Karnataka, India, Karnataka Acqueon Full time

    About the JobWe are building a Customer Data Platform (CDP) designed to unlock the full potential of customer experience (CX) across our products and services. This role offers the opportunity to design and scale a platform that unifies customer data from multiple sources, ensures data quality and governance, and provides a single source of truth for...