Data Engineer

7 days ago

bangalore district, India Mastek Full time

Data Engineer – NiFi / Cloudera / Iceberg / Snowflake / Databricks Overview We are seeking a Data Engineer with strong Apache NiFi expertise to design and implement pipelines that move and transform data from Cloudera (HDFS/Hive/Impala) into Apache Iceberg tables, with downstream integration into Snowflake and Databricks. The ideal candidate will have hands-on experience with modern data lakehouse architectures and will play a critical role in enabling scalable, governed, and high-performance data platforms. Key Responsibilities: Data Ingestion & Pipeline Development Design, configure, and maintain NiFi data flows to extract, transform, and load data from Cloudera into Iceberg tables. Implement streaming and batch ingestion pipelines with NiFi processors and custom scripting where needed. Optimize NiFi workflows for scalability, reliability, and monitoring. Data Lakehouse Enablement Build and manage Apache Iceberg-based datasets for structured, semi-structured, and unstructured data. Ensure schema evolution, partitioning, and metadata management in Iceberg. Develop integration flows from Iceberg to Snowflake and Databricks for analytics, ML, and reporting use cases. Integration & Orchestration Work with Snowflake to ingest curated data from Iceberg for enterprise reporting and commercial insights. Collaborate with Databricks teams to enable advanced analytics and machine learning use cases. Integrate NiFi pipelines with orchestration tools (Airflow, Oozie, or AWS/Azure/GCP schedulers). Performance, Security & Governance Tune NiFi flows and Snowflake/Databricks ingestion for performance and cost optimization. Implement role-based security and ensure compliance (HIPAA, GDPR, SOX if applicable). Work with governance teams to enable lineage, metadata tracking, and auditability. Qualifications: Bachelor’s degree in Computer Science, Information Systems, or related field. 5+ years of data engineering experience, with at least 2+ years working with Apache NiFi. Strong experience with Cloudera ecosystem (HDFS, Hive, Impala, Spark). Hands-on expertise with Apache Iceberg (schema evolution, time travel, partitioning, compaction). Working knowledge of Snowflake and Databricks integration patterns. Proficiency in SQL and one programming language (Python, Java, or Scala). Understanding of data lakehouse architectures and ETL/ELT best practices.

  • Lead Data Engineer

    1 week ago

    bangalore, India Eucloid Data Solutions Full time

    Eucloid is looking for a Lead Data Engineer to join our Data Platform team supporting various business applications. The ideal candidate will support development of data infrastructure on Databricks for our clients by participating in activities which may include starting from up- stream and down-stream technology selection to designing and building of...

  • Lead Data Engineer

    1 week ago

    bangalore, India Eucloid Data Solutions Full time

    Eucloid is looking for a Lead Data Engineer to join our Data Platform team supporting various business applications. The ideal candidate will support development of data infrastructure on Databricks for our clients by participating in activities which may include starting from up- stream and down-stream technology selection to designing and building of...

  • Data Engineer

    2 weeks ago

    bangalore district, India People Tech Group Inc Full time

    We are Hiring - Data Engineer Bengaluru! Job Title: AWS Data Engineer Location: Bengaluru Experience: 5+Years Notice : Immediate Joiners Job Summary: We are looking for a skilled and experienced Data Engineer with over 5 years of experience in data engineering and data migration projects. The ideal candidate should possess strong expertise in SQL, Python,...

  • Data Engineer

    7 days ago

    bangalore district, India Acqueon Full time

    About the Job We are building a Customer Data Platform (CDP) designed to unlock the full potential of customer experience (CX) across our products and services. This role offers the opportunity to design and scale a platform that unifies customer data from multiple sources, ensures data quality and governance, and provides a single source of truth for...

  • Data Engineer

    2 weeks ago

    bangalore district, India CentricaSoft Full time

    Title: Data Engineer - PySpark About CentricaSoft: CentricaSoft is a data-driven technology partner delivering end-to-end data solutions for our clients. We design, build, and scale modern data platforms that empower business decisions. We’re growing our data engineering team and seeking a hands-on PySpark Developer who thrives in a fast-paced,...

  • Data Engineer

    2 weeks ago

    bangalore district, India Networth Corp Full time

    Responsibilities: Working along with architects for planning, designing, integrating, and implementing complex solutions in Azure Cloud computing services End to end development lifecycle of complex large-scale and multi-user applications Implement Azure Cloud data warehouses, Azure and No-SQL databases and hybrid data scenarios Big Data ETL considering...

  • Data Engineer

    2 weeks ago

    bangalore district, India Recro Full time

    Role - Data Engineer Experience - 2+ Yrs Location - Bangalore Roles and Responsibilities: Create and maintain optimal data pipeline architecture Create and maintain events/streaming based architecture/design Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for...

  • Data Engineer

    2 weeks ago

    bangalore district, India LTIMindtree Full time

    GCP Big Data Engineer Location -Bangalore & Gurgaon Leadership role with 8-10 yrs experience Skills Set GCP SQL PySpark ETL knowledge MUST Skills Mandatory Skills : GCP Storage,GCP BigQuery,GCP DataProc,GCP Cloud Composer,GCP DMS,Apache airflow,Java,Python,Scala,GCP Datastream,Google Analytics Hub,GCP Workflows,GCP Dataform,GCP Datafusion,GCP...

  • Senior Data Engineer

    2 weeks ago

    bangalore district, India Avathon Full time

    Who We Are & Why Join Us Avathon is revolutionizing industrial AI with a powerful platform that enables businesses to harness the full potential of their operational data. Our technology seamlessly integrates and contextualizes siloed datasets, providing a 360-degree operational view that enhances decision-making and efficiency. With advanced capabilities...

  • Senior Data Engineer

    7 days ago

    bangalore district, India Guidewire Software Full time

    Responsibilities: Design and Development: Architect, design, and develop robust, scalable, and efficient data pipelines. Design and manage platform solutions to support data engineering needs to ensure seamless integration and performance. Write clean, efficient, and maintainable code. Leadership and Collaboration: Lead and mentor a team of Data engineers,...