Freelance Apache NiFi GCP Data Engineer

3 hours ago

india, IN ThreatXIntel Full time

Company Description

ThreatXIntel is a startup cybersecurity company specializing in protecting businesses and organizations from cyber threats. We offer services including cloud security, web and mobile security testing, DevSecOps, and cloud security assessments to deliver customized, affordable solutions tailored to client needs. Committed to proactive security, we continuously monitor and test digital environments to mitigate vulnerabilities before they can be exploited. Our mission is to empower businesses of all sizes with high-quality cybersecurity services to protect their digital assets and enable business growth.


Role Description

We are seeking an experienced Freelance Data Engineer with strong expertise in Apache NiFi and the Google Cloud Platform (GCP) ecosystem. The consultant will design and implement real-time data ingestion pipelines, integrate with GCP services, and ensure secure, scalable, and high-performance data delivery for analytics and downstream platforms.

Key Responsibilities

  • Design and build real-time ingestion pipelines using Apache NiFi for structured and unstructured datasets.
  • Integrate NiFi pipelines with the GCP ecosystem, including BigQuery, Pub/Sub, Dataflow, Dataproc, Cloud Storage, and Composer (Airflow).
  • Implement low-latency, high-throughput data movement across ingestion, storage, and processing layers.
  • Develop and optimize SQL queries for BigQuery and relational systems.
  • Build Python scripts for data transformations, automation tasks, and custom NiFi processors.
  • Apply data quality, validation, deduplication, and error-handling frameworks within ingestion flows.
  • Work with analysts, data scientists, and platform teams to deliver clean, reliable, analytics-ready datasets.
  • Enforce GCP IAM, VPC Service Controls, encryption policies, and secure-by-design practices.
  • Monitor, troubleshoot, and tune NiFi processor performance and GCP data pipelines.
  • Document data flows, architecture standards, and operational best practices.

Required Skills & Qualifications

  • Strong hands-on experience with Apache NiFi for real-time ingestion, routing, transformation, and flow orchestration.
  • Deep expertise across the GCP ecosystem:
  • Pub/Sub (event ingestion & streaming)
  • BigQuery (data warehousing & SQL optimization)
  • Dataflow / Apache Beam (stream & batch processing)
  • Cloud Storage (data lake)
  • Dataproc (Spark/Hadoop processing)
  • Composer/Airflow (workflow orchestration)
  • Strong proficiency in SQL (analytic functions, optimization, partitioning, clustering).
  • Strong Python development for automation and data processing.
  • Solid understanding of real-time streaming concepts: windowing, late data handling, deduplication, event ordering.
  • Knowledge of data architecture patterns (ETL/ELT, data lakes, data warehouses).
  • Experience with CI/CD, Git-based workflows, and infrastructure-as-code (nice to have).
  • Understanding of cloud security, IAM roles, governance controls, and compliance in GCP environments.


  • Senior Data Engineer

    2 hours ago

    Bangalore Urban district, India, IN AuxoAI Full time

    AuxoAI is seeking a Senior Data Engineer to lead the design, development, and optimization of modern data pipelines and cloud-native platforms using Google Cloud Platform (GCP). This role is ideal for someone with deep experience building scalable batch and streaming data workflows, strong hands-on engineering skills, and a drive to mentor junior...

  • GCP data Engineer

    2 hours ago

    Gurugram, Haryana, India, IN Impetus Full time

    Location: Gurgaon and BangaloreExperience: 8+ years (data engineering / analytics engineering), with previous lead responsibilitiesJob Descriptions for Big data or Cloud EngineerWe are looking for candidates with hands on experience in pyspark with GCP cloud.Qualifications3-10 years of IT experience range is preferred.Able to effectively use GCP managed...

  • Data Engineer

    2 hours ago

    Gurugram, Haryana, India, IN Sirius AI Full time

    Data Engineer Job Description About Sirius AI Sirius AI is a US headquartered AI Consulting services and products company with operations in India. Sirius AI focuses on Financial Services enterprises and solutions / services delivered across multiple geographies. We are an innovation-driven AI and data driven consulting services firm with a high focus on...

  • GCP Data Engineer

    3 hours ago

    Bangalore Urban, Karnataka, India, IN Impetus Full time

    Job Title: GCP Data EngineerLocation: BengaluruExperience: 3-8 yearsAbout the Role:Design, build, and maintain large-scale data pipelines on BigQuery and other Google Cloud Platform (GCP) services.Use Python and PySpark/Spark to transform, clean, aggregate and prepare data for analytics/ML.Orchestrate workflows using Cloud Composer (Airflow) to schedule,...

  • Freelance Data Quality Engineer

    2 hours ago

    india, IN Leading MNC Part time

    Looking for a Freelance Data Quality Engineer to join a team of rockstar developers. The candidate should have a minimum of 8+ yrs. of experience.There are multiple openings. If you're looking for freelance/ part time opportunity (along with your day job) & a chance to work with the top 0.1% of developers in the industry, this one is for you! You will report...

  • Lead GCP Data Engineer

    2 hours ago

    Gurugram, Haryana, India, IN Impetus Full time

    Job Title: Lead Data Engineer – GCP (BigQuery • Composer • Python • PySpark)Location: GurgaonExperience: 8+ years (data engineering / analytics engineering), with previous lead responsibilitiesAbout the Role:You will lead the design, build and operation of large-scale data platforms on the Google Cloud Platform. You will manage a team of data...

  • GCP Data Engineer

    3 hours ago

    , India, IN HCLTech Full time

    Looking for 5+ Years of experiencenotice period- immediate to 60 daysSkill required - GCP - Data Side, Storage Classes, Dataflow, Big query, Pyspark / Python, Airflow

  • Freelance Machine Learning Engineer

    3 hours ago

    india, IN Leading MNC Part time

    Looking for a Freelance Machine Learning Engineer to join a team of rockstar developers. The candidate should have a minimum of 8+ yrs. of experience.There are multiple openings. If you're looking for freelance/ part time opportunity (along with your day job) & a chance to work with the top 0.1% of developers in the industry, this one is for you! You will...

  • GCP data Engineer

    3 hours ago

    Bangalore Urban, Karnataka, India, IN Impetus Full time

    Provides technical leadership in Big Data space (Hadoop Stack like Spark, M/R, HDFS, Hive, HBase etc) and contributes to open source Big Data technologies.ResponsibilitiesMust have: Operating knowledge of cloud computing platforms (GCP, especially Big Query, Dataflow, Dataproc, Storage, VMs, Networking, Pub Sub, Cloud Functions, Composer services).Should be...

  • AI Data Engineer

    3 hours ago

    Bangalore Urban, Karnataka, India, IN Tata Consultancy Services Full time

    Job Title: AI Data EngineerExp- 4 to 8 yearsLocation- PAN IndiaJob DescriptionKey Responsibilities:• Build and maintain data infrastructure: Design and construct scalable, reliable data pipelines, storage, and processing systems in the cloud.• Ensure data quality: Clean, transform, and enrich raw data to create "business truth" that AI models can use for...