Freelance – Apache NiFi

Company DescriptionThreatXIntel is a dynamic cybersecurity startup dedicated to protecting businesses and organizations against cyber threats. Specializing in services such as cloud security, DevSecOps, and vulnerability testing, the company offers tailored, cost-effective solutions to meet clients' unique requirements. Focused on startups and small businesses, ThreatXIntel ensures that high-quality cybersecurity services are accessible to safeguard digital assets. With a proactive approach, the team continuously monitors and tests digital environments to identify vulnerabilities before threats arise. ThreatXIntel is committed to delivering peace of mind, allowing clients to prioritize business growth.Role DescriptionWe are seeking a Freelance Data Engineer with strong expertise in Apache NiFi and the Google Cloud Platform (GCP) ecosystem to design and build real-time data ingestion and processing pipelines. The ideal candidate will develop scalable solutions leveraging NiFi and GCP services such as BigQuery, Pub/Sub, Dataflow, Dataproc, Cloud Storage, and Composer. This role requires hands-on technical execution, collaboration with cross-functional teams, and a deep understanding of data engineering best practices.Key ResponsibilitiesDesign and implement real-time data ingestion pipelines using Apache NiFi for both structured and unstructured datasets.Integrate ingestion and processing pipelines with the GCP ecosystem , including BigQuery, Pub/Sub, Dataflow, Dataproc, Cloud Storage, and Composer (Airflow).Ensure low-latency and high-throughput data movement across ingestion, processing, and storage layers.Develop and optimize SQL queries for BigQuery and relational databases to support analytics and reporting.Build Python scripts for data transformation, automation, and developing custom NiFi processors.Implement data quality checks , validation rules, deduplication, late-arriving data handling, and robust error-handling frameworks.Collaborate with data analysts, data scientists, and platform engineering teams to deliver analytics-ready datasets.Apply data security, GCP IAM policies, VPC Service Controls, and encryption standards to maintain compliance.Monitor, troubleshoot, and optimize pipeline performance to ensure stability and reliability.Document data flows, architecture diagrams, transformation logic, and operational best practices.Required Skills & QualificationsStrong hands-on experience with Apache NiFi for real-time ingestion, routing, transformation, and orchestration.Deep knowledge of key GCP services , including:Pub/Sub (event ingestion & streaming)BigQuery (warehouse, SQL optimization)Dataflow / Apache Beam (data processing pipelines)Dataproc (Spark/Hadoop processing)Cloud Storage (data lake)Composer / Airflow (workflow orchestration)Advanced SQL skills for querying, optimization, and data modeling.Proficiency in Python for automation, ETL tasks, and scripting custom processors.Strong understanding of real-time streaming concepts such as windowing, watermarks, deduplication, and event-time processing.Knowledge of data architecture principles , including ETL/ELT design, data lakes, and warehouse best practices.Experience with CI/CD , Git-based version control, and collaborative development workflows.Familiarity with security and governance practices in cloud-based data environments.Ideal Candidate ProfileHands-on engineer comfortable owning end-to-end ingestion pipeline development.Strong communicator capable of collaborating with diverse technical teams.Self-driven, detail-oriented, and capable of delivering high-quality work in fast-paced environments.Ability to solve complex data engineering challenges using modern cloud-native tools.