Freelance – Apache NiFi

Company Description ThreatXIntel is a dynamic cybersecurity startup dedicated to protecting businesses and organizations against cyber threats. Specializing in services such as cloud security, DevSecOps, and vulnerability testing, the company offers tailored, cost-effective solutions to meet clients' unique requirements. Focused on startups and small businesses, ThreatXIntel ensures that high-quality cybersecurity services are accessible to safeguard digital assets. With a proactive approach, the team continuously monitors and tests digital environments to identify vulnerabilities before threats arise. ThreatXIntel is committed to delivering peace of mind, allowing clients to prioritize business growth.Role Description We are seeking a Freelance Data Engineer with strong expertise inApache NiFiand theGoogle Cloud Platform (GCP)ecosystem to design and build real-time data ingestion and processing pipelines. The ideal candidate will develop scalable solutions leveraging NiFi and GCP services such as BigQuery, Pub/Sub, Dataflow, Dataproc, Cloud Storage, and Composer. This role requires hands-on technical execution, collaboration with cross-functional teams, and a deep understanding of data engineering best practices. Key Responsibilities Design and implementreal-time data ingestion pipelinesusing Apache NiFi for both structured and unstructured datasets. Integrate ingestion and processing pipelines with theGCP ecosystem , including BigQuery, Pub/Sub, Dataflow, Dataproc, Cloud Storage, and Composer (Airflow). Ensurelow-latency and high-throughputdata movement across ingestion, processing, and storage layers. Develop and optimizeSQL queriesfor BigQuery and relational databases to support analytics and reporting. BuildPython scriptsfor data transformation, automation, and developing custom NiFi processors. Implementdata quality checks , validation rules, deduplication, late-arriving data handling, and robust error-handling frameworks. Collaborate with data analysts, data scientists, and platform engineering teams to deliver analytics-ready datasets. Apply data security, GCP IAM policies, VPC Service Controls, and encryption standards to maintain compliance. Monitor, troubleshoot, and optimize pipeline performance to ensure stability and reliability. Document data flows, architecture diagrams, transformation logic, and operational best practices. Required Skills & Qualifications Strong hands-on experience withApache NiFifor real-time ingestion, routing, transformation, and orchestration. Deep knowledge of keyGCP services , including: Pub/Sub(event ingestion & streaming) BigQuery(warehouse, SQL optimization) Dataflow / Apache Beam(data processing pipelines) Dataproc(Spark/Hadoop processing) Cloud Storage(data lake) Composer / Airflow(workflow orchestration) AdvancedSQLskills for querying, optimization, and data modeling. Proficiency inPythonfor automation, ETL tasks, and scripting custom processors. Strong understanding ofreal-time streaming conceptssuch as windowing, watermarks, deduplication, and event-time processing. Knowledge ofdata architecture principles , including ETL/ELT design, data lakes, and warehouse best practices. Experience withCI/CD , Git-based version control, and collaborative development workflows. Familiarity withsecurity and governance practicesin cloud-based data environments. Ideal Candidate Profile Hands-on engineer comfortable owning end-to-end ingestion pipeline development. Strong communicator capable of collaborating with diverse technical teams. Self-driven, detail-oriented, and capable of delivering high-quality work in fast-paced environments. Ability to solve complex data engineering challenges using modern cloud-native tools.