Freelance – Apache NiFi

2 weeks ago

india, IN ThreatXIntel Full time

Company DescriptionThreatXIntel is a dynamic cybersecurity startup dedicated to protecting businesses and organizations against cyber threats. Specializing in services such as cloud security, DevSecOps, and vulnerability testing, the company offers tailored, cost-effective solutions to meet clients' unique requirements. Focused on startups and small businesses, ThreatXIntel ensures that high-quality cybersecurity services are accessible to safeguard digital assets. With a proactive approach, the team continuously monitors and tests digital environments to identify vulnerabilities before threats arise. ThreatXIntel is committed to delivering peace of mind, allowing clients to prioritize business growth.Role DescriptionWe are seeking a Freelance Data Engineer with strong expertise in Apache NiFi and the Google Cloud Platform (GCP) ecosystem to design and build real-time data ingestion and processing pipelines. The ideal candidate will develop scalable solutions leveraging NiFi and GCP services such as BigQuery, Pub/Sub, Dataflow, Dataproc, Cloud Storage, and Composer. This role requires hands-on technical execution, collaboration with cross-functional teams, and a deep understanding of data engineering best practices.Key ResponsibilitiesDesign and implement real-time data ingestion pipelines using Apache NiFi for both structured and unstructured datasets.Integrate ingestion and processing pipelines with the GCP ecosystem, including BigQuery, Pub/Sub, Dataflow, Dataproc, Cloud Storage, and Composer (Airflow).Ensure low-latency and high-throughput data movement across ingestion, processing, and storage layers.Develop and optimize SQL queries for BigQuery and relational databases to support analytics and reporting.Build Python scripts for data transformation, automation, and developing custom NiFi processors.Implement data quality checks, validation rules, deduplication, late-arriving data handling, and robust error-handling frameworks.Collaborate with data analysts, data scientists, and platform engineering teams to deliver analytics-ready datasets.Apply data security, GCP IAM policies, VPC Service Controls, and encryption standards to maintain compliance.Monitor, troubleshoot, and optimize pipeline performance to ensure stability and reliability.Document data flows, architecture diagrams, transformation logic, and operational best practices.Required Skills & QualificationsStrong hands-on experience with Apache NiFi for real-time ingestion, routing, transformation, and orchestration.Deep knowledge of key GCP services, including:Pub/Sub (event ingestion & streaming)BigQuery (warehouse, SQL optimization)Dataflow / Apache Beam (data processing pipelines)Dataproc (Spark/Hadoop processing)Cloud Storage (data lake)Composer / Airflow (workflow orchestration)Advanced SQL skills for querying, optimization, and data modeling.Proficiency in Python for automation, ETL tasks, and scripting custom processors.Strong understanding of real-time streaming concepts such as windowing, watermarks, deduplication, and event-time processing.Knowledge of data architecture principles, including ETL/ELT design, data lakes, and warehouse best practices.Experience with CI/CD, Git-based version control, and collaborative development workflows.Familiarity with security and governance practices in cloud-based data environments.Ideal Candidate ProfileHands-on engineer comfortable owning end-to-end ingestion pipeline development.Strong communicator capable of collaborating with diverse technical teams.Self-driven, detail-oriented, and capable of delivering high-quality work in fast-paced environments.Ability to solve complex data engineering challenges using modern cloud-native tools.

  • Freelance Apache NiFi GCP Data Engineer

    2 weeks ago

    india, IN ThreatXIntel Full time

    Company DescriptionThreatXIntel is a startup cybersecurity company specializing in protecting businesses and organizations from cyber threats. We offer services including cloud security, web and mobile security testing, DevSecOps, and cloud security assessments to deliver customized, affordable solutions tailored to client needs. Committed to proactive...

  • Freelance GCP Data Engineer

    2 weeks ago

    india, IN ThreatXIntel Full time

    Company DescriptionThreatXIntel is a startup specializing in cybersecurity, offering tailored and cost-effective solutions for businesses and organizations of all sizes. Our expertise spans cloud security, web and mobile security testing, cloud security assessments, DevSecOps, and more. Driven by a proactive approach, we monitor and test our clients' digital...

  • Senior Data Engineer

    2 weeks ago

    , India, IN Northern Arc Capital Full time

    We are looking for a Lead Data Engineer — immediate joiners only.QUALIFICATIONS, EXPERIENCE, & COMPETENCIES•9+ years of experience in similar role•Bachelor's or master’s degree in computer science, Information Technology, or a related field.•Proven experience in data engineering, ETL development, and data integration.•Proficiency in data pipeline...

  • Freelance Back-End Java Engineer

    6 days ago

    india, IN ThreatXIntel Full time

    Company DescriptionThreatXIntel is a startup cyber security company specializing in protecting businesses and organizations from evolving cyber threats. We provide customized and affordable solutions, including cloud security, web and mobile security testing, cloud security assessment, and DevSecOps. Focused on proactive security, we continuously monitor and...

  • Freelance Agentic AI

    1 week ago

    india, IN ThreatXIntel Full time

    Company DescriptionThreatXIntel is a cybersecurity startup focused on safeguarding businesses and organizations from evolving cyber threats. Our team specializes in cloud security, web and mobile security testing, and DevSecOps, offering tailor-made and cost-effective solutions to meet the unique needs of our clients. With a proactive approach, we...

  • Freelance Real-Time Data Processing Specialist

    2 weeks ago

    india beBeeDataEngineer Full time

    Real-Time Data EngineerWe are seeking a skilled Freelance Data Engineer with expertise in Apache NiFi and Google Cloud Platform (GCP) to design and implement real-time data ingestion and processing pipelines.Design and implement efficient real-time data ingestion pipelines using Apache NiFi for both structured and unstructured datasets.Integrate ingestion...

  • Data Pipeline Specialist

    1 week ago

    india beBeeEngineer Full time

    Job OpportunityWe are seeking a skilled Data Engineer to join our team.This is a freelance position that involves designing and building real-time data ingestion and processing pipelines using Apache NiFi and the Google Cloud Platform (GCP) ecosystem.The ideal candidate will develop scalable solutions using NiFi and GCP services such as BigQuery, Pub/Sub,...

  • Cloud Data Expert

    2 weeks ago

    india beBeeDataEngineering Full time

    Freelance Data Engineering PositionWe seek a skilled Freelance Data Engineer to design and implement real-time data pipelines using the Google Cloud Platform ecosystem.The ideal candidate will possess expertise in Apache NiFi, SQL, and API integrations, as well as experience with BigQuery, Pub/Sub, Dataflow, Dataproc, Cloud Storage, and...

  • Real-Time Data Engineering Specialist

    1 week ago

    india beBeeDataEngineer Full time

    ThreatXIntel is a dynamic cybersecurity startup dedicated to protecting businesses and organizations against cyber threats.As a Freelance Data Engineer, you will design and build real-time data ingestion and processing pipelines using Apache NiFi and the Google Cloud Platform (GCP) ecosystem. This includes BigQuery, Pub/Sub, Dataflow, Dataproc, Cloud...

  • Senior Real-Time Data Engineer

    2 weeks ago

    india beBeeData Full time

    ThreatXIntel is seeking a skilled Freelance GCP Data Engineer to join their team.Job Overview:Design, implement, and optimize real-time data pipelines for structured and unstructured data using Apache NiFi. Collaborate with data analysts, scientists, and engineers to deliver high-quality datasets while ensuring reliability, accuracy, and security within the...

  • Optimizing Data Flows

    2 weeks ago

    india beBeeData Full time

    At the forefront of cybersecurity, our esteemed organization offers bespoke solutions tailored to address the complex challenges faced by businesses in this domain. Our expertise encompasses a broad spectrum, spanning cloud security, web and mobile security testing, as well as comprehensive assessments of cloud infrastructure. Furthermore, we possess a deep...

  • Senior Software Engineer

    3 days ago

    Pune, MH, India Mastercard Full time

    Our PurposeMastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation,...

  • Data Engineer II

    6 days ago

    Pune, India Mastercard Full time

    Job Description Our Purpose Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we're helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and...

  • AWS Data Engineer

    2 weeks ago

    Noida, India Coforge Full time

    Job Description We are seeking a highly skilled and experienced AWS Data Engineer with strong expertise in Streaming Data and Big Data Technologies, AWS, Python, & Spark. The ideal candidate will have hands-on experience in architecting and implementing scalable data solutions using modern cloud-native tools and frameworks. Key Responsibilities: - Architect...

  • BizOps Engineer I

    2 weeks ago

    Pune, MH, India Mastercard Full time

    Our PurposeMastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation,...