Expert Data Pipeline Developer

4 days ago

Varanasi, Uttar Pradesh, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000
Remote Data Engineering Opportunity

ThreatXIntel is a cyber security startup that specializes in protecting businesses and organizations from cyber threats. Our team provides various services, such as cloud security, web and mobile security testing, and DevSecOps. We aim to deliver customized and affordable solutions tailored to meet the specific needs of our clients.

The Freelance Data Engineer will design, develop, and optimize data pipelines using AWS Glue, PySpark, and Step Functions. The role requires proficiency in pytest for testing, Git/GitHub for version control, and clear technical documentation.

  • Design and develop scalable data pipelines using AWS Glue, PySpark, and Step Functions.
  • Implement ETL/ELT processes for ingesting, transforming, and delivering data from multiple sources.
  • Manage and optimize storage solutions with Amazon S3.
  • Write reusable and maintainable Python code for data engineering workflows.
  • Implement unit and integration tests using pytest to ensure data quality and pipeline reliability.
  • Collaborate using Git/GitHub for version control, code reviews, and CI/CD integration.
  • Create clear technical documentation for pipelines, workflows, and design decisions.
  • Troubleshoot and resolve issues related to data pipelines and AWS services.

Key Qualifications:

  • 5–10 years of data engineering experience, with at least 5 years in AWS ecosystem.
  • Strong coding skills in Python with experience in PySpark.
  • Hands-on experience with AWS Glue, Step Functions, and S3.
  • Proficiency in pytest for test automation.
  • Solid understanding of Git/GitHub workflows.
  • Strong communication skills with a focus on documentation and knowledge sharing.

  • Expert Data Pipeline Developer

    1 day ago

    Varanasi, Uttar Pradesh, India beBeeDataEngineer Full time US$ 90,000 - US$ 1,20,000

    Job Title: Data EngineerWe are looking for a highly skilled Data Engineer to join our team. The successful candidate will have expertise in the Google Cloud stack and be able to work with BigQuery, SQL, and Python to design, build, test, and deploy large-scale data processing systems.Responsibilities:Design and develop scalable data pipelines using tools...

  • Data Pipelines Expert

    5 days ago

    Varanasi, Uttar Pradesh, India beBeedataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job TitleA Data Engineer is needed to design, build and maintain high-quality, secure and scalable data pipelines using various technologies.

  • Senior Data Engineer

    5 days ago

    Varanasi, Uttar Pradesh, India beBeeData Full time ₹ 15,00,000 - ₹ 21,00,000

    Job OpportunityWe are seeking a skilled data expert to take on the challenge of designing, building, and maintaining robust data pipelines for financial data.

  • Cloud Data Pipelines Specialist

    5 days ago

    Varanasi, Uttar Pradesh, India beBeeDataEngineer Full time US$ 1,80,000 - US$ 2,00,000

    Job Title:">Data Engineer - Cloud Integration">Description:">We are seeking a skilled professional to design, build, and maintain scalable data pipelines using Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake.">This role requires expertise in data governance frameworks, ensuring data quality across all pipelines.">The ideal candidate will...

  • Data Pipeline Architect

    1 day ago

    Varanasi, Uttar Pradesh, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title:Data Pipeline ArchitectCreate and implement scalable, event-driven data pipelines that support real-time clinical workflows.Design and integrate systems with EHR/EMR platforms (e.g., Epic) using FHIR and HL7 standards.Develop high-quality systems to ingest, track, and update clinical events in real-timeEnsure downstream analytics systems receive...

  • Data Pipeline Specialist

    6 days ago

    Varanasi, Uttar Pradesh, India beBeeDataEngineer Full time US$ 1,20,000 - US$ 1,50,000

    Job Description:As a skilled data engineer, you will be responsible for designing and implementing robust data pipelines using Python and SQL. You will work with modern DevOps/configuration management tools to manage multi-environment deployments.Design, implement, and manage CI/CD pipelines using Github Actions or equivalent tools.Configure and maintain...

  • Principal Data Pipeline Specialist

    7 days ago

    Varanasi, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Overview:We are seeking an experienced Data Engineer to design, develop, and maintain large-scale data pipelines using Azure Databricks and PySpark. This role involves collaboration with cross-functional teams to integrate data engineering solutions with business requirements, ensuring data quality, security, and compliance throughout the data...

  • Data Engineering Expert

    6 days ago

    Varanasi, Uttar Pradesh, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Data Engineering ExpertThe ideal candidate will have a strong background in data engineering with experience in designing and maintaining cloud-based data architectures.Key Responsibilities:Design and Develop Scalable Data SolutionsPerform Advanced Data Pre-Processing TasksMaintain Cloud-Based Data ArchitecturesDeploy and Optimize Data Solutions...

  • Data Pipelines Engineer

    3 days ago

    Varanasi, Uttar Pradesh, India beBeeSoftware Full time ₹ 90,00,000 - ₹ 1,50,00,000

    Job Description">Overview:We are seeking a skilled Python Developer to join our team. This full-time position will involve building automated data pipelines, implementing analytical algorithms, and developing REST APIs.Key Responsibilities:Design and implement automated data pipelines using Google Cloud Platform.Develop analytical algorithms to drive...

  • Data Pipeline Specialist

    5 days ago

    Varanasi, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 35,00,000

    Oil and Gas Data Engineer OpportunityJob Description:We are seeking a seasoned Oil and Gas Data Engineer with expertise in designing, building, and optimizing scalable data pipelines and cloud solutions. The ideal candidate will have hands-on experience with AWS cloud services, including S3, Redshift, Glue, Athena, EMR, Lake Formation, and Lambda.Key...