Data Engineer for Large-Scale Data Pipelines

1 day ago


Surat, Gujarat, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

At ThreatXIntel, we take a proactive approach to security by continuously monitoring and testing clients' digital environments to identify vulnerabilities before they can be exploited.

About the Role
  • We are seeking an experienced Freelance Data Engineer with expertise in Python, SQL, and AWS to design, build, and optimize large-scale data pipelines.
Key Responsibilities
  • Design, develop, and maintain Extract, Transform, Load (ETL)/Load, Transform, Extract (ELT) pipelines using PySpark, AWS, and Databricks.
  • Perform data profiling, validation, and transformation using Python and SQL.
  • Optimize and monitor workflows for performance, scalability, and cost efficiency.
  • Build automation scripts using Linux and Bash for data pipeline orchestration.
  • Collaborate with analytics teams to surface trustworthy Key Performance Indicators (KPIs) and metrics.
  • Troubleshoot and resolve data inconsistencies, configuration issues, and pipeline errors.
  • Work closely with cross-functional teams (analysts, product managers, engineers) to define and implement data requirements.
Qualifications
  • 5+ years of experience in data engineering with a strong background in Python and SQL.
  • Proven expertise in PySpark, AWS services, and Databricks.
  • Proficiency in Linux/Bash scripting for automation and data operations.
  • Hands-on experience with ETL/ELT pipelines and large-scale integrations.
  • Familiarity with Agile and Continuous Integration/Continuous Deployment (CI/CD) practices.
  • Strong problem-solving and communication skills, with experience in client-facing roles.


  • Surat, Gujarat, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Transforming Data into InsightsAbout This RoleWe are seeking an experienced data engineer to drive the development and maintenance of efficient data pipelines. The ideal candidate will possess a strong background in extract, transform, and load (ETL) processes, data engineering, and experience working with cloud-based data platforms.Data Pipeline...


  • Surat, Gujarat, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Senior Data EngineerJob Description:We are looking for a skilled Big Data Engineer to join our team. As a key member of the Professional Services department, you will design, build, and maintain high-performance data pipelines that process massive amounts of data.Key Responsibilities:Design and implement scalable batch processing systems using...


  • Surat, Gujarat, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Our team is seeking a highly skilled Data Engineer to join our ranks. As a key member of our organization, you will be responsible for designing, developing, and maintaining complex data pipelines using PySpark.Key Responsibilities:The ideal candidate will havea strong background in PySpark, with hands-on experience in building scalable data...


  • Surat, Gujarat, India beBeeDataEngineer Full time ₹ 40,00,000 - ₹ 50,00,000

    Seeking a seasoned Data Architect to design, build, and deploy scalable data pipelines on AWS cloud infrastructure.This is an exciting opportunity for a talented individual who can collaborate with cross-functional teams to optimize data pipelines for performance, reliability, and cost-effectiveness.We are looking for someone with:8+ years of experience...


  • Surat, Gujarat, India beBeeData Full time ₹ 1,80,00,000 - ₹ 2,00,00,000

    Job Title: Data ArchitectA company in the data engineering sector is looking for a highly skilled professional with expertise in Scala, Spark, Python, NiFi, and Apache Kafka to join its team.Key Responsibilities:Design, develop, and optimize large-scale data pipelines and distributed data processing systems.Work extensively with Scala, Spark (PySpark), and...


  • Surat, Gujarat, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Summary:We are seeking a skilled Data Engineer to design, develop and maintain large-scale data processing systems on GCP.Key Responsibilities:Data Engineering Expertise: Develop and implement efficient data pipelines, utilizing GCP's comprehensive suite of tools and services.Collaboration and Integration: Work closely with cross-functional teams to...


  • Surat, Gujarat, India beBeeData Full time ₹ 18,00,000 - ₹ 19,20,000

    We are modernizing our demand forecasting capabilities through a cutting-edge AI platform. We are seeking a highly skilled Data Scientist to design and operationalize advanced forecasting solutions at scale.Job Responsibilities:Create, build, and deploy production-grade prediction modelsDevelop end-to-end machine learning pipelines for data ingestion,...


  • Surat, Gujarat, India beBeeDataEngineering Full time ₹ 9,00,000 - ₹ 12,00,000

    Data Engineering SpecialistWe are seeking a highly skilled Data Engineering Specialist to join our team. The ideal candidate will have extensive experience in designing, implementing, and maintaining large-scale data processing systems on Azure.Key ResponsibilitiesDevelop scalable data pipelines using Azure Data Factory, Apache Kafka, and other relevant...


  • Surat, Gujarat, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 25,00,000

    Job DescriptionWe are seeking a skilled Data Engineer with expertise in designing and implementing scalable, high-performance data pipelines. The role involves building and optimizing ETL/ELT workflows, managing big data environments, and ensuring seamless data availability for analytics, AI/ML, and business intelligence use cases.

  • Big Data Engineer

    4 days ago


    Surat, Gujarat, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Big Data EngineerOverview:We are seeking an experienced Big Data Engineer to join our team. The ideal candidate will have a strong background in designing, building, and maintaining large-scale data processing systems.Key Responsibilities:Design and develop scalable big data architectures using technologies like AWS, Python, and PySpark.Build and...