High-Performance Data Pipeline Specialist

2 days ago

Surat, Gujarat, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

Job Title: Senior Data Engineer

Job Description:

We are looking for a skilled Big Data Engineer to join our team. As a key member of the Professional Services department, you will design, build, and maintain high-performance data pipelines that process massive amounts of data.

Key Responsibilities:

  1. Design and implement scalable batch processing systems using Python and big data technologies.
  2. Optimize database performance, focusing on slow-running queries and latency improvements.
  3. Use Python profilers and performance monitoring tools to identify bottlenecks.
  4. Reduce P95 and P99 latency metrics across our data platform.
  5. Build efficient ETL pipelines that can handle large-scale data processing.
  6. Collaborate with data scientists and product teams to understand data requirements.
  7. Monitor and troubleshoot data pipeline issues in production.
  8. Implement data quality checks and validation mechanisms.
  9. Document data architecture and engineering processes.
  10. Stay current with emerging big data technologies and best practices.

Qualifications Required:

  • Bachelor's degree in Computer Science, Engineering, or related technical field.
  • 4+ years of experience in data engineering roles.
  • Strong Python programming skills with focus on data processing libraries.
  • Experience with big data technologies (Spark, Hadoop, etc.).
  • Proven experience optimizing database performance (SQL or NoSQL).
  • Knowledge of data pipeline orchestration tools (Airflow, Luigi, etc.).
  • Understanding of performance optimization techniques and profiling tools.
  • Experience with cloud platforms (AWS, GCP, or Azure).
  • Ability to translate business requirements into technical specifications.
  • Ability to work effectively in a collaborative team environment.

Preferred Qualifications:

  • Master's degree in Computer Science or related field.
  • Experience with SEO data or web crawling systems.
  • Experience with Clickhouse Database.
  • Knowledge of distributed systems and microservices architecture.
  • Familiarity with container orchestration (Kubernetes, Docker).
  • Experience with real-time data processing.
  • Contributions to open-source projects.
  • Experience with machine learning operations.

Skills and Abilities:

  • Strong analytical thinking and problem-solving skills.
  • Excellent attention to detail and commitment to data quality.
  • Good communication skills to explain complex technical concepts.
  • Self-motivated with the ability to work independently.
  • Passion for performance optimization and efficiency.

  • Senior Data Engineer

    2 days ago

    Surat, Gujarat, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Transforming Data into InsightsAbout This RoleWe are seeking an experienced data engineer to drive the development and maintenance of efficient data pipelines. The ideal candidate will possess a strong background in extract, transform, and load (ETL) processes, data engineering, and experience working with cloud-based data platforms.Data Pipeline...

  • High-Performance Data Engineer

    18 hours ago

    Surat, Gujarat, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Job OverviewWe are seeking a highly skilled data engineer to design, build and optimize scalable data pipelines and lakehouse architectures on Microsoft Azure. The ideal candidate will have expertise in Azure Data Services, strong proficiency in SQL and advanced Python skills.Key ResponsibilitiesDesign and develop scalable data pipelines using Azure Data...

  • Senior Data Pipeline Developer

    3 hours ago

    Surat, Gujarat, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Our team is seeking a highly skilled Data Engineer to join our ranks. As a key member of our organization, you will be responsible for designing, developing, and maintaining complex data pipelines using PySpark.Key Responsibilities:The ideal candidate will havea strong background in PySpark, with hands-on experience in building scalable data...

  • Digital Performance Marketing Specialist

    5 days ago

    Surat, Gujarat, India beBeePerformance Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title:Digital Performance Marketing SpecialistJob DescriptionWe are seeking a Digital Performance Marketing Specialist to manage high-performing campaigns across Google Ads, Meta & TikTok.The ideal candidate will have experience with account ownership, data analysis, and campaign optimization.Responsibilities include running Search, Display, and YouTube...

  • High-Performance Backend Developer

    3 hours ago

    Surat, Gujarat, India beBeeBackend Full time ₹ 25,00,000 - ₹ 35,00,000

    Job OverviewWe are seeking a skilled backend developer to join our team.The ideal candidate will have extensive experience in developing high-performance backend services using Golang and a strong understanding of concurrency, memory management, and performance tuning.The role will involve designing, developing, and maintaining scalable microservices and...

  • Data Architecture Specialist

    2 days ago

    Surat, Gujarat, India beBeeData Full time ₹ 15,00,000 - ₹ 20,25,000

    Job Title: Data ArchitectAbout the Role:We are seeking a skilled professional to fill the role of Data Architect. The ideal candidate will possess expertise in designing and implementing scalable data architectures, ensuring seamless data flow and efficient processing.Key Responsibilities:Design and Develop Scalable Data Pipelines:Create efficient ETL...

  • Azure Data Engineer Position

    17 hours ago

    Surat, Gujarat, India beBeeDataEngineering Full time ₹ 9,00,000 - ₹ 12,00,000

    Data Engineering SpecialistWe are seeking a highly skilled Data Engineering Specialist to join our team. The ideal candidate will have extensive experience in designing, implementing, and maintaining large-scale data processing systems on Azure.Key ResponsibilitiesDevelop scalable data pipelines using Azure Data Factory, Apache Kafka, and other relevant...

  • Senior Data Engineer

    5 hours ago

    Surat, Gujarat, India HealthyMe Ai Full time

    About the RoleWe are seeking a Senior Data Engineer with deep expertise in building data pipelines and integrating healthcare data. You will report to the Head of AI and Engineering and be responsible for designing, building, and operating the backbone of our AI-powered applications. This is a critical role where you will create scalable pipelines that...

  • High Performance Optimization Specialist

    17 hours ago

    Surat, Gujarat, India beBeePerformance Full time ₹ 40,00,000 - ₹ 43,00,000

    About Us:We are a global digital solutions company known for our innovative approach and friendly culture.Our team guides clients in unlocking the value of their data and applications to solve digital challenges, achieving outcomes that benefit both business and society.Key Services Include:Digital TransformationEnterprise AIData & AnalyticsProduct...

  • Senior Cloud Data Specialist

    20 hours ago

    Surat, Gujarat, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    High-Performance Data EngineerWe are seeking a highly skilled data engineer with strong hands-on experience in Databricks and AWS to join our dynamic team.Key Responsibilities:Design, develop, and maintain scalable data pipelines using Databricks and PySpark.Implement and manage data solutions on AWS, including services like S3, Glue, Lambda, EMR, and...