High-Performance Data Pipeline Specialist

2 days ago

Kanpur, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

About Us

We're looking for a talented Data Engineer to help us scale and optimize our data processing capabilities.

Role Overview

You will design, build, and maintain high-performance data pipelines that process terabytes of data.

Key Responsibilities

  • Design and implement scalable batch processing systems using Python and big data technologies
  • Optimize database performance, focusing on slow-running queries and latency improvements
  • Use Python profilers and performance monitoring tools to identify bottlenecks
  • Reduce P95 and P99 latency metrics across our data platform
  • Build efficient ETL pipelines that can handle large-scale data processing
  • Collaborate with data scientists and product teams to understand data requirements
  • Monitor and troubleshoot data pipeline issues in production
  • Implement data quality checks and validation mechanisms
  • Document data architecture and engineering processes
  • Stay current with emerging big data technologies and best practices

Qualifications Required

  • Bachelor's degree in Computer Science, Engineering, or related technical field
  • 4+ years of experience in data engineering roles
  • Strong Python programming skills with focus on data processing libraries
  • Experience with big data technologies (Spark, Hadoop, etc.)
  • Proven experience optimizing database performance (SQL or NoSQL)
  • Knowledge of data pipeline orchestration tools (Airflow, Luigi, etc.)
  • Understanding of performance optimization techniques and profiling tools
  • Experience with cloud platforms (AWS, GCP, or Azure)
  • Ability to translate business requirements into technical specifications
  • Ability to work in a collaborative team environment
  • Good communication skills to explain complex technical concepts
  • Self-motivated with the ability to work independently
  • Possession of a valid UK driving license is not required

Benefits

  • A competitive salary and benefits package
  • Opportunities for career growth and professional development
  • A collaborative and dynamic work environment
  • The chance to work with a talented team of professionals

Others

  • Prior experience working with SEO data or web crawling systems is desirable
  • Prior experience with Clickhouse Database is desirable
  • Familiarity with distributed systems and microservices architecture is desirable
  • Familiarity with container orchestration (Kubernetes, Docker) is desirable
  • Knowledge of real-time data processing is desirable
  • Contributions to open-source projects are desirable
  • Prior experience with machine learning operations is desirable

  • High-Performance Data Pipelines Expert

    13 hours ago

    Kanpur, Uttar Pradesh, India beBeeDataEngineering Full time ₹ 18,00,000 - ₹ 25,00,000

    Are you a data engineering expert looking for a new challenge? This is an excellent opportunity to leverage your skills and experience in designing, developing, and maintaining scalable and high-performance data pipelines using Databricks and PySpark.About the RoleThis is a Senior Data Engineer position where you will be working on the design, development,...

  • Data Pipeline Specialist

    9 hours ago

    Kanpur, Uttar Pradesh, India beBeeData Full time ₹ 15,00,000 - ₹ 17,50,000

    About the RoleWe're seeking a highly skilled Data Engineer to build robust pipelines that transform large, messy technical datasets into high-quality structured data.

  • Pipeline Intervention Specialist

    2 days ago

    Kanpur, Uttar Pradesh, India beBeePipeline Full time ₹ 80,00,000 - ₹ 1,50,00,000

    Job Title: Pipeline Intervention SpecialistThe pipeline industry is seeking a highly skilled professional to conduct hot tapping and pipeline isolation services on-site.Key Responsibilities:Conduct hot tapping and pipeline isolation services in accordance with industry standards and regulations.Collaborate with project teams to ensure safe and efficient...

  • Data Pipeline Specialist

    13 hours ago

    Kanpur, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Job OpportunityWe are seeking a skilled professional to take on the role of Data Engineer, responsible for designing, developing, and maintaining data pipelines that ingest data from various sources into our enterprise-wide data warehouse system.Utilize cutting-edge technologies like AWS services (AWS Lambda, AWS DMS, and AWS Glue) for data ingestion and...

  • Data Pipeline Architect

    5 days ago

    Kanpur, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    About the Role:The ideal candidate will have a solid foundation in data engineering with a minimum of 7 years of experience in development on data-centric projects.Mandatory Skills:Expertise in working with real-time data and Kafka framework, including kSQL/Mirror Maker etc.Proficiency in at least one programming language: Groovy or Java.Thorough...

  • Senior Data Pipeline Architect

    2 days ago

    Kanpur, Uttar Pradesh, India beBeeEngineeringManager Full time ₹ 20,00,000 - ₹ 25,00,000

    About the RoleWe are seeking an experienced Engineering Manager to lead a team of data engineers in designing and delivering scalable data pipelines and analytics platforms using Big Data platforms, Microsoft Azure, and Databricks.Main ResponsibilitiesLead, mentor, and grow a team of data engineers working on large-scale distributed data systems.Architect...

  • High-Performance Backend Engineer

    2 days ago

    Kanpur, Uttar Pradesh, India beBeeBackendDeveloper Full time ₹ 1,80,00,000 - ₹ 2,00,00,000

    Skillful Backend Developer WantedWe are looking for an accomplished software developer to design and implement high-performance backend systems.Key Responsibilities:Create robust, scalable, and efficient backend solutions.Implement algorithms and data models to support business workflows.Collaborate with cross-functional teams to deliver high-quality...

  • Data Ingestion Specialist

    4 days ago

    Kanpur, Uttar Pradesh, India beBeeDataIngestion Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title:We are seeking a talented Data Ingestion Specialist to design and develop data pipelines for integrating multiple sources into Databricks.The ideal candidate will have hands-on experience with automation, infrastructure management, and scalable data workflows.Design and develop efficient data ingestion pipelines for integrating various data sources...

  • Data Architecture Specialist

    1 day ago

    Kanpur, Uttar Pradesh, India beBeeArchitecture Full time ₹ 1,20,00,000 - ₹ 1,80,00,000

    About the RoleWe are seeking an experienced Data Architecture Specialist to join our team. In this role, you will design and develop scalable, robust, and secure data pipelines using Azure Data Factory, Databricks, Synapse Analytics, Azure Data Lake Gen2, Event Hub, and Azure Functions.Your key responsibilities will include developing and maintaining ETL/ELT...

  • Data Innovation Specialist

    11 hours ago

    Kanpur, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Unlock Your PotentialYour skills and expertise are the key to unlocking a world of possibilities.At our organization, you will have the opportunity to unleash your potential and make a meaningful impact in a fast-paced environment.We are seeking an experienced Data Engineer with deep knowledge in data engineering principles, tools, and technologies to...