High-Performance Data Pipelines Engineer

1 day ago

Alwar, Rajasthan, India beBeeDataEngineer Full time US$ 1,60,000 - US$ 2,20,000
Job Title

Senior Data Engineer Position

">Job Description

We are seeking a talented Senior Data Engineer to design, build and maintain high-performance data pipelines that process large amounts of data.

Our platform processes massive amounts of data to provide actionable insights to clients. We're looking for a skilled Big Data Engineer who can help us scale and optimize our data processing capabilities.

The ideal candidate will have strong Python programming skills with a focus on data processing libraries, experience with big data technologies (Spark, Hadoop, etc.), and proven experience optimizing database performance (SQL or NoSQL).

Key Responsibilities:

  • Design and implement scalable batch processing systems using Python and big data technologies.
  • Optimize database performance, focusing on slow-running queries and latency improvements.
  • Use Python profilers and performance monitoring tools to identify bottlenecks.
  • Reduce P95 and P99 latency metrics across our data platform.
  • Build efficient ETL pipelines that can handle large-scale data processing.
  • Collaborate with data scientists and product teams to understand data requirements.
  • Monitor and troubleshoot data pipeline issues in production.
  • Implement data quality checks and validation mechanisms.
  • Document data architecture and engineering processes.

Qualifications Required:

  • Bachelor's degree in Computer Science, Engineering, or related technical field.
  • 4+ years of experience in data engineering roles.
  • Strong Python programming skills with focus on data processing libraries.
  • Experience with big data technologies (Spark, Hadoop, etc.).
  • Proven experience optimizing database performance (SQL or NoSQL).
  • Knowledge of data pipeline orchestration tools (Airflow, Luigi, etc.).
  • Understanding of performance optimization techniques and profiling tools.
  • Experience with cloud platforms (AWS, GCP, or Azure).
  • Ability to translate business requirements into technical specifications.
  • Ability to work in a collaborative team environment.

Preferred Qualifications:

  • Master's degree in Computer Science or related field.
  • Experience with SEO data or web crawling systems.
  • Experience with Clickhouse Database.
  • Knowledge of distributed systems and microservices architecture.
  • Familiarity with container orchestration (Kubernetes, Docker).
  • Experience with real-time data processing.
  • Contributions to open-source projects.
  • Experience with machine learning operations.

Skills and Abilities:

  • Strong analytical thinking and problem-solving skills.
  • Excellent attention to detail and commitment to data quality.
  • Ability to work effectively in a collaborative team environment.
  • Good communication skills to explain complex technical concepts.
  • Self-motivated with the ability to work independently.
  • Passion for performance optimization and efficiency.

  • High-Performance Data Pipeline Specialist

    2 weeks ago

    Alwar, Rajasthan, India beBeeDataEngineer Full time ₹ 28,00,000 - ₹ 32,00,000

    Big Data Engineer JobA Big Data Engineer is needed to work on designing, building and maintaining high-performance data pipelines that process terabytes of data. This role requires the ability to optimize existing systems, identify and resolve performance bottlenecks, and implement solutions that improve the overall efficiency of our platform.Key...

  • High-Performance Data Solutions Expert

    1 week ago

    Alwar, Rajasthan, India beBeeDataEngineer Full time ₹ 10,00,000 - ₹ 15,00,000

    Senior Data Engineer PositionWe are seeking a seasoned data professional to design and implement scalable, high-performance data pipelines.The ideal candidate will have strong expertise in cloud services and Databricks, with experience in building and optimizing workflows, managing big data environments, and ensuring seamless data availability for analytics,...

  • High-Performance Data Strategist

    2 weeks ago

    Alwar, Rajasthan, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Senior Data EngineerWe are seeking an accomplished data professional to lead the design, development, and maintenance of our modern data platform.Key Responsibilities:Design and implement efficient, scalable ETL/ELT pipelines and data flowsDevelop transformation models using dbt and orchestrate pipelines via AirflowOptimize large-scale processing jobs in...

  • Large Scale Data Pipeline Leader

    2 weeks ago

    Alwar, Rajasthan, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    **Job Title:** Senior Data EngineerWe are seeking a highly skilled data engineer to lead the design, development and maintenance of large-scale data pipelines. As a senior data engineer, you will be responsible for developing and maintaining efficient ETL pipelines that meet client needs.**Key Responsibilities:• Develop and Maintain ETL Pipelines: Utilize...

  • Data Pipeline Specialist

    2 weeks ago

    Alwar, Rajasthan, India beBeeData Full time ₹ 5,00,000 - ₹ 10,00,000

    Job Title: Data EngineerWe are seeking a skilled professional to design, build, and optimize scalable data pipelines.The ideal candidate will be responsible for enabling seamless data integration, transformation, and analytics to support business intelligence and advanced analytics initiatives.Key Responsibilities:Design, build, and maintain ETL/ELT...

  • Data Pipeline Architect

    2 weeks ago

    Alwar, Rajasthan, India beBeeIngestion Full time ₹ 20,00,000 - ₹ 25,00,000

    We are seeking a highly skilled Data Ingestion Specialist to join our team. This role requires a strong understanding of data ingestion processes, focusing on integrating various data sources into Databricks.The ideal candidate will have hands-on experience with data pipelines, automation, and infrastructure management to support scalable and efficient...

  • High-Performance Data Solutions Architect

    7 days ago

    Alwar, Rajasthan, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Role:Data Engineer - Big Data SolutionsWe are seeking a seasoned Data Engineer to join our team. The ideal candidate will be responsible for designing, building, and maintaining high-performance data pipelines that support business growth.The successful candidate will collaborate with cross-functional teams to develop and implement data models, ETL...

  • AWS Data Engineer

    2 weeks ago

    Alwar, Rajasthan, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Our team specializes in crafting robust data solutions to drive business insights.About the RoleIn this key position, you will develop and implement scalable data pipelines using AWS, Python, and SQL to support informed decision-making. You will ensure the integrity and performance of our data processing and storage systems, optimize data flows within AWS,...

  • Data Pipeline Tester

    2 weeks ago

    Alwar, Rajasthan, India beBeeTestEngineer Full time ₹ 10,00,000 - ₹ 20,00,000

    Senior Test EngineerAs a key member of our team, you will play a crucial role in ensuring the accuracy, integrity, and performance of ETL pipelines deployed into Azure cloud environments.Design and develop comprehensive test cases to validate data flows, transformations, and system behaviors using Python scripting and Pytest framework.Validate data...

  • Senior Data Engineer

    3 days ago

    Alwar, Rajasthan, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Lead Data Integration SpecialistWe are seeking a seasoned professional with expertise in designing, developing, and maintaining high-performing data pipelines using Matillion ELT.Candidates should possess excellent collaboration skills to work with cross-functional teams in delivering robust data solutions.Key Responsibilities:Design, develop, and maintain...