High-Performance Data Pipeline Architect

6 days ago

Thoothukudi, Tamil Nadu, India beBeeDataEngineer Full time ₹ 1,20,00,000 - ₹ 2,00,00,000
Big Data Engineer Position

We are seeking a skilled Big Data Engineer to design, build, and maintain high-performance data pipelines that process massive amounts of data. This role is critical in ensuring our data infrastructure can handle increasing volumes of data while maintaining exceptional performance standards.

Key Responsibilities:
  • Design and implement scalable batch processing systems using Python and big data technologies
  • Optimize database performance, focusing on slow-running queries and latency improvements
  • Use Python profilers and performance monitoring tools to identify bottlenecks
  • Reduce P95 and P99 latency metrics across our data platform
  • Build efficient ETL pipelines that can handle large-scale data processing
  • Collaborate with data scientists and product teams to understand data requirements
  • Monitor and troubleshoot data pipeline issues in production
  • Implement data quality checks and validation mechanisms
  • Document data architecture and engineering processes

Qualifications Required:

  • Bachelor's degree in Computer Science, Engineering, or related technical field
  • 4+ years of experience in data engineering roles
  • Strong Python programming skills with focus on data processing libraries
  • Experience with big data technologies (Spark, Hadoop, etc.)
  • Proven experience optimizing database performance (SQL or NoSQL)
  • Knowledge of data pipeline orchestration tools (Airflow, Luigi, etc.)
  • Understanding of performance optimization techniques and profiling tools
  • Experience with cloud platforms (AWS, GCP, or Azure)

Preferred Qualifications:

  • Master's degree in Computer Science or related field
  • Experience with SEO data or web crawling systems
  • Experience with Clickhouse Database
  • Knowledge of distributed systems and microservices architecture
  • Familiarity with container orchestration (Kubernetes, Docker)
  • Experience with real-time data processing
  • Contributions to open-source projects

Skills and Abilities:

  • Strong analytical thinking and problem-solving skills
  • Excellent attention to detail and commitment to data quality
  • Ability to work effectively in a collaborative team environment
  • Good communication skills to explain complex technical concepts
  • Self-motivated with the ability to work independently
  • Passion for performance optimization and efficiency

  • High-Performance Data Solutions Architect

    5 days ago

    Thoothukudi, Tamil Nadu, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    We are seeking a skilled Senior Data Engineer with experience in Databricks and cloud infrastructure to join our data engineering team.Key Responsibilities:Data Pipeline Design: Develop scalable, high-performance data pipelines using Databricks and PySpark.Cloud Infrastructure Management: Implement and manage data solutions on AWS, including services like...

  • Data Architect Specialist

    3 days ago

    Thoothukudi, Tamil Nadu, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Data Architect SpecialistAbout the RoleThe ideal candidate will be responsible for designing, building and optimizing scalable data pipelines and architectures. You will work closely with cross-functional teams to deliver robust, secure and high-performance data solutions that empower analytics and data science initiatives.

  • Data Architect

    4 days ago

    Thoothukudi, Tamil Nadu, India beBeeData Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Role: Data ArchitectJob Summary:We are seeking an experienced and skilled Data Architect to design, develop, and maintain scalable data architectures that process and transform large volumes of structured and unstructured data. The ideal candidate will have a strong background in data engineering, cloud technologies, and data architecture, collaborating...

  • Senior High-Performance Application Architect

    5 days ago

    Thoothukudi, Tamil Nadu, India beBeeBackend Full time ₹ 15,00,000 - ₹ 25,00,000

    High-Performance Backend DeveloperWe are seeking an expert in Node.js and NestJS to design and deliver the high-performance backend powering our trading platform. This is a hands-on, builder role: you will create systems that handle real-time financial data, low-latency execution pipelines, and mission-critical APIs that integrate with frontend apps, AI...

  • Data Pipeline Architect

    6 days ago

    Thoothukudi, Tamil Nadu, India beBeeInformatician Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Profile:">Lead Informatica Developer">We are seeking a highly skilled and experienced Informatica ETL Developer with expertise in designing, developing, testing, and maintaining large-scale ETL applications. The ideal candidate will have a strong background in data pipeline development and take ownership in delivering scalable solutions that meet...

  • Chief Data Pipeline Architect

    3 days ago

    Thoothukudi, Tamil Nadu, India beBeeDataEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Senior Data Engineer PositionWe are seeking an experienced professional to lead our data engineering team. As a key member, you will be responsible for designing, developing, and optimizing large-scale data pipelines using Apache Spark, AWS Glue, and Azure Data Factory.Key Responsibilities:Design, develop, and optimize ETL/ELT data pipelines using Apache...

  • Expert Data Pipeline Architect

    4 days ago

    Thoothukudi, Tamil Nadu, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team.The successful candidate will have expertise in designing, implementing and optimizing data pipelines using Python and cloud-based technologies.Key responsibilities include:Designing and implementing Extract, Transform, Load (ETL) processes to move data from diverse...

  • High Performance Storage Architect

    2 days ago

    Thoothukudi, Tamil Nadu, India beBeeSoftware Full time ₹ 30,00,000 - ₹ 35,00,000

    We're seeking a seasoned software architect to spearhead the development of high-performance storage solutions on Linux.Design and implement cutting-edge storage features using C/C++.Leverage expertise in Linux kernel development, filesystems, and virtualization to drive innovation.To succeed, you'll require:Strong proficiency in C/C++ programming.Hands-on...

  • Data Systems Architect

    6 days ago

    Thoothukudi, Tamil Nadu, India beBeeDatabase Full time ₹ 21,00,000 - ₹ 25,50,000

    Job Title: Data Systems ArchitectAbout the Role:We are seeking a skilled and experienced Data Systems Architect to design, implement, and maintain databases that can efficiently store and retrieve data.Responsibilities:Develop and implement ETL processes using various tools and technologies such as MySQL, Oracle SQL, and SSIS (SQL Server Integration...

  • Senior Data Pipeline Engineer

    10 hours ago

    Thoothukudi, Tamil Nadu, India beBeeDataIntegration Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Data Integration SpecialistAs a skilled professional, you will play a vital role in building scalable and efficient data pipelines using industry-leading technologies.Key Responsibilities:Design, develop, and maintain ETL pipelines for ingestion from varied sources like flat files, APIs, Oracle, DB2, etc.Build and optimize data flows for loading...