High-Performance Data Pipeline Architect

2 days ago

Thrissur, Kerala, India beBeeDataEngineer Full time ₹ 1,80,00,000 - ₹ 2,20,00,000

We're seeking a highly skilled Big Data Engineer to help us scale and optimize our data processing capabilities.

As a key member of our team, you'll design, build, and maintain high-performance data pipelines that process terabytes of data. Your focus will be on optimizing existing systems, identifying performance bottlenecks, and implementing solutions to improve overall efficiency.

Key Responsibilities:

  • Design scalable batch processing systems using Python and big data technologies
  • Optimize database performance, focusing on slow-running queries and latency improvements
  • Use Python profilers and performance monitoring tools to identify issues
  • Reduce P95 and P99 latency metrics across our data platform
  • Build efficient ETL pipelines for large-scale data processing
  • Collaborate with data scientists and product teams to understand data requirements
  • Monitor and troubleshoot data pipeline issues in production
  • Implement data quality checks and validation mechanisms
  • Document data architecture and engineering processes

Qualifications Required:

  • Bachelor's degree in Computer Science, Engineering, or related technical field
  • 4+ years of experience in data engineering roles
  • Strong Python programming skills with focus on data processing libraries
  • Experience with big data technologies (Spark, Hadoop, etc.)
  • Proven experience optimizing database performance (SQL or NoSQL)
  • Knowledge of data pipeline orchestration tools (Airflow, Luigi, etc.)
  • Understanding of performance optimization techniques and profiling tools
  • Experience with cloud platforms (AWS, GCP, or Azure)

Preferred Qualifications:

  • Master's degree in Computer Science or related field
  • Experience with SEO data or web crawling systems
  • Experience with Clickhouse Database
  • Knowledge of distributed systems and microservices architecture
  • Familiarity with container orchestration (Kubernetes, Docker)

Skills and Abilities:

  • Strong analytical thinking and problem-solving skills
  • Excellent attention to detail and commitment to data quality
  • Ability to work effectively in a collaborative team environment
  • Good communication skills to explain complex technical concepts
  • Self-motivated with the ability to work independently

  • High-Performance Data Architect

    8 hours ago

    Thrissur, Kerala, India beBeeDatabricks Full time ₹ 25,00,000 - ₹ 27,00,000

    Senior Databricks Data EngineerOur organization seeks a Senior Databricks Data Engineer with hands-on experience in Databricks and AWS to manage our data engineering team's infrastructure and applications.Key Responsibilities:Design, develop, and maintain scalable and high-performance data pipelines using Databricks and PySpark.Implement and manage data...

  • Chief Data Pipeline Architect

    2 days ago

    Thrissur, Kerala, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our growing team of consultants.Design, develop, and maintain scalable and robust data pipelines that collect, process, and store large volumes of structured and unstructured data.Architect and implement data warehouses, data lakes, and other storage solutions that support...

  • Chief Data Pipeline Architect

    4 days ago

    Thrissur, Kerala, India beBeeDataEngineer Full time ₹ 10,00,000 - ₹ 20,00,000

    Key Data Engineer PositionWe are seeking a highly skilled data professional with 4+ years of experience in designing, building, and optimizing ETL pipelines for large-scale data processing workflows. The ideal candidate will have hands-on experience working with Azure Cloud services including Azure Data Factory, Databricks, and related...

  • High-Performance Data Solutions Specialist

    4 days ago

    Thrissur, Kerala, India beBeeDataEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Seeking an experienced Data Engineer to lead the design and implementation of robust data solutions on Microsoft Fabric and Azure platforms.About the Role:Collaborate with stakeholders to architect technical solutions that meet business objectivesDevelop end-to-end data pipelines using Microsoft Fabric and Azure servicesDesign and optimize data lake zones...

  • Senior Data Architect

    11 hours ago

    Thrissur, Kerala, India beBeeData Full time ₹ 18,00,000 - ₹ 25,00,000

    **Job Overview**We are seeking an expert in designing and implementing efficient, scalable ETL/ELT pipelines and data flows using SQL and ETL/ELT design principles to build and maintain our modern data platform.Key Responsibilities:Develop transformation models using dbt and orchestrate pipelines via Airflow.Optimize large-scale processing jobs in Apache...

  • Chief Data Architect

    2 days ago

    Thrissur, Kerala, India beBeeEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Cloud Engineering LeadershipWe are seeking a highly experienced and strategic Cloud Engineering Manager with deep expertise in Big Data platforms, Microsoft Azure, and Databricks.The ideal candidate will lead a team of talented cloud engineers to design and deliver scalable data pipelines and analytics platforms, ensuring high performance, reliability, and...

  • Master Data Architect

    2 days ago

    Thrissur, Kerala, India beBeeDataArchitect Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Are you a master data architect seeking a challenging role where you can leverage your expertise to design and implement cutting-edge Databricks solutions?In this key position, you will be responsible for architecting scalable, secure, and efficient Databricks solutions that meet client requirements. You will work closely with clients to understand their...

  • Senior Cloud Data Architect

    8 hours ago

    Thrissur, Kerala, India beBeeData Full time ₹ 15,00,000 - ₹ 18,00,000

    Cloud Data ArchitectRole OverviewWe are seeking an experienced cloud data architect to design, develop, and maintain scalable data pipelines and architectures that support business analytics initiatives. This role will focus on building production-ready workflows for ingestion, transformation, quality checks, lineage capture, access auditing, cost usage...

  • Lead Data Solutions Architect

    2 days ago

    Thrissur, Kerala, India beBeeData Full time ₹ 1,00,00,000 - ₹ 1,50,00,000

    We are currently seeking a skilled Data Architect to design, develop, and govern our data architecture and pipelines.The ideal candidate will be collaborative, well-organised, and able to think outside the box. Most importantly, this role is for an individual who is passionate about making a difference through healthcare.Key Responsibilities:Data...

  • Senior Data Engineer and Snowflake Architect

    3 days ago

    Thrissur, Kerala, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    About the JobWe are seeking a seasoned data engineering professional to lead full lifecycle implementations of Snowflake platforms, focusing on secure platform setup, cloud integration, and high-performance data pipeline development.Lead the setup and provisioning of Snowflake environments, ensuring robust resource optimization and access control.Design and...