Principal Data Engineer

17 hours ago

Kannur, Kerala, India beBeeEngineering Full time ₹ 36,00,000 - ₹ 42,00,000

Job Title: Technical Lead - Data Engineering

About the Role

We are seeking a highly skilled technical lead to spearhead our data engineering efforts. This individual will be responsible for designing and implementing large-scale data pipelines using Apache Beam and Databricks.

  • Develop and maintain a shared component library/SDK for data ingestion, processing, and publishing.
  • Create pluggable interfaces to enable seamless integration with multiple teams and tools.
  • Define patterns/templates for Apache Beam pipelines and Databricks jobs, ensuring standardization of configuration, packaging, versioning, CI/CD, and documentation.
  • Own end-to-end profiling and tuning, leveraging tools such as cProfile, py-spy, and line_profiler.
  • Instrument services with Elastic APM and correlate traces/metrics with Splunk logs, building dashboards and runbooks.
  • Implement concurrency best practices, including asyncio for I/O-bound tasks and ThreadPool/ProcessPool for CPU-bound tasks.
  • Establish SLOs/alerts for throughput, latency, error rates, and set up dead letter queues (DLQs) and recovery patterns.
Required Skills and Qualifications

7+ years of experience in Python, with strong depth in performance and concurrency, profiling, and memory tuning.

  • Observability expertise: Elastic APM instrumentation and dashboarding; Splunk for logs and correlation.
  • Must have implemented Large Language Model (LLM) based solutions and supported them in production.
  • API engineering for high-throughput integrations, resilience patterns, and secure handling of sensitive data.
  • Strong architecture/design skills: clean interfaces, packaging shared libraries, versioning, CI/CD, testing.
  • 3+ years building large-scale data pipelines with Apache Beam and/or Spark, including hands-on Databricks experience.
  • Document processing: OCR, PDF parsing, text normalization.
  • LLM/SLM integration experience, prompt/eval frameworks, PII redaction/guardrails.
  • Cloud and tooling: AWS/Azure/GCP, Dataflow/Flink, Terraform, Docker; cost/performance tuning on Databricks.
  • Security/compliance mindset, secrets management, least-privilege access.
Benefits

As a technical lead, you will have opportunities to mentor junior engineers, lead design reviews, codify best practices, and write clear documentation and examples.

You will also partner with machine learning engineers on future LLM/SLM paths, evaluating harnesses, safety/PII, and cost/performance.

Our ideal candidate is a seasoned professional with a passion for innovation and collaboration, who can drive technical excellence and deliver results-oriented solutions.

  • Principal Data Engineer

    1 hour ago

    Kannur, Kerala, India beBeeDataEngineer Full time ₹ 1,20,00,000 - ₹ 2,40,00,000

    Geektrust Overview:A leading retail and e-commerce organization is seeking a seasoned expert with 5-16 years of experience in large-scale technology project management.Role OverviewWe are looking for an accomplished professional to lead high-availability systems handling complex supply chain operations, order management, and personalized customer experiences...

  • Expert Aviation Civil Engineering Leadership Opportunity

    22 hours ago

    Kannur, Kerala, India beBeeAirfield Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title:Aviation Airfield Engineering ManagerThe WSP Aviation team is seeking an experienced Airfield Engineer to be the Principal Engineer within our office, with the potential to become the GCC Airfield Engineering Team Lead as the team grows and develops.The successful candidate will be the first member of this team and will form a crucial role in its...

  • Principal Big Data Specialist

    22 hours ago

    Kannur, Kerala, India beBeeDataEngineer Full time ₹ 12,00,000 - ₹ 20,00,000

    Job Description: Data Engineer

  • Leading Data Engineer

    6 days ago

    Kannur, Kerala, India beBeeDataEngineer Full time ₹ 10,00,000 - ₹ 20,25,000

    Data Engineer OpportunityWe are seeking a skilled Data Engineer to join our team. As a key member of our data engineering group, you will play a critical role in designing, building, and maintaining large-scale data processing systems.Key Responsibilities:Design and implement data pipelines using Informatica DEICollaborate with cross-functional teams to...

  • Data Engineering Specialist

    2 days ago

    Kannur, Kerala, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title: Data Engineering SpecialistWe are seeking a highly skilled and experienced data engineer to join our team.About the RoleThe ideal candidate will have a strong background in data engineering, with experience working on large-scale data pipelines and a proven track record of delivering high-quality solutions.The role will involve designing,...

  • Data Engineering Lead

    2 days ago

    Kannur, Kerala, India beBeeDataEngineering Full time ₹ 12,00,000 - ₹ 21,60,000

    Job Title: Data Engineering LeadTo succeed in this pivotal role, you will be responsible for crafting data ingestion and transformation frameworks that fuel real-time and batch analytics across our data platform.The ideal candidate will have extensive expertise in building large-scale data pipelines, implementing enterprise-grade data governance, security,...

  • Big Data Engineer

    2 weeks ago

    Kannur, Kerala, India Nice Software Solutions Pvt. Ltd. Full time

    Big Data Engineer (PySpark)Location: Pune/Nagpur (WFO)Experience: 8 - 12 YearsEmployment Type: Full-timeJob OverviewWe are looking for an experienced Big Data Engineer with strong expertise in PySpark and Big Data ecosystems. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines while ensuring high...

  • Expert Data Engineer

    18 hours ago

    Kannur, Kerala, India beBeeDataEngineering Full time ₹ 18,00,000 - ₹ 24,00,000

    Data Engineering ExpertWe are seeking a highly skilled Data Engineer to join our team.This is an opportunity for remote collaboration, with a preference for candidates in the same region for occasional in-person meetings.Key Requirements:Mandatory Skills:PowerBI (Essential)Snowflake (Essential)Python (Essential)Advanced SQL Query Writing – Experience with...

  • Data Engineering Expert

    22 hours ago

    Kannur, Kerala, India beBeeDataEngineer Full time ₹ 2,50,00,000 - ₹ 3,00,00,000

    Job DescriptionThe ideal candidate will have a strong background in data engineering, data architecture, and data integration. We are seeking an expert who can design, develop, and implement scalable data pipelines to drive business growth.Required Skills and QualificationsOver 6 years experience as a Data Engineer or in a similar role with expertise in data...

  • Enterprise Data Engineer

    2 days ago

    Kannur, Kerala, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    As a highly skilled data professional, you will play a key role in designing, building, and maintaining large-scale data processing systems. This includes architecting, developing, and deploying data pipelines that ensure high-performance, scalability, and reliability.The ideal candidate will have expertise in Azure-based technologies such as Data Factory...