Senior Data Engineer

4 days ago

Palakkad, Kerala, India beBeeDataEngineering Full time ₹ 1,00,00,000 - ₹ 1,50,00,000
Job Description

As a senior data engineering professional, you will design and implement large-scale data pipelines using Azure Databricks and Azure Data Lake. Your primary focus will be on building production-ready workflows for data ingestion, transformation, quality checks, lineage capture, access auditing, cost usage analysis, retention tracking, and metadata integration.

Key Responsibilities:
  • Pipeline Development: Develop robust ETL/ELT pipelines in Databricks (PySpark, SQL, Delta Lake) to ingest, transform, and curate governance and operational metadata from multiple sources landed in Databricks.
  • Granular Data Quality Capture: Implement profiling logic to capture issue-level metadata (source table, column, timestamp, severity, rule type) to support drill-down from dashboards into specific records and enable targeted remediation.
  • Governance Metrics Automation: Develop data pipelines to generate metrics for dashboards covering data quality, lineage, job monitoring, access & permissions, query cost, usage & consumption, retention & lifecycle, policy enforcement, sensitive data mapping, and governance KPIs.
  • Microsoft Purview Integration: Automate asset onboarding, metadata enrichment, classification tagging, and lineage extraction for integration into governance reporting.
  • Data Retention & Policy Enforcement: Implement logic for retention tracking and policy compliance monitoring (masking, RLS, exceptions).
  • Job & Query Monitoring: Build pipelines to track job performance, SLA adherence, and query costs for cost and performance optimization.
  • Metadata Storage & Optimization: Maintain curated Delta tables for governance metrics, structured for efficient dashboard consumption.
  • Testing & Troubleshooting: Monitor pipeline execution, optimize performance, and resolve issues quickly.
  • Collaboration: Work closely with the lead engineer, QA, and reporting teams to validate metrics and resolve data quality issues.
  • Security & Compliance: Ensure all pipelines meet organizational governance, privacy, and security standards.
Required Qualifications

To succeed in this role, you should have a Bachelor's degree in Computer Science, Engineering, Information Systems, or related field, and 4+ years of hands-on experience with Azure Databricks and Azure Data Lake.

  • Senior Data Engineer

    3 days ago

    Palakkad, Kerala, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job OverviewAchieve success as a Senior Data Engineer with expertise in large-scale data processing and system design.Key Responsibilities:• Develop high-performance ETL workflows using Python and Pandas to handle complex supply chain operations and personalized customer experiences.• Implement data ingestion and transformation processes using Hive SQL...

  • Data Engineering Leader

    1 week ago

    Palakkad, Kerala, India beBeeDataEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title: Senior Data EngineerAre you an expert data engineer looking to build large-scale data platforms on Google Cloud? We're seeking a talented professional with experience in designing and building robust data pipelines for ingestion, transformation, and analytics.Key Responsibilities:Design and develop efficient data pipelines using modern tools and...

  • Data Engineer Leader

    9 hours ago

    Palakkad, Kerala, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Senior Data Engineering Specialist Job Title: Senior Data Engineering SpecialistExperience Level: 5+ YearsKey Responsibilities:Develop and optimize ETL/ELT pipelines using Azure Data Factory (ADF), Databricks, and Synapse Analytics.Design and implement scalable data architectures following Medallion Architecture and best practices.Perform data ingestion,...

  • Senior Data Engineer

    3 days ago

    Palakkad, Kerala, India beBeeDataEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title: Data Engineering SpecialistAbout the Role:We are seeking a highly skilled data engineer to join our team. As a Data Engineering Specialist, you will be responsible for designing and developing end-to-end data integration solutions, optimizing workflows, and collaborating with business and technical stakeholders to deliver scalable and...

  • Senior Data Architect

    3 days ago

    Palakkad, Kerala, India beBeeDataEngineer Full time US$ 1,80,000 - US$ 2,30,000

    Unlock the Potential of Data EngineeringWe are seeking a highly skilled Senior Data Engineer to join our team. As a key member of our data engineering group, you will play a critical role in designing, building, and maintaining high-performance data pipelines that support advanced analytics and machine learning.Key Responsibilities:Lead the optimization and...

  • Senior Network Engineer

    7 days ago

    Palakkad, Kerala, India beBeeNetwork Full time ₹ 12,50,000 - ₹ 14,50,000

    Job Title: Senior Network Engineer - Data Center SpecialistWe are seeking a seasoned network professional with extensive experience in designing, implementing, and managing large-scale data center networks.Key Responsibilities:Provide Level 2 operational support for Cisco Tetration platforms, including day-to-day monitoring, troubleshooting, and maintenance...

  • Senior Data Engineering Leader

    6 days ago

    Palakkad, Kerala, India beBeeData Full time ₹ 1,80,00,000 - ₹ 2,20,00,000

    Engineering Manager RoleWe are seeking an experienced Engineering Manager to lead our team of data engineers in designing and delivering scalable data pipelines and analytics platforms.

  • Senior Cloud Data Engineer

    4 days ago

    Palakkad, Kerala, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    As a seasoned Data Engineer, you will play a pivotal role in shaping our data infrastructure within AWS.Data Engineer with AWS IIThis is a contract-to-hire position that requires overlap hours in IST and UK. Your responsibilities will include optimizing data flows within AWS, developing scalable ETL processes and APIs, troubleshooting data issues, and...

  • Senior Data Engineer Wanted

    3 days ago

    Palakkad, Kerala, India beBeeData Full time ₹ 10,00,000 - ₹ 15,00,000

    Job OpportunityData Engineer PositionWe are seeking a highly skilled Data Engineer to design, develop, and maintain large-scale data systems using Google Cloud Platform (GCP).Key Responsibilities:Data Ingestion: Design and implement efficient data pipelines to ingest data from various sources into GCP.Data Processing: Develop and optimize data processing...

  • Senior Data Leader

    4 days ago

    Palakkad, Kerala, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Senior Director, Real World DataDescription:The Senior Director (Real World Data) is a senior leadership position within our organization. This role involves owning and managing a significant body of work, as well as providing support to the RWD team for other parts of the organization.Key responsibilities include leading a team of analysts who support the...