Senior Data Engineer

2 days ago

Kollam, Kerala, India beBeeData Full time ₹ 8,00,000 - ₹ 15,00,000
Expert Data Pipeline Developer Needed

We are seeking an experienced and skilled data pipeline developer to design, build, and deploy robust ETL/ELT pipelines in Databricks. The ideal candidate will have a strong background in data engineering, Azure Databricks, and Azure Data Lake.

Key Responsibilities:
  • Pipeline Design and Development: Create complex data pipelines using PySpark, SQL, and Delta Lake to ingest, transform, and curate governance and operational metadata.
  • Granular Data Quality Monitoring: Implement profiling logic to capture issue-level metadata to support drill-down from dashboards into specific records and enable targeted remediation.
  • Governance Metrics Automation: Develop data pipelines to generate metrics for dashboards covering data quality, lineage, job monitoring, access & permissions, query cost, usage & consumption, retention & lifecycle, policy enforcement, sensitive data mapping, and governance KPIs.
  • Microsoft Purview Integration: Automate asset onboarding, metadata enrichment, classification tagging, and lineage extraction for integration into governance reporting.
  • Retention Tracking and Policy Compliance: Implement logic for retention tracking and policy compliance monitoring (masking, RLS, exceptions).
  • Job and Query Performance Monitoring: Build pipelines to track job performance, SLA adherence, and query costs for cost and performance optimization.
  • Metadata Storage and Optimization: Maintain curated Delta tables for governance metrics, structured for efficient dashboard consumption.
Requirements:
  • Bachelor's degree in Computer Science, Engineering, Information Systems, or related field.
  • 4+ years of hands-on data engineering experience, with Azure Databricks and Azure Data Lake.
  • Proficiency in PySpark, SQL, and ETL/ELT pipeline design.
  • Demonstrated experience building granular data quality checks and integrating governance logic into pipelines.
  • Working knowledge of Microsoft Purview for metadata management, lineage capture, and classification.
  • Experience with Azure Data Factory or equivalent orchestration tools.
Preferred Qualifications:
  • Microsoft certification in Azure Data Engineering.
  • Experience in governance-heavy or regulated environments.
  • Exposure to Power BI or other BI tools as a data source consumer.

  • Data Scientist/ML Engineer

    5 days ago

    Kollam, Kerala, India Quant-data Full time

    We're Hiring: Machine Learning Engineer / Data Engineer (Remote | Full-Time) Build AI-powered credit decisioning systems on Microsoft AzureWe're looking for a Machine Learning Engineer / Data Engineer with 5+ years of experience to join our AI-driven credit lending platform team. In this role, you'll design and deploy scalable ML solutions that power loan...

  • Senior Data Engineer

    24 hours ago

    Kollam, Kerala, India beBeeData Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title: Senior Data EngineerThis senior data engineer position offers the opportunity to develop and maintain scalable data pipelines using Python and AWS services (S3, Redshift, Lambda, Glue) ensuring efficient data processing and transfer.The selected candidate will be responsible for implementing and optimizing Extract, Transform, Load (ETL) processes...

  • Senior Data Engineering Professional

    4 days ago

    Kollam, Kerala, India beBeeDataEngineer Full time ₹ 12,00,000 - ₹ 20,10,000

    Job Title: Senior Data EngineerWe are seeking an experienced professional to join our organization in the role of Senior Data Engineer. In this position, you will be responsible for developing and implementing data engineering solutions using Python and Pyspark.Key Responsibilities:Design and implement scalable data architectures to support large-scale data...

  • Senior Data Engineering Leader

    2 days ago

    Kollam, Kerala, India beBeeDataEngineering Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Senior Data Engineering Leader">The Technical Lead-Data Engineer is a senior-level position responsible for leading data engineering teams and projects. The role requires a deep understanding of data warehousing concepts, OLAP design, and enterprise-level data engineering principles.">Must-Have Skills and Qualifications:">">8+ years of experience...

  • Leader in Data Engineering

    24 hours ago

    Kollam, Kerala, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 40,00,000

    Senior Data Engineer PositionAt Tredence, we're looking for a skilled Senior Data Engineer to fill a key role in our data engineering team. The successful candidate will be responsible for designing and implementing large-scale data systems that enable clients to make informed business decisions.Responsibilities:Develop modern data warehouse solutions using...

  • Senior Cloud Data Engineer

    2 days ago

    Kollam, Kerala, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Data Engineer Job DescriptionWe are seeking a seasoned and proficient Senior Data Engineer with substantial experience in cloud technologies.As a pivotal member of our data engineering team, you will play a crucial role in designing, implementing, and optimizing data pipelines, ensuring seamless integration with cloud platforms.Key Responsibilities:Design,...

  • Data engineer

    2 days ago

    Kollam, Kerala, India Centrilogic Full time

    Data EngineerPurpose:Over 15 years, we have become a premier global provider of multi-cloud management, cloud-native application development solutions, and strategic end-to-end digital transformation services.Headquartered in Canada and with regional headquarters in the U. S. and the United Kingdom, Centrilogic delivers smart, streamlined solutions to...

  • Senior Cloud Data Architect

    2 days ago

    Kollam, Kerala, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Senior Software Engineer/Technical SpecialistJob DescriptionWe are seeking a skilled software engineer to join our team. As a senior software engineer, you will be responsible for designing and implementing scalable data architectures using Azure Data Factory (ADF), Databricks, and Synapse Analytics.You will work with large datasets, developing...

  • Senior Data Engineering Specialist

    2 days ago

    Kollam, Kerala, India beBeeData Full time ₹ 18,00,000 - ₹ 25,00,000

    Job Opportunity:We are seeking a skilled Data Engineer to join our organization.The ideal candidate will have a strong background in data engineering, with expertise in building scalable data pipelines using PySpark and Apache Airflow.Proficiency in using Spark SQL, DataFrame, and RDD APIs to implement complex business logic is essential.A solid foundation...

  • Senior Data Engineering Strategist

    1 day ago

    Kollam, Kerala, India beBeeDataEngineering Full time ₹ 19,49,900 - ₹ 24,99,900

    Lead Data Engineer-Solution Architect RoleThis is a leadership position for an experienced Data Engineer who will be responsible for designing and implementing large-scale data architecture solutions.The ideal candidate will have a strong background in data architecture, software engineering, and solution delivery.A Bachelor's degree in Computer Science or...