Data Engineer

3 days ago

Anantapur, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 80,00,000 - ₹ 1,50,00,000
Data Engineer: Automating Governance and Analytics Initiatives

We are seeking a hands-on data engineer to develop, optimize, and maintain automated data pipelines supporting data governance and analytics initiatives. This role requires expertise in designing, building, and deploying robust ETL/ELT pipelines in Databricks (PySpark, SQL, Delta Lake) to ingest, transform, and curate governance and operational metadata from multiple sources landed in Databricks.

  • Pipeline Development

Key Responsibilities:

  • Pipeline Development - Design, build, and deploy robust ETL/ELT pipelines in Databricks (PySpark, SQL, Delta Lake) to ingest, transform, and curate governance and operational metadata from multiple sources landed in Databricks.
  • Granular Data Quality Capture - Implement profiling logic to capture issue-level metadata (source table, column, timestamp, severity, rule type) to support drill-down from dashboards into specific records and enable targeted remediation.
  • Governance Metrics Automation - Develop data pipelines to generate metrics for dashboards covering data quality, lineage, job monitoring, access & permissions, query cost, usage & consumption, retention & lifecycle, policy enforcement, sensitive data mapping, and governance KPIs.
  • Microsoft Purview Integration - Automate asset onboarding, metadata enrichment, classification tagging, and lineage extraction for integration into governance reporting.
  • Data Retention & Policy Enforcement - Implement logic for retention tracking and policy compliance monitoring (masking, RLS, exceptions).
  • Job & Query Monitoring - Build pipelines to track job performance, SLA adherence, and query costs for cost and performance optimization.
  • Metadata Storage & Optimization - Maintain curated Delta tables for governance metrics, structured for efficient dashboard consumption.
  • Testing & Troubleshooting - Monitor pipeline execution, optimize performance, and resolve issues quickly.
  • Collaboration - Work closely with the lead engineer, QA, and reporting teams to validate metrics and resolve data quality issues.
  • Security & Compliance - Ensure all pipelines meet organizational governance, privacy, and security standards.

Benefits:

  • Opportunity to work with cutting-edge technologies and architectures.
  • Chance to collaborate with experienced professionals in the field.
  • Professional growth and development opportunities.

Requirements:

  • Strong understanding of data engineering principles and practices.
  • Proficiency in designing, building, and deploying ETL/ELT pipelines in Databricks.
  • Experience with data governance and analytics initiatives.
  • Excellent problem-solving and analytical skills.
  • Strong communication and collaboration skills.

Please submit your application, including your resume and a cover letter outlining your experience and qualifications.

  • Data engineer

    5 days ago

    Anantapur, Andhra Pradesh, India SourceFuse Full time

    Source Fuse Technologies hiring for Data Engineer with 3+ years for experience Job Summary:As a Data Engineer, you'll manage the invisible backbone of our analytics, building scalable data pipelines that fuel real-time insights for 630+ ships worldwide. Your work will turn raw data points into actionable intelligence, propelling machine learning models and...

  • Data Engineer

    2 days ago

    Anantapur, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job DescriptionAt our organization, we are seeking an experienced ETL Developer to join our team. The successful candidate will be responsible for designing and developing scalable ETL workflows using Informatica BDM/DEI.This is a full-time on-site role based in Bengaluru for one of our esteemed overseas clients. The role involves understanding requirements,...

  • Strategic Data Engineer

    12 hours ago

    Anantapur, Andhra Pradesh, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Strategic Data EngineerWe are seeking an experienced data engineer to lead our team's data engineering efforts. As a senior data engineer, you will play a critical role in designing, developing, and maintaining scalable data pipelines and data warehouse solutions on Google Cloud Platform (GCP).About the Position:This is an exceptional opportunity...

  • Data Engineering Specialist

    7 days ago

    Anantapur, Andhra Pradesh, India beBeeDataEngineering Full time ₹ 10,00,000 - ₹ 12,00,000

    Job Title: Data EngineerWe are seeking a skilled professional to join our team as a Data Engineer.The ideal candidate will have experience working with data platforms, designing and developing efficient and scalable data systems, and collaborating with cross-functional teams to drive business outcomes.Key Responsibilities:Design and develop data pipelines...

  • Cloud Data Engineer

    5 days ago

    Anantapur, Andhra Pradesh, India beBee Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title: Cloud Data EngineerAzure Databricks Engineer with Injection experienceJob DescriptionWe are seeking an experienced Cloud Data Engineer to lead our data ingestion and pipeline development efforts. The ideal candidate will have expertise in CI/CD, Kubernetes, and cloud platforms.The role requires a strong understanding of data ingestion processes,...

  • Data Engineer

    4 days ago

    Anantapur, Andhra Pradesh, India Innodata Inc. Full time

    We are looking for a Data Engineer with strong experience in CI/CD practices, Databricks (Spark), Python, Github and SQL. The ideal candidate should have hands-on expertise in building and automating data pipelines, managing multi-environment deployments, and working with modern DevOps/configuration management tools.Key ResponsibilitiesDesign, implement, and...

  • Data Engineering Specialist

    5 days ago

    Anantapur, Andhra Pradesh, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Data Engineering SpecialistWe are seeking a skilled Data Engineer to help implement an application for large retailers. The ideal candidate will have hands-on expertise in Python and PySpark, as well as experience with Telemetry, Big Data, and backend engineering.Key Responsibilities:Design, develop, and deploy data processing pipelines using...

  • Cloud Data Engineer

    2 days ago

    Anantapur, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 60,00,000 - ₹ 1,20,00,000

    Job Title: Cloud Data EngineerWe are seeking an experienced Cloud Data Engineer to join our team.Key Responsibilities:Design, develop, and maintain ETL/ELT pipelines using cloud-based technologies such as PySpark, AWS, and Databricks.Perform data profiling, validation, and transformation using Python and SQL.Optimize and monitor workflows for performance,...

  • Big data engineer

    2 hours ago

    Anantapur, Andhra Pradesh, India Idyllic Services Full time

    Job Title: Big Data Engineer – ScalaLocation: Bangalore, Chennai, Gurgaon, Pune, Mumbai.Experience: 7–10 Years (Minimum 3+ years in Scala)Notice Period: Immediate to 30 DaysMode of Work: Hybrid ???? Click the link below to learn more about the role and take the AI Interview to begin your application journey: Pe NRole OverviewWe are looking for a highly...

  • Data Quality Engineer

    3 days ago

    Anantapur, Andhra Pradesh, India beBeeInformatica Full time ₹ 18,00,000 - ₹ 25,00,000

    Seeking a skilled Data Quality Engineer to develop high-quality solutions using Informatica IDMC. Key responsibilities include designing, developing and implementing data quality rules and transformations.The ideal candidate will have 6-10 years of experience in ETL solution development using IICS/IDMC (CDI & CDQ modules) and possess strong knowledge of...