Azure Data Pipeline Architect

1 day ago

Mysore, Karnataka, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 21,00,000

Job Overview:

We are seeking a skilled professional to design and deploy robust data pipelines that support data governance and analytics initiatives. This role will focus on building production-ready workflows for ingestion, transformation, quality checks, lineage capture, access auditing, cost usage analysis, retention tracking, and metadata integration, primarily using Azure Databricks, Azure Data Lake, and Microsoft Purview.

Key Responsibilities:

  • Pipeline Development – Design, build, and deploy ETL/ELT pipelines in Databricks (PySpark, SQL, Delta Lake) to ingest, transform, and curate governance and operational metadata from multiple sources landed in Databricks.
  • Granular Data Quality Capture – Implement profiling logic to capture issue-level metadata (source table, column, timestamp, severity, rule type) to support drill-down from dashboards into specific records and enable targeted remediation.
  • Governance Metrics Automation – Develop data pipelines to generate metrics for dashboards covering data quality, lineage, job monitoring, access & permissions, query cost, usage & consumption, retention & lifecycle, policy enforcement, sensitive data mapping, and governance KPIs.
  • Microsoft Purview Integration – Automate asset onboarding, metadata enrichment, classification tagging, and lineage extraction for integration into governance reporting.
  • Data Retention & Policy Enforcement – Implement logic for retention tracking and policy compliance monitoring (masking, RLS, exceptions).
  • Job & Query Monitoring – Build pipelines to track job performance, SLA adherence, and query costs for cost and performance optimization.
  • Metadata Storage & Optimization – Maintain curated Delta tables for governance metrics, structured for efficient dashboard consumption.
  • Testing & Troubleshooting – Monitor pipeline execution, optimize performance, and resolve issues quickly.
  • Collaboration – Work closely with the lead engineer, QA, and reporting teams to validate metrics and resolve data quality issues.
  • Security & Compliance – Ensure all pipelines meet organizational governance, privacy, and security standards.

Requirements:

  • Bachelor's degree in Computer Science, Engineering, Information Systems, or related field.
  • 4+ years of hands-on data engineering experience, with Azure Databricks and Azure Data Lake.
  • Proficiency in PySpark, SQL, and ETL/ELT pipeline design.
  • Demonstrated experience building granular data quality checks and integrating governance logic into pipelines.
  • Working knowledge of Microsoft Purview for metadata management, lineage capture, and classification.
  • Experience with Azure Data Factory or equivalent orchestration tools.
  • Understanding of data modeling, metadata structures, and data cataloging concepts.
  • Strong debugging, performance tuning, and problem-solving skills.
  • Ability to document pipeline logic and collaborate with cross-functional teams.

Desirable Qualifications:

  • Microsoft certification in Azure Data Engineering.
  • Experience in governance-heavy or regulated environments (e.g., finance, healthcare, hospitality).
  • Exposure to Power BI or other BI tools as a data source consumer.
  • Familiarity with DevOps/CI-CD for data pipelines in Azure.
  • Experience integrating both cloud and on-premises data sources into Azure.

  • Senior Data Architect Azure Databricks

    2 days ago

    Mysore, Karnataka, India beBeeAzureDatabricksExpert Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title: Senior Data Architect Azure DatabricksWe are seeking a seasoned data architect to design and implement cutting-edge data solutions using Azure Databricks. As a key member of our team, you will be responsible for developing and optimizing complex data pipelines, ensuring data quality, security, and scalability.The ideal candidate has 8+ years of...

  • Data Pipeline Architect

    2 days ago

    Mysore, Karnataka, India beBeeDataPipelineDeveloper Full time ₹ 37,50,000 - ₹ 52,50,000

    Senior Data Pipeline DeveloperJob OverviewAs a Senior Data Pipeline Developer, you will be responsible for designing and implementing high-performance data pipelines using advanced ETL concepts and methodologies.Key ResponsibilitiesDesign and implement complex data pipelines utilizing ETL frameworks and tools such as SSIS and AWS GlueDevelop and maintain...

  • Data Engineer – Azure

    2 weeks ago

    Mysore, Karnataka, India vueverse. Full time

    Hybrid Role: Blr/HYD/Pune/ChennaiJob DescriptionWe are looking for a skilled Data Engineer with strong expertise in the Azure Data Platform to design, develop, and optimize large-scale data pipelines and analytics solutions. The ideal candidate should have deep hands-on experience with Azure Databricks, PySpark, Azure Data Factory, and Synapse Analytics,...

  • Cloud Data Architect

    1 day ago

    Mysore, Karnataka, India beBeeDataEngineering Full time ₹ 10,00,000 - ₹ 12,00,000

    As a Data Architect on the Microsoft Azure platform, you will be responsible for designing and implementing scalable data architectures that meet business objectives.Key Responsibilities:Data Pipeline Development: Design and implement efficient data pipelines using Azure Data Factory, Apache Beam, or other relevant tools to ensure high-quality data...

  • Data Pipeline Architect

    1 day ago

    Mysore, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Senior Data EngineerAbout the RoleWe are seeking a seasoned professional to build data pipelines and integrate healthcare data.This role is critical in creating scalable pipelines that process clinical encounter data, retrieve associated medical documents, and deliver validated information to machine learning systems for inference.You will work...

  • Lead Azure Data

    2 days ago

    Mysore, Karnataka, India beBeeAzure Full time ₹ 18,00,000 - ₹ 25,00,000

    Drive Innovation in Azure Data & AIWe are seeking a highly skilled professional to lead the development of technical training content on Azure's modern data and AI stack. As a Subject Matter Expert (SME), you will be responsible for defining and reviewing presentations, labs, and real-world scenarios to ensure instructional value and accuracy.Define and...

  • Cloud Data Engineer

    2 days ago

    Mysore, Karnataka, India beBeeData Full time ₹ 10,00,000 - ₹ 20,00,000

    Cloud Data Engineer (Azure Databricks)This position involves designing and developing data ingestion pipelines for integrating multiple sources into Databricks. As a key team member, you will be responsible for implementing and maintaining Continuous Integration/Continuous Deployment (CI/CD) pipelines for data workflows.Key Responsibilities:Develop and...

  • Senior Data Solutions Architect

    2 days ago

    Mysore, Karnataka, India beBeeDataSolutions Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Senior Data Solutions ArchitectAbout the RoleWe are seeking a highly skilled and experienced Senior Data Solutions Architect to join our team. As a key member of our data engineering group, you will be responsible for designing, developing, and maintaining large-scale data pipelines and ETL processes on Databricks.Key ResponsibilitiesDesign and...

  • Distinguished Data Architect

    9 hours ago

    Mysore, Karnataka, India beBeeData Full time ₹ 80,00,000 - ₹ 1,50,00,000

    Job Title: Senior Software Engineer/Technical SpecialistThis is a highly technical role involving the design, development, and optimization of ETL/ELT pipelines using Azure Data Factory (ADF), Databricks, and Synapse Analytics.Develop and maintain scalable data architectures following Medallion Architecture and best practices.Write and optimize SQL queries...

  • Advanced Data Pipeline Architect

    13 hours ago

    Mysore, Karnataka, India beBeeDataEngineer Full time US$ 1,00,000 - US$ 1,40,000

    **Job Summary**As a skilled data engineer, you will be responsible for designing and implementing efficient data pipelines using Python and AWS technologies. Your primary goal will be to ensure seamless data flow across various systems.",