Expert Cloud Data Solutions Developer

1 week ago

Coimbatore, Tamil Nadu, India beBeeDataEngineer Full time ₹ 60,00,000 - ₹ 1,00,00,000

Job Description:

We are seeking a skilled Data Engineer to develop and maintain automated data pipelines for analytics initiatives.

This role will focus on building production-ready workflows for ingestion, transformation, quality checks, lineage capture, access auditing, cost usage analysis, retention tracking, and metadata integration using Azure Databricks, Azure Data Lake, and Microsoft Purview.

Key Responsibilities:
  • Pipeline Development – Design, build, and deploy robust ETL/ELT pipelines in Databricks (PySpark, SQL, Delta Lake) to ingest, transform, and curate governance and operational metadata from multiple sources landed in Databricks.
  • Granular Data Quality Capture – Implement profiling logic to capture issue-level metadata (source table, column, timestamp, severity, rule type) to support drill-down from dashboards into specific records and enable targeted remediation.
  • Governance Metrics Automation – Develop data pipelines to generate metrics for dashboards covering data quality, lineage, job monitoring, access & permissions, query cost, usage & consumption, retention & lifecycle, policy enforcement, sensitive data mapping, and governance KPIs.
  • Microsoft Purview Integration – Automate asset onboarding, metadata enrichment, classification tagging, and lineage extraction for integration into governance reporting.
  • Data Retention & Policy Enforcement – Implement logic for retention tracking and policy compliance monitoring (masking, RLS, exceptions).
  • Job & Query Monitoring – Build pipelines to track job performance, SLA adherence, and query costs for cost and performance optimization.
  • Metadata Storage & Optimization – Maintain curated Delta tables for governance metrics, structured for efficient dashboard consumption.
  • Testing & Troubleshooting – Monitor pipeline execution, optimize performance, and resolve issues quickly.
  • Collaboration – Work closely with the lead engineer, QA, and reporting teams to validate metrics and resolve data quality issues.
  • Security & Compliance – Ensure all pipelines meet organizational governance, privacy, and security standards.

Requirements:

  • A bachelor's degree in Computer Science, Engineering, Information Systems, or related field.
  • 4+ years of hands-on data engineering experience, with Azure Databricks and Azure Data Lake.
  • Proficiency in PySpark, SQL, and ETL/ELT pipeline design.
  • Demonstrated experience building granular data quality checks and integrating governance logic into pipelines.
  • Working knowledge of Microsoft Purview for metadata management, lineage capture, and classification.
  • Experience with Azure Data Factory or equivalent orchestration tools.
  • Understanding of data modeling, metadata structures, and data cataloging concepts.
  • Strong debugging, performance tuning, and problem-solving skills.
  • Ability to document pipeline logic and collaborate with cross-functional teams.

Preferred Qualifications:

  • Microsoft certification in Azure Data Engineering.
  • Experience in governance-heavy or regulated environments (e.g., finance, healthcare, hospitality).
  • Exposure to Power BI or other BI tools as a data source consumer.
  • Familiarity with DevOps/CI-CD for data pipelines in Azure.
  • Experience integrating both cloud and on-premises data sources into Azure.

  • Cloud Development Expert

    2 days ago

    Coimbatore, Tamil Nadu, India beBeeCloudExpert Full time ₹ 1,80,00,000 - ₹ 2,10,00,000

    Job Title: Cloud Development ExpertAbout the Role:We are seeking a skilled Cloud Development Expert to contribute to our scrum team and collaborate on Fusion Manufacturing Data Model (MFGDM). MFGDM serves as the foundation for Manufacturing Cloud and core Data Framework for Cloud Flagship Product.Responsibilities:Design, implement, and validate MFGDM...

  • Data Solutions Expert

    1 week ago

    Coimbatore, Tamil Nadu, India beBeeDataDriven Full time ₹ 20,00,000 - ₹ 25,00,000

    About UsWe provide data-driven solutions to empower businesses. Our team of experts designs and implements scalable, secure, and innovative data architecture.Key ResponsibilitiesWe design end-to-end data architecture solutions that meet business needs.Our team architects, builds, and optimizes data pipelines and ETL workflows using industry-leading tools.We...

  • Data Engineering Manager

    13 hours ago

    Coimbatore, Tamil Nadu, India Cloud Destinations Infotech Full time ₹ 7,00,000 - ₹ 12,00,000 per year

    Company DescriptionCloud Destinations, headquartered in Silicon Valley, specializes in digital transformation and enterprise application development for the Healthcare, Retail, and Technology sectors. We excel in cloud migrations, multi-cloud operations, DevOps, security operations, and IoT. Our globally distributed team includes skilled technical experts...

  • Cloud Data Migration Expert

    2 days ago

    Coimbatore, Tamil Nadu, India beBeeDataMigration Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Cloud Data Migration ExpertWe are seeking an experienced professional to lead the migration of large-scale SQL workloads from on-premise environments to Azure. The ideal candidate will have a strong foundation in data integration, transformation, and migration, along with a passion for working on complex data migration projects.Key...

  • Cloud Solutions Expert for IoT Integration

    2 days ago

    Coimbatore, Tamil Nadu, India beBeeExpert Full time ₹ 15,00,000 - ₹ 25,00,000

    Cloud Solutions Expert for IoT IntegrationThis position involves designing and implementing scalable, secure, and efficient cloud solutions using Azure. The focus will be on IoT device connectivity and integration.Collaboration with cross-functional teams is essential to understand business requirements and translate them into technical solutions.The ideal...

  • Expert Cloud Solution Architect

    6 days ago

    Coimbatore, Tamil Nadu, India beBeeCloud Full time ₹ 20,00,000 - ₹ 25,00,000

    Senior Cloud Architect Role Summary">This is an exciting opportunity for a seasoned Senior Cloud Architect to join our organization and lead the design, development, and implementation of our next-generation cloud-native architecture.">We are seeking a highly skilled and experienced professional with a deep understanding of Kubernetes, microservices, TOGAF,...

  • Big Data Expert

    4 days ago

    Coimbatore, Tamil Nadu, India beBeeData Full time ₹ 15,00,000 - ₹ 35,00,000

    Job Title: Big Data ExpertPrimary Skills:Big Data Analytics, Machine Learning, Data ScienceCloud Computing, Artificial Intelligence, Data EngineeringKey Responsibilities:Design and develop complex data pipelines utilizing big data analytics frameworks.Orchestrate large-scale data workflows ensuring seamless integration across cloud-based services.Develop...

  • Cloud Data Solutions Specialist

    4 days ago

    Coimbatore, Tamil Nadu, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Cloud Data Solutions EngineerWe are seeking a highly skilled AWS data professional with strong expertise in Python development and cloud-based data solutions. The ideal candidate will have a solid background in data migration, API ecosystem development, and modern data warehousing practices.Design, build, and maintain scalable and reliable data pipelines on...

  • Cloud Data Solutions Architect

    1 week ago

    Coimbatore, Tamil Nadu, India beBeeData Full time ₹ 18,00,000 - ₹ 20,00,000

    Job Title: Cloud Data Solutions Architect We are seeking a highly skilled and experienced Cloud Data Solutions Architect to lead the design and implementation of scalable and secure data solutions. The ideal candidate will have a deep understanding of Microsoft Azure services, including Power BI, Synapse, and Data Factory.Responsibilities:Design and oversee...

  • Cloud Data Solutions Architect

    4 days ago

    Coimbatore, Tamil Nadu, India beBeeData Full time ₹ 18,00,000 - ₹ 25,00,000

    We are seeking a highly skilled Cloud Data Solutions Architect to design and build scalable data pipelines in the cloud.About This Role:This role involves collaborating with cross-functional teams to support data-driven initiatives, using AWS services such as EMR, Glue, and PySpark. The ideal candidate will have hands-on experience with cloud-based...