Principal Data Engineer

1 day ago

Vijayawada, Andhra Pradesh, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000
Job Title: Data Engineering Professional

Overview:

Develop, optimize and maintain scalable data pipelines to support business intelligence and analytics initiatives.

Description:

Key Responsibilities
  1. Pipeline Development: Design, build and deploy robust ETL/ELT pipelines in Azure Databricks (PySpark, SQL, Delta Lake) to ingest, transform and curate governance and operational metadata from multiple sources landed in Databricks.
  2. Granular Data Quality Capture: Implement profiling logic to capture issue-level metadata (source table, column, timestamp, severity, rule type) to support drill-down from dashboards into specific records and enable targeted remediation.
  3. Governance Metrics Automation: Develop data pipelines to generate metrics for dashboards covering data quality, lineage, job monitoring, access & permissions, query cost, usage & consumption, retention & lifecycle, policy enforcement, sensitive data mapping, and governance KPIs.
  4. Microsoft Purview Integration: Automate asset onboarding, metadata enrichment, classification tagging, and lineage extraction for integration into governance reporting.
  5. Data Retention & Policy Enforcement: Implement logic for retention tracking and policy compliance monitoring (masking, RLS, exceptions).
  6. Job & Query Monitoring: Build pipelines to track job performance, SLA adherence, and query costs for cost and performance optimization.
  7. Metadata Storage & Optimization: Maintain curated Delta tables for governance metrics, structured for efficient dashboard consumption.
  8. Testing & Troubleshooting: Monitor pipeline execution, optimize performance, and resolve issues quickly.
  9. Collaboration: Work closely with the lead engineer, QA, and reporting teams to validate metrics and resolve data quality issues.
  10. Security & Compliance: Ensure all pipelines meet organizational governance, privacy, and security standards.
Required Qualifications
  • Bachelor's degree in Computer Science, Engineering, Information Systems, or related field.
  • 4+ years of hands-on data engineering experience, with Azure Databricks and Azure Data Lake.
  • Proficiency in PySpark, SQL, and ETL/ELT pipeline design.
  • Demonstrated experience building granular data quality checks and integrating governance logic into pipelines.
  • Working knowledge of Microsoft Purview for metadata management, lineage capture, and classification.
  • Experience with Azure Data Factory or equivalent orchestration tools.
  • Understanding of data modeling, metadata structures, and data cataloging concepts.
  • Strong debugging, performance tuning, and problem-solving skills.
  • Ability to document pipeline logic and collaborate with cross-functional teams.

  • Data Scientist/ML Engineer

    5 days ago

    Vijayawada, Andhra Pradesh, India Quant-data Full time

    We're Hiring: Machine Learning Engineer / Data Engineer (Remote | Full-Time) Build AI-powered credit decisioning systems on Microsoft AzureWe're looking for a Machine Learning Engineer / Data Engineer with 5+ years of experience to join our AI-driven credit lending platform team. In this role, you'll design and deploy scalable ML solutions that power loan...

  • Principal Software Quality Assurance Engineer

    6 days ago

    Vijayawada, Andhra Pradesh, India beBeeTest Full time ₹ 15,00,000 - ₹ 25,00,000

    Senior Test Automation LeadWe are seeking a seasoned Principal QA Engineer to spearhead our testing strategy, elevating product quality through expert leadership and strategic direction.This role offers the chance to work on diverse projects, driving both manual and automated testing in a dynamic environment.Leverage your expertise to lead test strategy,...

  • Principal Data Scientist

    13 hours ago

    Vijayawada, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 35,00,000

    Job Title: Big Data EngineerWe are seeking a highly skilled engineer to join our team as a Big Data Engineer. The ideal candidate will have expertise in Scala, Spark, Python, NiFi, and Apache Kafka.The successful candidate will be responsible for designing, developing, and optimizing large-scale data pipelines and distributed data processing systems.Key...

  • Data Engineer

    7 hours ago

    Vijayawada, Andhra Pradesh, India Moutups Full time

    Job Title: Data Engineer (Remote)Job Type: Full-TimeAbout the Role: We are looking for a highly motivated and detail-oriented Data Engineer to join our remote team. In this role, you will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure that support real-time and batch data processing. You will collaborate...

  • Data Engineer

    23 hours ago

    Vijayawada, Andhra Pradesh, India beBeeIngestion Full time ₹ 15,00,000 - ₹ 20,00,000

    Ingestion engineers are crucial for our organization's data strategy, responsible for crafting efficient data ingestion pipelines that integrate multiple sources into Databricks.Key Responsibilities:Data Ingestion Pipelines: Develop and optimize data ingestion pipelines for integrating multiple sources into Databricks.CI/CD Pipeline Implementation: Implement...

  • Data Engineer

    23 hours ago

    Vijayawada, Andhra Pradesh, India VSquare Systems Pvt. Ltd. Full time

    Job Description:We are looking for a highly skilled Databricks PySpark Developer to join our data platform implementation team. In this role, you will be instrumental in designing, developing, and maintaining ETL processes to ensure efficient extraction, transformation, and loading of data from various sources into data lake and data warehouse. You will work...

  • Senior Data Engineer

    13 hours ago

    Vijayawada, Andhra Pradesh, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    About the RoleAt our organization, we are seeking a skilled Senior Software Engineer/Technical Specialist to join our team. This role is ideal for an individual with expertise in data engineering, including experience with Azure Data Factory (ADF), Databricks, and Synapse Analytics.As a Senior Software Engineer/Technical Specialist, you will be responsible...

  • Cloud Data Engineering Leader

    3 days ago

    Vijayawada, Andhra Pradesh, India beBeeCloudEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Unlock your potential as a Cloud Data Engineering Leader at our innovative organization. We are seeking an experienced professional to spearhead the development of scalable data pipelines and analytics platforms in the cloud.The ideal candidate will lead a team of talented data engineers, guiding them to design and deliver high-performance data solutions...

  • Spatial Data Engineer

    1 day ago

    Vijayawada, Andhra Pradesh, India beBeeData Full time ₹ 10,00,000 - ₹ 15,00,000

    Job Title: Spatial Data EngineerWe are seeking an experienced Spatial Data Engineer to join our team. This role is crucial in the production and management of geospatial data across various projects.The position encompasses tasks such as digitization, geocoding, and conducting surveys when necessary.Key Responsibilities:Geospatial Data Production: Engage in...

  • Senior Data Engineering Strategist

    12 hours ago

    Vijayawada, Andhra Pradesh, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Senior Data Engineering Strategist A senior data engineering strategist is required to lead and manage a team of experienced data engineers. The ideal candidate should have a strong background in enterprise-level data engineering roles with expertise in OLAP design, data warehousing concepts, and Snowflake. Key Responsibilities: Leverage...