Lead - Data Engineer (Databricks)

2 weeks ago

Hyderabad, Telangana, India Anblicks Full time

Lead Data Engineer (Databricks) | 8+ Years Experience

Location : Hyderabad / Ahmedabad

Immediate or Early Joiners Preferred

Job Summary:

We are looking for an experienced Lead Data Engineer with 8+ years of expertise in designing, building, and optimizing data pipelines and ETL workflows. The ideal candidate will have strong hands-on experience with Databricks, PySpark, SQL, and Azure-based data engineering while also leading data engineering teams and projects.

Key Responsibilities:

  • Lead and mentor a team of data engineers, ensuring best practices in development and deployment.
  • Oversee project execution, define technical roadmaps, and ensure timely delivery.
  • Collaborate with cross-functional teams including data scientists, analysts, and DevOps.
  • Conduct code reviews, performance assessments, and knowledge-sharing sessions.
  • Design, develop, and optimize ETL/ELT workflows using Databricks, PySpark, SQL, and Azure Data Factory (ADF).
  • Implement incremental data loads, partitioning, and performance tuning.
  • Work with Azure Data Lake, Delta Lake, and Blob Storage for data storage and processing.
  • Integrate structured & unstructured data from APIs, databases, and streaming sources (Event Hubs/Kafka).
  • Build distributed data processing pipelines in Databricks (Apache Spark, PySpark/SQL).
  • Optimize Spark jobs, Delta Lake queries, and data processing performance.
  • Manage and optimize Azure Synapse, BigQuery, Cosmos DB, or Snowflake.
  • Tune SQL queries with indexing, caching, and partitioning for optimal performance.
  • Ensure data quality, lineage, and compliance using Azure Purview.
  • Implement RBAC, encryption, data masking, and security policies.
  • Monitor and troubleshoot data pipelines using Azure Monitor, Log Analytics.
  • Automate testing, validation, and deployment using Terraform & Python.
  • Implement CI/CD pipelines with GitHub Actions, Azure DevOps, or Jenkins.

Required Skills & Experience (8+ Years)

Must-Have:

  • Leadership & Project Management: Experience leading data engineering teams and managing end-to-end project execution.
  • Data Engineering & ETL: Hands-on expertise in Databricks, PySpark, SQL, ADF, Synapse, and Delta Lake.
  • Programming: Strong experience with Python (PySpark) and SQL for data processing.
  • Cloud Data Services: Proficiency in Azure Data Lake, Blob Storage, Synapse, and ADF.
  • Data Modeling & Performance Tuning: Expertise in SQL query optimization, indexing, and partitioning.
  • Streaming & Integration: Experience with Event Hubs, Kafka, APIs, and data ingestion frameworks.
  • Security & Compliance: Knowledge of RBAC, encryption, Azure Purview, and data masking.

Good-to-Have:

  • Experience with Snowflake, BigQuery, AWS Redshift, or GCP Cloud Storage.
  • Exposure to real-time data processing (Spark Streaming, Kafka, Flink).

Education:

BE/B.Tech, MCA, MSc, M.Tech in Computer Science, Information Technology, or a related field.

  • Data Engineering Team Lead with Databricks Expertise

    7 days ago

    Hyderabad, Telangana, India Anblicks Full time

    Job Summary:We are seeking an experienced Data Engineering Team Lead with 8+ years of expertise in designing, building, and optimizing data pipelines and ETL workflows using Databricks, PySpark, SQL, and Azure-based data engineering.Main Responsibilities:Lead and mentor a team of data engineers, ensuring best practices in development and deployment.Oversee...

  • Databricks System Engineer

    1 week ago

    Hyderabad, Telangana, India Wipro Full time

    In this role, you will work closely with the Team Lead, other Databricks Administrators, System Administrators, and Data Engineers/Scientists/Architects/Modelers/Analysts to ensure the smooth operation of the Databricks platform.Key ResponsibilitiesDesign, implement, and maintain the Databricks platform to meet business requirementsCollaborate with the data...

  • Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India NuStar Technologies Full time

    We are looking for a talented Data Engineer - Azure Databricks to lead the design and implementation of our big data solutions. As a key member of our team, you will work closely with cross-functional groups to develop scalable and efficient data pipelines.Responsibilities:Developing and maintaining large-scale data processing systemsDesigning and...

  • Databricks Engineer

    2 days ago

    Hyderabad, Telangana, India UNIQUEIT ENTERPRISE SOLUTIONS PRIVATE LIMITED Full time

    What You'll Do :- Design and develop data processing pipelines and analytics solutions using Databricks.- Architect scalable and efficient data models and storage solutions on the Databricks platform.- Collaborate with architects and other teams to migrate current solution to use Databricks.- Optimize performance and reliability of Databricks clusters and...

  • Databricks Engineer

    4 weeks ago

    Hyderabad, Telangana, India ShyftLabs Full time

    Job DescriptionPosition Overview:ShyftLabs is seeking a skilled Databricks Engineer to support in designing, developing, and optimizing big data solutions using the Databricks Unified Analytics Platform. This role requires strong expertise in Apache Spark, SQL, Python, and cloud platforms (AWS/Azure/GCP). The ideal candidate will collaborate with...

  • Senior Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India Enable Data Incorporated Full time

    Enable Data Incorporated is seeking a highly skilled Senior Data Engineer with expertise in cloud technologies, Spark, and Databricks to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, we leverage modern solutions to drive increased value across our clients' business ecosystem.As a Senior Data...

  • Data Engineering Team Lead Position

    2 weeks ago

    Hyderabad, Telangana, India Enable Data Incorporated Full time

    Job Overview">We are seeking a highly skilled senior data engineer to join our dynamic team at Enable Data Incorporated.">Key Responsibilities">Design, develop, and maintain scalable and efficient data solutions in the cloud using Apache Spark and Databricks.Gather and analyze data requirements from business stakeholders and identify opportunities for...

  • Databricks Engineering Expert

    2 weeks ago

    Hyderabad, Telangana, India Tata Consultancy Services Full time

    Job Description:We are seeking an experienced Azure Databricks Lead to join our team of data engineering experts. As a key member of our team, you will be responsible for designing, building, and maintaining large-scale data architectures on the Azure Databricks platform.

  • Senior Data Engineer/ Architect

    2 weeks ago

    Hyderabad, Telangana, India Enable Data Incorporated Full time

    Enable Data Incorporated is seeking a highly skilled Senior Data Engineer with expertise in cloud technologies, Spark, and Databricks to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, we leverage modern solutions to drive increased value across our clients' business ecosystem.As a Senior Data...

  • AidenAI - Data Engineer - Azure Databricks

    4 weeks ago

    Hyderabad, Telangana, India AIDEN AI PRIVATE LIMITED Full time

    Data Engineer (Teradata to Snowflake and Databricks on Azure Cloud). Position Overview : We are looking for a talented Data Engineer with strong expertise in data migration from Teradata to Snowflake and Databricks on Azure Cloud. This position will be pivotal in migrating legacy data systems to cloud-based solutions, ensuring a smooth transition with...