Data Engineering Lead

4 days ago

Gulbarga, Karnataka, India beBeeData Full time ₹ 1,60,00,000 - ₹ 2,30,00,000
Job Overview

We are seeking an experienced Data Engineer Leader to lead the development of a scalable data ingestion framework while ensuring high data quality and validation.

The successful candidate will also be responsible for designing and implementing robust APIs for seamless data integration.

This role is ideal for someone with deep expertise in building and managing big data pipelines using modern technologies, and who is passionate about driving quality and efficiency in data processing systems.

  • Data Ingestion Framework Design & Development
  • Use AWS services such as AWS Glue, Lambda, EMR, ECS, EC2, and Step Functions to build highly scalable, resilient, and automated data pipelines.
  • Data Quality & Validation
  • Develop and implement automated data quality checks, validation routines, and error-handling mechanisms to ensure the accuracy and integrity of incoming data.
  • Establish comprehensive monitoring, logging, and alerting systems to proactively identify and resolve data quality issues.
  • API Development
  • Architect and develop secure, high-performance APIs to enable seamless integration of data services with external applications and internal systems.
  • Create thorough API documentation and establish standards for API security, versioning, and performance optimization.
  • Cross-Functional Communication
  • Work closely with business stakeholders, data scientists, and operations teams to understand requirements and translate them into technical solutions.
  • Agile Development
  • Participate in sprint planning, code reviews, and agile ceremonies, while contributing to continuous improvement initiatives and CI/CD pipeline development (using tools like GitLab).
Required Skills and QualificationsExperience & Technical Skills

Analytical platform development experience required.

  • Programming Skills
  • Proficiency in Python and/or PySpark, SQL for developing ETL processes and handling large-scale data manipulation.
  • AWS Expertise
  • Extensive experience using AWS services including AWS Glue, Lambda, Step Functions, and S3 to build and manage data ingestion frameworks.
  • Data Platforms
  • Familiarity with big data systems (e.g., AWS EMR, Apache Spark, Apache Iceberg) and databases like DynamoDB, Aurora, Postgres, or Redshift.
  • CI/CD & Agile
  • Hands-on experience with CI/CD pipelines (preferably with GitLab) and Agile development methodologies.

Soft Skills: Strong problem-solving abilities and attention to detail. Excellent communication and interpersonal skills with the ability to work independently and collaboratively.

Additional Requirements
  • Kinesis, Firehose, and SQS experience preferred. Data lakehouse architectures and modern data quality frameworks familiarity recommended. Prior experience in a proactive data quality management and API-driven integrations in complex environments advantageous.

  • Data Engineering Lead

    2 days ago

    Gulbarga, Karnataka, India beBeeEngineering Full time ₹ 30,00,000 - ₹ 40,00,000

    Job Opportunity: Data Engineering LeadWe're seeking an experienced data engineering professional to lead our Commercial Data team in Advertising. This role is ideal for a skilled developer who thrives in a collaborative environment and enjoys tackling technical challenges.About the Role:The purpose of the Commercial Data team is to develop leading property...

  • Data Engineering Lead

    1 day ago

    Gulbarga, Karnataka, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Unlock your data engineering potential with this exciting roleJob OverviewThis is a unique opportunity to join our team as a senior data engineer and contribute to the development of intricate ETL pipelines. You will be responsible for refining high-volume data processing systems, ensuring precision, coherence, and performance.Key ResponsibilitiesDesign and...

  • Senior Data Engineering Project Lead

    2 days ago

    Gulbarga, Karnataka, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job DescriptionWe are seeking an experienced Technical Project Manager to lead our data engineering projects.The successful candidate will have a proven track record of managing data engineering, analytics, or AI/ML projects end-to-end.Responsibilities include:Developing comprehensive project plans, roadmaps, and timelines for data ingestion, transformation,...

  • Data Engineering Lead

    5 days ago

    Gulbarga, Karnataka, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    A career-defining opportunity awaits a seasoned data engineer to assume a leadership role in our cutting-edge technology division. About UsWe are a forward-thinking company that has been pioneering innovative solutions for the digital advertising landscape. Our platform empowers major publishers and businesses worldwide by providing transparent and efficient...

  • Data Architecture Lead

    1 week ago

    Gulbarga, Karnataka, India beBeeData Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    Job TitleJob Overview:The Data Engineering Team Lead is a high-level leadership role that oversees the entire data engineering process. The position requires expertise in designing, developing, and maintaining large-scale data architectures.Key Responsibilities:Developing and implementing data architecture plans to support business growth and...

  • Senior Data Engineer

    2 days ago

    Gulbarga, Karnataka, India beBeeDataEngineering Full time ₹ 35,00,000

    Job Title: Data Engineering LeadWe are seeking a highly skilled and experienced Data Engineering Lead to join our organization. The successful candidate will be responsible for leading the development and implementation of data engineering solutions, ensuring high-quality delivery and optimal performance.Experience in team handling and leadershipAt least 14...

  • Lead Big Data Engineer

    4 days ago

    Gulbarga, Karnataka, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 24,00,000

    About the Role:This position requires a highly skilled data engineer to develop efficient and scalable data pipelines, perform integration testing, and provide estimates for development, testing & deployments.Key Responsibilities:Design and implement high-quality data pipelines that support increasing data volume and complexityCollaborate with analytics and...

  • Data Architect Lead

    7 days ago

    Gulbarga, Karnataka, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Senior Data ArchitectAbout the Role:We are seeking an experienced Senior Data Architect to lead our data warehouse team in designing and implementing innovative data-intensive applications on Amazon Web Services.About the Team:You'll be part of a dynamic team focusing on building and improving data warehouse offerings applying marketing suites.Your Key...

  • Lead Data Pipeline Architect

    1 week ago

    Gulbarga, Karnataka, India beBeeData Full time ₹ 1,20,00,000 - ₹ 1,60,00,000

    About the RoleWe are seeking an experienced Data Engineer to lead our data pipeline architecture and collaborate with data scientists and analysts.

  • Lead Financial Data Architect

    1 day ago

    Gulbarga, Karnataka, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Principal EngineerWe are seeking a seasoned principal engineer to lead the modernization of our finance data platform.This is an exceptional opportunity to shape next-generation financial data architecture and drive business growth through accurate and transparent financial insights.You will partner closely with finance and accounting stakeholders to design...