Lead Big Data Engineer

5 days ago

Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000
Job Description

We are seeking a senior data engineer to lead our big data engineering team. As a key member of the team, you will be responsible for developing architectures that meet business needs and providing technical leadership.

The role involves setting up cloud-based data lakes and implementing standardized data models. You will develop data pipelines for new sources, data transformations within the data lake, implementing GraphQL, working on NoSQL databases, CI/CD, and data delivery as per business requirements.

You will build scalable data pipelines to bring in various data from multiple sources within the organization and public data sources. Collaborate with cross-functional teams to source data and make it available for downstream consumption.

Work with the team to provide effective solution designs to meet business needs. Ensure regular communication with stakeholders and understand any key concerns in how the initiative is being delivered or any risks/issues that have either not yet been identified or are not being progressed.

Manage dependencies and challenges (risks) are escalated and managed. Escalate critical issues to the Sponsor and/or Head of Data Engineering.

Manage timelines (milestones, decisions, and delivery) to ensure value is achieved without compromising quality and within budget. Develop an appropriate communications plan for initiative execution and delivery, both internal and external.

Key Responsibilities
  • Develop large-scale data processing jobs using best practices.
  • Collaborate with cross-functional teams to source data and implement efficient solutions.
  • Design and implement scalable data architecture to meet business needs.
  • Manage dependencies and risks associated with project delivery.
Requirements
  • Strong problem analysis skills and experience with Azure Databricks platform.
  • At least one Cloud Infra provider (Azure/AWS).
  • Apache Spark (Spark SQL, Dataframe API) or Hive query language (HQL).
  • Streaming data pipeline using Apache Spark Structured Streaming or Apache Flink on Kafka & Delta Lake.
  • NoSQL databases, Cosmos DB, Restful APIs, and GraphQL.
  • Hive and Hadoop file formats (Avro / Parquet / ORC).
  • Basic knowledge of scripting (shell / bash).
  • CI/CD tools such as Jenkins, JIRA, Bitbucket, Artifactory, Bamboo, and Azure Dev-ops.
  • DevOps practices using Git version control.
Benefits

A team player with excellent problem analysis skills and experience in big data engineering. Good understanding of cloud infrastructure, data pipelines, and NoSQL databases. Ability to debug, fine-tune, and optimize large-scale data processing jobs.

  • Lead Data Engineering Specialist

    6 days ago

    Allahabad, Uttar Pradesh, India beBeeData Full time ₹ 18,00,000 - ₹ 25,00,000

    Job OpportunityWe are seeking an experienced Senior Data Engineer to lead our data engineering efforts. As a Senior Data Engineer, you will be responsible for designing and building robust data pipelines for ingestion, transformation, and analytics using the best of modern data engineering tools and practices on Google Cloud (GCP).The ideal candidate will...

  • Big Data Expert

    2 days ago

    Allahabad, Uttar Pradesh, India beBeeData Full time ₹ 20,00,000 - ₹ 40,00,000

    Greetings from our organizationOur Team is Looking for a Talented Big Data ProfessionalYears of Experience: 6 - 12 yearsLocation: Pune/ Mumbai/ ChennaiNotice Period: 0 - 45 daysJob Description:We are Seeking Highly Skilled Professionals to Join Our TeamPySparkHiveKey Skills:SparkHBaseData Quality toolsAgile Scrum methodologiesData ingestion from multiple...

  • Data Engineering Specialist

    16 hours ago

    Allahabad, Uttar Pradesh, India beBeeBigData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title: Data Engineering SpecialistJob Overview:As a Data Engineering Specialist, you will play a pivotal role in designing and developing large-scale data processing systems using cutting-edge technologies.This challenging position requires expertise in big data technologies, including PySpark and Hive. Familiarity with Spark, HBase, and data quality...

  • Big Data Solutions Specialist

    1 day ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Big Data EngineerWe are seeking a highly skilled professional with expertise in Scala, Spark, Python, NiFi, and Apache Kafka to join our data engineering team.Key Responsibilities:Design, develop, and optimize large-scale data pipelines and distributed data processing systems.Work extensively with Scala, Spark (PySpark), and Python for data...

  • Freelance Big Data Specialist

    19 hours ago

    Allahabad, Uttar Pradesh, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job TitleWe are seeking a highly skilled Freelance Data Engineer with expertise in designing, building, and optimizing large-scale data pipelines using Python, SQL, and cloud-based technologies.Data Engineering Expertise: Develop and maintain ETL/ELT pipelines using PySpark, AWS, and Databricks to extract, transform, and load data from various sources.Big...

  • Big Data Developer

    19 hours ago

    Allahabad, Uttar Pradesh, India beBeeDataInfrastructure Full time ₹ 12,50,000 - ₹ 17,50,000

    Job Title: Data Infrastructure SpecialistSeeking a seasoned data architect to design and build scalable data systems that handle high-volume financial information.This role involves leading the development of robust data pipelines that ensure data integrity, reliability, and real-time availability across our platforms.The ideal candidate will have expertise...

  • Senior Data Architect

    4 days ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Find Meaningful WorkWe are seeking an accomplished Data Engineering Manager to lead a talented team of data engineers and design scalable data pipelines and analytics platforms.

  • Chief Data Infrastructure Engineer

    19 hours ago

    Allahabad, Uttar Pradesh, India beBeeData Full time ₹ 1,20,00,000 - ₹ 1,50,00,000

    Cloud Data Architect PositionJob Overview:We are seeking an experienced Cloud Data Architect to design, develop, and manage enterprise data solutions on Google Cloud Platform (GCP).Key Responsibilities:Design end-to-end data solutions on GCP, aligning with business and technical requirements.Define data models, storage strategies, data ingestion, processing,...

  • Big Data Specialist

    4 days ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time US$ 90,000 - US$ 1,20,000

    Data Engineering RoleJob Title: Data EngineerWe are seeking a skilled Data Engineer to support data-driven initiatives.This role involves designing scalable pipelines, working with streaming data, and collaborating with data scientists and stakeholders to deliver high-quality solutions.Key Responsibilities:Design, build, and optimize data pipelines and ETL...

  • Data Engineer

    2 days ago

    Allahabad, Uttar Pradesh, India beBeeInformatica Full time ₹ 1,80,00,000 - ₹ 2,40,00,000

    Job Title: Data Engineer - Informatica SpecialistAbout the RoleThe successful candidate will be responsible for designing and developing scalable ETL workflows using Informatica BDM/DEI. This will involve integrating data from structured and unstructured sources, including big data platforms.The ideal candidate will have strong data warehousing concepts to...