Senior Data Engineering Specialist

2 days ago

Pune, Maharashtra, India beBeeData Full time ₹ 15,00,000 - ₹ 20,00,000

Job Overview

We are seeking an experienced Senior Data Engineer to lead our big data team. The ideal candidate will have a strong background in big data engineering and be able to work closely with clients and team members to understand their requirements and develop architectures that meet their needs.

The successful candidate will support the Data Engineering team in setting up the Data Lake on Cloud and implementing standardized Data Models, single views of customers. They will also develop data pipelines for new sources, data transformations within the Data Lake, and implement GraphQL, NoSQL databases, CI/CD, and data delivery as per business requirements.

Main Responsibilities:

  • Data Pipeline Development: Design, build, and maintain large-scale data pipelines to bring in various data from multiple sources within the organization as well as from social media and public data sources.
  • Collaboration and Communication: Collaborate with cross-functional teams to source data and make it available for downstream consumption. Ensure regular communication with key stakeholders, understand any key concerns in how the initiative is being delivered or any risks/issues that have either not yet been identified or are not being progressed.
  • Team Leadership: Lead the big data team in providing an effective solution design to meet business needs. Manage dependencies and challenges (risks) and escalate critical issues to the Sponsor and/or Head of Data Engineering.
  • Project Management: Manage timelines (milestones, decisions, and delivery) to achieve the value of the initiative without compromising quality and within budget. Develop an appropriate and coordinated communications plan for initiative execution and delivery, both internal and external.
  • Post-Implementation Review: Ensure final handover of the initiative to business-as-usual processes and carry out a post-implementation review (as necessary) to ensure initiative objectives have been delivered and any lessons learned are fed into future initiative management processes.

Required Skills and Qualifications

  • Teamwork: Work effectively as part of a team.
  • Problem Analysis: Excellent problem analysis skills to identify and resolve complex data engineering issues.
  • Cloud Infrastructure: Good experience with Azure Databricks platform and at least one Cloud Infrastructure provider (Azure/AWS).
  • Big Data Processing: Experience in building data pipelines using batch processing with Apache Spark (Spark SQL, Dataframe API) or Hive Query Language (HQL). Knowledge of streaming data processing technologies such as Apache Spark Structured Streaming or Apache Flink on Kafka & Delta Lake.
  • NoSQL Databases: Knowledge of NoSQL databases, including Cosmos DB, Restful APIs, and GraphQL.
  • ETL Processing: Knowledge of Big Data ETL processing tools, Data modeling, and Data mapping. Experience with Hive and Hadoop file formats (Avro / Parquet / ORC).
  • Scripting: Basic knowledge of scripting (shell/bash). Experience of working with multiple data sources including relational databases (SQL Server / Oracle / DB2 / Netezza), NoSQL/document databases, flat files.
  • CICD Tools: Basic understanding of CI/CD tools such as Jenkins, JIRA, Bitbucket, Artifactory, Bamboo, and Azure Dev-ops. Ability to debug, fine-tune, and optimize large-scale data processing jobs.

  • Data Engineering Specialist

    1 day ago

    Pune, Maharashtra, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 24,00,000

    Data Engineering SpecialistThe role of a data engineer is pivotal in designing, implementing, and optimizing data pipelines for efficient data modeling and transformation.

  • Senior QE Engineer

    2 weeks ago

    Pune, Maharashtra, India NTT Data Full time

    Job DescriptionReq ID:330977NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Senior QE Engineer to join our team in Pune, Mahrshtra (IN-MH), India (IN).Senior Quality Engineer I...

  • Senior Data Engineer

    4 weeks ago

    Pune, Maharashtra, India NielsenIQ Full time

    We love data GfK Entertainment is globally recognized for outstanding market research in the entertainment sector music video games books Our data forms the basis for crucial decisions made by the largest entertainment companies both nationally and globally We create the official music charts in 10 countries and present the German No 1 Music Award ...

  • Data Engineering Specialist

    1 day ago

    Pune, Maharashtra, India beBeeDataEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job OverviewData Engineering SpecialistWe are seeking a skilled Data Engineering Specialist to join our organization.The ideal candidate will have a strong background in SQL, ETL, and data warehousing, with a minimum of 5 years of experience.ResponsibilitiesDesign, develop, and maintain large-scale data warehouses and ETL processes.Collaborate with...

  • Senior Software Engineer

    3 weeks ago

    Pune, Maharashtra, India Data Dynamics Full time

    About the Role : We are seeking a Senior Software Engineer to join our team in Pune. As a key member of our engineering group, you will play a crucial role in the entire software development lifecycle, from initial design to hands-on development. You will work with cross-functional teams to create innovative solutions that are scalable and reliable. This...

  • Senior Data Analytics Specialist

    5 days ago

    Pune, Maharashtra, India beBeeRecommender Full time ₹ 15,00,000 - ₹ 25,00,000

    Job OpportunitySenior Data Analytics SpecialistWe are seeking a highly skilled and experienced Senior Data Analytics Specialist to join our team.At least 6 years of experience in developing and implementing AI-driven solutions, with a focus on NLP and recommender systems.Proficiency in working with large datasets and performing data preprocessing using...

  • Senior Data Engineer

    4 days ago

    Pune, Maharashtra, India SPIRO Full time

    Position: Senior Data EngineerLocation: PuneExperience: 5+ yearsJob Summary:We are seeking an experienced Senior Data Engineer with a strong background in PySpark, Spark, and Big Data technologies. The ideal candidate will have over five years of hands-on experience designing, building, and optimizing large-scale data processing systems and pipelines.Key...

  • A135 - Data Mining and Material Compliance Specialist

    3 weeks ago

    Pune, Maharashtra, India Speed Engineering Solutions Full time

    Job DescriptionKey Responsibilities- Data mining and material specialist.- Candidate shall have worked on any data management tool.- Good understanding of overall design process.- Knows engineering change request. Good at MS-Excel.- Should be able to create BOM.- Aware about material specifications.Skills And Experience Required- Experience - 1 to 3 Years-...

  • Senior Data Engineer

    4 weeks ago

    Pune, Maharashtra, India Kivor Full time

    Company DescriptionKivor is revolutionizing enterprise application support with Gen AI-powered autonomous agents. Our platform transforms data from logs, tickets, source code, and documentation into an application-specific knowledge graph, enabling faster issue detection and seamless autonomous resolutions. This innovation reduces support costs by over 50%...

  • Senior Data Engineer

    3 weeks ago

    Pune, Maharashtra, India Kivor Full time

    Company Description Kivor is revolutionizing enterprise application support with Gen AI-powered autonomous agents. Our platform transforms data from logs, tickets, source code, and documentation into an application-specific knowledge graph, enabling faster issue detection and seamless autonomous resolutions. This innovation reduces support costs by over 50%...