Data Pipeline Architect

2 days ago

Pune, Maharashtra, India beBeeData Full time ₹ 1,04,000 - ₹ 1,30,878
Senior Big Data Engineer">

This role will oversee a team of employees and contractors in their professional development. Key responsibilities include mentoring, problem resolution, and team building to ensure success. The position will also lead software engineering projects in developing high-quality, secure, scalable applications and pipelines based on technical requirements and design artifacts.

The team is cross-functional, comprising developers, testers, product managers, and architects with diverse skills. The successful candidate will provide ongoing guidance on professionalism, career growth, development standards, automation tools, and Agile methodologies.

  • Develop strategic technical plans for both software engineering and data engineering solutions.
  • Collaborate with product management and engineering teams to translate product use cases and requirements into industry-leading designs, architectures, and implementations.
  • Own the end-to-end development of full-stack Spark Scala applications, including programming, unit testing, integration testing, documentation, maintenance, and support.
  • Collaborate with other engineers and the engineering team to execute baseline architecture implementations and design activities.
  • Work with business and technical partners to identify and scope opportunities, outline potential value and ROI, and recommend solutions for emerging issues.
  • Document software programs according to best practices and follow quality assurance and control processes.
  • Support defect management, code versioning, and deployments.
  • Ensure robust test coverage for full-stack applications, including unit testing and integration testing.
  • Collaborate with data engineers to optimize ETL pipelines, data processing workflows, and database design for analytics needs.
  • Regularly communicate with management about risk analysis and design trade-offs.
  • Participate in product kick-off sessions and requirement workshops for direction and course correction.
  • Contribute to the development and code review of key functionalities.

Essential Knowledge, Skills, and Experience

  • Proven experience leading full-stack software development teams in a distributed environment.
  • A track record demonstrating ability to build a diverse and strong team.
  • Proven expertise in leading real-time data pipelines and large-scale distributed systems.
  • Hands-on experience with Hadoop, Apache Spark, Kafka, NiFi, and Big Data ecosystems.
  • Strong knowledge of data security, compliance, and governance standards.
  • Proficiency in programming languages such as Java, Python, or Scala.
  • Experience providing innovative analytical solutions with lasting impact.
  • Knowledge of data science concepts, including structured and unstructured data and analytics-driven features.

Desirable Qualifications

  • Minimum 8 years of experience in Data & Software Engineering, with 2-4 years of experience in managing and leading teams.
  • Extensive familiarity with cloud platforms and container orchestration tools.
  • Experience with Agile/Scrum environments preferred.

  • Chief Data Pipeline Architect

    3 days ago

    Pune, Maharashtra, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Lead Data Solutions ArchitectThe Role:This Senior Data Engineer position plays a pivotal role in the design, development, and maintenance of large-scale data pipelines using Azure services such as Databricks and Data Factory. The ideal candidate will possess proficiency in programming languages like SQL, Python, R, or Scala to create efficient data...

  • Data Pipeline Architect

    3 days ago

    Pune, Maharashtra, India beBeeData Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    Senior Data Engineer - AWS ExpertWe are seeking a seasoned professional to spearhead the design, development, and deployment of end-to-end data pipelines on AWS cloud infrastructure.Key Responsibilities:Design scalable data processing workflows using Apache Spark and SQL for analytics and reporting requirements.Build and maintain orchestration workflows to...

  • Senior Data Pipeline Expert

    3 days ago

    Pune, Maharashtra, India beBeeData Full time ₹ 1,80,00,000 - ₹ 2,00,00,000

    Job Title: Data Architect SpecialistDescription:We are looking for a skilled Data Architect Specialist to craft and maintain robust data pipelines, utilizing Python and SQL, to ensure efficient extraction, transformation, and loading (ETL) of data. This role involves building and optimizing data pipelines to facilitate seamless data flow across systems and...

  • Data Pipeline Architect

    15 hours ago

    Pune, Maharashtra, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    PySpark Developer Job Description We are seeking a highly skilled PySpark developer to join our team in an offshore capacity. This role focuses on designing, building, and optimizing large-scale data pipelines and processing solutions on the Databricks Unified Analytics Platform. Main Responsibilities: Develop and manage data pipelines using PySpark on...

  • Senior Data Pipeline Specialist

    1 day ago

    Pune, Maharashtra, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Job OverviewWe are seeking a seasoned Data Engineer to join our team. This role involves crafting and maintaining robust data pipelines using Python and SQL to ensure efficient extraction, transformation, and loading (ETL) of data.The ideal candidate will have expertise in building and optimizing data pipelines to facilitate seamless data flow across systems...

  • Senior Data Pipeline Architect

    23 hours ago

    Pune, Maharashtra, India beBeeData Full time US$ 1,80,000 - US$ 2,20,000

    Senior Data Engineer RoleWe are seeking a talented and passionate Senior Data Engineer to join our team.This role involves building and scaling our data infrastructure, enabling business-driven decision-making across the organization.The ideal candidate will design, develop, and maintain efficient and reliable data pipelines for both ELT (Extract, Load,...

  • Data Architect

    3 days ago

    Pune, Maharashtra, India beBeeData Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    Senior Data Solutions ArchitectWe are seeking a highly skilled and experienced Senior Data Solutions Architect to design, build, and maintain robust data pipelines using Azure cloud services.The ideal candidate will have 9-12 years of experience in designing, building, and maintaining scalable data solutions for large-scale organizations.Key...

  • Data Architect

    2 days ago

    Pune, Maharashtra, India beBeeDataArchitecture Full time ₹ 15,00,000 - ₹ 25,50,000

    Data ArchitectWe are seeking a highly skilled Data Architect to join our team. As a Data Architect, you will be responsible for designing, developing, and maintaining data solutions that facilitate data generation, collection, and processing.Your typical day will involve creating data pipelines, ensuring data quality, and implementing ETL processes to...

  • Chief Data Pipeline Architect

    23 hours ago

    Pune, Maharashtra, India beBeeSnowflake Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Snowflake Data Engineer Job SummaryWe are seeking a highly skilled Snowflake data engineer with at least 5 years of experience in building and optimizing data pipelines, transforming data, and enabling advanced analytics solutions.

  • Data Architect Lead

    7 days ago

    Pune, Maharashtra, India beBeeArchitect Full time ₹ 15,00,000 - ₹ 20,00,000

    Key Role in Data ArchitectureWe are looking for a highly skilled data architect to lead our team. As a data architect, you will be responsible for designing and implementing large-scale data architectures that facilitate seamless data flow across systems and platforms.Your Key Responsibilities:Design and Implement Data Pipelines: Develop and implement...