Senior Data Pipeline Architect

2 weeks ago

Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000
Data Engineer Role

We are seeking a highly skilled and experienced Data Engineer to join our team. The ideal candidate will have a strong background in designing and developing scalable data pipelines that process massive datasets efficiently.

Key Responsibilities:

  • Data Pipeline Design & Development: Utilize Python and leverage GCP services such as Vertex AI, BigQuery, GKE, Cloud Run, PostgreSQL, Dataflow, and Pub/Sub to build efficient data pipelines.
  • Data Filtering & Curation: Optimize data ingestion and transformation for downstream models, implementing intelligent scanning mechanisms to filter out relevant data for AI/ML models.
  • AI/ML Integration: Leverage GenAI models under the Vertex AI umbrella, build and fine-tune custom models when LLMs are cost-prohibitive or less effective, and design multi-layered model pipelines balancing accuracy, cost, and performance.
  • Cloud Infrastructure: Architect scalable solutions primarily on GCP, with experience in Azure a plus but not mandatory.
  • Performance Optimization: Ensure real-time, low-latency processing for high-volume workloads, optimizing GPU/CPU-intensive environments to manage cost and efficiency.
  • Collaboration & Ownership: Work closely with data scientists, ML engineers, and solution architects, taking end-to-end ownership of assigned tasks and deliverables, working independently where needed.
Required Skills & Qualifications
  • Programming: Expert-level Python (mandatory).
  • Data Engineering: Hands-on experience with streaming pipelines, and large-scale data ingestion frameworks.
  • Ai/ml Expertise: Experience integrating and fine-tuning LLMs and GenAI models, comfortable working with custom model development where LLMs cannot be applied, familiarity with OpenAI, Gemini, Vertex AI, or equivalent platforms.
  • Cloud: GCP (mandatory) — experience with Vertex AI, BigQuery, GKE, Cloud Run, PostgreSQL, Dataflow, Pub/Sub, etc., Azure experience is a plus.
  • Scalability & Low Latency: Proven ability to design and implement high-performance, distributed data solutions.
  • High-Volume Data Processing: Experience processing massive datasets efficiently within strict timeframes.

  • Strategic Data Pipeline Developer

    2 weeks ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Senior Data Engineer Job DescriptionOur organization is seeking a skilled Senior Data Engineer to design, develop, and maintain data pipelines that extract data from Oracle Symphony via APIs, process, and store it in the Databricks Lakehouse platform. The successful candidate will integrate the data into Oracle EPM (Enterprise Performance Management) to...

  • Chief Data Pipeline Architect

    2 weeks ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 24,00,000

    Data Engineering RoleThis is an exciting opportunity for a Data Engineer to join our organization. As a key member of the team, you will be responsible for designing, developing, and maintaining data warehouses and data pipelines using Snowflake DB.The ideal candidate will have a strong background in data engineering and management, with experience in...

  • Chief Data Pipeline Architect

    1 week ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    About Our Data Engineering RoleWe are seeking a highly skilled Senior Data Engineer with expertise in building scalable, event-driven data pipelines and integrating healthcare data.Design and implement efficient data pipelines to support real-time clinical workflows.Integrate with Electronic Health Record (EHR) systems using standards like FHIR and...

  • Senior Data Pipeline Engineer

    1 week ago

    Allahabad, Uttar Pradesh, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Seeking a skilled and driven Senior ETL Developer to join our data engineering team. The successful candidate will design and implement scalable and efficient data pipelines using cutting-edge technologies.">Design and develop large-scale data pipelines using IBM DataStage, AWS Glue, and SnowflakeCollaborate with cross-functional teams to ensure seamless...

  • Senior Data Pipeline Specialist

    2 weeks ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,10,000

    About Our OpportunityWe are looking for a talented Data Engineer to join our team and play a key role in designing, building, and maintaining scalable data pipelines and infrastructure.You will collaborate closely with developers, architects, analysts, and data scientists to ensure smooth, secure, and efficient data delivery across multiple products and...

  • Senior Data Architect

    2 weeks ago

    Allahabad, Uttar Pradesh, India beBeeData Full time ₹ 18,00,000 - ₹ 25,00,000

    Job TitleWe are seeking a highly skilled Data Engineer to join our team in Pune. As a Senior Data Architect, you will be responsible for designing, deploying, and managing data pipelines to integrate data from various sources.Daily tasks include using data integration tools, performing data modeling, ensuring data quality, and providing analytical insights....

  • Lead Data Pipeline Specialist

    2 weeks ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Senior AWS Data EngineerAs a Senior AWS Data Engineer, you will design and deploy end-to-end data pipelines on the AWS cloud infrastructure. You will utilize services such as Amazon S3, AWS Glue, AWS Lambda, Amazon Redshift, etc.Develop and implement data processing and transformation workflows using Apache Spark and SQL to support analytics and reporting...

  • Chief Data Architect

    2 weeks ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time US$ 1,08,300 - US$ 1,64,441

    Senior Data Engineer PositionThe Mastermind Group is seeking a seasoned professional to lead the development and implementation of intelligent data infrastructure. This role requires designing robust ETL pipelines, building scalable business intelligence solutions, and driving data-powered outcomes that enhance client value across telecom operations and IT...

  • Data Pipeline Engineer

    1 week ago

    Allahabad, Uttar Pradesh, India beBeeBackend Full time ₹ 10,00,000 - ₹ 15,00,000

    Senior Backend Data & Integration Engineer PositionWe are seeking an experienced and skilled Senior Backend Data & Integration Engineer to join our team. As a key member of our engineering team, you will play a crucial role in designing and implementing robust data pipelines.The ideal candidate will have 4-8 years of experience in backend/data engineering...

  • Cloud Data Architect

    1 week ago

    Allahabad, Uttar Pradesh, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Cloud Data ArchitectDescription:We are seeking an experienced Data Engineer to join our team and drive the development of large-scale data acquisition pipelines, utilizing Google Cloud tools such as Python, Data Flow, and BigQuery.Create complex data pipelines for batch and real-time processing in the Google Cloud ecosystem.Build and manage large...