Chief Data Pipeline Architect

1 week ago

Mumbai, Maharashtra, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000
Job Title

A Data Engineer is required to design, develop, and maintain large-scale data pipelines using big data technologies.

  • Create and maintain optimal data pipeline architecture
  • Create and maintain events/streaming based architecture/design
  • Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
  • Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS 'big data' technologies.
  • Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
  • Work with data scientists to strive for greater functionality in our data systems.
Mandatory Qualifications:
  • Proficiency in either Scala or Python, expertise in Spark, and experience with performance tuning and optimization. DBT is also Mandatory.
  • Experience with big data tools: HDFS/S3, Spark/Flink,Hive,Hbase, Kafka/Kinesis, etc.
  • Experience with relational SQL and NoSQL databases, including Elasticsearch and Cassandra/Mongodb.
  • Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.
  • Experience with AWS/GCP cloud services
  • Experience with stream-processing systems: Spark-Streaming/Flink etc.
  • Experience with object-oriented/object function scripting languages: Java, Scala, etc.
  • Experience building and optimizing 'big data' data pipelines, architectures and data sets.
  • Strong analytic skills related to working with structured/unstructured datasets.
  • Build processes supporting data transformation, data structures, dimensional modeling, metadata, dependency, schema registration/evolution and workload management.
  • Working knowledge of message queuing, stream processing, and highly scalable 'big data' data stores.
  • Experience supporting and working with cross-functional teams in a dynamic environment
  • Have a few weekend side projects up on GitHub
  • Have contributed to an open-source project
  • Have worked at a product company
  • Have a working knowledge of a backend programming language
  • Programming Language (Mandatory) → Candidate should be really good in either Scala OR Python (not necessarily both).
  • Used mainly for writing data pipelines and transformations.
  • Big Data Framework (Mandatory) → Strong hands-on expertise in Apache Spark.
  • Must know how to write Spark jobs and also tune/optimize them for better performance (not just basic Spark usage).
  • Performance Tuning & Optimization (Mandatory) →
  • Able to handle large datasets efficiently.
  • Optimize Spark jobs (partitions, shuffles, caching, memory usage, etc.)
  • DBT (Mandatory) →
  • Must have hands-on experience with dbt (Data Build Tool) for data transformations and data modeling.
  • This is often used on top of warehouses like BigQuery, Snowflake, Redshift.

  • Data Pipeline Architect

    1 week ago

    Mumbai, Maharashtra, India beBeePipeline Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title: Data Pipeline ArchitectWe are seeking an experienced professional to design, develop and manage scalable data pipelines.Key Responsibilities:Design and Develop Pipelines: Utilize expertise in designing and developing data pipelines using Oracle Data Flow or Apache Spark (PySpark) to create efficient and reliable data solutions.Ensure Seamless...

  • Senior Recruiter

    2 weeks ago

    Mumbai, Maharashtra, India Nap Chief Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Senior Recruiter JD for Nap Chief — concise, engaging, and platform-friendly:Job Title: Senior RecruiterLocation: Andheri, Mumbai (On-site)Company: Nap Chief – Premium Kidswear BrandAre you passionate about finding great talent and building winning teams? About Nap Chief:Nap Chief is India's fastest-growing premium kidswear brand, known for its...

  • Chief Data Architect

    2 weeks ago

    Mumbai, Maharashtra, India beBeeDataArchitect Full time ₹ 15,00,000 - ₹ 20,10,000

    Data Architect and EngineerAs a Data Architect and Engineer, you will play a vital role in designing, developing, and maintaining robust data solutions for our organization. We are seeking an experienced professional to join our team and contribute to the development of innovative data pipelines and ETL processes.Key Responsibilities:Create scalable and...

  • Data Pipeline Architect

    7 days ago

    Mumbai, Maharashtra, India beBeeCloud Full time ₹ 80,00,000 - ₹ 1,50,00,000

    Job OverviewWe are seeking an experienced Cloud Data Information Architect to design and implement cloud data pipelines using core skillsets.The ideal candidate will have hands-on experience with AWS Databricks, Snowflake, Python, Fivetran, and other tools listed below.Key ResponsibilitiesData Domain Identification: Identify and define foundational business...

  • Data Pipeline Architect

    4 days ago

    Mumbai, Maharashtra, India beBeeDataEngineer Full time ₹ 8,00,000 - ₹ 12,00,000

    Job DescriptionWe are seeking a skilled Data Engineer to manage, expand and optimize our data pipeline architecture.The successful candidate will be responsible for creating and maintaining optimal data pipeline architecture, assembling complex data sets that meet business requirements, and building infrastructure for efficient data extraction,...

  • Lead Data Architect: Snaplogic Pipeline Expert

    6 days ago

    Mumbai, Maharashtra, India beBeePipeline Full time ₹ 1,04,000 - ₹ 1,30,878

    Lead Data Architect: Snaplogic Pipeline ExpertWe are seeking an experienced Lead Data Architect with expertise in designing, building and implementing complex data pipelines using Snaplogic. The ideal candidate will have a strong background in data manipulation, error handling, scheduling tasks, fault tolerance and error percolation.Key Responsibilities:•...

  • Data Pipeline Architect

    2 weeks ago

    Mumbai, Maharashtra, India beBeeETL Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job OverviewThe ETL Developer role encompasses the design, development, and automation of data pipelines in a cloud-based environment. Key responsibilities include crafting efficient and scalable data transformation processes, ensuring data quality and integrity, optimizing workflow performance, and implementing security measures.Key...

  • Chief Data Architect

    7 days ago

    Mumbai, Maharashtra, India beBeeData Full time ₹ 18,00,000 - ₹ 20,00,000

    Job Summary:We are seeking a skilled data architect to design and implement scalable, secure, and aligned data architectures.The ideal candidate will have experience in architecting enterprise-level data solutions and proficiency with tools such as Databricks, Azure Synapse, Delta Lake, and Unity Catalog.Key Responsibilities:Design and implement data...

  • Shopify Specialist

    4 weeks ago

    Mumbai, Maharashtra, India Nap Chief Full time

    Join the Nap Chief Crew If Shopify is your playground and you love blending creativity with e-commerce, this is your chance We're on the hunt for a Shopify Specialist who can make our online store as delightful as our kidswear. Location: Andheri East, Mumbai (On-site) Experience: 2–3 Years About Nap Chief Nap Chief is a premium kidswear...

  • Shopify Specialist

    4 weeks ago

    Mumbai, Maharashtra, India Nap Chief Full time

    Join the Nap Chief Crew If Shopify is your playground and you love blending creativity with e-commerce, this is your chance We're on the hunt for a Shopify Specialist who can make our online store as delightful as our kidswear. Location: Andheri East, Mumbai (On-site) Experience: 2–3 Years About Nap Chief Nap Chief is a premium kidswear...