Building Scalable Data Pipelines

2 days ago

Mysore, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000
Job Description

We are looking for a skilled Data Engineer to join our team. The ideal candidate will have expertise in designing and implementing data pipeline architectures, creating and maintaining events/streaming based architecture/design, and identifying, designing, and implementing internal process improvements.

The Data Engineer will be responsible for building the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and 'big data' technologies. They will also create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.

In addition, the Data Engineer will work with data scientists to strive for greater functionality in our data systems.

Required Skills and Qualifications
  1. Proficiency in either Scala or Python: The candidate should be proficient in one of these programming languages and have hands-on experience in writing data pipelines and transformations.
  2. Expertise in Apache Spark: Strong hands-on expertise in Apache Spark is mandatory, including knowledge of how to write Spark jobs and tune/optimize them for better performance.
  3. Able to handle large datasets efficiently: The candidate should be able to handle large datasets efficiently and optimize Spark jobs (partitions, shuffles, caching, memory usage, etc.)
  4. Hands-on experience with dbt (Data Build Tool): Must have hands-on experience with dbt for data transformations and data modeling.
  5. Experience with big data tools: Experience with big data tools like HDFS/S3, Spark/Flink,Hive,Hbase, Kafka/Kinesis, etc., relational SQL and NoSQL databases, and data pipeline and workflow management tools is highly desirable.
  6. Experience with AWS/GCP cloud services: Familiarity with AWS/GCP cloud services is highly desirable.
  7. Experience supporting and working with cross-functional teams: The candidate should have experience supporting and working with cross-functional teams in a dynamic environment.
Benefits
  • Familiarity with object-oriented/object function scripting languages is highly desirable.
  • Familiarity with stream-processing systems like Spark-Streaming/Flink, etc.
Others

The ideal candidate will have 2+ years of experience in a Data Engineer role and have contributed to an open-source project. A strong analytic skills related to working with structured/unstructured datasets is highly desirable. Also, the candidate should have a few weekend side projects up on GitHub and a working knowledge of a backend programming language.

Key Responsibilities:

  • Design and implement data pipeline architectures.
  • Create and maintain events/streaming based architecture/design.
  • Identify, design, and implement internal process improvements.
  • Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and 'big data' technologies.
  • Create data tools for analytics and data scientist team members.
  • Work with data scientists to strive for greater functionality in our data systems.

  • Data Pipeline Architect

    1 week ago

    Mysore, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Senior Data EngineerAbout the RoleWe are seeking a seasoned professional to build data pipelines and integrate healthcare data.This role is critical in creating scalable pipelines that process clinical encounter data, retrieve associated medical documents, and deliver validated information to machine learning systems for inference.You will work...

  • Scalable Data Solutions Architect

    5 days ago

    Mysore, Karnataka, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Cloud Data EngineerAs a skilled Cloud Data Engineer, you will be responsible for designing and implementing scalable data engineering solutions on Google Cloud Platform.The ideal candidate will have hands-on experience in GCP Data Engineering (BigQuery, Dataflow, DataProc, PySpark, Python, Cloud Composer) and proficiency with Tableau or MicroStrategy for...

  • Scalable Data Architect

    2 weeks ago

    Mysore, Karnataka, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Cloud Data EngineerDesign and build robust data solutions on Microsoft Fabric and Azure platforms, leveraging expertise in PySpark, T-SQL, Stored Procedures, and Azure Service Bus.Key Responsibilities:Develop end-to-end data pipelines using Microsoft Fabric, collaborating with stakeholders to convert business needs into technical designs.Implement scalable...

  • Cloud Data Pipeline Specialist

    19 hours ago

    Mysore, Karnataka, India beBeeInformatician Full time ₹ 20,00,000 - ₹ 30,00,000

    Job Description:We are seeking a highly experienced Informatica Engineer to design and develop cloud-based ETL pipelines using Azure services and integrate data into AWS Datalake or other modern data platforms.This role involves working on cutting-edge cloud technologies to design, develop, and maintain scalable ETL pipelines using Azure Data Factory and...

  • Data Pipeline Specialist

    6 days ago

    Mysore, Karnataka, India beBeeetldeveloper Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Are you a data pipeline specialist looking for a new challenge?We have an exciting opportunity for an ETL Developer to join our team of talented professionals.In this role, you will be responsible for designing, developing, and maintaining efficient data pipelines using IBM DataStage and AWS Glue/Lambda. You will work closely with architects, business...

  • AWS Data Pipeline Engineer

    2 days ago

    Mysore, Karnataka, India beBeeData Full time ₹ 18,00,000 - ₹ 20,00,000

    Dear Candidate, we are seeking a highly skilled professional to join our data engineering team as an AWS Data Architect.The ideal candidate will have deep expertise in building scalable data pipelines using Apache Spark, PySpark, SQL, and Python, along with hands-on experience in the AWS ecosystem.Our requirements include:Designing, Developing, and...

  • Chief Data Pipeline Architect

    5 days ago

    Mysore, Karnataka, India beBeeDataFlowSpecialist Full time ₹ 80,00,000 - ₹ 1,50,00,000

    Job Overview:As a seasoned professional in data management and processing, we seek an accomplished Data Flow Consultant to spearhead the design and implementation of efficient data pipelines.Key Responsibilities:Design and develop scalable data pipelines using Oracle Data Flow or Apache Spark (PySpark) with expertise in Python and SQL.Collaborate with...

  • Data Pipeline Engineer

    3 days ago

    Mysore, Karnataka, India beBeeETLPipeline Full time ₹ 10,00,000 - ₹ 16,00,000

    Job SummaryWe are seeking a talented and motivated professional to join our data engineering team. As an ETL Developer, you will work on building scalable and efficient data pipelines using IBM DataStage (on Cloud Pak for Data), AWS Glue, and Snowflake.Key ResponsibilitiesDesign, develop, and maintain ETL pipelines using IBM DataStage (CP4D) and AWS...

  • Scalable Data Solutions Architect

    3 days ago

    Mysore, Karnataka, India beBeeDataSolutions Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    About this roleWe are seeking a skilled Data Solutions Architect to design, build, and optimize scalable data systems using Google Cloud Platform (GCP) and AtScale.Key Responsibilities:Design and develop large-scale data analytics solutions using AtScale and GCP services.Develop and deploy data platforms and pipelines on GCP that integrate with...

  • Data Pipeline Specialist

    2 days ago

    Mysore, Karnataka, India beBeeDataEngineer Full time ₹ 75,00,000 - ₹ 1,20,00,000

    Data Engineering is a vital function within our organization, driving business growth and informing strategic decision-making.We're on a mission to revolutionize the food and beverage industry with our powerful cloud-based platform.As a Data Engineer, you'll design and optimize distributed data pipelines, infrastructure, and tools that power insights across...