Lead Data Pipeline Specialist

12 hours ago

Bengaluru, Karnataka, India beBeeDataEngineer Full time US$ 1,04,000 - US$ 1,30,878
Job Title: Senior Data Engineer

About the Role:

This is an exceptional opportunity to join our organization as a Senior Data Engineer. As a key member of our data team, you will be responsible for designing, building, and optimizing scalable data pipelines using cloud-native technologies.

Key Responsibilities:

  • Design and architect high-throughput ETL pipelines using AWS Glue, Lambda, and EMR to handle large datasets and complex data workflows.
  • Implement, monitor, and maintain a cloud-native data infrastructure using AWS services like S3, Redshift, and EMR.
  • Develop highly performant data transformation processes using Apache Spark on EMR for distributed data processing and parallel computation.
  • Design and implement real-time data ingestion and streaming systems using AWS Kinesis or Apache Kafka.
  • Use Apache Airflow to schedule and orchestrate complex ETL workflows.
  • Optimize data models, schemas, and queries in Amazon Redshift to ensure low-latency querying and scalable analytics.
  • Leverage Docker to containerize data engineering applications and use AWS Fargate for running containerized applications in a serverless environment.
  • Build automated monitoring and alerting systems to proactively detect and troubleshoot pipeline issues.

Requirements:

  • 5+ years of hands-on experience in building, maintaining, and optimizing data pipelines in a cloud-native environment.
  • Solid understanding of ETL/ELT processes and experience with tools like AWS Glue.
  • Deep experience working with AWS cloud services including S3, Glue, Lambda, Redshift, EMR, and Big Data Technologies.
  • Expertise in using Apache Spark for distributed data processing at scale.
  • Strong experience with Apache Airflow or similar workflow orchestration tools.
  • Proficiency in Python programming language and knowledge of standard processes in coding for high performance, maintainability, and reliability.
  • Advanced knowledge of SQL and experience in query optimization, partitioning, and indexing.
  • Experience with version control systems like Git and implementing CI/CD pipelines using tools like Terraform or AWS CloudFormation.

Preferred Qualifications:

  • Experience in designing and building real-time data streaming solutions using Kafka or Kinesis.
  • Familiarity with data governance practices, data cataloging, and data lineage tools.
  • Knowledge of supporting machine learning pipelines and building data systems that can scale to meet the requirements of AI/ML workloads.
  • AWS certification.

  • Data Pipeline Specialist

    5 days ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time

    Job Title: Azure Databricks EngineerAs a skilled Data Engineer, you will play a pivotal role in our dynamic team.High Impact Required: Immediate joiners only. Please note that candidates with less than 12 months of experience will not be considered.Key Responsibilities:Our ideal candidate will have strong expertise in building and optimizing data pipelines...

  • Lead Data Pipeline Developer

    10 hours ago

    Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Data EngineerWe are seeking a skilled Data Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing, developing and maintaining ETL/ELT data pipelines that ingest data from various sources into our data warehouse.Pipeline Development: Design and develop efficient data pipelines using AWS...

  • Position: Data Pipeline Lead

    3 days ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 9,00,000 - ₹ 12,00,000

    Senior Data Engineer Position">We are looking for a seasoned Senior Data Engineer to join our team and lead the design, development, and maintenance of scalable data pipelines using Google Cloud Platform (GCP), BigQuery, DataProc (PySpark), and Informatica.]"The ideal candidate will have hands-on expertise in GCP, including BigQuery, DataProc, Cloud Storage,...

  • Data Pipeline Specialist

    2 days ago

    Bengaluru, Karnataka, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    About the JobAre you ready for a new career challenge?We are currently seeking an ETL Developer to join our team.The successful candidate will have at least 5 years of experience in designing and implementing data pipelines using Databricks (PySpark/Scala/SQL).The role involves assessing current-state Informatica ETL jobs, mappings, workflows, and...

  • High Performance Data Pipelines Specialist

    4 days ago

    Bengaluru, Karnataka, India beBeeData Full time US$ 1,80,000 - US$ 2,40,000

    Databricks Lead - High Performance Data PipelinesNTT DATA is seeking a skilled Databricks Lead to join our team. The ideal candidate will have extensive experience in data engineering, with a strong focus on high-performance data pipelines.Job DescriptionWe are looking for a technically hands-on leader who can design and implement scalable data architectures...

  • Principal Data Pipeline Architect

    12 hours ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time US$ 1,25,000 - US$ 1,75,000

    About the RoleWe are seeking a Senior Data Engineer with expertise in building scalable data pipelines and integrating healthcare data. Reporting to the Head of AI and Engineering, you will design, build, and operate the backbone of our AI-powered applications.You will work closely with software engineers, ML specialists, and clinical partners to bring our...

  • Data Pipeline Specialist

    1 day ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    We're seeking a skilled Data Engineer to support our team.Key Responsibilities:Design, build, and maintain robust, scalable data pipelines for large volume data from various sources.Integrate data from multiple sources, ensuring consistency and reliability.Develop and maintain efficient data models and schemas.Manage and optimize databases for performance,...

  • Data Pipeline Specialist

    3 days ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 9,00,000 - ₹ 12,00,000

    Job TitleSenior Data EngineerJob DescriptionAs a Senior Data Engineer, you will design, implement, and optimize data pipelines for large-scale data processing. You will work with data scientists, analysts, and engineers to ensure data availability and quality.Key responsibilities include developing and maintaining ETL/ELT workflows using Spark, Hadoop, Hive,...

  • Senior Data Pipeline Specialist

    5 days ago

    Bengaluru, Karnataka, India beBeeAutomatedTesting Full time

    Job Opportunity:The role of Senior Consultant involves designing, developing and implementing automated test cases for complex data pipelines.Ability to design, develop, and implement automated test cases for complex data pipelines is a mustAWS experience is essentialPharma expertise is also requiredProficiency in programming languages such as Python, R, or...

  • Databricks Lead Urgent

    4 days ago

    Bengaluru, Karnataka, India NTT Data Full time

    Job DescriptionNTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Databricks Lead to join our team in Bangalore, Karntaka (IN-KA), India (IN).Job Duties: Wyndham Group of Hotels is...