Senior Data Pipeline Engineer

3 days ago

Bengaluru, Karnataka, India Steady Rabbit Full time US$ 1,50,000 - US$ 2,00,000 per year

Experience: 6+ years

Work Location: Bangalore

Shift - 1 to 10pm IST

Job Summary:

We are looking for a highly skilledAWS Data Engineer with strong experience in data architecture, DevOps practices, and code quality tools like SonarQube. You will be responsible for designing scalable data storage and access patterns using AWS services such asDynamo DB, Redis, Glue, Lambda, and Kafka. This role focuses on ensuring efficient, secure, and maintainable data solutions to support both short-term and long-term consumption across our platforms.

Key Responsibilities:

  • Design and implement robust data architectures that support scalability, high availability, and performance within theAWS ecosystem.
  • Develop optimized data access patterns for DynamoDB, Redis, and PostgreSQL based on usage and business needs.
  • Architect short-term and long-term data storage strategies aligned with consumption patterns.
  • Integrate and utilize AWS services, including API Gateway, Lambda, Glue, Kafka, and CloudWatch, to enable real-time and batch data processing.
  • Collaborate with engineering and pipeline teams to ensure best practices in data modeling,partitioning, and storage optimization.
  • Monitor and correct improper storage practices, particularly in DynamoDB and streaming systems.
  • Build reusable Lambda functions and manage their deployment using DevOps pipelines.
  • Set up and maintain CI/CD workflows using GitHub Actions and integrate with quality gates viaSonarQube.
  • Apply DevOps principles to infrastructure provisioning, automated testing, monitoring, and continuous delivery.
  • Ensure code quality and maintainability by incorporating SonarQube analysis into the development lifecycle.
  • Use Kafka for building real-time data ingestion pipelines and managing event-driven architecture.
  • Maintain and integrate PostgreSQL databases with AWS-native and hybrid environments.
  • Analyze stored data and expose insights securely via API Gateway endpoints.

Required Skills and Experience:

  • 3+ years of hands-on experience as a Data Engineer or Data Architect in an AWS environment.
  • Deep knowledge of AWS services: DynamoDB, Redis, Lambda, Glue, API Gateway, and Kafka.
  • Strong experience with data modeling, access pattern design, and schema optimization.
  • Experience with CI/CD pipelines, DevOps tools, and infrastructure automation.
  • Hands-on experience with GitHub, GitHub Actions, and integrating withSonarQube for code quality assurance.
  • Solid understanding of PostgreSQL and its integration into scalable architectures.
  • Strong problem-solving skills and the ability to work independently or in a cross-functional team.
  • Clear understanding of security, data governance, and monitoring best practices in cloud environments.

  • Data Pipeline Engineer

    5 days ago

    Bengaluru, Karnataka, India beBeeData Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    The Role of Senior Data Integration SpecialistJob Overview:We are seeking an experienced Senior Data Integration Specialist to join our team. This is a key position that will play a vital role in the smooth operation of our enterprise data integration platform.As a Senior Data Integration Specialist, you will be responsible for designing, implementing, and...

  • Position: Data Pipeline Lead

    1 week ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 9,00,000 - ₹ 12,00,000

    Senior Data Engineer Position">We are looking for a seasoned Senior Data Engineer to join our team and lead the design, development, and maintenance of scalable data pipelines using Google Cloud Platform (GCP), BigQuery, DataProc (PySpark), and Informatica.]"The ideal candidate will have hands-on expertise in GCP, including BigQuery, DataProc, Cloud Storage,...

  • Senior Pipeline Engineer

    3 days ago

    Bengaluru, Karnataka, India AVIN ENGINEERING CONSULTANCY Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Company DescriptionAVIN Engineering, based in Bangalore, India, specializes in consultancy services for both offshore and onshore pipeline engineering, subsea services, and project management within the oil and gas industry. Our team of dedicated engineers excels in conceptual design, FEED, detailed engineering design, construction, and commissioning,...

  • Senior Pipeline Engineer

    4 days ago

    Bengaluru, Karnataka, India AVIN ENGINEERING CONSULTANCY Full time

    Job DescriptionCompany DescriptionAVIN Engineering, based in Bangalore, India, specializes in consultancy services for both offshore and onshore pipeline engineering, subsea services, and project management within the oil and gas industry. Our team of dedicated engineers excels in conceptual design, FEED, detailed engineering design, construction, and...

  • Senior Data Pipeline Architect

    5 days ago

    Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 1,04,000 - ₹ 1,30,878

    Lead Data EngineeringThe Lead Data Engineer is a key member of our engineering team, responsible for leading and mentoring data engineers to ensure alignment with technical and business objectives.As a senior data engineer, you will architect, design, and implement scalable and high-performance data pipelines using AWS technologies. You will collaborate with...

  • Senior Pipeline Engineer

    6 days ago

    Bengaluru, Karnataka, India DNEG Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Brahma is a pioneering enterprise AI company developing Astras, AI-native products built to help enterprises and creators innovate at scale. Brahma enables teams to break creative bottlenecks, accelerate storytelling, and deliver standout content with speed and efficiency. Part of the DNEG Group, Brahma brings together Hollywood's leading creative...

  • Data Senior Engineer

    3 days ago

    Bengaluru, Karnataka, India NTT DATA Full time US$ 80,000 - US$ 1,20,000 per year

    Req ID: 337669NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Senior Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN). Job Duties: 8+ years of experience...

  • Senior Data Scientist

    3 days ago

    Bengaluru, Karnataka, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Senior Data Scientist - Scalable Data PipelinesJob Title: Senior Data ScientistRole Overview:We are seeking a highly skilled and experienced Senior Data Scientist to lead the design, development, and maintenance of scalable data pipelines that process and transform large volumes of structured and unstructured data. The ideal candidate will have a strong...

  • Data Engineer for Large-Scale Data Pipelines

    60 minutes ago

    Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 18,00,000 - ₹ 26,00,000

    Job Title: Senior Data EngineerDescription:We are seeking a highly skilled data engineer to design, develop, and maintain large-scale data pipelines that extract data from Oracle Symphony via APIs, process and store it in the Databricks Lakehouse platform, and then integrate it into Oracle EPM for financial planning, reporting, and analytics.Key...

  • Expert Data Pipeline Architect

    2 hours ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 19,00,000

    Senior Data Engineer PositionA senior data engineer with expertise in Airflow, Python, and Snowflake to design and develop scalable data pipelines.Key Responsibilities:Develop and maintain complex data workflows using AirflowCollaborate with cross-functional teams to identify business requirements and implement data solutionsDesign and optimize data...