Big Data Engineer

24 hours ago

kottayam, India beBeeDataEngineer Full time

Senior Data EngineerWe are building the future of healthcare analytics.Our goal: pipelines that are reliable, observable, and continuously improving in production.This is a fully remote role, open to candidates based in Europe or India, with periodic team gatherings in Mountain View, California.Data Pipeline Design and Development: Design, build, and maintain scalable ETL pipelines using Python (Pandas, PySpark) and SQL, orchestrated with Airflow (MWAA).Data Lake/Lakehouse Management: Develop and maintain the SAIVA Data Lake/Lakehouse on AWS, ensuring quality, governance, scalability, and accessibility.Distributed Data Processing: Run and optimize distributed data processing jobs with Spark on AWS EMR and/or EKS.Batch and Streaming Ingestion: Implement batch and streaming ingestion frameworks (APIs, databases, files, event streams).Validation and Quality Checks: Enforce validation and quality checks to ensure reliable analytics and ML readiness.Pipeline Monitoring and Troubleshooting: Monitor and troubleshoot pipelines with CloudWatch, integrating observability tools like Grafana, Prometheus, or Datadog.Infrastructure Provisioning: Automate infrastructure provisioning with Terraform, following AWS best practices.Database Integration: Manage SQL Server, PostgreSQL, and Snowflake integrations into the Lakehouse.On-call Support: Participate in an on-call rotation to support pipeline health and resolve incidents quickly.Coding and Collaboration: Write production-grade code, and contribute to design/code reviews and engineering best practices.Key ResponsibilitiesDevelop, implement, and maintain scalable data pipelines.Manage data lake and lakehouse architecture on AWS.Design and optimize distributed data processing jobs.Implement batch and streaming ingestion frameworks.Enforce validation and quality checks for analytics and ML readiness.Monitor and troubleshoot pipelines with CloudWatch and observability tools.Automate infrastructure provisioning with Terraform.Integrate databases into the Lakehouse.Participate in on-call rotation for pipeline health and incident resolution.Write production-grade code and contribute to design/code reviews.Requirements5+ years in data engineering, ETL pipeline development, or data platform roles.Experience designing and operating data lakes or Lakehouse architectures on AWS.Strong SQL skills with PostgreSQL, SQL Server, and at least one AWS cloud warehouse (Snowflake or Redshift).Proficiency in Python (Pandas, PySpark); Scala or Java a plus.Hands-on with Spark on AWS EMR and/or EKS for distributed processing.Strong background in Airflow (MWAA) for workflow orchestration.Expertise with AWS services: S3, Glue, Lambda, Athena, Step Functions, ECS, CloudWatch.Proficiency with Terraform for IaC; familiarity with Docker, ECS, and CI/CD pipelines.Experience building monitoring, validation, and alerting into pipelines with CloudWatch, Grafana, Prometheus, or Datadog.Strong communication skills and ability to collaborate with data scientists, analysts, and product teams.A track record of delivering production-ready, scalable AWS pipelines.

  • Gcp senior data engineers

    3 weeks ago

    Kottayam, India Tata Consultancy Services Full time

    TCS presents an excellent Opportunity For "GCP Senior Data Engineers" Role Job Description 1 Role - GCP Senior Data Engineers 2 Location of Requirement - Chennai, Kochi, Bangalore, Hyderabad, Pune, Kolkata, Delhi & Bhubaneswar 3 Experience Range - 5 to 10 Years Desired Competencies Experience with Google Cloud Platform , Data Integration, orchestration...

  • Cloud Data Solutions Engineer

    3 days ago

    kottayam, India beBeeData Full time

    Job DescriptionWe are seeking a skilled professional to join our team as a Data Engineer. As a key member of our engineering team, you will be responsible for designing, building, and maintaining large-scale data systems.Your primary focus will be on supporting ongoing platform activities and project efforts as needed. This includes triaging identified...

  • High-Level Data Engineering Professional

    3 days ago

    kottayam, India beBeeDataEngineer Full time

    Senior Data Engineer Job DescriptionWe are seeking a seasoned Data Engineer to join our team. As a key member, you will be responsible for designing and implementing scalable data pipelines and cloud-based data solutions using Python, SQL, Big Data ecosystems, and Snowflake.Duties and Responsibilities:Design, Build, and Optimize Large-Scale Data Pipelines:...

  • Senior Data Engineer

    2 weeks ago

    Kottayam, India Donyati Full time

    Senior Data Engineer Job Title: Senior Data EngineerLocation: Remote/ Willing to TravelJob Type: Full-timeExperience Level: 8+ yearsAbout the Role:We are seeking a highly skilled Senior Data Engineer to join our team in building a modern data platform on AWS. You will play a key role in transitioning from legacy systems to a scalable, cloud-native...

  • Senior Data Engineer

    2 weeks ago

    Kottayam, India Donyati Full time

    Senior Data Engineer Job Title: Senior Data EngineerLocation: Remote/ Willing to TravelJob Type: Full-timeExperience Level: 8+ yearsAbout the Role:We are seeking a highly skilled Senior Data Engineer to join our team in building a modern data platform on AWS. You will play a key role in transitioning from legacy systems to a scalable, cloud-native...

  • Senior Big Data Specialist

    1 day ago

    kottayam, India beBeeDataEngineer Full time

    Key Roles and ResponsibilitiesWe are seeking a skilled Data Engineer to support our migration initiative from Teradata to BigQuery. This role requires a hands-on developer who can collaborate closely with data and reporting teams to ensure smooth repointing and validation of Power BI reports.Support the repointing and validation of reporting data sources...

  • Senior Revenue Operations Specialist: Integration Architect

    3 days ago

    kottayam, India beBeeintegration engineer Full time

    Integration ArchitectAs a key member of our Revenue Operations team, you will play a critical role in shaping the architecture of our integrations.Our ideal candidate is a highly skilled technical professional with a strong background in integration engineering and a passion for designing scalable, governed integration patterns.We are seeking an experienced...

  • High-Performance Data Engineer

    23 hours ago

    kottayam, India beBeeData Full time

    Job Description:We are seeking a skilled professional to lead the design, development, and optimization of high-performance data pipelines and systems. The ideal candidate will have strong experience with cloud platforms, modern ETL/ELT tools, and deep technical skills in Python, SQL, and distributed data frameworks.Key Responsibilities:Design and develop...

  • Ai Data Engineer

    1 week ago

    Kottayam, India Whatjobs IN C2 Full time

    Job Title: AI Data Engineer Exp- 4 to 8 years Location- PAN India Job Description Key Responsibilities: - Build and maintain data infrastructure: Design and construct scalable, reliable data pipelines, storage, and processing systems in the cloud. - Ensure data quality: Clean, transform, and enrich raw data to create "business truth" that AI models can use...

  • Executive Data Architect

    3 days ago

    kottayam, India beBeeDataEngineer Full time

    Senior Data Engineer - Key ResponsibilitiesThe Senior Data Engineer – Data Acquisition is accountable for designing, developing and maintaining robust data acquisition frameworks using Azure Data Platform, Databricks, Snowflake, and Kafka.Data Ingestion PipelinesKafka-based Real-time Data StreamingETL/ELT Pipeline DevelopmentSQL Query OptimizationAzure...