Data Pipeline Architect

3 days ago

bikaner, India beBeeDataEngineering Full time

Job Title: Senior Data EngineerSenior Data EngineerWe are building the future of healthcare analytics. Join us to design, build and scale robust data pipelines that power nationwide analytics and support our machine learning systems.This role is fully remote with open worldwide availability and periodic team gatherings in Mountain View, California.Design, build and maintain scalable ETL pipelines using Python (Pandas, PySpark) and SQL, orchestrated with Airflow (MWAA).Develop and maintain the SAIVA Data Lake/Lakehouse on AWS ensuring quality, governance scalability and accessibility.Run and optimize distributed data processing jobs with Spark on AWS EMR and/or EKS.Implement batch and streaming ingestion frameworks APIs databases files event streams.Enforce validation and quality checks to ensure reliable analytics and ML readiness.Monitor and troubleshoot pipelines with CloudWatch integrating observability tools like Grafana Prometheus or Datadog.Automate infrastructure provisioning with Terraform following AWS best practices.Manage SQL Server PostgreSQL and Snowflake integrations into the Lakehouse.Participate in an on-call rotation to support pipeline health and resolve incidents quickly.Requirements:Strong SQL skills with PostgreSQL SQL Server and at least one AWS cloud warehouse Snowflake or Redshift.Proficiency in Python Pandas PySpark Scala or Java a plus.Hands-on with Spark on AWS EMR and/or EKS for distributed processing.Strong background in Airflow (MWAA) for workflow orchestration.Expertise with AWS services S3 Glue Lambda Athena Step Functions ECS CloudWatch.Proficiency with Terraform for IaC familiarity with Docker ECS and CI/CD pipelines.Experience building monitoring validation and alerting into pipelines with CloudWatch Grafana Prometheus or Datadog.Strong communication skills and ability to collaborate with data scientists analysts and product teams.A track record of delivering production-ready scalable AWS pipelines not just prototypes.Key Skills:Python Pandas PySparkSQL PostgreSQL SQL ServerSpark on AWS EMR/EKSAirflow MWAAAWS services S3 Glue LambdaTerraform for IaCDocker ECS and CI/CD pipelines

  • Experienced Data Pipeline Developer

    3 days ago

    bikaner, India beBeeData Full time

    We are seeking an experienced Data Engineer to join our team. The ideal candidate will have a strong background in designing and developing data pipelines, as well as experience with AWS services such as Redshift, Glue, S3, and Athena.Key Responsibilities:Design, develop, and optimize data pipelines and ETL/ELT workflows on AWS.Migrate legacy data solutions...

  • Data Architect Specialist

    15 hours ago

    bikaner, India beBeeData Full time

    Data Architect Specialist plays a vital role in designing, developing and maintaining scalable data integration systems to support business operations.Develops, implements and maintains large-scale data pipelines using various technologies.Collaborates with cross-functional teams to gather requirements and ensure successful data system deployment.Sets up...

  • Data Pipeline Specialist

    3 days ago

    bikaner, India beBeeMACHINELEARNING Full time

    Machine Learning Engineer - Data Pipeline SpecialistWe are seeking a skilled Machine Learning Engineer to join our team as a Data Pipeline Specialist. This is a unique opportunity to build and deploy machine learning systems that drive business outcomes.The ideal candidate will have experience shipping DS/ML systems to production, with expertise in Python,...

  • Chief Data Architect

    3 days ago

    bikaner, India beBeeDataEngineer Full time

    Job DescriptionAs a key member of our team, you will be responsible for designing and developing robust data pipelines that power our large language model development.This involves building scalable systems to transform raw datasets into pristine formats. You'll collaborate closely with AI researchers and ML engineers to understand data requirements, define...

  • Data Pipeline Engineer

    3 days ago

    bikaner, India beBeeTechnical Full time

    Technical Data SpecialistThe ideal candidate will serve as the primary point of contact for technical data-related inquiries and concerns, ensuring high system stability through proactive monitoring and root-cause analysis.Support escalation point for issues with data ingestion, consumption, pipelines.Monitoring: Proactively monitor data pipeline health to...

  • Chief Data Engineering Architect

    15 hours ago

    bikaner, India beBeeDataEngineer Full time

    Senior Data Engineer RoleWe are seeking a highly skilled data professional to join our organization in building a modern data platform on AWS. You will play a key role in transitioning from legacy systems to a scalable, cloud-native architecture using technologies like Apache Iceberg, AWS Glue, Redshift, and Atlan for governance.This role requires hands-on...

  • Principal Data Architect

    3 days ago

    bikaner, India beBeeDataEngineer Full time

    Senior Oracle Data EngineerWe are seeking an experienced professional to fill the role of Senior Oracle Data Engineer.Job Description:This position involves architecting and implementing robust data pipelines across on-prem and cloud environments, leveraging Oracle Data Integrator (ODI) and in-depth knowledge of ETL and ELT frameworks. Key responsibilities...

  • ETL Specialist

    3 days ago

    bikaner, India beBeeInformatica Full time

    Job SummaryWe are seeking a seasoned QA professional to join our team as an ETL Specialist. The ideal candidate will possess hands-on experience in data pipeline testing and API testing, with expertise in Informatica tools.Key Responsibilities:End-to-end testing of data pipelines and validation of source-to-target mappingsComplex data transformations and SCD...

  • Quality Assurance Specialist for Data Pipelines

    16 hours ago

    bikaner, India beBeeETL Full time

    Job Opportunity: ETL Quality Assurance ExpertWe are seeking an experienced QA Consultant to join our team, focusing on testing ETL pipelines and MDM systems.The ideal candidate will possess hands-on experience in ETL/Data Pipeline Testing, API Testing, and strong SQL skills. A good understanding of data flows is also necessary.Main responsibilities include...

  • Cloud Engineer

    19 hours ago

    bikaner, India beBeeData Full time

    Job OverviewWe are seeking a skilled Freelance Data Engineer to develop and implement real-time data pipelines using Apache NiFi for structured and unstructured data.The ideal candidate has expertise in the Google Cloud Platform (GCP) ecosystem, including BigQuery, Pub/Sub, Dataflow, Dataproc, Cloud Storage, and Composer/Airflow.Create efficient data...