Data Engineer for Large-Scale Data Pipelines

1 day ago

Ahmedabad, Gujarat, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

Job Title: Data Engineer

About the Role

We are seeking an experienced data engineer to design, build and maintain large-scale data pipelines in Google Cloud Platform. The ideal candidate will have a strong background in data engineering with expertise in GCP.

The primary responsibility of this role is to develop, construct, test and maintain data acquisition pipelines for large volumes of structured and unstructured data. This includes batch and real-time processing in Google Cloud.

Key Responsibilities
  • Develop complex datasets based on business requirements
  • Construct 'big data' pipeline architecture using tools such as Python, Data Flow, DataStream, CDC (Change Data Capture), Cloud Functions, Cloud Run, Pub Sub, BigQuery, Cloud Storage
  • Leverage a variety of tools in the Google Cloud Ecosystem to integrate systems and data pipelines
  • Use logs & alerts to effectively monitor pipelines
  • Develop JSON messaging structure for integrating with various applications
  • Leverage DevOps and CI/CD practices (GitHub, Terraform) to ensure the reliability and scalability of data pipelines
Requirements
  • Requires 5-7 years of experience in a data engineering role – with core GCP experience of 3-4 years
  • Strong communication skills and ability to work with stakeholders and business clients
  • Able to translate business needs to technical requirements
Work Arrangements

Remote work arrangement available. Work hours will be in CST, until 2 am IST on most days. Occasional flexibility may be required to stretch work hours by 1-2 hours every few weeks.

  • Expert in Large Scale Data Engineering

    3 days ago

    Ahmedabad, Gujarat, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    **Job Overview**We are seeking an experienced Senior Data Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing, implementing, and maintaining large-scale data pipelines using Azure Data Factory (ADF) and Databricks.",

  • Large Scale Data Pipeline Developer

    3 days ago

    Ahmedabad, Gujarat, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 30,00,000

    Senior Data EngineerDevelop and maintain large-scale data pipelines, integrating structured and unstructured data from multiple sources.Design, implement, and optimize ETL processes using Big Data ecosystems like Hadoop, Spark, Hive, or similar technologies.Collaborate with data analysts, data scientists, and business stakeholders to deliver reliable...

  • Data Engineering Specialist

    6 days ago

    Ahmedabad, Gujarat, India beBeeDataEngineer Full time US$ 80,000 - US$ 1,00,000

    Job Title: Data Engineer">Description Summary: We have an immediate requirement for an experienced professional to develop and maintain large-scale data pipelines.">">Design, develop, and maintain large-scale data pipelines.">Develop scalable data processing solutions using PySpark.">Work on high-performance data architectures.">">About Us: Our organization...

  • Freelance Data Engineer

    5 days ago

    Ahmedabad, Gujarat, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 20,00,000

    About Freelance Data Engineering RolesWe are seeking an experienced freelance data engineer to join our network of professionals. The ideal candidate will design, build, and optimize large-scale data pipelines using PySpark, AWS, and Databricks.Key Responsibilities:Develop and maintain ETL/ELT pipelines for robust, scalable, and high-performance data...

  • Data Engineering Opportunity

    7 days ago

    Ahmedabad, Gujarat, India beBeeData Full time US$ 62,400 - US$ 76,800

    Data Engineering OpportunityWe are seeking a seasoned Data Engineer to drive innovation on a global engineering project. With at least 5-8 years of experience in Data Engineering, leveraging skills in:Python: Develop and maintain large-scale data pipelines.Pyspark: Design and implement scalable data processing solutions.Palantir Foundry: Architect and deploy...

  • Senior Data Engineer

    1 day ago

    Ahmedabad, Gujarat, India beBeeDataEngineer Full time ₹ 14,68,800 - ₹ 19,12,000

    Job OpportunityA senior data engineer with expertise in Airflow, Python, and Snowflake is required to design, develop, and maintain complex data pipelines.Able to leverage distributed systems, containerization, and cloud-based technologies.The ideal candidate will have experience working on scalable data processing solutions using Apache Airflow, implement...

  • Anblicks - Lead Data Engineer - Snowflake Migration

    2 days ago

    Ahmedabad, Gujarat, India ANBLICKS CLOUD DATA ENGG PRIVATE LIMITED Full time

    Job Title : Lead Data Engineer Snowflake MigrationLocation : Hyderabad / AhmedabadExperience : 7+ YearsBe a Trailblazer in Our Data Modernization Center of Excellence (COE)At Anblicks, we are on a mission to modernize legacy data ecosystems and help enterprises unlock the power of the cloud. Our Data Modernization COE is at the forefront of this...

  • Master Data Pipeline Architect

    3 days ago

    Ahmedabad, Gujarat, India beBeeDataEngineer Full time US$ 1,17,000 - US$ 1,63,000

    Key Data Engineer RoleThe ideal candidate will possess a deep understanding of data infrastructure, data modeling, and data governance. They should be well-versed in cloud computing platforms, ELT/ETL processes, and data warehousing concepts.We are seeking an experienced Data Engineer to join our team. The successful candidate will be responsible for...

  • Chief Data Pipeline Architect

    1 day ago

    Ahmedabad, Gujarat, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Job OverviewWe are seeking a skilled Data Engineer to design, build, and optimize scalable data pipelines. As a key member of our team, you will be responsible for managing and processing large datasets, building real-time streaming applications, and collaborating with data scientists to integrate AI/ML models into production.

  • Senior Data Engineer

    2 days ago

    Ahmedabad, Gujarat, India Krish Technolabs Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    The Senior Data Engineer is a key technical leader responsible for architecting and building large-scale, enterprise-level data platforms. You will mentor junior engineers, define best practices, and drive the strategic vision for our data infrastructure.What Youll Be DoingDesign and architect complex, scalable, and reliable data pipelines and data...