Data Engineer

3 days ago

Erode, Tamil Nadu, India AgileEngine Full time
Senior Data Engineer (AI/ML |
Python | GCP ) - WFH

THIS IS FULLY REMOTE WORKING OPPORTUNITY.

WE NEED IMMEDIATE JOINERS or someone who can join in less than 1 month.

If you are interested and fulfill the below mentioned criteria then please share below information.

1. EMAIL ID
2. YEARS OF RELEVANT EXPERIENCE.
3. UPDATED RESUME.
4. CCTC/ECTC
5. Notice period

*Key Responsibilities*

1. Design & Develop Scalable Data Pipelines
  • Build high-throughput, low-latency pipelines to process massive datasets efficiently.
  • Ensure pipelines can handle 5–10 TB of structured and unstructured data at scale.
2. Data Filtering & Curation
  • Optimize data ingestion and transformation for downstream models.
  • Implement intelligent scanning mechanisms to filter out relevant data for AI/ML models
3. AI/ML Integration
  • Leverage GenAI models (e.g., Gemini Pro, OpenAI, GPT) under the Vertex AI umbrella.
  • Build and fine-tune custom models when LLMs are cost-prohibitive or less effective.
  • Design multi-layered model pipelines balancing accuracy, cost, and performance.
4. Cloud Infrastructure
  • Architect scalable solutions primarily on GCP (Vertex AI, BigQuery, GKE, Cloud run, PostgreSQL, Dataflow, Pub/Sub).
  • Azure knowledge is a plus but not mandatory.
  • Experienced in building robust monitoring system.
5. Performance Optimization
  • Ensure real-time, low-latency processing for high-volume workloads.
  • Optimize GPU/CPU-intensive environments to manage cost and efficiency.

6. Collaboration & Ownership
  • Work closely with data scientists, ML engineers, and solution architects.
  • Take end-to-end ownership of assigned tasks and deliverables, working independently where needed.
  • Collaborate with global teams and manage 3–4 hours of timezone overlap when required.
*Required Skills & Qualifications*

1. Programming

* Expert-level Python (mandatory).

2. Data Engineering:

* Hands-on experience with streaming pipelines, and large-scale data ingestion frameworks.

3. AI/ML Expertise:

* Experience integrating and fine-tuning LLMs and GenAI models.
* Comfortable working with custom model development where LLMs cannot be applied.
* Familiarity with OpenAI, Gemini, Vertex AI, or equivalent platforms.

4. Cloud:

* GCP (mandatory) — experience with Vertex AI, BigQuery, GKE, Cloud Run, PostgreSQL, Dataflow, Pub/Sub, etc.
* Azure experience is a plus.

5. Scalability & Low Latency:

* Proven ability to design and implement high-performance, distributed data solutions.

6. High-Volume Data Processing:

* Experience processing massive datasets efficiently within strict timeframes.

*Good to Have*
  • Experience with MLOps and deploying models at scale.
  • Knowledge of data governance and compliance when working with sensitive data.
  • Hands-on exposure to event-driven architectures and microservices.

  • Data Engineer

    1 week ago

    Erode, Tamil Nadu, India Firstsource Full time

    Key Skills and Responsibilities: We are seeking a Senior Data Engineer with a strong background in cloud-native data engineering, primarily on Microsoft Azure, and familiarity with AWS and GCP. The ideal candidate will have deep expertise in building scalable data pipelines, implementing enterprise-grade data governance, security, and AI-powered engineering...

  • Data Engineering Expert

    20 hours ago

    Erode, Tamil Nadu, India beBeeData Full time US$ 90,000 - US$ 1,20,000

    As a skilled data engineer, you will be responsible for designing and developing robust data pipelines using industry-leading tools.Key Responsibilities:Design and develop robust data pipelines using industry-leading tools.Develop skills in data engineering using Python, EMR/Spark, and Kafka/Kinesis streaming.Collaborate with cross-functional teams to...

  • Data Engineer Specialist

    1 week ago

    Erode, Tamil Nadu, India beBeeDataEngineering Full time ₹ 40,00,000 - ₹ 50,00,000

    Job Title: Data Engineering Specialist">We are looking for a highly skilled data engineering specialist to join our team. In this role, you will be responsible for designing, constructing, and optimizing large-scale data processing systems that enable advanced analytics and business intelligence.You will collaborate closely with data scientists, analysts,...

  • Big Data Engineer

    1 week ago

    Erode, Tamil Nadu, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Big Data EngineerWe are seeking a highly skilled Big Data Engineer to join our team. As a Big Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data processing systems.Key Responsibilities:Design and implement data pipelines using PySpark, Hive, and other big data tools.Analyze and optimize system performance to...

  • Data Engineering Specialist

    6 days ago

    Erode, Tamil Nadu, India beBeeDataEngineering Full time US$ 80,000 - US$ 1,50,000

    Job TitleWe are seeking an experienced Data Engineering specialist to design and develop robust data pipelines handling high-volume financial data. This role is critical in ensuring data integrity, scalability, and real-time availability across our platforms.Main ResponsibilitiesData Pipeline Development: Design, develop, and manage end-to-end data pipelines...

  • Data Engineering Position

    4 days ago

    Erode, Tamil Nadu, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Data Engineer RoleAbout this role. We're looking for a skilled Data Engineer to join our team and help us scale and optimize our data processing capabilities.This is a critical role that requires strong technical skills and experience with big data technologies.

  • Data Engineering Specialist

    2 days ago

    Erode, Tamil Nadu, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Senior Data Testing EngineerAre you a data enthusiast looking to take your testing skills to the next level? We are seeking an experienced Senior Data Testing Engineer to join our team.This role involves designing, developing, and executing automated and manual test cases for data validation and transformation processes. You will work closely with data...

  • Data Engineering Expert

    20 hours ago

    Erode, Tamil Nadu, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Senior Data EngineerJoin a dynamic team as a Senior Data Engineer and play a crucial role in designing, developing, and operating data warehouses and data platform functions.Job Description:Viasat has been shaping how consumers, businesses, governments, and militaries around the globe communicate for over three decades. As a member of our data...

  • Azure Data Engineer

    1 week ago

    Erode, Tamil Nadu, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 26,00,000

    Job TitleAzure Data Engineer (Data Architect)We are seeking a seasoned Data Engineer to join our team. This is an exciting opportunity for a professional with expertise in Azure DataBricks and Pyspark to drive data-driven decision making.

  • Data Engineer

    1 week ago

    Erode, Tamil Nadu, India Innodata Inc. Full time

    We are looking for a Data Engineer with strong experience in CI/CD practices, Databricks (Spark), Python, Github and SQL. The ideal candidate should have hands-on expertise in building and automating data pipelines, managing multi-environment deployments, and working with modern DevOps/configuration management tools.Key ResponsibilitiesDesign, implement, and...