Senior Data Engineer

4 days ago

india DeepRec.ai Full time

Senior Data Engineer – ML Services and Data Pipelines

Location: Remote

Employment Type: Contract

Experience: 5+ years

3 month contract + multiple extensions possible


Our client is based in the USA but employs a remote workforce. You should be available for team meetings at 8am PT and have some cross over for team collaboration in PST/ET timezone. Happy to hire remote worldwide.


We are seeking a skilled and experienced Data Engineer to design and implement scalable and extensible Machine Learning services and data pipelines within our AWS/Kubernetes environment. In this role, you will be responsible for setting up infrastructure to support the ingestion, processing, and indexing of text-based data. You will collaborate closely with our ML and software engineering teams to build robust pipelines and enable efficient data processing and document indexing.


Key Responsibilities


  • ML Infrastructure Setup: Design, implement, and maintain scalable ML infrastructure on AWS and Kubernetes to support current and future project needs. Tools like Amazon SageMaker for model development, deployment, and monitoring, and AWS Lambda for serverless computing will be key.
  • Data Pipeline Development: Develop, deploy, and manage data pipelines that process large volumes of data for machine learning use cases, with a focus on efficient text data processing. Use AWS Glue for ETL jobs, Amazon Kinesis for real-time data streaming, and AWS Step Functions to coordinate workflows.
  • Vector Database Integration: Set up and maintain vector databases to support natural language processing (NLP) models, ensuring efficient and accurate text-based data retrieval and analysis. Amazon OpenSearch Service and Amazon DynamoDB can be leveraged for indexing and storing large volumes of vectorized data.
  • Document Indexing System: Design and implement a system for document ingestion and indexing, providing seamless access to data for downstream ML and analytics processes. Tools like Amazon S3 for scalable storage and AWS Lambda for automation and processing will play a critical role.
  • Development Pipeline Setup: Establish a development pipeline for document ingestion, collaborating with DevOps and data science teams to ensure continuous integration and deployment practices using AWS CodePipeline and AWS CodeBuild.


Required Skills & Qualifications



  • Experience: 5+ years of experience in data engineering with a focus on ML infrastructure and data pipelines.
  • Technical Expertise:
  • Strong background in AWS and Kubernetes for deploying and managing scalable ML and data solutions.
  • Proficiency in data pipeline tools and frameworks such as AWS Glue, Amazon Kinesis, AWS Step Functions, or similar.
  • Experience with text-based data processing, NLP techniques, and vector databases (e.g., Amazon OpenSearch Service, Amazon DynamoDB, or third-party vector databases like Pinecone, Weaviate, or FAISS).
  • Programming Skills: Advanced skills in Python or similar languages for data engineering and ML pipeline development.
  • Data Handling & Storage: Proficiency in data storage solutions, including Amazon S3, AWS Redshift, Amazon RDS, and AWS Data Lake.
  • Problem-Solving Abilities: Ability to troubleshoot and resolve issues within ML pipelines and data processing environments.


Preferred Qualifications


  • Hands-on experience with ML Ops frameworks and practices.
  • Familiarity with continuous integration/continuous deployment (CI/CD) for data engineering workflows using AWS CodePipeline, AWS CodeBuild, and AWS CloudFormation.

  • Senior Data Engineer

    3 weeks ago

    India Mindteck Full time

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team at Mindteck. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data systems and architectures.Key Responsibilities:Design and implement data pipelines and architectures to support business...

  • Senior Data Engineer

    3 weeks ago

    India iO Associates - UKEU Full time

    Job Title: Senior Data Engineer Location: Mohali Salary: Negotiable on the current package iO Associates is looking for a Senior Data Engineer in Databricks, Python, Scala, Azure Synapse, and Azure Data Factory to join our dynamic team. Responsibilities include developing and troubleshooting complex data pipelines using Databricks, PySpark, Synapse,...

  • Senior Data Engineer

    3 weeks ago

    india iO Associates - UKEU Full time

    Job Title: Senior Data EngineerLocation: MohaliSalary: Negotiable on the current packageiO Associates is looking for a Senior Data Engineer in Databricks, Python, Scala, Azure Synapse, and Azure Data Factory to join our dynamic team. Responsibilities include developing and troubleshooting complex data pipelines using Databricks, PySpark, Synapse, ADF, ADLS...

  • Senior Data Engineer

    3 weeks ago

    India Apptad Inc. Full time

    Job Title: Senior Python Data EngineerWe are seeking a highly skilled Senior Python Data Engineer to join our team at Apptad Inc. as we continue to drive innovation in data engineering and analytics.Key Responsibilities:Design and implement scalable data architectures to support business intelligence and advanced analytics.Develop and maintain data pipelines...

  • Senior Data Engineer

    1 day ago

    India iO Associates - UKEU Full time

    Job Title: Senior Data Engineer Location: Mohali Salary: up to 30 LPA iO Associates is looking for a Senior Data Engineer in Databricks, Python, Scala, Azure Synapse, and Azure Data Factory to join our dynamic team. Responsibilities include developing and troubleshooting complex data pipelines using Databricks, PySpark, Synapse, ADF, ADLS...

  • Senior Data Engineer

    1 month ago

    india E-Solutions Full time

    Hi All, Hope you are doing well ! Job Title: Senior Data Engineer Experience: 4 - 8 years Location: Chennai Job Type: Full-time (5 days office) Job Description: We are seeking a skilled Senior Data Engineer with over 4+ years of experience to join our dynamic team in Chennai. In this role, you will be responsible for designing, building, and maintaining...

  • Senior Data Engineer

    1 month ago

    india E-Solutions Full time

    Hi All,Hope you are doing well !Job Title: Senior Data EngineerExperience: 4 - 8 yearsLocation: ChennaiJob Type: Full-time (5 days office)Job Description:We are seeking a skilled Senior Data Engineer with over 4+ years of experience to join our dynamic team in Chennai. In this role, you will be responsible for designing, building, and maintaining data...

  • Senior Data Engineer

    3 weeks ago

    India Recmatrix Consulting Full time

    Job Title: Senior Data Engineer Location: Remote within India Employment Type: Contract Experience Level: 6+ years Position Overview: We are seeking a Senior Data Warehouse Engineer to join our Digital Business team. This role involves building and managing data warehouses to support various business functions such as Sales, Marketing, and Content...

  • Senior data engineer

    3 weeks ago

    India Recmatrix Consulting Full time

    Job Title: Senior Data Engineer Location: Remote within India Employment Type: Contract Experience Level: 6+ years Position Overview: We are seeking a Senior Data Warehouse Engineer to join our Digital Business team. This role involves building and managing data warehouses to support various business functions such as Sales, Marketing, and...

  • Senior Data Engineer

    1 month ago

    india SR analytics Full time

    Job Title : Senior Data Engineer Location: Pune Company: SR Analytics Number of positions: 2 Experience: 4-9 years Job Location : Pune (Balewadi)- In-Office position About Us : We at SR Analytics, a Data Analytics and Business Intelligence Services firm, are looking to add a Senior Data Engineer to our team. If you are passionate about Azure Cloud...

  • Senior data engineer

    3 days ago

    India Linuxpect Private Limited Full time

    We are seeking an experienced and skilled Senior Data Engineer with 5 to 10 years of hands-on experience in building and managing data pipelines and big data infrastructure. The ideal candidate will have a strong technical background in data engineering tools, big data ecosystems, and programming languages. This role will focus on designing, implementing,...

  • Senior Data Engineer

    1 month ago

    india ValueLabs Full time

    Dear Aspirants, We at ValueLabs have an Opening for Senior Azure Data Engineer role. Below is the JD for the same. Role: Sr Data Engineer Experience: 5+ Years Preferable Immediate Joiners The Azure Data Engineer is responsible for the design, development, and maintenance of data pipelines, data warehouses, and data lakes for the organization. This role...

  • AWS Data Engineer

    3 months ago

    India ITI Data Full time

    Job DescriptionWe are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources. There...

  • AWS Data Engineer

    2 months ago

    India ITI Data Full time

    Job Description We are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources....

  • AWS Data Engineer

    4 days ago

    india ITI Data Full time

    Job Description We are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources....

  • Senior Data Engineer

    3 weeks ago

    india Recmatrix Consulting Full time

    Job Title: Senior Data EngineerLocation: Remote within IndiaEmployment Type: ContractExperience Level: 6+ yearsPosition Overview:We are seeking a Senior Data Warehouse Engineer to join our Digital Business team. This role involves building and managing data warehouses to support various business functions such as Sales, Marketing, and Content teams, along...

  • Senior data infrastructure engineer

    3 days ago

    India SourceBae Full time

    Job Role Senior Data Infrastructure Engineer Experience 5+ years Location: Remote Duration:3 - 6 months Job Description Title: Senior Data Infrastructure Engineer About the role As a Senior Data Infrastructure engineer, you will be working with Data Engineering, Analyst and Operations teams to build and maintain a secure, scalable and...

  • Senior Data Engineer

    3 days ago

    india Varcus infinity Full time

    Job Title: Senior Data Engineer   Location: India (Permanent Work From Home)   Salary: ₹30 - ₹40 LPA (flexible for exceptional candidates based on experience)   Experience Level: Minimum 5 years, with preference for candidates with e-commerce experience   (Note: Candidate must be available to work in PST time zone and able to join within a...

  • Senior Data Engineer

    3 days ago

    India Varcus infinity Full time € 30,00,000 - € 40,00,000

    Job Title: Senior Data Engineer Location: India (Permanent Work From Home) Salary: ₹30 - ₹40 LPA (flexible for exceptional candidates based on experience) Experience Level: Minimum 5 years, with preference for candidates with e-commerce experience (Note: Candidate must be available to work in PST time zone and able to join within a month.) ...

  • Senior Data Engineer

    2 weeks ago

    India Recmatrix Consulting Full time

    Job Title: Senior Data EngineerLocation: Remote within IndiaEmployment Type: ContractExperience Level: 6+ yearsPosition Overview:We are seeking a Senior Data Warehouse Engineer to join our Digital Business team. This role involves building and managing data warehouses to support various business functions such as Sales, Marketing, and Content teams, along...