Senior Data Engineer

3 days ago


Malappuram, India DeepRec.ai Full time

Senior Data Engineer – ML Services and Data Pipelines

Location: Remote

Employment Type: Contract

Experience: 5+ years

3 month contract + multiple extensions possible


Our client is based in the USA but employs a remote workforce. You should be available for team meetings at 8am PT and have some cross over for team collaboration in PST/ET timezone. Happy to hire remote worldwide.


We are seeking a skilled and experienced Data Engineer to design and implement scalable and extensible Machine Learning services and data pipelines within our AWS/Kubernetes environment. In this role, you will be responsible for setting up infrastructure to support the ingestion, processing, and indexing of text-based data. You will collaborate closely with our ML and software engineering teams to build robust pipelines and enable efficient data processing and document indexing.


Key Responsibilities


  • ML Infrastructure Setup: Design, implement, and maintain scalable ML infrastructure on AWS and Kubernetes to support current and future project needs. Tools like Amazon SageMaker for model development, deployment, and monitoring, and AWS Lambda for serverless computing will be key.
  • Data Pipeline Development: Develop, deploy, and manage data pipelines that process large volumes of data for machine learning use cases, with a focus on efficient text data processing. Use AWS Glue for ETL jobs, Amazon Kinesis for real-time data streaming, and AWS Step Functions to coordinate workflows.
  • Vector Database Integration: Set up and maintain vector databases to support natural language processing (NLP) models, ensuring efficient and accurate text-based data retrieval and analysis. Amazon OpenSearch Service and Amazon DynamoDB can be leveraged for indexing and storing large volumes of vectorized data.
  • Document Indexing System: Design and implement a system for document ingestion and indexing, providing seamless access to data for downstream ML and analytics processes. Tools like Amazon S3 for scalable storage and AWS Lambda for automation and processing will play a critical role.
  • Development Pipeline Setup: Establish a development pipeline for document ingestion, collaborating with DevOps and data science teams to ensure continuous integration and deployment practices using AWS CodePipeline and AWS CodeBuild.


Required Skills & Qualifications



  • Experience: 5+ years of experience in data engineering with a focus on ML infrastructure and data pipelines.
  • Technical Expertise:
  • Strong background in AWS and Kubernetes for deploying and managing scalable ML and data solutions.
  • Proficiency in data pipeline tools and frameworks such as AWS Glue, Amazon Kinesis, AWS Step Functions, or similar.
  • Experience with text-based data processing, NLP techniques, and vector databases (e.g., Amazon OpenSearch Service, Amazon DynamoDB, or third-party vector databases like Pinecone, Weaviate, or FAISS).
  • Programming Skills: Advanced skills in Python or similar languages for data engineering and ML pipeline development.
  • Data Handling & Storage: Proficiency in data storage solutions, including Amazon S3, AWS Redshift, Amazon RDS, and AWS Data Lake.
  • Problem-Solving Abilities: Ability to troubleshoot and resolve issues within ML pipelines and data processing environments.


Preferred Qualifications


  • Hands-on experience with ML Ops frameworks and practices.
  • Familiarity with continuous integration/continuous deployment (CI/CD) for data engineering workflows using AWS CodePipeline, AWS CodeBuild, and AWS CloudFormation.


  • Full Stack Engineer

    1 month ago


    Malappuram, India Equity Data Science (EDS) Full time

    Senior Software Developer (Full stack)Job Type: Full TimeJob Location: Malad MumbaiExperience: 5 Years+About the Role:We are looking for a Senior Developer to join our team and deliver a streamlined user experience. Proven experience as a Senior Developer having an edge on front end development. We want a team member who strives to develop a first-class...

  • Data Engineer II

    3 months ago


    Malappuram, India tsworks Full time

    Who We Aretsworks Technologies India Private Limited (subsidiary of The Software Works, Inc, USA) is a technology product and services company. Our mission is to provide domain expertise, innovative solutions and thought leadership to empower businesses to thrive in a digital world. We value our employees, take pride in providing best value in customer...


  • Malappuram, Kerala, India Aptonet Inc Full time

    Job Title: Master Data Management EngineerAt Aptonet Inc, we are seeking a highly skilled Master Data Management Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing and building Master Data Management functionality that enables better data quality and trust across our organization.Key...


  • Malappuram, Kerala, India Applicantz Full time

    Job Title: Senior Performance EngineerJob Summary:We are seeking a highly skilled Senior Performance Engineer to join our team at Applicantz. As a key member of our software development team, you will be responsible for ensuring the high performance and scalability of our web-based APIs.Key Responsibilities:Develop and maintain performance testing strategies...


  • Malappuram, Kerala, India Sysol System Solutions Pvt. Ltd. Full time

    Job Title: Senior Dotnet Software EngineerJob Overview: We are seeking a talented and dedicated Senior Dotnet Software Engineer to join our team at Sysol System Solutions Pvt. Ltd.Job Responsibilities:Design and develop software applications using the .NET framework.Collaborate with cross-functional teams to define, design, and ship new features.Participate...


  • Malappuram, Kerala, India JobStars HR Solutions Pvt Ltd Full time

    We are seeking a Senior Software Engineer with experience in backend development to join our team. The ideal candidate will have a strong understanding of software design patterns and be able to develop high-quality, scalable code.Developing and maintaining large-scale software systemsDesigning and implementing software architectureCollaborating with...


  • Malappuram, India Real Time Data Services Full time

    Key Roles and Functional Responsibilities: • Configuring and monitoring firewalls, switches, and routers. • Deep knowledge of Spanning Tree, VLans, VTP • Good knowledge on various tunnelling technologies e.g., VPN, GRE • Upgrade, install and troubleshoot networks, networking hardware devices and software. • Good experience in setting up and...

  • Senior Data Analyst

    3 months ago


    Malappuram, India MIRÓR Full time

    Company Overview: Miror is a unique Femtech platform dedicated to providing 360-degree care to women in their Perimenopause and Menopause journey. Over the past year, we have achieved significant traction and growth. Our WhatsApp community on this subject is India’s largest, our social media presence has grown exponentially, and we have established...

  • Azure Data Engineer

    13 hours ago


    Malappuram, India DigiHelic Solutions Pvt. Ltd. Full time

    Expereince- 5+ years of experience in data engineeringStrong understanding of data warehousing and ETL conceptsExperience with Azure cloud technologies, including Azure Data Factory, Azure Data Lake Storage, and Azure Synapse AnalyticsExperience with programming languages such as Python , Pyspark , SQLExcellent problem-solving and analytical skillsAbility to...


  • malappuram, India Coinbase Full time

    At Coinbase, our mission is to increase economic freedom around the world, and we couldn’t do this without hiring the best people. We’re a group of hard-working overachievers who are deeply focused on building the future of finance and Web3 for our users across the globe, whether they’re trading, storing, staking or using crypto. Know those people who...

  • Lead Data Scientist

    1 month ago


    malappuram, India YASH Technologies Full time

    What you should have? Bachelor’s degree in physics, Mathematics, Engineering, Metallurgy or Computer Science. MSc in relevant field - Physics, Mathematics, Engineering, Computer Science, Chemistry or Metallurgy. 8+ years of experience in Data science and Analytics delivery Deep knowledge of machine learning, statistics, optimization and related fields...


  • Malappuram, Kerala, India JobStars HR Solutions Pvt Ltd Full time

    Job Title: Senior Software Engineer - Frontend DevelopmentWe are seeking an experienced Frontend Developer to join our team at "Company". The successful candidate will be responsible for designing and developing innovative Frontend solutions using the latest technologies.Responsibilities:* Design and develop Frontend components and features* Collaborate with...

  • Senior Data Scientist

    13 hours ago


    Malappuram, India EverestDX Inc Full time

    About the Company:Everest DX – We are Digital Platform Services company, headquartered in Stamford. Our Platform/Solution includes Orchestration, Intelligent operations with BOTs’, AI-powered analytics for Enterprise IT. Our vision is to enable Digital Transformation for enterprises to deliver seamless customer experience, business efficiency and...

  • Lead Data Scientist

    2 months ago


    Malappuram, India Turing Full time

    Data Science Lead/ManagerAbout Turing:We are a pioneering organization at the forefront of the AI and Machine Learning industry, specializing in training Large Language Models (LLMs) through sophisticated Software Engineering, Data Science, and Machine Learning techniques. Our work encompasses a range of technical tasks aimed at enhancing the capabilities of...


  • Malappuram, Kerala, India NeenOpal Inc. Full time

    Job Title: Data Analytics ManagerNeenOpal Inc. is seeking a highly skilled Data Analytics Manager to join our team. As a Data Analytics Manager, you will be responsible for leading cross-functional teams to pilot and implement client solutions with a focus on client satisfaction across a solution area and industry verticals.Key Responsibilities:Leverage and...


  • Malappuram, Kerala, India Aspect Software Full time

    Job Title: Senior Quality Assurance EngineerAt Aspect Software, we are seeking a highly skilled Senior Quality Assurance Engineer to join our team.Key Responsibilities:Collaborate closely with Developers in the Scrum Team to design and execute test approaches that support software application development.Participate in sprint standups, retrospective, and...

  • Azure Engineer

    7 days ago


    Malappuram, India Tata Consultancy Services Full time

    Dear Candidate,Greetings from TATA CONSULTANCY SERVICES LIMITED!!!TCS Inviting applications for walk-in interview in Hyderabad Location.AWS Data EngineerAWS Terraform EngineerJava Spring Boot MicroservicesAzure Data bricks (Py spark), Azure data FactoryAzure+ .NETAzure DevOps/ Terraform KubernetesGen AIExperience Range- 4 to 8 yearsWalk-in Location: Kohinoor...


  • Malappuram, Kerala, India Borneo Full time

    Job Title: Engineering Program ManagerBorneo is seeking an experienced Engineering Program Manager to collaborate with cross-functional teams, make strategic decisions, and deliver high-quality products that contribute to our mission of transforming data security and privacy.Key Responsibilities:Oversee the entire software development lifecycle, from...


  • Malappuram, India Start.io Full time

    Start.io is a mobile marketing and audience platform. Start.io empowers the mobile app ecosystem and simplifies mobile marketing, audience building, and mobile monetization. Start.io's direct integration with over 500,000 monthly active mobile apps provides access to unprecedented levels of global first-party data, which can be leveraged to understand and...

  • Executive Engineer

    2 months ago


    Malappuram, India Talent Corner HR Services Pvt Ltd Full time

    Job Requirements1.Qualification: B.E Mechanical, MBA Operations (Preferred)2.Years of Experience: 2-3 years3.Technical Skills: Problem-solving skills, Attention to detail, Pressure management, Quality assurance, Technology Proficiency, Logical Thinking, Risk Assessment4.Behavioral Skills: (1) Team Work (2) Management skills (3) Operations Management (4)...