Senior Data Engineer

4 weeks ago

Bengaluru, India Ai Palette Full time

About Us

Ai Palette, a SaaS company established in 2018 and headquartered in Singapore, serves Consumer Packaged Goods brands worldwide, such as Nestle, Mondelez, and L’Oreal, by facilitating the launch of highly successful products tailored to their target audience.


Traditionally, brands like Nestle would commission surveys to gauge consumer preferences, a process fraught with challenges including time constraints, data accuracy, and high costs. Ai Palette addresses these issues by harnessing the power of artificial intelligence to analyze vast amounts of internet data, identifying emerging trends across various segments. This approach provides brands with invaluable insights into upcoming trends, enabling them to capitalize on significant growth opportunities efficiently.


Moreover, Ai Palette offers a comprehensive platform where brands can not only access trend data but also develop and test product concepts. Currently, the platform tracks an extensive database of 61 billion data points sourced from over 150 outlets across 12 categories, covering approximately 87% of the world's GDP. Its advanced Natural Language Processing algorithm supports 18 languages, including various Asian languages, distinguishing it as a unique and globally accessible solution.


Ai Palette's success and credibility are further underscored by its partnerships with leading food companies, including Fortune 500 giants like Kellogg's, Nestle, and Olam. The company has garnered support from prominent investors, including SGInnovate, AGFunder, Exfinity Ventures, Pi Ventures, Tinmen, and Orzon Ventures, with the most recent funding injection received in March 2024.


About the job

We are looking for a passionate Data Engineer who will be working in the Data Engineering division and core development of our AI Platform that will ideate consumer products of the future. As the Data Engineer, you will be working hand in hand with the Data Science and Full Stack team on the toughest and most challenging problems in Data Engineering and Cloud Computing handling millions of data points that include social media. You will get the opportunity to work and scale a growing Data Platform.

You don’t want a job that just pays the bills: you want a job to get out of bed for. You take more enjoyment from solving problems than your friends think is normal. If you are looking to find your “ikigai” , then your search stops here right with us


RESPONSIBILITIES

  1. Build the data collection pipelines that can acquire and handle millions of public data points from various sources using APIs and web extraction techniques at scale
  2. Build the data cleaning, quality and integrity pipeline in the platform leveraging the Apache Spark Python and AWS services
  3. Architect and develop distributed systems that can handle large-scale data processing
  4. Programming Language: Python/Java, Apache Spark, Apache Flink (Good to have)
  5. NoSQL Database: Elasticsearch, Dynamo DB/Mongo DB
  6. Work closely with the Data Science Team for the preprocessing steps required for the AI models in Production Environment
  7. Scale and automate Data Platform Collection layer to handle the consistently incoming data
  8. Implement data quality checks and validation processes
  9. Troubleshoot and resolve performance issues
  10. Collaborate with cross-functional teams to support data-related initiatives
  11. Document data pipelines, data models, and other technical processes


REQUIREMENTS


  1. 3-6 years of experience in building data engineering pipeline
  2. Have hand ons experience on Apache Spark, PySpark, SQL, Python/Java programming
  3. Worked with NoSQL database like Elasticsearch, Dynamo DB/Mongo DB, Cassandra - Anyone
  4. Strong experience in AWS Cloud Platform - S3, EC2, Lambda, Elasticsearch etc
  5. Quick learner, excellent communication and team player


IDEAL


  • Deep technical understanding of AWS PaaS and IaaS services
  • Previous experience with Airflow, Spark, PySpark, Python, ElasticSearch, Web Crawling and Docker
  • Prior experience with social media data (twitter, reddit, blogs, etc.)


BENEFITS

  • Own what you do with access to our Employee Share Options Pool (ESOP)
  • Great progression opportunities - we want you to grow with us
  • Look after yourself with health insurance including Hospital/Surgical, Major Medical and Outpatient
  • Learn new skills with sponsored training
  • Flexible work hours: Early birds and midnight-oil-burners rejoice
  • Tax Benefits

  • Sr data engineer

    3 months ago

    Bengaluru, India NTT DATA Full time

    Job Description Req ID: 286326 NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Sr data engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN).Senior data...

  • Data Specialist

    3 weeks ago

    Bengaluru, India NTT DATA Services Full time

    Req ID: 287764 NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Specialist / Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN).Data EngineerDesign...

  • Data Engineer

    2 days ago

    Bengaluru, India NTT DATA Full time

    Job DescriptionData Engineer Grade 07ResponsibilitiesAssure that data is cleansed, mapped, transformed, and otherwise optimised for storage and use according to business and technical requirements using Azure data factory and Azure databricksDevelop and maintain innovative Azure Azure data factory and Azure databricksSolution design using Microsoft Azure...

  • Senior Data Engineer

    3 weeks ago

    Bengaluru, India INFONET Full time

    Company DescriptionInfoNet provides customized software solutions for businesses of all sizes, from startups to Forbes 500 enterprises. Since 1996, we've been dedicated to helping companies streamline operations, enhance customer experiences, and drive growth with cutting-edge technology. Our team of developers, engineers, and consultants delivers tailored...

  • Data Engineer

    3 months ago

    Bengaluru, India NTT DATA Full time

    Job Description Data Engineer Grade 07ResponsibilitiesAssure that data is cleansed, mapped, transformed, and otherwise optimised for storage and use according to business and technical requirements using Azure data factory and Azure databricksDevelop and maintain innovative Azure Azure data factory and Azure databricksSolution design using Microsoft Azure...

  • Data Engineer

    2 weeks ago

    Bengaluru, India NTT DATA Services Full time

    About NTT DATA   NTT DATA is a $30+ billion trusted global innovator of business and technology services. We serve 75% of the Fortune Global 100 and are committed to helping clients innovate, optimize and transform for long-term success. As a Global Top Employer, we have diverse experts in more than 50 countries and a robust partner ecosystem of...

  • Data Engineer

    2 weeks ago

    Bengaluru, India NTT DATA Services Full time

    About NTT DATA NTT DATA is a $30+ billion trusted global innovator of business and technology services. We serve 75% of the Fortune Global 100 and are committed to helping clients innovate, optimize and transform for long-term success. As a Global Top Employer, we have diverse experts in more than 50 countries and a robust partner ecosystem of established...

  • Senior Data Engineer

    1 week ago

    Bengaluru, India RapidBrains Full time

    Job Title:Senior Data EngineerExperience:5-9 YearsLocation:BangaloreNotice Period:0 - 30 DaysJob Description:As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines, working with cutting-edge tools like Python, DBT, and Databricks. Your primary focus will be transforming raw data into meaningful...

  • Senior Data Engineer

    4 weeks ago

    Bengaluru, India NI Full time

    Senior Data Engineer – Data Engineering About the role   We are seeking a Senior Data Engineer to join our team and play an integral part in building and maintaining our data infrastructure. The ideal candidate will have experience in designing and implementing complex data models and pipeline solutions and have a good understanding of database...

  • Senior Data Engineer

    1 week ago

    Bengaluru, India RapidBrains Full time

    Job Title: Senior Data EngineerExperience: 5-9 YearsLocation: BangaloreNotice Period: 0 - 30 DaysJob Description:As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines, working with cutting-edge tools like Python, DBT, and Databricks. Your primary focus will be transforming raw data into...

  • Data Engineer

    3 months ago

    Bengaluru, India NTT DATA Services Full time

    Data Engineer Responsibilities Assure that data is cleansed, mapped, transformed, and otherwise optimised for storage and use according to business and technical requirements using Azure data factory and Azure databricks Develop and maintain innovative Azure Azure data factory and Azure databricks Solution design using Microsoft Azure services...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, India Mitchell Martin Inc. Full time

    Title: Data EngineerPosition Type: Full-Time Location: Bangalore, India Work Experience: Senior Data Engineer: 7 to 9 years of experience in architecting, designing and building Data Engineering solutions and Data Platforms.Experience in building Data Warehouses/Data Platforms on Databricks.Experience working with data acquisition and transformation tools...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, India Mitchell Martin Inc. Full time

    Title: Data EngineerPosition Type: Full-Time Location: Bangalore, India Work Experience: Senior Data Engineer: 7 to 9 years of experience in architecting, designing and building Data Engineering solutions and Data Platforms.Experience in building Data Warehouses/Data Platforms on Databricks.Experience working with data acquisition and transformation tools...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, India Mitchell Martin Inc. Full time

    Title: Data Engineer Position Type: Full-Time Location: Bangalore, India Work Experience: Senior Data Engineer: 7 to 9 years of experience in architecting, designing and building Data Engineering solutions and Data Platforms. Experience in building Data Warehouses/Data Platforms on Databricks. Experience working with data acquisition and...

  • Senior Engineer

    3 months ago

    Bengaluru, India NTT DATA Full time

    Job Description Req ID: 284013 NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Senior Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN).Once You Are...

  • Senior Lead Engineer

    3 months ago

    Bengaluru, India NTT DATA Full time

    Job Description Req ID: 286689 NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Senior Lead Engineer to join our team in Remote, Karnātaka (IN-KA), India (IN).Salesforce...

  • Senior Data Engineer

    1 month ago

    Bengaluru, India Procore Technologies Full time

    Job DescriptionWe’re looking for a Senior Data Engineer to join Procore’s Data Division. In this role, you’ll help build Procore’s next-generation construction data platform for others to build upon including Procore developers, analysts, partners, and customers.  As a Senior Data Engineer, you’ll partner with other engineers and product managers...

  • Senior Data Engineer

    1 month ago

    Bengaluru, India Procore Full time

    Job DescriptionWe’re looking for a Senior Data Engineer to join Procore’s Data Division. In this role, you’ll help build Procore’s next-generation construction data platform for others to build upon including Procore developers, analysts, partners, and customers.  As a Senior Data Engineer, you’ll partner with other engineers and product managers...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, India ScaleOrange Technologies Pvt Ltd Full time

    Company Description : ScaleOrange Technologies Pvt Ltd. Role Description : This is a full-time on-site role for a Senior Data Engineer with 5+ years of experience at ScaleOrange Technologies Pvt Ltd in Hyderabad. The Senior Data Engineer will be responsible for data engineering, data modeling, ETL processes, data warehousing, and data analytics on a...

  • Senior Data Engineer

    6 days ago

    Bengaluru, India Quantiphi Full time

    Experience:3 to 6 YearsLocation:Mumbai/Bangalore/Trivandrum (Hybrid)Job Role -Senior Data Engineer/ Associate Architect - DataAs a Sr. Data Engineer at Quantiphi, you will solve big data challenges and will develop big data solutions for migration, storage and processing. For building and assisting the production environment at large-scale the role of a data...