Senior Data Engineer

6 days ago

Bangalore, India Ai Palette Full time

About Us

Ai Palette, a SaaS company established in 2018 and headquartered in Singapore, serves Consumer Packaged Goods brands worldwide, such as Nestle, Mondelez, and L’Oreal, by facilitating the launch of highly successful products tailored to their target audience.

Traditionally, brands like Nestle would commission surveys to gauge consumer preferences, a process fraught with challenges including time constraints, data accuracy, and high costs. Ai Palette addresses these issues by harnessing the power of artificial intelligence to analyze vast amounts of internet data, identifying emerging trends across various segments. This approach provides brands with invaluable insights into upcoming trends, enabling them to capitalize on significant growth opportunities efficiently.

Moreover, Ai Palette offers a comprehensive platform where brands can not only access trend data but also develop and test product concepts. Currently, the platform tracks an extensive database of 61 billion data points sourced from over 150 outlets across 12 categories, covering approximately 87% of the world's GDP. Its advanced Natural Language Processing algorithm supports 18 languages, including various Asian languages, distinguishing it as a unique and globally accessible solution.

Ai Palette's success and credibility are further underscored by its partnerships with leading food companies, including Fortune 500 giants like Kellogg's, Nestle, and Olam. The company has garnered support from prominent investors, including SGInnovate, AGFunder, Exfinity Ventures, Pi Ventures, Tinmen, and Orzon Ventures, with the most recent funding injection received in March 2024.

About the job

We are looking for a passionate Data Engineer who will be working in the Data Engineering division and core development of our AI Platform that will ideate consumer products of the future. As the Data Engineer, you will be working hand in hand with the Data Science and Full Stack team on the toughest and most challenging problems in Data Engineering and Cloud Computing handling millions of data points that include social media. You will get the opportunity to work and scale a growing Data Platform.

You don’t want a job that just pays the bills: you want a job to get out of bed for. You take more enjoyment from solving problems than your friends think is normal. If you are looking to find your “ikigai” , then your search stops here right with us

RESPONSIBILITIES

  1. Build the data collection pipelines that can acquire and handle millions of public data points from various sources using APIs and web extraction techniques at scale
  2. Build the data cleaning, quality and integrity pipeline in the platform leveraging the Apache Spark Python and AWS services
  3. Architect and develop distributed systems that can handle large-scale data processing
  4. Programming Language: Python/Java, Apache Spark, Apache Flink (Good to have)
  5. NoSQL Database: Elasticsearch, Dynamo DB/Mongo DB
  6. Work closely with the Data Science Team for the preprocessing steps required for the AI models in Production Environment
  7. Scale and automate Data Platform Collection layer to handle the consistently incoming data
  8. Implement data quality checks and validation processes
  9. Troubleshoot and resolve performance issues
  10. Collaborate with cross-functional teams to support data-related initiatives
  11. Document data pipelines, data models, and other technical processes

REQUIREMENTS

  1. 3-6 years of experience in building data engineering pipeline
  2. Have hand ons experience on Apache Spark, PySpark, SQL, Python/Java programming
  3. Worked with NoSQL database like Elasticsearch, Dynamo DB/Mongo DB, Cassandra - Anyone
  4. Strong experience in AWS Cloud Platform - S3, EC2, Lambda, Elasticsearch etc
  5. Quick learner, excellent communication and team player

IDEAL

  • Deep technical understanding of AWS PaaS and IaaS services
  • Previous experience with Airflow, Spark, PySpark, Python, ElasticSearch, Web Crawling and Docker
  • Prior experience with social media data (twitter, reddit, blogs, etc.)

BENEFITS

  • Own what you do with access to our Employee Share Options Pool (ESOP)
  • Great progression opportunities - we want you to grow with us
  • Look after yourself with health insurance including Hospital/Surgical, Major Medical and Outpatient
  • Learn new skills with sponsored training
  • Flexible work hours: Early birds and midnight-oil-burners rejoice
  • Tax Benefits

  • Senior Mechanical Engineer

    4 months ago

    bangalore, India Abha Engineer Full time

    We are looking for a Senior Mechanical EngineerRoles are described below.1. Manpower Planning.2. Preparing of Project Cost.3. Schedule wise work execution.4. As Drawing & quality work execution.5. Client & Third Party Manage.6. Working Team Manage & Review.7. Reporting to Management.8. ROB & FOB Fabrication & Erection Work Knowledge.

  • Sr data engineer

    1 month ago

    Bangalore, India NTT DATA Services Full time

    Req ID: 286326  NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Sr data engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN). Senior data...

  • Data Engineer

    2 days ago

    bangalore, India NTT DATA Full time

    Job Description Data Engineer Grade 07 ResponsibilitiesAssure that data is cleansed, mapped, transformed, and otherwise optimised for storage and use according to business and technical requirements using Azure data factory and Azure databricksDevelop and maintain innovative Azure Azure data factory and Azure databricksSolution design using Microsoft...

  • Senior Engineer

    2 days ago

    bangalore, India NTT DATA Full time

    Job Description Req ID:  284013  NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Senior Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN). Once...

  • Data Engineer

    4 weeks ago

    Bangalore, India NTT DATA Services Full time

    Req ID: 287747  NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN). Mandatory Skills...

  • Senior Data Engineer

    1 week ago

    bangalore, India Visa Full time

    Job DescriptionVisa Consulting & Analytics (VCA) team is a key part of the Global Solutions organization, a high-performing team of data engineer, scientists, data analysts and statisticians helping major organizations adapt and evolve to meet the changes taking place in technology, finance, and commerce, with cutting-edge, creative and advanced analytic...

  • Senior Data Engineer

    4 months ago

    bangalore, India tsworks Full time

    Who We Aretsworks Technologies India Private Limited (subsidiary of The Software Works, Inc, USA) is a technology product and services company. Our mission is to provide domain expertise, innovative solutions and thought leadership to empower businesses to thrive in a digital world. We value our employees, take pride in providing best value in customer...

  • Senior Data Engineer

    4 months ago

    bangalore, India Tredence Inc. Full time

    Data Engineering - Associate Manager (6-10 Years) / Manager (10-14 Years)This position requires someone with good problem solving, business understanding and client presence.Overall professional experience of the candidate should be above 10 years. A minimum of 6 years of experience in leading and managing a client portfolio in Data Engineering space....

  • Senior Engineering Manager, Data Engineering

    4 months ago

    bangalore, India Postman Full time

    Who Are We? Postman is the world’s leading API platform, used by more than 30 million developers and 500,000 organizations, including 98% of the Fortune 500. Postman is helping developers and professionals across the globe build the API-first world by simplifying each step of the API lifecycle and streamlining collaboration—enabling users to...

  • Senior Data Engineer

    6 days ago

    Bangalore, India Quantiphi Full time

    Quantiphi is an award-winning AI-first digital engineering company driven by the desire to solve transformational problems at the heart of the business. Quantiphi solves the toughest and most complex business problems by combining deep industry experience, disciplined cloud and data-engineering practices, and cutting-edge artificial intelligence research to...

  • Senior Data Engineer

    1 week ago

    bangalore, India Mastek Full time

    Min Work Experience 7-10 YearsRole : Senior Data EngineerExtensive experience in Big Data space (Hadoop Stack like M/R, HDFS, Pig, Hive, HBase, Flume, Sqoop, NoSQL stores like Cassandra, HBase etc.) across Fractal and contributes to open-source Big Data technologies.Write and tune complex Java, MapReduce, and Hive jobs.Experience leading a...

  • Senior Data Engineer

    6 days ago

    Bangalore, India StockX Full time

    Why you'll love this role As a Senior Data Engineer, you will be empowered to leverage data to drive amazing customer experiences and business results. You will own the end to end development of data engineering solutions to support analytical needs of the business. The ideal candidate will be passionate about working with disparate datasets and be...

  • Senior Data Engineer

    3 months ago

    bangalore, India StockX Full time

    Why you'll love this role As a Senior Data Engineer, you will be empowered to leverage data to drive amazing customer experiences and business results. You will own the end to end development of data engineering solutions to support analytical needs of the business. The ideal candidate will be passionate about working with disparate datasets and be someone...

  • Senior Engineering Manager

    4 months ago

    bangalore, India Hevo Data Full time

    About HevoAt Hevo, we are changing the way companies leverage data to drive user experience, growth, and business processes. There has been a fundamental change in the amount of data companies are generating on a day-to-day basis. More and more users in an organization are now looking to use data to drive business decisions. Data is no longer a second-class...

  • Senior Engineer

    3 months ago

    Bangalore, India NTT DATA Services Full time

    Req ID: 283990  NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Senior Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN). Once You Are...

  • Senior Engineering Manager

    6 days ago

    Bangalore, India Hevo Data Full time

    About Hevo At Hevo, we are changing the way companies leverage data to drive user experience, growth, and business processes. There has been a fundamental change in the amount of data companies are generating on a day-to-day basis. More and more users in an organization are now looking to use data to drive business decisions. Data is no longer a...

  • Senior Data Engineer II

    1 week ago

    bangalore, India MakeMyTrip Full time

    Position: Senior Data Engineer I / Senior Data Engineer II Experience: Must have 4+ years of experience About Role: We are looking for experienced Data engineers with excellent problem-solving skills to develop machine-learning powered Data Products design to enhance customer experiences. About us: Nurtured from the seed of a single great idea - to empower...

  • Senior Consultant

    1 week ago

    bangalore, India Capco Full time

    About Us“Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Companies for Women in India 2022 by Avtar & Seramount . With our presence across 32 cities across globe, we support 100+ clients across banking, financial and Energy...

  • Senior Data Engineer

    6 days ago

    Bangalore, India KarmaLifeAI Full time

    WHAT WE DO? KarmaLife , we are revolutionizing the gig economy by providing Earned Wage Access (EWA) and other small-ticket, short-term credit solutions to empower millions of workers in India. Our tech-driven approach evaluates current behavior instead of financial history, ensuring financial stability and access to funds during emergencies....

  • Senior Data Engineer

    34 minutes ago

    Bangalore City, India Visa Full time

    Job DescriptionVisa Consulting & Analytics (VCA) team is a key part of the Global Solutions organization, a high-performing team of data engineer, scientists, data analysts and statisticians helping major organizations adapt and evolve to meet the changes taking place in technology, finance, and commerce, with cutting-edge, creative and advanced analytic...