Senior Big Data Engineer

5 days ago

Remote, India UMENIT SOLUTIONS LLP Full time ₹ 2,40,000 - ₹ 6,00,000 per year

Job Title: Senior Big Data EngineerLocation: Remote / Hybrid (Preferred timezone overlap with [your region])

Type: Full-time / ContractAbout the ProjectWe're building an intelligent data pipeline that brings SAP AO (Analysis Office) data into a central database, which will serve as the foundation for an AI-powered chatbot. This chatbot will help users interact with complex enterprise data through natural language queries.We already have a data science team in-house. What we need now is a Big Data Engineer who can own the data extraction, transformation, and storage process from SAP AO to a scalable, queryable data layer.Key Responsibilities

  • Design, build, and manage data pipelines to extract, transform, and load (ETL/ELT) data from SAP AO (and related SAP systems such as BW or HANA) into a database or data warehouse.
  • Ensure data quality, consistency, and completeness during the migration process.
  • Set up a data lake or data warehouse (e.g., BigQuery, Snowflake, Redshift, or Azure Synapse) optimized for large-scale analytical queries.
  • Work closely with data scientists to prepare data for downstream AI and analytics use cases, including chatbot and natural language query systems.
  • Implement data modeling, partitioning, and indexing strategies for performance optimization.
  • Automate and monitor ETL pipelines using tools such as Apache Airflow, AWS Glue, Databricks, or similar orchestration frameworks.
  • Collaborate with AI/ML engineers to enable embedding and semantic search pipelines (RAG architecture).
  • Ensure compliance with data governance, access control, and security best practices.

Required Skills & Experience

  • 5+ years of experience as a Big Data Engineer / Data Engineer.
  • Strong experience in ETL/ELT pipeline design and implementation.
  • Hands-on experience with SAP data integration — particularly SAP AO, SAP BW, or SAP HANA data extraction.
  • Proficiency in Python / PySpark / SQL.
  • Experience with big data frameworks such as Spark, Hadoop, or Databricks.
  • Deep understanding of data warehousing concepts (fact/dimension modeling, partitioning, schema design).
  • Experience with cloud data ecosystems (AWS, GCP, or Azure).
  • Familiarity with data APIs, connectors, or SDKs for SAP.
  • Strong problem-solving skills and the ability to work collaboratively with AI and software teams.

Nice-to-Have

  • Experience integrating data pipelines with AI/LLM-based systems.
  • Knowledge of vector databases (Pinecone, Weaviate, FAISS) or RAG pipelines.
  • Exposure to chatbot architecture and natural language query systems.
  • Experience setting up CI/CD for data pipelines.

What You'll Work On

  • Building a robust bridge between SAP AO data and our internal database.
  • Creating a scalable data foundation for AI-driven insights and chatbot experiences.
  • Collaborating with data scientists and AI engineers to bring structured enterprise data to life through conversational interfaces.

Job Type: Contract

Pay: ₹ ₹50.00 per hour

Expected hours: 40 per week

Experience:

  • Big Data Engineer / Data Engineer.: 5 years (Required)
  • ETL/ELT pipeline design and implementation.: 5 years (Required)
  • SAP data integration : 3 years (Required)
  • Python / PySpark / SQL.: 5 years (Required)
  • CI/CD for data pipelines.: 3 years (Required)
  • data pipelines with AI/LLM-based systems.: 3 years (Required)

  • Big Data

    2 weeks ago

    Remote, India Technology Next Full time

    **Big Data AI/ML Engineer** **Experience**: 6 to 9 Yrs **Location**:Remote **Tenure: - **6 Months **Salary: - ** 80K-90K In hand **Notice**:Immediate Joiner We are seeking a talented Data Engineer with AI & ML knowledge to join our team. As Data Engineer or ML Ops Engineer, your primary responsibility will be to develop & integrate ML solutions that...

  • Big Data with Pyspark

    2 weeks ago

    Remote, India Achutha Associates Full time

    **Job Title**: Big Data Engineer (PySpark) **Location**: Bengaluru, India **Experience**: 5+ years **Employment Type**: Full-time **Job Summary**: **Key Responsibilities**: - Design, develop, and maintain scalable **big data pipelines** using **PySpark** and other big data technologies. - Work with **Hadoop, Spark, Kafka, Hive, and other distributed data...

  • Big Data Architect

    2 weeks ago

    Remote, India Databricks Full time

    As a **Big Data Architect** in our Professional Services team you will work with clients on short to medium term customer engagements on their big data challenges using the Databricks platform. You will provide data engineering, data science, and cloud technology projects which require integrating with client systems, training, and other technical tasks to...

  • Sr. Big Data Engineer

    15 hours ago

    Remote - India Databricks Full time ₹ 5,00,000 - ₹ 10,00,000 per year

    CSQ326R34As a Sr. Big Data Engineer in our Professional Services team, you will work with clients on short to medium-term customer engagements on their big data challenges using the Databricks Platform. You will provide data engineering, data science, and cloud technology projects that require integrating with client systems, training, and other technical...

  • Big Data Architect

    1 week ago

    Remote, India Databricks Full time

    CSQ225R24 As a** Resident Solutions Architect** in our Professional Services team you will work with clients on short to medium term customer engagements on their **big data challenges using the Databricks platform**. You will provide data engineering, data science, and cloud technology projects which require integrating with client systems, training, and...

  • Senior Data Engineer

    2 days ago

    Remote, India AABM Cloud Data Solutions Full time ₹ 20,00,000 per year

    Senior Data Engineer (Remote, India)10+ Years Data Engineer Experience RequiredSQL, Python, Snowflake - Mandatory hands-on experienceExperience with any ETL Tools (SSIS Preferred), Cloud Environments, and DB2/DBT is a plus.Job Type: Full-timePay: From ₹2,000,000.00 per yearApplication Question(s):How many years of Python Experience do you have?How many...

  • Subject Matter Expert Specialist

    2 weeks ago

    Remote, India Data Engineer Academy LLP Full time ₹ 7,20,000 - ₹ 12,00,000 per year

    We're Hiring: Subject Matter Expert (SME) – Snowflake & Databricks on AWS CloudRemote | Part Time | Flexible Timings | Pay: ₹600 – ₹1000/HourAbout the OpportunityWe are seeking a highly experienced SME with extensive expertise in Snowflake, Databricks, and AWS cloud platforms.What We're Looking ForMinimum 10 years of experience in cloud data...

  • Big Data Eng_Roji_Tookiaki

    21 hours ago

    Remote, India Coders Brain Technology Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    Job Title: BigData Engineer Reporting to: Technical Lead Location: India Requirements: Our ideal candidate will have the following responsibilities: ● Owning and delivering installation and configuration of AI products. ● Participating in solution architecture discussions to estimate hardware sizing. ● Analysing defects and conducting effective...

  • Data Architect

    1 week ago

    Remote, India Alphawizz Technologies Pvt. Ltd. Full time

    **Job Title**: Data Architect (Cloud | Big Data | Analytics) **Location**: Remote **Experience**: Minimum 10+ years in data architecture, engineering, or related roles **Tech Stack Includes**: AWS Redshift | GCP BigQuery | Azure Synapse | Apache Spark | Airflow | dbt | Hadoop | Snowflake | PostgreSQL | MySQL | MongoDB | Kafka | Flink | Tableau | Power BI |...

  • Senior Data Engineer

    2 weeks ago

    Remote, India Ipeople Infosysteams LLC Full time ₹ 20,00,000 - ₹ 35,00,000 per year

    Senior Data EngineerXometry (NASDAQ: XMTR) powers the industries of today and tomorrow by connecting the people with big ideas to the manufacturers who can bring them to life. Xometry's digital marketplace gives manufacturers the critical resources they need to grow their business while also making it easy for buyers at Fortune 1000 companies to tap into...