AI - Senior Data Engineer - Kafka/Spark

3 weeks ago

Bengaluru, Karnataka, India Observe Full time

Job Description :

We are looking for a Senior Data Engineer with strong hands-on experience in building scalable data pipelines and real-time processing systems. You will be part of a high-impact team focused on modernizing our data architecture, enabling self-serve analytics, and delivering high-quality data products. This role is ideal for engineers who love solving complex data challenges, have a growth mindset, and are excited to work on both batch and streaming systems.

Responsibilities :

- Build and maintain real-time and batch data pipelines using tools like Kafka, Spark, and Airflow.

- Contribute to the development of a scalable LakeHouse architecture using modern data formats such as Delta Lake, Hudi, or Iceberg.

- Optimize data ingestion and transformation workflows across cloud platforms (AWS, GCP, or Azure).

- Collaborate with Analytics and Product teams to deliver data models, marts, and dashboards that drive business insights.

- Support data quality, lineage, and observability using modern practices and tools.

- Participate in Agile processes (Sprint Planning, Reviews) and contribute to team knowledge sharing and documentation.

- Contribute to building data products for inbound (ingestion) and outbound (consumption) use cases across the organization.

Requirements :

- 5-8 years of experience in data engineering or backend systems with a focus on large-scale data pipelines.

- Hands-on experience with streaming platforms (e. g., Kafka) and distributed processing tools (e. g., Spark or Flink).

- Working knowledge of LakeHouse formats (Delta/Hudi/Iceberg) and columnar storage like Parquet.

- Proficient in building pipelines on AWS, GCP, or Azure using managed services and cloud-native tools.

- Experience in Airflow or similar orchestration platforms.

- Strong in data modeling and optimizing data warehouses like Redshift, BigQuery, or Snowflake.

- Exposure to real-time OLAP tools like ClickHouse, Druid, or Pinot.

- Familiarity with observability tools such as Grafana, Prometheus, or Loki.

- Some experience integrating data with MLOps tools like MLflow, SageMaker, or Kubeflow.

- Ability to work with Agile practices using JIRA, Confluence, and participate in engineering ceremonies.

(ref:hirist.tech)

  • Data Engineer – Scala + Spark

    2 weeks ago

    Bengaluru, Karnataka, India Axheleon IT Services Private Limited Full time

    Job DescriptionJob Title: Data Engineer Scala + SparkLocation: Bangalore (Hybrid/On-site)Experience: 5 to 8 Years (Minimum 4 Years of Relevant Experience in Scala & Spark)Budget: Up to 25 LPANotice Period: Immediate to 30 Days MaxAbout the Role:We are looking for a skilled and experienced Data Engineer with strong expertise in Scala, Apache Spark, and SQL to...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 28,00,000

    Job Summary:We are seeking a Senior Data Engineer with expertise in Apache Spark to join our team. As a Senior Data Engineer, you will be responsible for designing and developing scalable data processing solutions using AWS EMR and Apache Spark.About the Role:This is an exciting opportunity to work with a talented team of data engineers and scientists to...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Weekday AI Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    This role is for one of the Weekday's clientsMin Experience: 3 yearsLocation: BangaloreJobType: full-timeWe are seeking a highly skilled and motivated Data Engineer to join our data team. In this role, you will design, build, and maintain scalable data infrastructure that powers data-driven decision-making across the organization.Key Responsibilities Manage...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Weekday AI Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    This role is for one of the Weekday's clientsMin Experience: 3 yearsLocation: BangaloreJobType: full-timeWe are seeking a highly skilled and motivated Data Engineer to join our data team. In this role, you will design, build, and maintain scalable data infrastructure that powers data-driven decision-making across the organization.RequirementsKey...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Recro Full time

    Key Responsibilities• Design and implement cloud-native Data Lakes and Lakehouse optimized for AI/MLworkloads and real-time analytics.• Develop real-time and event-driven data pipelines using Py Spark, Python and cloud services(Azure, AWS, or GCP).• Enable decentralized data ownership and domain-driven design as part of Data Meshinitiatives.•...

  • Senior Data Engineer

    4 days ago

    Bengaluru, Karnataka, India Rezolve AI Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    We are seeking a hands-on Senior Data Engineer to build and operate our real-time data platform that powers metrics, SLAs, and dashboards for our enterprise SaaS product. You will design pipelines that ingest flexible JSON data from transactional databases, compute state/queue/assignment metrics in real time, and serve low-latency, drillable analytics across...

  • Senior data engineer

    2 weeks ago

    Bengaluru, Karnataka, India Recro Full time

    Key Responsibilities• Design and implement cloud-native Data Lakes and Lakehouse optimized for AI/MLworkloads and real-time analytics.• Develop real-time and event-driven data pipelines using Py Spark, Python and cloud services(Azure, AWS, or GCP).• Enable decentralized data ownership and domain-driven design as part of Data Meshinitiatives.•...

  • ai - Data Engineer

    3 weeks ago

    Bengaluru, Karnataka, India Sprouts Full time

    Job Summary :We are looking for an experienced Data Engineer to design, build, and maintain scalable data pipelines and processing systems that power our modern data platform. The ideal candidate will have hands-on expertise in distributed data processing, data lake/lakehouse systems, and workflow orchestration using modern tools such as Spark, PySpark,...

  • Data Engineering

    4 weeks ago

    Bengaluru, Karnataka, India ScaleneWorks People Solutions Full time

    Job Title Middleware Engineer Position Data Engineer Experience 5-6yrs Category IT Infrastructure Main location India Karnataka Bangalore Employment Type Full Time Qualification Bachelor s degree in Computer Science or related field or higher Roles and ResponsibilitiesData Engineer - 5-6 years experience Responsibilities Design develop and...

  • AI Engineer

    2 weeks ago

    Bengaluru, Karnataka, India DEEPJOB TECHNOLOGIES PRIVATE LIMITED Full time

    We are hiring an experienced AI Engineer (with a Data Science background) to lead the design and deployment of AI-driven workflow automation systems.Key Responsibilities :- Build and deploy ML models for classification, prediction & NLP- Automate internal and external business processes using AI- Develop and scale APIs and microservices for production...