Data Streaming

14 hours ago

Bengaluru, India ACE Talent Consulting Full time

Job Description:
Spark/Scala/PySpark developer who knows how to fully exploit the potential of our Spark cluster.

Should have ability to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data.

**Responsibilities**:
Create Scala/Spark/Pyspark jobs for data transformation and aggregation

Produce unit tests for Spark transformations and helper methods

Write Scaladoc-style documentation with all code

Design data processing pipelines

**Skills**:Pyspark

Scala (with a focus on the functional programming paradigm)

Apache Spark 2.x, 3.x
- Apache Spark RDD API
- Apache Spark SQL DataFrame API
- Apache Spark Streaming API

Spark query tuning and performance optimization

SQL database integration (Postgres, and/or MySQL)

Experience working with HDFS, AWS ( S3, Redshift, EMR, IAM, Polices, Routing)

CI-CD Pipleline, Jenkins, Gitlab /Bitbucket

Deep understanding of distributed systems (e.g. partitioning, replication, consistency, and consensus)

**Salary**: ₹300,000.00 - ₹1,500,000.00 per year

Schedule:

- Day shift

Ability to commute/relocate:

- Bengaluru, Karnataka: Reliably commute or planning to relocate before starting work (required)

**Experience**:

- total work: 3 years (preferred)

Work Location: One location

  • Data Streaming Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Innova ESI Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Role: Data Streaming EngineerExperience: 6+yearsLocation: Pan IndiaImmediate joiners onlyWe are looking for a Data Streaming Engineer to design, develop, and test real-time data pipelines leveraging Solace, Kafka, Databricks Delta Live Tables (DLT), and Apache Spark Structured Streaming.The role involves building ingestion frameworks, streaming data readers,...

  • Data streaming Engineer

    5 days ago

    Bengaluru, India Xebia Full time

    Role Description - We’re seeking a hands-on engineer with deep expertise in Confluent Cloud Kafka and Apache Flink. Candidate should be able to design, build, and operate real-time streaming pipelines and analytics, write and optimize complex Flink SQL queries, and administer Confluent Cloud environments w.r.t to managing topics, schema governance. Key...

  • Data streaming Engineer

    5 days ago

    Bengaluru, India Xebia Full time

    Role Description - We’re seeking a hands-on engineer with deep expertise in Confluent Cloud Kafka and Apache Flink. Candidate should be able to design, build, and operate real-time streaming pipelines and analytics, write and optimize complex Flink SQL queries, and administer Confluent Cloud environments w.r.t to managing topics, schema governance. Key...

  • Data streaming Engineer

    3 days ago

    Bengaluru, India Xebia Full time

    Role Description - We’re seeking a hands-on engineer with deep expertise in Confluent Cloud Kafka and Apache Flink. Candidate should be able to design, build, and operate real-time streaming pipelines and analytics, write and optimize complex Flink SQL queries, and administer Confluent Cloud environments w.r.t to managing topics, schema governance. Key...

  • Data streaming Engineer

    3 days ago

    Bengaluru, India Xebia Full time

    Role Description -We’re seeking a hands-on engineer with deep expertise in Confluent Cloud Kafka and Apache Flink. Candidate should be able to design, build, and operate real-time streaming pipelines and analytics, write and optimize complex Flink SQL queries, and administer Confluent Cloud environments w.r.t to managing topics, schema governance.Key...

  • Data streaming engineer

    3 days ago

    Bengaluru, India Xebia Full time

    Role Description -We’re seeking a hands-on engineer with deep expertise in Confluent Cloud Kafka and Apache Flink. Candidate should be able to design, build, and operate real-time streaming pipelines and analytics, write and optimize complex Flink SQL queries, and administer Confluent Cloud environments w.r.t to managing topics, schema governance.Key...

  • Data streaming Engineer

    5 days ago

    Bengaluru, India Xebia Full time

    Role Description -We’re seeking a hands-on engineer with deep expertise in Confluent Cloud Kafka and Apache Flink. Candidate should be able to design, build, and operate real-time streaming pipelines and analytics, write and optimize complex Flink SQL queries, and administer Confluent Cloud environments w.r.t to managing topics, schema governance.Key...

  • Data streaming Engineer

    5 days ago

    Bengaluru, India Xebia Full time

    Role Description -We’re seeking a hands-on engineer with deep expertise in Confluent Cloud Kafka and Apache Flink. Candidate should be able to design, build, and operate real-time streaming pipelines and analytics, write and optimize complex Flink SQL queries, and administer Confluent Cloud environments w.r.t to managing topics, schema governance.Key...

  • Data streaming Engineer

    3 days ago

    Bengaluru, India Xebia Full time

    Role Description -We’re seeking a hands-on engineer with deep expertise in Confluent Cloud Kafka and Apache Flink. Candidate should be able to design, build, and operate real-time streaming pipelines and analytics, write and optimize complex Flink SQL queries, and administer Confluent Cloud environments w.r.t to managing topics, schema governance.Key...

  • Data streaming Engineer

    4 days ago

    Bengaluru, India Xebia Full time

    Role Description -We’re seeking a hands-on engineer with deep expertise in Confluent Cloud Kafka and Apache Flink. Candidate should be able to design, build, and operate real-time streaming pipelines and analytics, write and optimize complex Flink SQL queries, and administer Confluent Cloud environments w.r.t to managing topics, schema governance.Key...