Data Streaming

6 days ago

Pune, India ACE Talent Consulting Full time

Job Description:
Spark/Scala/PySpark developer who knows how to fully exploit the potential of our Spark cluster.

Should have ability to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data.

**Responsibilities**:
Create Scala/Spark/Pyspark jobs for data transformation and aggregation

Produce unit tests for Spark transformations and helper methods

Write Scaladoc-style documentation with all code

Design data processing pipelines

**Skills**:Pyspark

Scala (with a focus on the functional programming paradigm)

Apache Spark 2.x, 3.x
- Apache Spark RDD API
- Apache Spark SQL DataFrame API
- Apache Spark Streaming API

Spark query tuning and performance optimization

SQL database integration (Postgres, and/or MySQL)

Experience working with HDFS, AWS ( S3, Redshift, EMR, IAM, Polices, Routing)

CI-CD Pipleline, Jenkins, Gitlab /Bitbucket

Deep understanding of distributed systems (e.g. partitioning, replication, consistency, and consensus)

**Salary**: ₹300,000.00 - ₹1,500,000.00 per year

Schedule:

- Day shift

Ability to commute/relocate:

- Pune, Maharashtra: Reliably commute or planning to relocate before starting work (required)

**Experience**:

- total work: 3 years (preferred)

Work Location: One location

  • Data Streaming Engineer_6+ Years_ PAN India

    7 days ago

    Pune, Maharashtra, India Innova ESI Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    We are looking for a Data Streaming Engineer to design, develop, and test real-time data pipelines leveraging Solace, Kafka, Databricks Delta Live Tables (DLT), and Apache Spark Structured Streaming. The role involves building ingestion frameworks, streaming data readers, raw data parsers, transformation pipelines from raw to bronze layers, and publishing...

  • Event Streaming

    1 week ago

    Pune, India NMS Consultant Full time

    Job SummaryWe are seeking an experienced Event Streaming & Cloud Integration Developer to design, build, and optimize real-time data streaming and integration solutions across cloud environments. The ideal candidate will have strong hands-on expertise in Kafka, AWS integration services, and cloud-native microservices architectures, with a focus on delivering...

  • Event Streaming

    1 week ago

    pune, India NMS Consultant Full time

    Job SummaryWe are seeking an experienced Event Streaming & Cloud Integration Developer to design, build, and optimize real-time data streaming and integration solutions across cloud environments. The ideal candidate will have strong hands-on expertise in Kafka, AWS integration services, and cloud-native microservices architectures, with a focus on delivering...

  • Event Streaming

    1 week ago

    Pune, India NMS Consultant Full time

    Job SummaryWe are seeking an experienced Event Streaming & Cloud Integration Developer to design, build, and optimize real-time data streaming and integration solutions across cloud environments. The ideal candidate will have strong hands-on expertise in Kafka, AWS integration services, and cloud-native microservices architectures, with a focus on delivering...

  • Event Streaming

    1 week ago

    Pune, India NMS Consultant Full time

    Job SummaryWe are seeking an experienced Event Streaming & Cloud Integration Developer to design, build, and optimize real-time data streaming and integration solutions across cloud environments. The ideal candidate will have strong hands-on expertise in Kafka, AWS integration services, and cloud-native microservices architectures, with a focus on delivering...

  • Event Streaming

    1 week ago

    Pune, India NMS Consultant Full time

    Job Summary We are seeking an experienced Event Streaming & Cloud Integration Developer to design, build, and optimize real-time data streaming and integration solutions across cloud environments. The ideal candidate will have strong hands-on expertise in Kafka , AWS integration services , and cloud-native microservices architectures, with a focus on...

  • Event Streaming

    1 week ago

    Pune, India NMS Consultant Full time

    Job Summary We are seeking an experienced Event Streaming & Cloud Integration Developer to design, build, and optimize real-time data streaming and integration solutions across cloud environments. The ideal candidate will have strong hands-on expertise in Kafka , AWS integration services , and cloud-native microservices architectures, with a focus on...

  • Event Streaming

    1 week ago

    Pune, India NMS Consultant Full time

    Job SummaryWe are seeking an experienced Event Streaming & Cloud Integration Developer to design, build, and optimize real-time data streaming and integration solutions across cloud environments. The ideal candidate will have strong hands-on expertise in Kafka, AWS integration services, and cloud-native microservices architectures, with a focus on delivering...

  • Event Streaming

    1 week ago

    Pune, India NMS Consultant Full time

    Job Summary We are seeking an experienced Event Streaming & Cloud Integration Developer to design, build, and optimize real-time data streaming and integration solutions across cloud environments. The ideal candidate will have strong hands-on expertise in Kafka, AWS integration services, and cloud-native microservices architectures, with a focus on...

  • Event streaming

    7 days ago

    Pune, India NMS Consultant Full time

    Job SummaryWe are seeking an experienced Event Streaming & Cloud Integration Developer to design, build, and optimize real-time data streaming and integration solutions across cloud environments. The ideal candidate will have strong hands-on expertise in Kafka, AWS integration services, and cloud-native microservices architectures, with a focus on delivering...