Data Streaming

5 days ago

Pune, India ACE Talent Consulting Full time

Job Description:
Spark/Scala/PySpark developer who knows how to fully exploit the potential of our Spark cluster.

Should have ability to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data.

**Responsibilities**:
Create Scala/Spark/Pyspark jobs for data transformation and aggregation

Produce unit tests for Spark transformations and helper methods

Write Scaladoc-style documentation with all code

Design data processing pipelines

**Skills**:Pyspark

Scala (with a focus on the functional programming paradigm)

Apache Spark 2.x, 3.x
- Apache Spark RDD API
- Apache Spark SQL DataFrame API
- Apache Spark Streaming API

Spark query tuning and performance optimization

SQL database integration (Postgres, and/or MySQL)

Experience working with HDFS, AWS ( S3, Redshift, EMR, IAM, Polices, Routing)

CI-CD Pipleline, Jenkins, Gitlab /Bitbucket

Deep understanding of distributed systems (e.g. partitioning, replication, consistency, and consensus)

**Salary**: ₹300,000.00 - ₹1,500,000.00 per year

Schedule:

- Day shift

Ability to commute/relocate:

- Pune, Maharashtra: Reliably commute or planning to relocate before starting work (required)

**Experience**:

- total work: 3 years (preferred)

Work Location: One location

  • Stream Data Engineer

    3 days ago

    Pune, Maharashtra, India beBeeKafka Full time ₹ 15,00,000 - ₹ 25,00,000

    **Job Title:** "Stream Data Engineer"">Job Summary">We are seeking an experienced Stream Data Engineer to design and implement real-time data processing systems using Apache Kafka. The ideal candidate will have a strong background in stream processing, data engineering, and software development.">Required Skills and Qualifications:">">Apache Kafka: Proven...

  • Data Streaming Specialist

    5 days ago

    Chennai, Bengaluru, Pune, India beBeeKafka Full time ₹ 15,00,000 - ₹ 25,00,000

    Kafka Professional About the Role We are seeking an experienced Kafka professional to join our team. As a key member of our data streaming solutions group, you will be responsible for managing and supporting the Kafka ecosystem. Key Responsibilities Ecosystem Management: Manage and support the Kafka ecosystem, focusing primarily on Level 2 (L2) support...

  • Data Streaming Specialist

    4 days ago

    Pune, Bengaluru / Bangalore, Chennai, India beBeeStreaming Full time ₹ 1,04,000 - ₹ 1,30,878

    Job Title: Data Streaming SpecialistWe are seeking a skilled Data Streaming Specialist to join our team. As a Data Streaming Specialist, you will be responsible for designing and implementing data streaming pipelines using Spark and Flink.The ideal candidate will have experience with Spark and Flink, as well as strong knowledge of data processing concepts...

  • Streaming Data Architect

    3 days ago

    Chennai, Bengaluru, Pune, India beBeeData Full time ₹ 10,00,000 - ₹ 15,00,000

    Job Posting for Data EngineerAbout the Role:We are looking for a highly skilled data engineer to join our team. The ideal candidate will have experience with streaming data and be able to design, develop, and implement data pipelines using Spark.Key Responsibilities:Design, develop, and maintain large-scale data processing systems using Apache...

  • Senior Data Streaming Specialist

    3 days ago

    Pune, Bengaluru / Bangalore, Chennai, India beBeeDataStreaming Full time ₹ 1,04,000 - ₹ 1,30,878

    **Job Overview**We are seeking an experienced professional to join our team as a Data Streaming Expert. As a key member of our organization, you will be responsible for managing and supporting the Kafka ecosystem.

  • Tcs data Streaming

    2 days ago

    Pashan, Pune, Maharashtra, India Response Informatics Full time

    JOB DESCRIPTION: Spark/Scala/PySpark developer who knows how to fully exploit the potential of our Spark cluster. Should have ability to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data. SKILLS: Pyspark Scala (with a focus on the functional programming paradigm) Apache Spark 2.x, 3.x -...

  • Data Streaming

    5 days ago

    Pune, India ACE Talent Consulting Full time

    ob Description: Spark/Scala/PySpark developer who knows how to fully exploit the potential of our Spark cluster. Should have ability to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data. **Responsibilities**: Create Scala/Spark/Pyspark jobs for data transformation and aggregation Produce...

  • Data Engineer

    4 weeks ago

    Pune, Maharashtra, India Maveric Systems Limited Full time

    Real-Time Data Engineer is responsible for designing and implementing scalable, real time data processing systems in Kafka.This role involves build of KAFKA cluster, ensuring high availability, and integrating with other data processing tools and platformsAs part of the CTO Data Ingestion Service, the incumbent will be required to:- Building scalable,...

  • Data Engineer

    2 weeks ago

    Pune, Maharashtra, India Maveric Systems Limited Full time

    Real-Time Data Engineer is responsible for designing and implementing scalable, real time data processing systems in Kafka. This role involves build of KAFKA cluster, ensuring high availability, and integrating with other data processing tools and platforms As part of the CTO Data Ingestion Service, the incumbent will be required to: Building scalable,...

  • Data Engineer

    4 days ago

    Pune, Maharashtra, India Maveric Systems Limited Full time

    Real-Time Data Engineer is responsible for designing and implementing scalable, real time data processing systems in Kafka. This role involves build of KAFKA cluster, ensuring high availability, and integrating with other data processing tools and platformsAs part of the CTO Data Ingestion Service, the incumbent will be required to:Building scalable,...