Data Streaming

2 days ago


Bengaluru, India ACE Talent Consulting Full time

Job Description:
Spark/Scala/PySpark developer who knows how to fully exploit the potential of our Spark cluster.

Should have ability to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data.

**Responsibilities**:
Create Scala/Spark/Pyspark jobs for data transformation and aggregation

Produce unit tests for Spark transformations and helper methods

Write Scaladoc-style documentation with all code

Design data processing pipelines

**Skills**:Pyspark

Scala (with a focus on the functional programming paradigm)

Apache Spark 2.x, 3.x
- Apache Spark RDD API
- Apache Spark SQL DataFrame API
- Apache Spark Streaming API

Spark query tuning and performance optimization

SQL database integration (Postgres, and/or MySQL)

Experience working with HDFS, AWS ( S3, Redshift, EMR, IAM, Polices, Routing)

CI-CD Pipleline, Jenkins, Gitlab /Bitbucket

Deep understanding of distributed systems (e.g. partitioning, replication, consistency, and consensus)

**Salary**: ₹300,000.00 - ₹1,500,000.00 per year

Schedule:

- Day shift

Ability to commute/relocate:

- Bengaluru, Karnataka: Reliably commute or planning to relocate before starting work (required)

**Experience**:

- total work: 3 years (preferred)

Work Location: One location



  • Bengaluru, Karnataka, India Innova ESI Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Role: Data Streaming EngineerExperience: 6+yearsLocation: Pan IndiaImmediate joiners onlyWe are looking for a Data Streaming Engineer to design, develop, and test real-time data pipelines leveraging Solace, Kafka, Databricks Delta Live Tables (DLT), and Apache Spark Structured Streaming.The role involves building ingestion frameworks, streaming data readers,...


  • Bengaluru, India BairesDev Full time

    Job Description At BairesDev, we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley. Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works on roles that drive significant impact worldwide. When you apply for this...

  • Kafka Developer

    2 weeks ago


    Bengaluru, India People Prime World Wide Full time

    Key Responsibilities :- Design, develop, and maintain scalable and reliable Apache Kafka-based data streaming platforms.- Utilize Kafka Connect, Kafka Streams, and Kafka client libraries to build real-time data pipelines and event-driven applications.- Independently resolve issues related to infrastructure deployment and application setup on cloud services...

  • Data Engineer

    2 weeks ago


    Bengaluru, India NTT DATA, Inc. Full time

    Req ID: We are currently seeking a Data Engineer (Talend &Pyspark) to join our team in Bangalore, Karntaka (IN-KA), India (IN). "Job Duties: Key Responsibilities: Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack...


  • Bengaluru, India BairesDev Full time

    Job Description At BairesDev, we've been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley. Our diverse 4,000+ team, composed of the world's Top 1% of tech talent, works on roles that drive significant impact worldwide. When you apply for this...


  • Bengaluru, India NTT DATA, Inc. Full time

    6+ years of experience in Data Engineering or related role. Hands-on experience with Snowflake (data modelling, performance tuning, query optimization, Snowpipe, Time Travel, Streams & Tasks). Strong expertise in AWS Glue for ETL job development, orchestration, and optimization. Proficiency with AWS services such as S3, Lambda, Redshift, Athena, Step...

  • Data Engineer

    2 weeks ago


    Bengaluru, India NTT Data Full time

    Job Description NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer to join our team in Bangalore, Karntaka (IN-KA), India (IN). - 6+ years of experience in Data...

  • Offshore Data Engineer

    17 hours ago


    Bengaluru, India NTT DATA Full time

    Job Description Req ID:     NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Offshore Data Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN). Key...

  • Offshore Data Engineer

    12 hours ago


    Bengaluru, India NTT DATA Full time

    Job Description Req ID:     NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Offshore Data Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN). Key...

  • Data Streaming

    2 days ago


    Bengaluru, India ACE Talent Consulting Full time

    ob Description: Spark/Scala/PySpark developer who knows how to fully exploit the potential of our Spark cluster. Should have ability to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data. **Responsibilities**: Create Scala/Spark/Pyspark jobs for data transformation and aggregation Produce...