Data Streaming

2 weeks ago


Bengaluru, India ACE Talent Consulting Full time

Job Description:
Spark/Scala/PySpark developer who knows how to fully exploit the potential of our Spark cluster.

Should have ability to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data.

**Responsibilities**:
Create Scala/Spark/Pyspark jobs for data transformation and aggregation

Produce unit tests for Spark transformations and helper methods

Write Scaladoc-style documentation with all code

Design data processing pipelines

**Skills**:Pyspark

Scala (with a focus on the functional programming paradigm)

Apache Spark 2.x, 3.x
- Apache Spark RDD API
- Apache Spark SQL DataFrame API
- Apache Spark Streaming API

Spark query tuning and performance optimization

SQL database integration (Postgres, and/or MySQL)

Experience working with HDFS, AWS ( S3, Redshift, EMR, IAM, Polices, Routing)

CI-CD Pipleline, Jenkins, Gitlab /Bitbucket

Deep understanding of distributed systems (e.g. partitioning, replication, consistency, and consensus)

**Salary**: ₹300,000.00 - ₹1,500,000.00 per year

Schedule:

- Day shift

Ability to commute/relocate:

- Bengaluru, Karnataka: Reliably commute or planning to relocate before starting work (required)

**Experience**:

- total work: 3 years (preferred)

Work Location: One location


  • Data Analyst

    Found in: Whatjobs IN C2 - 2 weeks ago


    Bengaluru, India NTT DATA Full time

    • Experience in implementing data storage solutions, managing, and developing data processing, and optimizing data solutions (Azure).• Expertise in integrating, transforming, and consolidating data from structured and semi-structured data systems suitable for analytics solutions.• Hands-on experience in integrating event based or streaming data into an...

  • Spark Streaming

    2 weeks ago


    Hyderabad / Secunderabad, Telangana, Bengaluru / Bangalore, India Staidlogic Software Private Limited Full time

    Total Experience - 4 years Relevant Experience - 4 years Location - PAN India Mandatory Skills: Flink and Java Spring Boot Job Description Specifically on Kafka + Flink + Java Spring Boot + Bigdata: - Design and implement broker and partition architecture - Develop consumable topics and corresponding API/microservices using Java Spring boot to render data...

  • Data Analyst

    Found in: Talent IN C2 - 2 weeks ago


    Bengaluru, India NTT DATA Services Full time

    •    Experience in implementing data storage solutions, managing, and developing data processing, and optimizing data solutions (Azure).  •    Expertise in integrating, transforming, and consolidating data from structured and semi-structured data systems suitable for analytics solutions.  •    Hands-on experience in integrating event based or...

  • Data Analyst

    Found in: beBee S IN - 2 weeks ago


    Bengaluru, India NTT DATA Services Full time

    •    Experience in implementing data storage solutions, managing, and developing data processing, and optimizing data solutions (Azure).   •    Expertise in integrating, transforming, and consolidating data from structured and semi-structured data systems suitable for analytics solutions.   •    Hands-on experience in integrating event based...

  • Digital Engineering Lead Engineer

    Found in: Talent IN C2 - 2 weeks ago


    Bengaluru, India NTT DATA Full time

    • Experience in implementing data storage solutions, managing, and developing data processing, and optimizing data solutions (Azure). • Expertise in integrating, transforming, and consolidating data from structured and semi-structured data systems suitable for analytics solutions. • Hands-on experience in integrating event based or streaming data into...

  • Digital Engineering Lead Engineer

    Found in: beBee S IN - 4 weeks ago


    Bengaluru, India NTT DATA Services Full time

    •    Experience in implementing data storage solutions, managing, and developing data processing, and optimizing data solutions (Azure).   •    Expertise in integrating, transforming, and consolidating data from structured and semi-structured data systems suitable for analytics solutions.   •    Hands-on experience in integrating event based...

  • Digital Engineering Lead Engineer

    Found in: beBee S IN - 4 weeks ago


    Bengaluru, India NTT DATA Services Full time

    •    Experience in implementing data storage solutions, managing, and developing data processing, and optimizing data solutions (Azure).   •    Expertise in integrating, transforming, and consolidating data from structured and semi-structured data systems suitable for analytics solutions.   •    Hands-on experience in integrating event based...

  • Digital Engineering Lead Engineer

    Found in: beBee S IN - 3 weeks ago


    Bengaluru, India NTT DATA Services Full time

    •    Experience in implementing data storage solutions, managing, and developing data processing, and optimizing data solutions (Azure).   •    Expertise in integrating, transforming, and consolidating data from structured and semi-structured data systems suitable for analytics solutions.   •    Hands-on experience in integrating event based...

  • Senior Software Engineer

    Found in: Whatjobs IN C2 - 2 days ago


    Bengaluru, India Sony Pictures Networks India Private Limited Full time

    Position:Senior Software Engineer - Streaming, Digital BusinessPurpose:Defining the platform architecture, technology strategy and implementation standardsEducation:Bachelor's degree in computer science, Computer Engineering, Electrical Engineering, or related disciplines; extensive, directly related experience may be considered in-lieu of...

  • Solution Architect

    Found in: Talent IN 2A C2 - 3 weeks ago


    Bengaluru, India DATA KAVERI Full time

    Responsibilities :1. Understand the IUDX architecture and support the deployment team on the development of Data models, adapters, APIs, Validation and operations and also do the hands on development and deployment as needed.2. Understand the city use cases, figure out the public/privately owned data sets needed to create the application, define the solution...

  • Senior Software Engineer

    Found in: Talent IN C2 - 3 weeks ago


    Bengaluru, India Sony Pictures Networks India Private Limited Full time

    Position: Senior Software Engineer - Streaming, Digital Business Purpose: Defining the platform architecture, technology strategy and implementation standards Education: Bachelor's degree in computer science, Computer Engineering, Electrical Engineering, or related disciplines; extensive, directly related experience may be considered in-lieu of...

  • Data Engineer

    Found in: Talent IN 2A C2 - 4 days ago


    Bengaluru, India Gethyr Full time

    We are seeking a talented and motivated Data Engineer stream processing, particularly utilizing Flink and Kafka, alongside expertise in Spark and AWS technologies. As a Data Engineer, you will play a crucial role in designing, implementing, and maintaining robust stream processing solutions to handle large volumes of data in real-time, ensuring high...


  • Bengaluru, India Skillety Technologies Full time

    **Job Openings** **Azure Data Architect****: **Job Description**: Role: Azure Data Architect with streaming experience Job description Total experience in data management area for 10 + years with Azure cloud data platform experience Architect with Azure stack (ADLS, AALS, Azure Data Bricks, Azure Streaming Analytics Azure Data Factory, cosmos DB & Azure...

  • Data Engineer

    Found in: Whatjobs IN C2 - 2 weeks ago


    Bengaluru, India Nityo Infotech Full time

    Experience with big data tools: AWS EC2, RDS, EKS, Kubernetes, etc.o Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.o Experience with data pipeline and workflow management tools: Argo, Airflow, etc.o Experience with Azure cloud services: Data Lake analytics, HDInsights, Data factoryo Experience with stream-processing...

  • Data Engineer

    Found in: Talent IN C2 - 3 weeks ago


    Bengaluru, India Nityo Infotech Full time

    Experience with big data tools: AWS EC2, RDS, EKS, Kubernetes, etc.o Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.o Experience with data pipeline and workflow management tools: Argo, Airflow, etc.o Experience with Azure cloud services: Data Lake analytics, HDInsights, Data factoryo Experience with stream-processing...

  • Scala Developer

    Found in: Talent IN 2A C2 - 3 days ago


    Bengaluru, India NUSTAR TECHNOLOGIES INDIA PRIVATE LIMITED Full time

    Job Description- Must have 4+ years of IT experience- Must have good experience in Spark and Scala- Develop and maintain big data applications using Spark and Scala, ensuring high performance, reliability, and scalability.- Implement Spark data processing techniques, performance tuning, memory management, fault tolerance mechanisms, and scalability...

  • Data Engineer

    Found in: Whatjobs IN C2 - 2 weeks ago


    Bengaluru, India Razorpay Full time

    The Role: We are looking for a savvy Data Engineer to join our growing team, responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up. The right...

  • Data Engineer

    Found in: Talent IN 2A C2 - 2 weeks ago


    Bengaluru, India Razorpay Full time

    The Role:We are looking for a savvy Data Engineer to join our growing team, responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up. The right...

  • Data Engineer

    Found in: Appcast Linkedin IN C2 - 2 weeks ago


    Bengaluru, India Razorpay Full time

    The Role:We are looking for a savvy Data Engineer to join our growing team, responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up. The right...

  • GCP Architect

    Found in: Talent IN 2A C2 - 3 weeks ago


    Bengaluru, India Change Leaders Full time

    Desired Skills & Responsibilities :- Strong understanding of data warehousing and data modeling techniques.- Proficient understanding of distributed computing principles - Hadoop v2, MapReduce, HDFS- Strong data engineering skills on GCP cloud platforms Airflow, Cloud Composer, Data Fusion, Data Flow, Data Proc, Big Query- Experience with building...