Data Streaming
2 weeks ago
Job Description:
Spark/Scala/PySpark developer who knows how to fully exploit the potential of our Spark cluster.
Should have ability to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data.
**Responsibilities**:
Create Scala/Spark/Pyspark jobs for data transformation and aggregation
Produce unit tests for Spark transformations and helper methods
Write Scaladoc-style documentation with all code
Design data processing pipelines
**Skills**:Pyspark
Scala (with a focus on the functional programming paradigm)
Apache Spark 2.x, 3.x
- Apache Spark RDD API
- Apache Spark SQL DataFrame API
- Apache Spark Streaming API
Spark query tuning and performance optimization
SQL database integration (Postgres, and/or MySQL)
Experience working with HDFS, AWS ( S3, Redshift, EMR, IAM, Polices, Routing)
CI-CD Pipleline, Jenkins, Gitlab /Bitbucket
Deep understanding of distributed systems (e.g. partitioning, replication, consistency, and consensus)
**Salary**: ₹300,000.00 - ₹1,500,000.00 per year
Schedule:
- Day shift
Ability to commute/relocate:
- Bengaluru, Karnataka: Reliably commute or planning to relocate before starting work (required)
**Experience**:
- total work: 3 years (preferred)
Work Location: One location
-
Data Analyst
Found in: Whatjobs IN C2 - 2 weeks ago
Bengaluru, India NTT DATA Full time• Experience in implementing data storage solutions, managing, and developing data processing, and optimizing data solutions (Azure).• Expertise in integrating, transforming, and consolidating data from structured and semi-structured data systems suitable for analytics solutions.• Hands-on experience in integrating event based or streaming data into an...
-
Spark Streaming
2 weeks ago
Hyderabad / Secunderabad, Telangana, Bengaluru / Bangalore, India Staidlogic Software Private Limited Full timeTotal Experience - 4 years Relevant Experience - 4 years Location - PAN India Mandatory Skills: Flink and Java Spring Boot Job Description Specifically on Kafka + Flink + Java Spring Boot + Bigdata: - Design and implement broker and partition architecture - Develop consumable topics and corresponding API/microservices using Java Spring boot to render data...
-
Data Analyst
Found in: Talent IN C2 - 2 weeks ago
Bengaluru, India NTT DATA Services Full time• Experience in implementing data storage solutions, managing, and developing data processing, and optimizing data solutions (Azure). • Expertise in integrating, transforming, and consolidating data from structured and semi-structured data systems suitable for analytics solutions. • Hands-on experience in integrating event based or...
-
Data Analyst
Found in: beBee S IN - 2 weeks ago
Bengaluru, India NTT DATA Services Full time• Experience in implementing data storage solutions, managing, and developing data processing, and optimizing data solutions (Azure). • Expertise in integrating, transforming, and consolidating data from structured and semi-structured data systems suitable for analytics solutions. • Hands-on experience in integrating event based...
-
Digital Engineering Lead Engineer
Found in: Talent IN C2 - 2 weeks ago
Bengaluru, India NTT DATA Full time• Experience in implementing data storage solutions, managing, and developing data processing, and optimizing data solutions (Azure). • Expertise in integrating, transforming, and consolidating data from structured and semi-structured data systems suitable for analytics solutions. • Hands-on experience in integrating event based or streaming data into...
-
Digital Engineering Lead Engineer
Found in: beBee S IN - 4 weeks ago
Bengaluru, India NTT DATA Services Full time• Experience in implementing data storage solutions, managing, and developing data processing, and optimizing data solutions (Azure). • Expertise in integrating, transforming, and consolidating data from structured and semi-structured data systems suitable for analytics solutions. • Hands-on experience in integrating event based...
-
Digital Engineering Lead Engineer
Found in: beBee S IN - 4 weeks ago
Bengaluru, India NTT DATA Services Full time• Experience in implementing data storage solutions, managing, and developing data processing, and optimizing data solutions (Azure). • Expertise in integrating, transforming, and consolidating data from structured and semi-structured data systems suitable for analytics solutions. • Hands-on experience in integrating event based...
-
Digital Engineering Lead Engineer
Found in: beBee S IN - 3 weeks ago
Bengaluru, India NTT DATA Services Full time• Experience in implementing data storage solutions, managing, and developing data processing, and optimizing data solutions (Azure). • Expertise in integrating, transforming, and consolidating data from structured and semi-structured data systems suitable for analytics solutions. • Hands-on experience in integrating event based...
-
Senior Software Engineer
Found in: Whatjobs IN C2 - 2 days ago
Bengaluru, India Sony Pictures Networks India Private Limited Full timePosition:Senior Software Engineer - Streaming, Digital BusinessPurpose:Defining the platform architecture, technology strategy and implementation standardsEducation:Bachelor's degree in computer science, Computer Engineering, Electrical Engineering, or related disciplines; extensive, directly related experience may be considered in-lieu of...
-
Solution Architect
Found in: Talent IN 2A C2 - 3 weeks ago
Bengaluru, India DATA KAVERI Full timeResponsibilities :1. Understand the IUDX architecture and support the deployment team on the development of Data models, adapters, APIs, Validation and operations and also do the hands on development and deployment as needed.2. Understand the city use cases, figure out the public/privately owned data sets needed to create the application, define the solution...
-
Senior Software Engineer
Found in: Talent IN C2 - 3 weeks ago
Bengaluru, India Sony Pictures Networks India Private Limited Full timePosition: Senior Software Engineer - Streaming, Digital Business Purpose: Defining the platform architecture, technology strategy and implementation standards Education: Bachelor's degree in computer science, Computer Engineering, Electrical Engineering, or related disciplines; extensive, directly related experience may be considered in-lieu of...
-
Data Engineer
Found in: Talent IN 2A C2 - 4 days ago
Bengaluru, India Gethyr Full timeWe are seeking a talented and motivated Data Engineer stream processing, particularly utilizing Flink and Kafka, alongside expertise in Spark and AWS technologies. As a Data Engineer, you will play a crucial role in designing, implementing, and maintaining robust stream processing solutions to handle large volumes of data in real-time, ensuring high...
-
Azure Data Architect
5 days ago
Bengaluru, India Skillety Technologies Full time**Job Openings** **Azure Data Architect****: **Job Description**: Role: Azure Data Architect with streaming experience Job description Total experience in data management area for 10 + years with Azure cloud data platform experience Architect with Azure stack (ADLS, AALS, Azure Data Bricks, Azure Streaming Analytics Azure Data Factory, cosmos DB & Azure...
-
Data Engineer
Found in: Whatjobs IN C2 - 2 weeks ago
Bengaluru, India Nityo Infotech Full timeExperience with big data tools: AWS EC2, RDS, EKS, Kubernetes, etc.o Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.o Experience with data pipeline and workflow management tools: Argo, Airflow, etc.o Experience with Azure cloud services: Data Lake analytics, HDInsights, Data factoryo Experience with stream-processing...
-
Data Engineer
Found in: Talent IN C2 - 3 weeks ago
Bengaluru, India Nityo Infotech Full timeExperience with big data tools: AWS EC2, RDS, EKS, Kubernetes, etc.o Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.o Experience with data pipeline and workflow management tools: Argo, Airflow, etc.o Experience with Azure cloud services: Data Lake analytics, HDInsights, Data factoryo Experience with stream-processing...
-
Scala Developer
Found in: Talent IN 2A C2 - 3 days ago
Bengaluru, India NUSTAR TECHNOLOGIES INDIA PRIVATE LIMITED Full timeJob Description- Must have 4+ years of IT experience- Must have good experience in Spark and Scala- Develop and maintain big data applications using Spark and Scala, ensuring high performance, reliability, and scalability.- Implement Spark data processing techniques, performance tuning, memory management, fault tolerance mechanisms, and scalability...
-
Data Engineer
Found in: Whatjobs IN C2 - 2 weeks ago
Bengaluru, India Razorpay Full timeThe Role: We are looking for a savvy Data Engineer to join our growing team, responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up. The right...
-
Data Engineer
Found in: Talent IN 2A C2 - 2 weeks ago
Bengaluru, India Razorpay Full timeThe Role:We are looking for a savvy Data Engineer to join our growing team, responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up. The right...
-
Data Engineer
Found in: Appcast Linkedin IN C2 - 2 weeks ago
Bengaluru, India Razorpay Full timeThe Role:We are looking for a savvy Data Engineer to join our growing team, responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up. The right...
-
GCP Architect
Found in: Talent IN 2A C2 - 3 weeks ago
Bengaluru, India Change Leaders Full timeDesired Skills & Responsibilities :- Strong understanding of data warehousing and data modeling techniques.- Proficient understanding of distributed computing principles - Hadoop v2, MapReduce, HDFS- Strong data engineering skills on GCP cloud platforms Airflow, Cloud Composer, Data Fusion, Data Flow, Data Proc, Big Query- Experience with building...