Data Streaming
2 weeks ago
Job Description:
Spark/Scala/PySpark developer who knows how to fully exploit the potential of our Spark cluster.
Should have ability to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data.
**Responsibilities**:
Create Scala/Spark/Pyspark jobs for data transformation and aggregation
Produce unit tests for Spark transformations and helper methods
Write Scaladoc-style documentation with all code
Design data processing pipelines
**Skills**:Pyspark
Scala (with a focus on the functional programming paradigm)
Apache Spark 2.x, 3.x
- Apache Spark RDD API
- Apache Spark SQL DataFrame API
- Apache Spark Streaming API
Spark query tuning and performance optimization
SQL database integration (Postgres, and/or MySQL)
Experience working with HDFS, AWS ( S3, Redshift, EMR, IAM, Polices, Routing)
CI-CD Pipleline, Jenkins, Gitlab /Bitbucket
Deep understanding of distributed systems (e.g. partitioning, replication, consistency, and consensus)
**Salary**: ₹300,000.00 - ₹1,500,000.00 per year
Schedule:
- Day shift
Ability to commute/relocate:
- Chennai, Tamil Nadu: Reliably commute or planning to relocate before starting work (required)
**Experience**:
- total work: 3 years (preferred)
Work Location: One location
-
Data Scientist
6 days ago
Chennai, India Crayon Data Full timeRole: Data Scientist Experience level: 3+ years Location: Chennai Who are we? Crayon Data is a leading provider of AI-led revenue acceleration solutions, headquartered in Singapore with a local presence in India and the UAE. The company was founded in 2012 with the vision of simplifying the world’s choices. Our flagship platform, maya.ai, helps...
-
Data Architect
3 weeks ago
Chennai, India iXceed Solutions Full time**Job Title**:Data Architect **Job Location**:WFH from India **Job Mode**: Remote (need to follow Europe Time zone) Primarily experts of **Spark 3.X, SCALA,** Delta lake implementation, streaming solution implementation for IOT in Spark streaming. Kafka expertise. Any experience with MFG BI, DWH and Datalake implementation would be a bonus. **Job Type**:...
-
Data Engineer
Found in: Whatjobs IN C2 - 3 weeks ago
Chennai, India Wenger & Watson Inc Full timeExp-3+ yrsLocation-Chennai(WFO)compulsory work from office (1st and last saturday working)notice period-30-immediate joinerJob DescriptionShould have a minimum 6+ years in Data Engineering, Data Analytics platform. Should have strong hands-on design and engineering background in AWS, across a wide range of AWS services with the ability to demonstrate...
-
Data Engineer
Found in: Whatjobs IN C2 - 3 weeks ago
Chennai, India Wenger & Watson Inc Full timeExp-3+ yrs Location-Chennai(WFO) compulsory work from office (1st and last saturday working) notice period-30-immediate joiner Job Description Should have a minimum 6+ years in Data Engineering, Data Analytics platform. Should have strong hands-on design and engineering background in AWS, across a wide range of AWS services with the ability to...
-
Data Engineer
Found in: Talent IN 2A C2 - 3 weeks ago
Chennai, India Wenger & Watson Inc Full timeExp-3+ yrsLocation-Chennai(WFO)compulsory work from office (1st and last saturday working)notice period-30-immediate joinerJob DescriptionShould have a minimum 6+ years in Data Engineering, Data Analytics platform. Should have strong hands-on design and engineering background in AWS, across a wide range of AWS services with the ability to demonstrate...
-
Data Engineer
Found in: Appcast Linkedin IN C2 - 3 weeks ago
Chennai, India Wenger & Watson Inc Full timeExp-3+ yrsLocation-Chennai(WFO)compulsory work from office (1st and last saturday working)notice period-30-immediate joinerJob DescriptionShould have a minimum 6+ years in Data Engineering, Data Analytics platform. Should have strong hands-on design and engineering background in AWS, across a wide range of AWS services with the ability to demonstrate...
-
AWS Data Engineer
Found in: Whatjobs IN C2 - 7 days ago
Chennai, India CliqHR Full timeJob Description : - Must have minimum 5+ years of experience as AWS Data EngineerNice to have : AWS Certified Data Analytics Specialty certification- Experience in designing and implementing data pipelines using AWS Glue and orchestration using Airflow for batch, streaming (Kafka/Kinesis) and API ingestion patterns- Work closely with DevOps engineer to...
-
AWS Data Engineer
Found in: Talent IN 2A C2 - 1 week ago
Chennai, India CliqHR Full timeJob Description :- Must have minimum 5+ years of experience as AWS Data EngineerNice to have : AWS Certified Data Analytics Specialty certification- Experience in designing and implementing data pipelines using AWS Glue and orchestration using Airflow for batch, streaming (Kafka/Kinesis) and API ingestion patterns- Work closely with DevOps engineer to...
-
AWS Data Engineer
Found in: Adzuna IN C2 - 7 days ago
Chennai, Tamil Nadu, India CliqHR Full timeJob Description :- Must have minimum 5+ years of experience as AWS Data EngineerNice to have : AWS Certified Data Analytics Specialty certification- Experience in designing and implementing data pipelines using AWS Glue and orchestration using Airflow for batch, streaming (Kafka/Kinesis) and API ingestion patterns- Work closely with DevOps engineer to...
-
Senior Data Engineer
Found in: Whatjobs IN C2 - 3 weeks ago
Chennai, India hubwork Full timeSummary :We are looking for a data engineer who has experience in building data pipelines for large data platforms. This role will be working with a team building a large data platform catering to the needs of a wealth manager for their application and analytics needs. This role will be augmented with a customer of Intellect and other third parties in...
-
Senior Data Engineer
Found in: Talent IN 2A C2 - 3 weeks ago
Chennai, India hubwork Full timeSummary :We are looking for a data engineer who has experience in building data pipelines for large data platforms. This role will be working with a team building a large data platform catering to the needs of a wealth manager for their application and analytics needs. This role will be augmented with a customer of Intellect and other third parties in...
-
Senior Data Engineer
Found in: Adzuna IN C2 - 2 weeks ago
Chennai, Tamil Nadu, India hubwork Full timeSummary :We are looking for a data engineer who has experience in building data pipelines for large data platforms. This role will be working with a team building a large data platform catering to the needs of a wealth manager for their application and analytics needs. This role will be augmented with a customer of Intellect and other third parties in...
-
Big Data Developer
Found in: Whatjobs IN C2 - 2 weeks ago
Chennai, India RiskInsight Consulting Pvt Ltd Full timeResponsibilitiesDesign, develop, and deploy scalable Big Data applications using Apache Spark.Collaborate with data scientists and business analysts to understand requirements and translate them into technical solutions.Write efficient and optimized code to process and analyze large volumes of data.Implement data ingestion processes from various data sources...
-
Big Data Tester
4 days ago
Chennai, India Durus Consulting Private Limited Full time**JOB DESCRIPTION FOR BIG DATA ENGINEER** **Job Role ** : Big Data Testing **Job Type** : FTE **Location** : Hybrid **Notice Period** : Immediate to 30 days **Experience ** : 3+ YRS **Desired Skills and Experience.** - Demonstrated testing & automation experience of 3-6 Years in Big data technology stack - Hadoop, HDFS & Spark - Good understanding of...
-
Data Engineer, VP
Found in: beBee S IN - 2 weeks ago
Chennai, India NatWest Digital X Full timeJoin us as a Data EngineerYou’ll be the voice of our customers, using data to tell their stories and put them at the heart of all decision-makingWe’ll look to you to drive the build of effortless, digital first customer experiencesIf you’re ready for a new challenge and want to make a far-reaching impact through your work, this could be the opportunity...
-
Data Engineer, VP
Found in: beBee jobs IN - 2 weeks ago
Chennai, Tamil Nadu, India NatWest Digital X Full timeJoin us as a Data EngineerYou'll be the voice of our customers, using data to tell their stories and put them at the heart of all decision-makingWe'll look to you to drive the build of effortless, digital first customer experiencesIf you're ready for a new challenge and want to make a far-reaching impact through your work, this could be the opportunity...
-
Big Data Developer
Found in: Whatjobs IN C2 - 4 days ago
Chennai, India RiskInsight Consulting Pvt Ltd. Full timeResponsibilitiesDesign, develop, and deploy scalable Big Data applications using Apache Spark Collaborate with data scientists and business analysts to understand requirements and translate them into technical solutions Write efficient and optimized code to process and analyze large volumes of data Implement data ingestion processes from various data sources...
-
Big Data Developer
Found in: Talent IN 2A C2 - 5 days ago
Chennai, India RiskInsight Consulting Pvt Ltd. Full timeResponsibilitiesDesign, develop, and deploy scalable Big Data applications using Apache SparkCollaborate with data scientists and business analysts to understand requirements and translate them into technical solutionsWrite efficient and optimized code to process and analyze large volumes of dataImplement data ingestion processes from various data sources to...
-
Big Data Developer
Found in: Appcast Linkedin IN C2 - 5 days ago
Chennai, India RiskInsight Consulting Pvt Ltd. Full timeResponsibilitiesDesign, develop, and deploy scalable Big Data applications using Apache SparkCollaborate with data scientists and business analysts to understand requirements and translate them into technical solutionsWrite efficient and optimized code to process and analyze large volumes of dataImplement data ingestion processes from various data sources to...
-
Big Data Developer
Found in: Talent IN C2 - 3 weeks ago
Chennai, India RiskInsight Consulting Pvt Ltd Full timeResponsibilities Design, develop, and deploy scalable Big Data applications using Apache Spark. Collaborate with data scientists and business analysts to understand requirements and translate them into technical solutions. Write efficient and optimized code to process and analyze large volumes of data. Implement data ingestion processes from various data...