Data Streaming

2 weeks ago


Chennai, India ACE Talent Consulting Full time

Job Description:
Spark/Scala/PySpark developer who knows how to fully exploit the potential of our Spark cluster.

Should have ability to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data.

**Responsibilities**:
Create Scala/Spark/Pyspark jobs for data transformation and aggregation

Produce unit tests for Spark transformations and helper methods

Write Scaladoc-style documentation with all code

Design data processing pipelines

**Skills**:Pyspark

Scala (with a focus on the functional programming paradigm)

Apache Spark 2.x, 3.x
- Apache Spark RDD API
- Apache Spark SQL DataFrame API
- Apache Spark Streaming API

Spark query tuning and performance optimization

SQL database integration (Postgres, and/or MySQL)

Experience working with HDFS, AWS ( S3, Redshift, EMR, IAM, Polices, Routing)

CI-CD Pipleline, Jenkins, Gitlab /Bitbucket

Deep understanding of distributed systems (e.g. partitioning, replication, consistency, and consensus)

**Salary**: ₹300,000.00 - ₹1,500,000.00 per year

Schedule:

- Day shift

Ability to commute/relocate:

- Chennai, Tamil Nadu: Reliably commute or planning to relocate before starting work (required)

**Experience**:

- total work: 3 years (preferred)

Work Location: One location


  • Data Scientist

    6 days ago


    Chennai, India Crayon Data Full time

    Role: Data Scientist Experience level: 3+ years Location: Chennai Who are we? Crayon Data is a leading provider of AI-led revenue acceleration solutions, headquartered in Singapore with a local presence in India and the UAE. The company was founded in 2012 with the vision of simplifying the world’s choices. Our flagship platform, maya.ai, helps...

  • Data Architect

    3 weeks ago


    Chennai, India iXceed Solutions Full time

    **Job Title**:Data Architect **Job Location**:WFH from India **Job Mode**: Remote (need to follow Europe Time zone) Primarily experts of **Spark 3.X, SCALA,** Delta lake implementation, streaming solution implementation for IOT in Spark streaming. Kafka expertise. Any experience with MFG BI, DWH and Datalake implementation would be a bonus. **Job Type**:...

  • Data Engineer

    Found in: Whatjobs IN C2 - 3 weeks ago


    Chennai, India Wenger & Watson Inc Full time

    Exp-3+ yrsLocation-Chennai(WFO)compulsory work from office (1st and last saturday working)notice period-30-immediate joinerJob DescriptionShould have a minimum 6+ years in Data Engineering, Data Analytics platform. Should have strong hands-on design and engineering background in AWS, across a wide range of AWS services with the ability to demonstrate...

  • Data Engineer

    Found in: Whatjobs IN C2 - 3 weeks ago


    Chennai, India Wenger & Watson Inc Full time

    Exp-3+ yrs Location-Chennai(WFO) compulsory work from office (1st and last saturday working) notice period-30-immediate joiner Job Description Should have a minimum 6+ years in Data Engineering, Data Analytics platform.  Should have strong hands-on design and engineering background in AWS, across a wide range of AWS services with the ability to...

  • Data Engineer

    Found in: Talent IN 2A C2 - 3 weeks ago


    Chennai, India Wenger & Watson Inc Full time

    Exp-3+ yrsLocation-Chennai(WFO)compulsory work from office (1st and last saturday working)notice period-30-immediate joinerJob DescriptionShould have a minimum 6+ years in Data Engineering, Data Analytics platform. Should have strong hands-on design and engineering background in AWS, across a wide range of AWS services with the ability to demonstrate...

  • Data Engineer

    Found in: Appcast Linkedin IN C2 - 3 weeks ago


    Chennai, India Wenger & Watson Inc Full time

    Exp-3+ yrsLocation-Chennai(WFO)compulsory work from office (1st and last saturday working)notice period-30-immediate joinerJob DescriptionShould have a minimum 6+ years in Data Engineering, Data Analytics platform. Should have strong hands-on design and engineering background in AWS, across a wide range of AWS services with the ability to demonstrate...

  • AWS Data Engineer

    Found in: Whatjobs IN C2 - 7 days ago


    Chennai, India CliqHR Full time

    Job Description : - Must have minimum 5+ years of experience as AWS Data EngineerNice to have : AWS Certified Data Analytics Specialty certification- Experience in designing and implementing data pipelines using AWS Glue and orchestration using Airflow for batch, streaming (Kafka/Kinesis) and API ingestion patterns- Work closely with DevOps engineer to...

  • AWS Data Engineer

    Found in: Talent IN 2A C2 - 1 week ago


    Chennai, India CliqHR Full time

    Job Description :- Must have minimum 5+ years of experience as AWS Data EngineerNice to have : AWS Certified Data Analytics Specialty certification- Experience in designing and implementing data pipelines using AWS Glue and orchestration using Airflow for batch, streaming (Kafka/Kinesis) and API ingestion patterns- Work closely with DevOps engineer to...

  • AWS Data Engineer

    Found in: Adzuna IN C2 - 7 days ago


    Chennai, Tamil Nadu, India CliqHR Full time

    Job Description :- Must have minimum 5+ years of experience as AWS Data EngineerNice to have : AWS Certified Data Analytics Specialty certification- Experience in designing and implementing data pipelines using AWS Glue and orchestration using Airflow for batch, streaming (Kafka/Kinesis) and API ingestion patterns- Work closely with DevOps engineer to...

  • Senior Data Engineer

    Found in: Whatjobs IN C2 - 3 weeks ago


    Chennai, India hubwork Full time

    Summary :We are looking for a data engineer who has experience in building data pipelines for large data platforms. This role will be working with a team building a large data platform catering to the needs of a wealth manager for their application and analytics needs. This role will be augmented with a customer of Intellect and other third parties in...

  • Senior Data Engineer

    Found in: Talent IN 2A C2 - 3 weeks ago


    Chennai, India hubwork Full time

    Summary :We are looking for a data engineer who has experience in building data pipelines for large data platforms. This role will be working with a team building a large data platform catering to the needs of a wealth manager for their application and analytics needs. This role will be augmented with a customer of Intellect and other third parties in...

  • Senior Data Engineer

    Found in: Adzuna IN C2 - 2 weeks ago


    Chennai, Tamil Nadu, India hubwork Full time

    Summary :We are looking for a data engineer who has experience in building data pipelines for large data platforms. This role will be working with a team building a large data platform catering to the needs of a wealth manager for their application and analytics needs. This role will be augmented with a customer of Intellect and other third parties in...

  • Big Data Developer

    Found in: Whatjobs IN C2 - 2 weeks ago


    Chennai, India RiskInsight Consulting Pvt Ltd Full time

    ResponsibilitiesDesign, develop, and deploy scalable Big Data applications using Apache Spark.Collaborate with data scientists and business analysts to understand requirements and translate them into technical solutions.Write efficient and optimized code to process and analyze large volumes of data.Implement data ingestion processes from various data sources...

  • Big Data Tester

    4 days ago


    Chennai, India Durus Consulting Private Limited Full time

    **JOB DESCRIPTION FOR BIG DATA ENGINEER** **Job Role ** : Big Data Testing **Job Type** : FTE **Location** : Hybrid **Notice Period** : Immediate to 30 days **Experience ** : 3+ YRS **Desired Skills and Experience.** - Demonstrated testing & automation experience of 3-6 Years in Big data technology stack - Hadoop, HDFS & Spark - Good understanding of...

  • Data Engineer, VP

    Found in: beBee S IN - 2 weeks ago


    Chennai, India NatWest Digital X Full time

    Join us as a Data EngineerYou’ll be the voice of our customers, using data to tell their stories and put them at the heart of all decision-makingWe’ll look to you to drive the build of effortless, digital first customer experiencesIf you’re ready for a new challenge and want to make a far-reaching impact through your work, this could be the opportunity...

  • Data Engineer, VP

    Found in: beBee jobs IN - 2 weeks ago


    Chennai, Tamil Nadu, India NatWest Digital X Full time

    Join us as a Data EngineerYou'll be the voice of our customers, using data to tell their stories and put them at the heart of all decision-makingWe'll look to you to drive the build of effortless, digital first customer experiencesIf you're ready for a new challenge and want to make a far-reaching impact through your work, this could be the opportunity...

  • Big Data Developer

    Found in: Whatjobs IN C2 - 4 days ago


    Chennai, India RiskInsight Consulting Pvt Ltd. Full time

    ResponsibilitiesDesign, develop, and deploy scalable Big Data applications using Apache Spark Collaborate with data scientists and business analysts to understand requirements and translate them into technical solutions Write efficient and optimized code to process and analyze large volumes of data Implement data ingestion processes from various data sources...

  • Big Data Developer

    Found in: Talent IN 2A C2 - 5 days ago


    Chennai, India RiskInsight Consulting Pvt Ltd. Full time

    ResponsibilitiesDesign, develop, and deploy scalable Big Data applications using Apache SparkCollaborate with data scientists and business analysts to understand requirements and translate them into technical solutionsWrite efficient and optimized code to process and analyze large volumes of dataImplement data ingestion processes from various data sources to...

  • Big Data Developer

    Found in: Appcast Linkedin IN C2 - 5 days ago


    Chennai, India RiskInsight Consulting Pvt Ltd. Full time

    ResponsibilitiesDesign, develop, and deploy scalable Big Data applications using Apache SparkCollaborate with data scientists and business analysts to understand requirements and translate them into technical solutionsWrite efficient and optimized code to process and analyze large volumes of dataImplement data ingestion processes from various data sources to...

  • Big Data Developer

    Found in: Talent IN C2 - 3 weeks ago


    Chennai, India RiskInsight Consulting Pvt Ltd Full time

    Responsibilities Design, develop, and deploy scalable Big Data applications using Apache Spark. Collaborate with data scientists and business analysts to understand requirements and translate them into technical solutions. Write efficient and optimized code to process and analyze large volumes of data. Implement data ingestion processes from various data...