Data Streaming

2 weeks ago

Pune, India ACE Talent Consulting Full time

Job Description:
Spark/Scala/PySpark developer who knows how to fully exploit the potential of our Spark cluster.

Should have ability to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data.

**Responsibilities**:
Create Scala/Spark/Pyspark jobs for data transformation and aggregation

Produce unit tests for Spark transformations and helper methods

Write Scaladoc-style documentation with all code

Design data processing pipelines

**Skills**:Pyspark

Scala (with a focus on the functional programming paradigm)

Apache Spark 2.x, 3.x
- Apache Spark RDD API
- Apache Spark SQL DataFrame API
- Apache Spark Streaming API

Spark query tuning and performance optimization

SQL database integration (Postgres, and/or MySQL)

Experience working with HDFS, AWS ( S3, Redshift, EMR, IAM, Polices, Routing)

CI-CD Pipleline, Jenkins, Gitlab /Bitbucket

Deep understanding of distributed systems (e.g. partitioning, replication, consistency, and consensus)

**Salary**: ₹300,000.00 - ₹1,500,000.00 per year

Schedule:

- Day shift

Ability to commute/relocate:

- Pune, Maharashtra: Reliably commute or planning to relocate before starting work (required)

**Experience**:

- total work: 3 years (preferred)

Work Location: One location

  • Data Engineer

    Found in: Talent IN 2A C2 - 3 weeks ago

    Pune, India gethyr Full time

    We are seeking a talented and motivated Data Engineer stream processing, particularly utilizing Flink and Kafka, alongside expertise in Spark and AWS technologies. As a Data Engineer, you will play a crucial role in designing, implementing, and maintaining robust stream processing solutions to handle large volumes of data in real-time, ensuring high...

  • Data Engineer

    Found in: Whatjobs IN C2 - 3 weeks ago

    Bangalore,Pune, India gethyr Full time

    We are seeking a talented and motivated Data Engineer stream processing, particularly utilizing Flink and Kafka, alongside expertise in Spark and AWS technologies. As a Data Engineer, you will play a crucial role in designing, implementing, and maintaining robust stream processing solutions to handle large volumes of data in real-time, ensuring high...

  • Data Engineer

    Found in: Whatjobs IN C2 - 3 days ago

    Bangalore/Pune, India Gethyr Full time

    We are seeking a talented and motivated Data Engineer stream processing, particularly utilizing Flink and Kafka, alongside expertise in Spark and AWS technologies. As a Data Engineer, you will play a crucial role in designing, implementing, and maintaining robust stream processing solutions to handle large volumes of data in real-time, ensuring high...

  • Data Engineer

    Found in: Adzuna IN C2 - 2 weeks ago

    Bangalore/Pune, IN Gethyr Full time

    We are seeking a talented and motivated Data Engineer stream processing, particularly utilizing Flink and Kafka, alongside expertise in Spark and AWS technologies. As a Data Engineer, you will play a crucial role in designing, implementing, and maintaining robust stream processing solutions to handle large volumes of data in real-time, ensuring high...

  • Data Engineer

    Found in: Whatjobs IN C2 - 2 weeks ago

    Pune, India Kiash Solutions LLP Full time

    Role-Data EngineerExperience Level: 5+Should have the following Mandatory skills:Good working experience in SOLR and Elastic SearchGood in Machine learning & NLP techniques with work experience.Excellent experience in data processing using PythonExperience with relational SQL, NoSQL DB, Streaming and ETL.Experience with integration of data from multiple data...

  • Senior Data Engineer

    Found in: Whatjobs IN C2 - 3 weeks ago

    Pune, India Synechron Full time

    We are hiringSenior Data Engineers (PySpark)at Synechron Technologies.Job Title : Senior Data Engineers (PySpark)Experience : 3+ YearsNotice Period: Looking for immediate to 30 days' joinersJob Location : Hinjewadi, Pune (Hybrid Work Model)Software Requirements:Proficient in Python, PySpark, and SQLExperience in the build and deployment of Big Data...

  • Data Engineer

    Found in: Talent IN C2 - 3 weeks ago

    Pune, India Kiash Solutions LLP Full time

    Role- Data EngineerExperience Level: 5+ Should have the following Mandatory skills: Good working experience in SOLR and Elastic Search Good in Machine learning & NLP techniques with work experience. Excellent experience in data processing using Python Experience with relational SQL, NoSQL DB,...

  • Principal Data Engineer

    Found in: Appcast Linkedin IN C2 - 3 weeks ago

    Pune, India AppZen Full time

    Must Haves:Solid understanding of data fundamentals and statistical processesAbility to interpret and analyze large data setsCandidate must have experience in RDBMS (Postgres, Oracle, MySQL) database and turning SQL queriesA great candidate will have broad understanding of of all these technologies and in depth knowledge of some:Expertise with data...

  • Principal Data Engineer

    Found in: Whatjobs IN C2 - 3 weeks ago

    Pune, India AppZen Full time

    Must Haves: Solid understanding of data fundamentals and statistical processes Ability to interpret and analyze large data sets Candidate must have experience in RDBMS (Postgres, Oracle, MySQL) database and turning SQL queries A great candidate will have broad understanding of of all these technologies and in depth knowledge of some: Expertise with data...

  • Principal Data Engineer

    Found in: Talent IN 2A C2 - 3 weeks ago

    Pune, India AppZen Full time

    Must Haves:Solid understanding of data fundamentals and statistical processesAbility to interpret and analyze large data setsCandidate must have experience in RDBMS (Postgres, Oracle, MySQL) database and turning SQL queriesA great candidate will have broad understanding of of all these technologies and in depth knowledge of some:Expertise with data...

  • Data Engineer

    Found in: Talent IN 2A C2 - 3 weeks ago

    Pune, India Huquo Full time

    Basic understanding of :- Scheduling and workflow management & working experience in either ADF, Informatica, Airflow, or Similar- Enterprise Data Modelling and Semantic Modelling & working experience in ERwin, ER/Studio, PowerDesigner, or Similar- Logical/Physical model on Big Data sets or modern data warehouse & working experience in ERwin, ER/Studio,...

  • Senior Data Engineer

    Found in: Whatjobs IN C2 - 14 hours ago

    pune, India Synechron Full time

    We are hiring Senior Data Engineers (PySpark) at Synechron Technologies.Job Title : Senior Data Engineers (PySpark)Experience : 3+ YearsNotice Period: Looking for immediate to 30 days' joinersJob Location : Hinjewadi, Pune (Hybrid Work Model)Software Requirements:Proficient in Python, PySpark, and SQLExperience in the build and deployment of Big Data...

  • Senior Data Engineer

    Found in: Whatjobs IN C2 - 2 weeks ago

    Pune, India Siemens Energy Full time

    Senior Data EngineerAbout the RoleLocationIndia Maharashtra Pune Company Siemens Energy Industrial Turbomachinery India Private Limited Organization Transformation of Industry Business Unit Compression Full / Part time Full-time Experience Level Mid-level Professional Profile Overview: Title: Senior Data Engineer Location: Pune, India Role Description:...

  • Data Architect

    Found in: Whatjobs IN C2 - 4 days ago

    Pune, India Celebal Technologies Full time

    Overview:As a Data Architect you will work with multiple teams to deliver solutions on the Azure Cloud using core cloud data warehouse tools (Azure Data Factory, Azure Databricks, Azure Synapse Analytics and other Big Data related technologies). In addition to building the next generation of application data platforms (not infrastructure) and/or improving...

  • Senior Data Engineer

    Found in: Whatjobs IN C2 - 3 weeks ago

    pune, India Siemens Energy Full time

    Senior Data Engineer About the Role Location India Maharashtra Pune Company Siemens Energy Industrial Turbomachinery India Private Limited Organization Transformation of Industry Business Unit Compression Full / Part time Full-time Experience Level Mid-level Professional Profile Overview: Title: Senior Data Engineer Location: Pune,...

  • Senior Data Engineer

    Found in: Talent IN C2 - 3 weeks ago

    Pune, India Siemens Energy Full time

    Senior Data Engineer About the Role Location India Maharashtra Pune Company Siemens Energy Industrial Turbomachinery India Private Limited Organization Transformation of Industry Business Unit Compression Full / Part time Full-time Experience Level Mid-level Professional Profile Overview: Title: Senior Data Engineer Location: Pune, India Role...

  • Senior Data Engineer

    Found in: Whatjobs IN C2 - 3 weeks ago

    Pune, India Synechron Full time

    We are hiring Senior Data Engineers (PySpark) at Synechron Technologies. Job Title : Senior Data Engineers (PySpark) Experience : 3+ Years Notice Period: Looking for immediate to 30 days' joiners Job Location : Hinjewadi, Pune (Hybrid Work Model) Software Requirements: Proficient in Python, PySpark, and SQL Experience in the build and deployment of Big...

  • Senior Data Engineer

    Found in: Talent IN 2A C2 - 3 weeks ago

    Pune, India Synechron Full time

    We are hiring Senior Data Engineers (PySpark) at Synechron Technologies.Job Title : Senior Data Engineers (PySpark)Experience : 3+ YearsNotice Period: Looking for immediate to 30 days' joinersJob Location : Hinjewadi, Pune (Hybrid Work Model)Software Requirements:Proficient in Python, PySpark, and SQLExperience in the build and deployment of Big Data...

  • Senior Data Engineer

    Found in: Appcast Linkedin IN C2 - 3 weeks ago

    Pune, India Synechron Full time

    We are hiring Senior Data Engineers (PySpark) at Synechron Technologies.Job Title : Senior Data Engineers (PySpark)Experience : 3+ YearsNotice Period: Looking for immediate to 30 days' joinersJob Location : Hinjewadi, Pune (Hybrid Work Model)Software Requirements:Proficient in Python, PySpark, and SQLExperience in the build and deployment of Big Data...

  • Principal Data Engineer

    Found in: Whatjobs IN C2 - 3 weeks ago

    Pune, India AppZen Full time

    Must Haves:Solid understanding of data fundamentals and statistical processesAbility to interpret and analyze large data setsCandidate must have experience in RDBMS (Postgres, Oracle, MySQL) database and turning SQL queriesA great candidate will have broad understanding of of all these technologies and in depth knowledge of some:Expertise with data...