PySpark With Scala

1 day ago

Bengaluru Hyderabad Pune, India Infosys Full time ₹ 5,00,000 - ₹ 25,00,000 per year

We are seeking an experienced Big Data Developer with strong expertise in Apache Spark, Scala, and Python (PySpark).

The ideal candidate will design and develop data processing pipelines, ETL workflows, and distributed applications, ensuring scalability and performance in large-scale data environments.

Key Responsibilities:

  • Develop and maintain data pipelines using Apache Spark, Scala, and PySpark.
  • Apply functional programming principles for clean and efficient Scala code.
  • Optimize Spark jobs for performance, cost efficiency, and scalability.
  • Collaborate with data engineers, data scientists, and cloud teams to deliver robust solutions.
  • Integrate Spark applications with data lakes, streaming platforms (Kafka), and cloud services.
  • Implement unit tests, perform code reviews, and follow best practices in software development.

Required Skills:

  • Strong proficiency in Scala and Python (PySpark).
  • Hands-on experience with Apache Spark (Core, SQL, Streaming).
  • Knowledge of functional programming concepts in Scala.
  • Familiarity with data lake architectures, HDFS, and Hive.
  • Proficiency in SQL and data modeling.
  • Experience with CI/CD pipelines, Git, and cloud platforms (AWS, Azure, GCP).

Preferred Skills:

  • Experience with Databricks and Delta Lake.
  • Knowledge of Akka, Play Framework, or reactive programming.
  • Exposure to containerization (Docker, Kubernetes).

Location: Bangalore, Hyderabad, Trivandrum, Pune, Bhubaneshwar

Experience: 3-20 Years

  • Data Engineer

    1 week ago

    Bengaluru, India BlueByte Technologies Full time

    Description :Key Responsibilities :- Design, develop, and optimize large-scale data pipelines using PySpark and Scala.- Implement ETL processes on big data platforms such as Hadoop and Azure Data Lake.- Work with Azure services like Azure Databricks, Azure Data Factory, Azure Synapse Analytics, and Azure Blob Storage.- Develop, test, and maintain data...

  • Spark, PySpark, Scala with Azure

    2 weeks ago

    Bengaluru, Chennai, Hyderabad, India Infosys Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Role & responsibilitiesEngage with Business / Stakeholders with status update on progress of development and issue fixSetup Standards related to Coding, Pipelines and DocumentationUnderstand the requirements from the business and translate it into an appropriate technical requirement.Define key business problems to be solved; formulate high level solution...

  • Python pyspark

    2 days ago

    Hyderabad, India Virtusa Full time

    Python pyspark - CREQ Description Design and develop on Hadoop applications. Hands on in developing Jobs in pySpark with Python/ SCALA (Preferred) or Java/ SCALA. Experience on Core Java, Experience on Map Reduce programs, Hive programming, Hive queries performance concepts. Experience on source code management with Git repositories. Secondary skills...

  • Pyspark developer

    1 week ago

    Bengaluru, India Tata Consultancy Services Full time

    The ideal candidate will be responsible for developing high-quality applications. They will also be responsible for designing and implementing testable and scalable code. Functional Skills: Experience in Credit Risk/Regulatory risk domain Technical Skills: Spark ,PySpark, Python, Hive, Scala, MapReduce, Unix shell scripting Good to Have Skills: Exposure to...

  • Pyspark developer

    6 days ago

    Bengaluru, India Tata Consultancy Services Full time

    The ideal candidate will be responsible for developing high-quality applications. They will also be responsible for designing and implementing testable and scalable code. Functional Skills: Experience in Credit Risk/Regulatory risk domain Technical Skills: Spark ,PySpark, Python, Hive, Scala, MapReduce, Unix shell scripting Good to Have Skills: Exposure to...

  • Pyspark Data Engineer

    1 week ago

    Hyderabad, Pune, India Krazy Mantra HR Solutions Pvt. Ltd Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    We are looking for a skilled Pyspark Data Engineer with 3 to 5 years of experience. The ideal candidate will have expertise in Hadoop, Spark/Pyspark, Hive, YARN, Scala development, and data modeling. This position is based in Pune and Hyderabad.Roles and ResponsibilityDesign and develop scalable data pipelines using Pyspark and Spark.Collaborate with...

  • Big Data Engineer

    4 days ago

    Hyderabad, India Sheryl strategic solutions Pvt. LTD. Full time

    Description : Position : Big Data EngineerLocation : Hyderabad, India (Hybrid, 2-3 times per week on any weekdays)Duration : 1 year+Job Summary : Experian FSD is seeking a skilled and motivated Big Data Engineer to join our dynamic data engineering team. This hybrid role based in Hyderabad offers the opportunity to work on cutting-edge data platforms and...

  • Python Pyspark

    1 week ago

    Bengaluru, Karnataka, India Tata Consultancy Services (TCS) Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    Greetings from Tata Consultancy Services (TCS)Location : Chennai, Bangalore, Pune, DelhiExperience : 5-10 YearsRole: Python Pyspark DeveloperRequired Skill set Python, Pyspark, Scala, SparkMust HaveStrong analytical skills in conducting sophisticated statistical analysis using bureau/vendor data, customer performances.Working experience in languages PySpark...

  • Pyspark developer

    2 weeks ago

    Bengaluru, India Tata Consultancy Services Full time

    The ideal candidate will be responsible for developing high-quality applications. They will also be responsible for designing and implementing testable and scalable code.Functional Skills: Experience in Credit Risk/Regulatory risk domain Technical Skills: Spark ,PySpark, Python, Hive, Scala, MapReduce, Unix shell scripting Good to Have Skills: Exposure to...

  • Pyspark developer

    1 week ago

    Bengaluru, India Tata Consultancy Services Full time

    The ideal candidate will be responsible for developing high-quality applications. They will also be responsible for designing and implementing testable and scalable code.Functional Skills: Experience in Credit Risk/Regulatory risk domain Technical Skills: Spark ,PySpark, Python, Hive, Scala, MapReduce, Unix shell scripting Good to Have Skills: Exposure to...