PySpark Data Engineer

3 days ago

Bengaluru, Karnataka, India BLS360 Full time

Role Overview

This is a contract remote role. We are seeking a highly motivated PySpark Data Engineer with 3–4 years of experience in building, optimizing, and maintaining large-scale data processing solutions. The ideal candidate will bring hands-on expertise in PySpark, Python, SQL, and real-time data streaming and will contribute to the design and implementation of scalable data pipelines in cloud environments such as Azure.

·         Location: India. Remote.

·         Job type: Contract

Key Responsibilities

  • Design, develop, and optimize data pipelines using PySpark, Spark, and Python.
  • Implement ETL/ELT solutions to integrate structured and unstructured data from multiple sources.
  • Develop and maintain Hive queries and ensure seamless data transformations.
  • Write and optimize SQL queries for large datasets.
  • Build, monitor, and support real-time streaming solutions using Apache Kafka.
  • Work on Databricks to create, manage, and optimize data workflows.
  • Deploy and manage solutions in Azure cloud services (e.g., Data Lake, Synapse, Azure Databricks).
  • Debug, troubleshoot, and optimize code for performance and scalability.
  • Collaborate with cross-functional teams to deliver data-driven solutions.
  • Adhere to best practices for coding, testing, and deployment.

Required Skills & Qualifications

  • 3–4 years of experience in data engineering with a focus on PySpark and Python.
  • Strong knowledge of Spark, Hive, SQL, and Unix shell scripting.
  • Experience with Databricks in a production environment.
  • Hands-on experience with Azure cloud services (preferred: Data Lake, Synapse, Databricks).
  • Experience working with Apache Kafka or other real-time streaming technologies.
  • Strong problem-solving and debugging skills.
  • Excellent written and verbal communication skills.
  • Ability to work independently in a fast-paced environment and adapt to new technologies quickly.

Nice-to-Have Skills

  • Exposure to CI/CD pipelines for data solutions.
  • Knowledge of data governance and security best practices.
  • Experience with Agile/Scrum methodology.

  • Data Engineer- PySpark

    1 week ago

    Bengaluru, Karnataka, India Valuelabs Full time US$ 90,000 - US$ 1,20,000 per year

    Job Title- PySpark Data EngineerWe're growing our Data Engineering team at ValueLabs and looking for a talented individual to build scalable data pipelines on Cloudera Data PlatformExperience- 5years to 9years.Pyspark Job Description:Data Pipeline Development: Design, develop, and maintain highly scalable and optimized ETL pipelines using PySpark on the...

  • Data Engineer, PySpark

    2 weeks ago

    Bengaluru, Karnataka, India RBS Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Join us as a Data Engineer, PySparkThis is an exciting opportunity to use your technical expertise to collaborate with colleagues and build effortless, digital first customer experiencesYou'll be simplifying the bank by developing innovative data driven solutions, using insight to be commercially successful, and keeping our customers' and the bank's data...

  • Data Engineer, PySpark

    1 week ago

    Bengaluru, Karnataka, India NatWest Group Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Join us as a Data Engineer, PySparkThis is an exciting opportunity to use your technical expertise to collaborate with colleagues and build effortless, digital first customer experiencesYou'll be simplifying the bank by developing innovative data driven solutions, using insight to be commercially successful, and keeping our customers' and the bank's data...

  • Experienced PySpark Engineer

    2 weeks ago

    Bengaluru, Karnataka, India beBeePySparkEngineer Full time ₹ 9,00,000 - ₹ 12,00,000

    Job Title: Experienced PySpark EngineerJob Summary:Our organization is seeking an experienced and highly skilled PySpark engineer to join our team. As a key member of the data engineering team, you will be responsible for designing, developing, and maintaining large-scale data processing systems using PySpark.Key Responsibilities:Design and implement complex...

  • Pyspark

    6 days ago

    Bengaluru, Karnataka, India Virtusa Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    highly skilled Data Engineer with 6+ years of experience in the IT industry, specializing in Data Engineering and Data Migration. The ideal candidate will have strong expertise in Oracle to PostgreSQL migration and be well-versed in AWS DMS (Database Migration Service). The role requires proficiency in Python and PySpark to work on various data migration...

  • Python Pyspark Data Engineer

    1 week ago

    Bengaluru, Karnataka, India DXC Technology Full time

    Python Pyspark Data Engineer Job Location Hyderabad Bangalore Chennai Kolkata Noida Gurgaon Pune Indore Mumbai Strong Python Skills - Proficient in Python for data manipulation automation and building reusable components Data Pipeline Development - Experience designing and maintaining ETL data pipelines using tools like Airflow or...

  • Pyspark Developer

    3 days ago

    Bengaluru, Karnataka, India Synechron Technologies Pvt. Ltd. Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Day-to-Day Activities:Design, develop, and maintain ETL pipelines using PySpark on CDP.Implement and manage data ingestion processes from various sources. Process, cleanse, and transform large datasets using PySpark. Conduct performance tuning and optimization of ETL processes. Implement data quality checks and validation routines. Automate data workflows...

  • Pyspark Architect

    2 weeks ago

    Bengaluru, Karnataka, India Talent Worx Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Job Description: PySpark ArchitectLocation: Bangalore,Pune,Kolkata Experience Required: 7 to 13 YearsRole Overview:We are seeking a highly skilled and motivated PySpark Manager to lead our dataengineering initiatives. The ideal candidate will have deep expertise in PySpark,Hadoop, AWS, Teradata, Scala, and data visualization tools. You will be responsible...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 1,04,000 - ₹ 1,30,878

    Job Title: Data Engineer (Spark & PySpark)About the RoleWe are seeking a highly skilled and experienced Data Engineer to join our team. As a key member of our data engineering group, you will be responsible for designing, developing, and maintaining large-scale data processing systems using Apache Spark and Python.Your primary focus will be on developing and...

  • Data Engineer

    4 days ago

    Bengaluru, Karnataka, India Golden Opportunities Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Job DescriptionJob title:Data Engineer ( Python + PySpark + SQL )Candidate Specification: Minimum 6 to 8 years of experience in Data EngineerJob DescriptionData Engineer with strong expertise in Python, PySpark, and SQL.Design, develop, and maintain robust data pipelines using PySpark and Python.Strong understanding of SQL and relational databases (e.g.,...