PySpark Developer

3 days ago

Chennai, Tamil Nadu, India Risk Resources LLP Full time ₹ 10,000 - ₹ 80,000 per year
Overview The PySpark Developer plays a crucial role in leveraging big data technologies to analyze and transform data, thereby driving informed business decisions. This position focuses on the design, development, and optimization of data processing applications using PySpark, a Python API for Apache Spark. In the fast-paced tech environment, the PySpark Developer will work alongside data engineers, data scientists, and other IT professionals to integrate large data sets from various sources, ensuring that data is processed efficiently and accurately. This role is vital for organizations looking to harness the power of big data, enabling them to gain insights and improve their operations through data-driven strategies. The ideal candidate will possess a strong technical background, excellent problem-solving skills, and a passion for working with data. Key Responsibilities
  • Design and implement scalable data processing applications using PySpark.
  • Develop ETL pipelines for data integration and transformation.
  • Collaborate with data architects to enhance data modeling for optimal performance.
  • Ensure data quality and integrity throughout the data lifecycle.
  • Integrate data from various sources, including structured and unstructured data.
  • Write efficient SQL queries for data retrieval and analysis.
  • Conduct performance tuning and optimization of Spark applications.
  • Monitor and troubleshoot data processing jobs to ensure reliability.
  • Work with cloud services (AWS, Azure, or GCP) for data processing and storage.
  • Participate in code reviews and provide feedback to peers.
  • Document processes, designs, and solutions for future reference.
  • Stay current with emerging technologies and best practices in big data.
  • Collaborate in an Agile team environment and participate in sprint planning.
  • Provide support for production data processing applications and resolve issues.
  • Train and mentor junior team members on PySpark best practices.
Required Qualifications
  • Bachelor's degree in Computer Science, Information Technology, or a related field.
  • 3+ years of experience in data engineering or software development.
  • Hands-on experience with PySpark and Apache Spark framework.
  • Proficiency in SQL and familiarity with relational databases.
  • Experience with ETL and data warehousing solutions.
  • Knowledge of cloud computing platforms (AWS, Azure, GCP).
  • Strong programming skills in Python and familiarity with Scala or Java.
  • Understanding of data processing frameworks and big data technologies.
  • Experience with version control systems (e.g., Git).
  • Familiarity with data visualization tools is a plus.
  • Strong analytical and problem-solving skills.
  • Excellent verbal and written communication skills.
  • Ability to work effectively in a collaborative team environment.
  • Experience with Agile development practices.
  • Knowledge of machine learning concepts is desirable.

  • Pyspark Developer

    5 days ago

    Chennai, Tamil Nadu, India viraaj hr solutions Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    Primary Job Title: PySpark DeveloperRole & ResponsibilitiesDesign, develop and maintain scalable PySpark ETL pipelines for batch and streaming workloads across data lakes and warehouses.Ingest and transform data from diverse sources (Kafka, S3, HDFS, RDBMS) into analytics-ready formats using PySpark and Spark SQL.Optimize Spark jobs for performance and...

  • Pyspark Developer

    1 week ago

    Chennai, Tamil Nadu, India Citi Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Discover your future at CitiWorking at Citi is far more than just a job. A career with us means joining a team of more than 230,000 dedicated people from around the globe. At Citi, you'll have the opportunity to grow your career, give back to your community and make a real impact.Job OverviewAt Citi we're not just building technology, we're building the...

  • Pyspark Developer

    1 day ago

    Chennai, Tamil Nadu, India Citi Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Discover your future at CitiWorking at Citi is far more than just a job. A career with us means joining a team of more than 230,000 dedicated people from around the globe. At Citi, you'll have the opportunity to grow your career, give back to your community and make a real impact.Job OverviewAt Citi we're not just building technology, we're building the...

  • Pyspark

    2 weeks ago

    Chennai, Tamil Nadu, India Cognizant Technology Solutions Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job SummaryAs a Sr. Developer you will play a pivotal role in optimizing and enhancing our data processing capabilities using PySpark and AWS technologies. Your expertise in Python and domain knowledge in Cards & Payments will drive impactful solutions. This hybrid role offers the flexibility of day shifts ensuring a balanced work-life...

  • AWS + Pyspark

    2 weeks ago

    Chennai, Tamil Nadu, India Cognizant Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Skills- AWS +PysparkExperience: 4 to 9 yearsLocation: AIA-PuneAs an AWS Data Engineer, you will design, build, and manage robust data pipelines on the AWS cloud platform. You will leverage your expertise in AWS data services and programming languages to ingest, process, transform, and store data efficiently, enabling data-driven insights and...

  • Pyspark, AWS

    2 weeks ago

    Chennai, Tamil Nadu, India Cognizant Technology Solutions Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Job SummaryWe are seeking a skilled Developer with 4 to 6 years of experience to join our team. The ideal candidate will have expertise in Amazon S3 and PySpark and a strong background in the Cards & Payments domain. This hybrid role offers the opportunity to work on cutting-edge projects in a dynamic environment contributing to the companys growth and...

  • Pyspark Developer(Chennai,Bangalore,Hyderabad,Kolkata,Pune)

    2 weeks ago

    Chennai, Tamil Nadu, India Tata Consultancy Servicess Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Skill : PySpark DeveloperExperience Range: 4 to 8Notice : (Immediate to 60 Days)Job Description:Technical Skills: Spark ,PySpark, Python, Hive, Scala, MapReduce, Unix shell scriptingob Description:4+ Years of experience with Developing/Fine tuning and implementing programs/applicationsUsing Python/PySpark/Scala on Big Data/Hadoop Platform.Roles and...

  • Pyspark Data Engineer

    22 hours ago

    Chennai, Tamil Nadu, India Prowesstics Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    We are looking for a skilledData Engineerwith hands-on experience inPySparkandDatabricksto design, develop, and optimize large-scale data pipelines. The ideal candidate should have strong problem-solving skills, experience working in cloud-based environments, and the ability to collaborate effectively with cross-functional teams.Key Responsibilities:Develop...

  • Pyspark - Machine Learning

    5 days ago

    Chennai, Tamil Nadu, India Virtusa Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    7+ years of experience in Big Data with strong expertise in Spark and ScalaMandatory Skills: Big Data Primarily Spark and ScalaStrong Knowledge in HDFS, Hive, Impala with knowledge on Unix , Oracle, Autosys,Good to Have : Agile Methodology and Banking ExpertiseStrong Communication SkillsNot limited to Spark batch, need Spark streaming experienceNo SQL DB...

  • PySpark developer_CBS

    5 days ago

    Chennai, Tamil Nadu, India Coders Brain Technology Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    Location : Chennai/Hyderabad/Bangalore Details JD:   1. Minimum 6+ years of experience in creating SPARK Jobs using Java/Scala  2. Should have very good experience in developing data loading and transformation tasks using external sources, merge data, perform data enrichment and load in to target data destinations 3. Must have good knowledge on Big data...