pyspark+gcp

2 weeks ago

Chennai, Tamil Nadu, India Cognizant Technology Solutions Full time ₹ 20,00,000 - ₹ 25,00,000 per year

Job Summary

The Sr. Developer role is pivotal in driving the success of our data-driven projects. With a focus on Big Data and AWS technologies the candidate will design and implement scalable solutions. This hybrid role requires expertise in Apache Spark and Python ensuring efficient data processing and storage. The position offers a dynamic work environment with a day shift schedule contributing significantly to our companys innovation and impact.

Responsibilities

  • Develop and implement scalable data processing solutions using Big Data technologies to enhance data-driven decision-making.
  • Collaborate with cross-functional teams to design and optimize data architectures on AWS EC2 and AWS EMR platforms.
  • Utilize Amazon S3 for efficient data storage and retrieval ensuring data integrity and accessibility.
  • Leverage Apache Spark to process large datasets improving data analysis and reporting capabilities.
  • Write and maintain Python scripts for data manipulation and automation streamlining workflows and increasing productivity.
  • Monitor and troubleshoot data pipelines to ensure seamless data flow and minimize downtime.
  • Conduct code reviews and provide constructive feedback to peers fostering a culture of continuous improvement.
  • Stay updated with the latest industry trends and technologies to propose innovative solutions and maintain a competitive edge.
  • Ensure compliance with data security and privacy regulations safeguarding sensitive information.
  • Document technical specifications and processes to facilitate knowledge sharing and onboarding of new team members.
  • Participate in agile development processes contributing to sprint planning and retrospective meetings.
  • Provide technical guidance and mentorship to junior developers enhancing team capabilities and performance.
  • Collaborate with stakeholders to gather requirements and translate them into technical specifications aligning with business objectives.

Qualifications

  • Possess a strong background in Big Data technologies with hands-on experience in AWS EC2 and AWS EMR.
  • Demonstrate proficiency in Amazon S3 for data storage solutions.
  • Have extensive experience with Apache Spark for data processing tasks.
  • Exhibit advanced skills in Python programming for data manipulation and automation.
  • Show a proven track record of developing scalable data processing solutions.
  • Display excellent problem-solving abilities and attention to detail.
  • Have a solid understanding of data security and privacy regulations.

Certifications Required

AWS Certified Big Data - Specialty Apache Spark Developer Certification

  • Pyspark - Machine Learning

    3 days ago

    Chennai, Tamil Nadu, India Virtusa Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    7+ years of experience in Big Data with strong expertise in Spark and ScalaMandatory Skills: Big Data Primarily Spark and ScalaStrong Knowledge in HDFS, Hive, Impala with knowledge on Unix , Oracle, Autosys,Good to Have : Agile Methodology and Banking ExpertiseStrong Communication SkillsNot limited to Spark batch, need Spark streaming experienceNo SQL DB...

  • GCP Data Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India Virtusa Referral Program Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Design and develop robust ETL pipelines using Python, PySpark, and GCP services.Build and optimize data models and queries in BigQuery for analytics and reporting.Ingest, transform, and load structured and semi-structured data from various sources.Collaborate with data analysts, scientists, and business teams to understand data requirements.Ensure data...

  • GCP Data Engineer

    3 days ago

    Chennai, Tamil Nadu, India Virtusa Full time ₹ 1,00,00,000 - ₹ 2,00,00,000 per year

    Design and develop robust ETL pipelines using Python, PySpark, and GCP services.Build and optimize data models and queries in BigQuery for analytics and reporting.Ingest, transform, and load structured and semi-structured data from various sources.Collaborate with data analysts, scientists, and business teams to understand data requirements.Ensure data...

  • Pyspark

    2 weeks ago

    Chennai, Tamil Nadu, India Cognizant Technology Solutions Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job SummaryAs a Sr. Developer you will play a pivotal role in optimizing and enhancing our data processing capabilities using PySpark and AWS technologies. Your expertise in Python and domain knowledge in Cards & Payments will drive impactful solutions. This hybrid role offers the flexibility of day shifts ensuring a balanced work-life...

  • GCP Data Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India Virtusa Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Design and develop robust ETL pipelines using Python, PySpark, and GCP services.Build and optimize data models and queries in BigQuery for analytics and reporting.Ingest, transform, and load structured and semi-structured data from various sources.Collaborate with data analysts, scientists, and business teams to understand data requirements.Ensure data...

  • AWS + Pyspark

    2 weeks ago

    Chennai, Tamil Nadu, India Cognizant Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Skills- AWS +PysparkExperience: 4 to 9 yearsLocation: AIA-PuneAs an AWS Data Engineer, you will design, build, and manage robust data pipelines on the AWS cloud platform. You will leverage your expertise in AWS data services and programming languages to ingest, process, transform, and store data efficiently, enabling data-driven insights and...

  • GCP data engineer

    2 weeks ago

    Chennai, Tamil Nadu, India Prodapt Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    OverviewDesign and implement complex ETL/ELT pipelines using PySpark and Airflow for large-scale data processing on GCP.Lead data migration initiatives, including automating the movement of Teradata tables to BigQuery, ensuring data accuracy and consistency.Develop robust frameworks to streamline batch and streaming data ingestion workflows, leveraging...

  • Pyspark Developer

    1 week ago

    Chennai, Tamil Nadu, India Citi Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Discover your future at CitiWorking at Citi is far more than just a job. A career with us means joining a team of more than 230,000 dedicated people from around the globe. At Citi, you'll have the opportunity to grow your career, give back to your community and make a real impact.Job OverviewAt Citi we're not just building technology, we're building the...

  • Pyspark, AWS

    2 weeks ago

    Chennai, Tamil Nadu, India Cognizant Technology Solutions Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Job SummaryWe are seeking a skilled Developer with 4 to 6 years of experience to join our team. The ideal candidate will have expertise in Amazon S3 and PySpark and a strong background in the Cards & Payments domain. This hybrid role offers the opportunity to work on cutting-edge projects in a dynamic environment contributing to the companys growth and...

  • Data Engineering

    6 hours ago

    Chennai, Tamil Nadu, India, Tamil Nadu EXL Full time

    Responsibilities:Work with stakeholders to understand the data requirements to design, develop, and maintain complex ETL processes.Create the data integration and data diagram documentation.Lead the data validation, UAT and regression test for new data asset creation.Create and maintain data models, including schema design and optimization.Create and manage...