Pyspark Engineer5 YearsHyderabad

2 months ago

Hyderabad, India Notus Solution Full time
Overview:The PySparkEngineer plays a crucial role in our organization by leveragingtheir expertise in PySpark and Big Data technologies to designdevelop and maintain scalable data pipelines and analyticssolutions. This role is essential in enabling datadrivendecisionmaking and ensuring the optimal performance of our datasystems.KeyResponsibilities:
  • Designing andimplementing PySparkbased data processing and analyticssolutions.
  • Optimizing and tuning PySpark jobsfor performance and scalability.
  • Collaboratingwith data scientists and analysts to understand their requirementsand implement efficient dataworkflows.
  • Developing and maintaining ETLprocesses using PySpark to integrate data from multiplesources.
  • Creating and managing data pipelinesfor realtime and batchprocessing.
  • Troubleshooting and resolvingissues related to data processing and pipelineexecution.
  • Implementing best practices for dataengineering and ensuring data quality andreliability.
  • Collaborating with crossfunctionalteams to support datarelated initiatives andprojects.
  • Participating in code reviews andproviding technical guidance to junior teammembers.
  • Staying updated with the latestadvancements in PySpark and Big Datatechnologies.
RequiredQualifications:
  • Bachelors or Mastersdegree in Computer Science Engineering or a relatedfield.
  • Minimum of 5 years of handson experiencein PySpark development and Big Datatechnologies.
  • Proficiency in Python programmingfor data analysis and manipulation.
  • Strongunderstanding of SQL and experience working with relationaldatabases.
  • Experience in building andoptimizing data pipelines for largescale dataprocessing.
  • Solid understanding of distributedcomputing principles and clustermanagement.
  • Knowledge of data warehousingconcepts and best practices.
  • Experience withcloud platforms such as AWS Azure or GCP is aplus.
  • Excellent problemsolving and analyticalskills with a detailoriented mindset.
  • Strongcommunication and collaboration abilities to work effectively in ateamenvironment.
MandatorySkillData ModellerawsPysparkArchitect

bigdata,python,sql,data engineering