PySpark ETL Developer

Position Purpose

The Senior Developer will be a part of the ISPL Mumbai IHC ETL projects team. The developer position will primarily work on Apache Spark(python), Spark SQL, ETL tools, Unix, Autosys and DB

Responsibilities

Direct Responsibilities

• Expertise on PySpark, database migration, transformation, and integration solutions for any Data warehousing project.
• Must have excellent knowledge in Apache Spark and Python programming experience.
• Deep experience in developing data processing tasks using PySpark such as reading data from external sources, merging data, performing data enrichment, and loading into target data destinations.
• Experience in deployment and operationalizing the code, knowledge of scheduling tools like Airflow, Control-M etc. is preferred.
• Understanding of Unix/Linux + Shell Scripting
• Data modeling experience using advanced statistical analysis, unstructured data processing.
• Hands-on project experience on Jupyter Notebook/ Zeppelin/ PyCharm etc. IDEs
• Hands-on experience with AWS S3 Filesystem operations

Contributing Responsibilities

• Good knowledge of Hadoop, Hive, and Cloudera/ Hortonworks Data Platform
• Strong hands-on experience in Processing Framework - Spark 2.x/3.x (Core, Spark SQL, Streaming) Language & Package - Python (Scripting & PySpark), Unix Shell, SQL Query (basic & advanced)
• Expertise in RDBM solutions (Postgres & Oracle) and NoSQL Databases
• Knowledge on Streaming Platform Apache Kafka, Spark Streaming
• Extensive hands-on experience in designing, building, and executing data pipeline using ETL/ELT tools.
• Big Data Hadoop - Detailed Knowledge on HDP/CDH Migration to new Cloudera CDP platform Data Storage HDFS (File Format Parquet, ORC, Avro, JSON), Hive (Schema, Partitioning), Data Lake (Object Store)
• Optimize and troubleshoot existing PySpark applications for performance improvements.

Technical & Behavioral Competencies

Minimum 5 years hands-on experience with PySpark, Kubernetes, Docker

Strong technical expertise in PySpark, Kubernetes, Docker and be in position to handle all technical difficulties

Strong in designing (data warehousing) concepts.

Good working knowledge in Unix/Ubuntu (Should be able to write wrapper scripts)

Capable of tuning the code to handle the huge data volume.

Responsible for translating/understanding the functional requirements to meet the specified technical requirements.

Rich experience involved in testing the PySpark modules, plans, deploys, and tests the ETL mappings, etc., to ensure that the clients remain satisfied

Involved in coding, testing, implementing, debugging, and documenting the complex programs.

Involved in creating proper technical documentation in the work assignments.

Understand the business needs and designs programs and systems that match the complex business requirements and records all the specifications that are involved in the development and coding process.

Ensures that all the standard requirements have been met and is involved in performing the technical analysis.

Responsible for assisting the project manager by compiling information from the current systems, analyzing the program requirements and ensuring that it meets the specified time requirements.

Resolves moderate problems associated with the designed programs and provides technical guidance on complex programming.

Behavioral Competencies

Excellent verbal and written communication skills.

Conduct meetings with global stakeholders, prepare minutes & summaries.

Assertiveness, Negotiation, Proactiveness & Prioritization skills are important.

Discipline in documenting, following up on issues and changes.

Experience in interacting with global stakeholders and independently managing discussions.

Specific Qualifications (if required)

Skills Referential

Behavioural Skills: (Please select up to 4 skills)

Communication skills - oral & written

Ability to synthetize / simplify

Attention to detail / rigor

Organizational skills

Transversal Skills: (Please select up to 5 skills)

Analytical Ability

Ability to understand, explain and support change

Ability to develop and adapt a process

Ability to manage / facilitate a meeting, seminar, committee, training

Ability to develop and leverage networks

Education Level:

Bachelor Degree or equivalent

Experience Level

At least 5 years

Other/Specific Qualifications (if required)

Additional knowledge on reporting tools is advantage