PySpark ETL Developer

5 days ago


Mumbai, Maharashtra, India BNP Paribas Full time ₹ 12,00,000 - ₹ 36,00,000 per year

Position Purpose

The Senior Developer will be a part of the ISPL Mumbai IHC ETL projects team. The developer position will primarily work on Apache Spark(python), Spark SQL, ETL tools, Unix, Autosys and DB

Responsibilities

Direct Responsibilities

  • Expertise on PySpark, database migration, transformation, and integration solutions for any Data warehousing project.
  • Must have excellent knowledge in Apache Spark and Python programming experience.
  • Deep experience in developing data processing tasks using PySpark such as reading data from external sources, merging data, performing data enrichment, and loading into target data destinations.
  • Experience in deployment and operationalizing the code, knowledge of scheduling tools like Airflow, Control-M etc. is preferred.
  • Understanding of Unix/Linux + Shell Scripting
  • Data modeling experience using advanced statistical analysis, unstructured data processing.
  • Hands-on project experience on Jupyter Notebook/ Zeppelin/ PyCharm etc. IDEs
  • Hands-on experience with AWS S3 Filesystem operations

Contributing Responsibilities

  • Good knowledge of Hadoop, Hive, and Cloudera/ Hortonworks Data Platform
  • Strong hands-on experience in Processing Framework - Spark 2.x/3.x (Core, Spark SQL, Streaming) Language & Package - Python (Scripting & PySpark), Unix Shell, SQL Query (basic & advanced)
  • Expertise in RDBM solutions (Postgres & Oracle) and NoSQL Databases
  • Knowledge on Streaming Platform Apache Kafka, Spark Streaming
  • Extensive hands-on experience in designing, building, and executing data pipeline using ETL/ELT tools.
  • Big Data Hadoop - Detailed Knowledge on HDP/CDH Migration to new Cloudera CDP platform Data Storage HDFS (File Format Parquet, ORC, Avro, JSON), Hive (Schema, Partitioning), Data Lake (Object Store)
  • Optimize and troubleshoot existing PySpark applications for performance improvements.

Technical & Behavioral Competencies

Minimum 5 years hands-on experience with PySpark, Kubernetes, Docker

Strong technical expertise in PySpark, Kubernetes, Docker and be in position to handle all technical difficulties

Strong in designing (data warehousing) concepts.

Good working knowledge in Unix/Ubuntu (Should be able to write wrapper scripts)

Capable of tuning the code to handle the huge data volume.

Responsible for translating/understanding the functional requirements to meet the specified technical requirements.

Rich experience involved in testing the PySpark modules, plans, deploys, and tests the ETL mappings, etc., to ensure that the clients remain satisfied

Involved in coding, testing, implementing, debugging, and documenting the complex programs.

Involved in creating proper technical documentation in the work assignments.

Understand the business needs and designs programs and systems that match the complex business requirements and records all the specifications that are involved in the development and coding process.

Ensures that all the standard requirements have been met and is involved in performing the technical analysis.

Responsible for assisting the project manager by compiling information from the current systems, analyzing the program requirements and ensuring that it meets the specified time requirements.

Resolves moderate problems associated with the designed programs and provides technical guidance on complex programming.

Behavioral Competencies

Excellent verbal and written communication skills.

Conduct meetings with global stakeholders, prepare minutes & summaries.

Assertiveness, Negotiation, Proactiveness & Prioritization skills are important.

Discipline in documenting, following up on issues and changes.

Experience in interacting with global stakeholders and independently managing discussions.

Specific Qualifications (if required)

Skills Referential

Behavioural Skills: (Please select up to 4 skills)

Communication skills - oral & written

Ability to synthetize / simplify

Attention to detail / rigor

Organizational skills

Transversal Skills: (Please select up to 5 skills)

Analytical Ability

Ability to understand, explain and support change

Ability to develop and adapt a process

Ability to manage / facilitate a meeting, seminar, committee, training

Ability to develop and leverage networks

Education Level:

Bachelor Degree or equivalent

Experience Level

At least 5 years

Other/Specific Qualifications (if required)

Additional knowledge on reporting tools is advantage


  • Pyspark Developer

    1 week ago


    Mumbai, Maharashtra, India Artech Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Role & responsibilitiesYou will lead the effort to design, build, and configure applications, acting as the primary point of contact. Your typical day will involve collaborating with various teams to ensure that project goals are met, facilitating discussions to address challenges, and guiding your team through the development process. You will also be...

  • ETL Data Engineer

    5 days ago


    Mumbai, Maharashtra, India Sourcebae Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    ETL Data EngineerSkill to Evaluate: ETL, Etl Developer, GCP, Big Data, Bigquery, Kafka, Hive, Data Modeling, Python, Pyspark, SQLExperience: 5 to 6 YearsLocation: Lower Parel, Mumbai (3 Days WFO)BGV: Education, Address, Employment, CriminalAbout the RoleWe are looking for a passionate and experienced Data Engineer to join our team andhelp build scalable,...


  • Mumbai, Maharashtra, India Hybrowlabs Technologies Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Company DescriptionHybrowlabs Technologies is dedicated to building software better, and faster. We explore every tool that hits the market to find the best stack of tools for software development. Our magical formula and stack of tools will accelerate your software development process. Contact us to learn more.Role DescriptionDesign and architect scalable,...

  • ETL Developer

    3 days ago


    Mumbai, Maharashtra, India Hatchtra Innotech Pvt. Ltd. Full time ₹ 1,32,000 - ₹ 1,50,000 per year

    Role -: ETL/SSIS DeveloperExp YearsBudget - 11 LPAShift Time: 1 pm to 10 pmLocation - Mumbai / Pune / ChandigarhNotice Period - Immediate Joiner - LWD Till 31 Oct ONLYJob Summary:We are seeking a skilled ETL/SSIS Developer to join our data engineering team. The ideal candidate will have hands-on experience in designing, developing, and maintaining ETL...

  • ETL Developer

    3 days ago


    Mumbai, Maharashtra, India Trivoli Digital Pvt. Ltd. Full time ₹ 8,00,000 - ₹ 12,00,000 per year

    Hello,Greetings from Trivoli Digital Pvt LtdWe are hiring forJob Title: ETL Migration Specialist – SSIS to GoAnywhereLocation: RemoteAbout the RoleWe are looking for experienced ETL professionals with hands-on expertise in Microsoft SSIS and GoAnywhere to support a migration project. The role involves analyzing, migrating, testing, and optimizing existing...


  • Mumbai, Maharashtra, India translab Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    We have an exciting opportunity for the position of ETL Developer with one of our reputed clients in Mumbai. Please find the job details below:Job Title: ETL DeveloperLocation: MumbaiExperience: 4+ YearsJob Description:We are looking for an experienced ETL Developer with strong expertise in Oracle SQL, Oracle, IBM DataStage, and other ETL tools. The...


  • Mumbai, Maharashtra, India SS&C TECHNOLOGIES Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Senior Python developer As a leading financial services and healthcare technology company based on revenue, SS&C is headquartered in Windsor, Connecticut, and has 27,000 employees in 35 countries. Some 20,000 financial services and healthcare organizations, from the world's largest companies to small and mid-market firms, rely on SS&C for expertise, scale,...

  • ETL Consultant

    4 days ago


    Mumbai, Maharashtra, India Global Talent Track Private Limited (Global Talent Track)(231) Full time ₹ 5,00,000 - ₹ 12,00,000 per year

    About the Job : The ideal candidate will be responsible for designing, developing, and maintaining robust ETL processes. The role requires expertise in handling large-scale data integration, ensuring data accuracy, and delivering scalable solutions aligned with business requirements. Key Responsibilities : - Design, develop, and implement...


  • Mumbai, Maharashtra, India Translab Full time ₹ 5,00,000 - ₹ 12,00,000 per year

    Role & responsibilitiesGreetings from Translab Technologies Pvt. Ltd.We have an exciting opportunity for the position of ETL Developer with one of our reputed clients in Mumbai. Please find the job details below:Job Title: ETL DeveloperLocation: MumbaiExperience: 4+ YearsJob Description:We are looking for an experienced ETL Developer with strong...


  • Mumbai, Maharashtra, India Bahwan Cybertek Group Full time ₹ 1,00,000 - ₹ 3,00,000 per year

    Bahwan CyberTek Group is looking for a talented PLSQL & ETL Developer with experience in the Banking domain to join our innovative team. You will leverage your expertise in PLSQL and ETL processes to design, develop, and implement high-quality solutions that meet the evolving needs of our banking and financial services clients. Your role will be critical in...