
PySpark ETL Developer
5 days ago
Position Purpose
The Senior Developer will be a part of the ISPL Mumbai IHC ETL projects team. The developer position will primarily work on Apache Spark(python), Spark SQL, ETL tools, Unix, Autosys and DB
Responsibilities
Direct Responsibilities
- Expertise on PySpark, database migration, transformation, and integration solutions for any Data warehousing project.
- Must have excellent knowledge in Apache Spark and Python programming experience.
- Deep experience in developing data processing tasks using PySpark such as reading data from external sources, merging data, performing data enrichment, and loading into target data destinations.
- Experience in deployment and operationalizing the code, knowledge of scheduling tools like Airflow, Control-M etc. is preferred.
- Understanding of Unix/Linux + Shell Scripting
- Data modeling experience using advanced statistical analysis, unstructured data processing.
- Hands-on project experience on Jupyter Notebook/ Zeppelin/ PyCharm etc. IDEs
- Hands-on experience with AWS S3 Filesystem operations
Contributing Responsibilities
- Good knowledge of Hadoop, Hive, and Cloudera/ Hortonworks Data Platform
- Strong hands-on experience in Processing Framework - Spark 2.x/3.x (Core, Spark SQL, Streaming) Language & Package - Python (Scripting & PySpark), Unix Shell, SQL Query (basic & advanced)
- Expertise in RDBM solutions (Postgres & Oracle) and NoSQL Databases
- Knowledge on Streaming Platform Apache Kafka, Spark Streaming
- Extensive hands-on experience in designing, building, and executing data pipeline using ETL/ELT tools.
- Big Data Hadoop - Detailed Knowledge on HDP/CDH Migration to new Cloudera CDP platform Data Storage HDFS (File Format Parquet, ORC, Avro, JSON), Hive (Schema, Partitioning), Data Lake (Object Store)
- Optimize and troubleshoot existing PySpark applications for performance improvements.
Technical & Behavioral Competencies
Minimum 5 years hands-on experience with PySpark, Kubernetes, Docker
Strong technical expertise in PySpark, Kubernetes, Docker and be in position to handle all technical difficulties
Strong in designing (data warehousing) concepts.
Good working knowledge in Unix/Ubuntu (Should be able to write wrapper scripts)
Capable of tuning the code to handle the huge data volume.
Responsible for translating/understanding the functional requirements to meet the specified technical requirements.
Rich experience involved in testing the PySpark modules, plans, deploys, and tests the ETL mappings, etc., to ensure that the clients remain satisfied
Involved in coding, testing, implementing, debugging, and documenting the complex programs.
Involved in creating proper technical documentation in the work assignments.
Understand the business needs and designs programs and systems that match the complex business requirements and records all the specifications that are involved in the development and coding process.
Ensures that all the standard requirements have been met and is involved in performing the technical analysis.
Responsible for assisting the project manager by compiling information from the current systems, analyzing the program requirements and ensuring that it meets the specified time requirements.
Resolves moderate problems associated with the designed programs and provides technical guidance on complex programming.
Behavioral Competencies
Excellent verbal and written communication skills.
Conduct meetings with global stakeholders, prepare minutes & summaries.
Assertiveness, Negotiation, Proactiveness & Prioritization skills are important.
Discipline in documenting, following up on issues and changes.
Experience in interacting with global stakeholders and independently managing discussions.
Specific Qualifications (if required)
Skills Referential
Behavioural Skills: (Please select up to 4 skills)
Communication skills - oral & written
Ability to synthetize / simplify
Attention to detail / rigor
Organizational skills
Transversal Skills: (Please select up to 5 skills)
Analytical Ability
Ability to understand, explain and support change
Ability to develop and adapt a process
Ability to manage / facilitate a meeting, seminar, committee, training
Ability to develop and leverage networks
Education Level:
Bachelor Degree or equivalent
Experience Level
At least 5 years
Other/Specific Qualifications (if required)
Additional knowledge on reporting tools is advantage
-
Pyspark Developer
1 week ago
Mumbai, Maharashtra, India Artech Full time ₹ 20,00,000 - ₹ 25,00,000 per yearRole & responsibilitiesYou will lead the effort to design, build, and configure applications, acting as the primary point of contact. Your typical day will involve collaborating with various teams to ensure that project goals are met, facilitating discussions to address challenges, and guiding your team through the development process. You will also be...
-
ETL Data Engineer
5 days ago
Mumbai, Maharashtra, India Sourcebae Full time ₹ 12,00,000 - ₹ 36,00,000 per yearETL Data EngineerSkill to Evaluate: ETL, Etl Developer, GCP, Big Data, Bigquery, Kafka, Hive, Data Modeling, Python, Pyspark, SQLExperience: 5 to 6 YearsLocation: Lower Parel, Mumbai (3 Days WFO)BGV: Education, Address, Employment, CriminalAbout the RoleWe are looking for a passionate and experienced Data Engineer to join our team andhelp build scalable,...
-
Pyspark Developer Lead
3 days ago
Mumbai, Maharashtra, India Hybrowlabs Technologies Full time ₹ 8,00,000 - ₹ 24,00,000 per yearCompany DescriptionHybrowlabs Technologies is dedicated to building software better, and faster. We explore every tool that hits the market to find the best stack of tools for software development. Our magical formula and stack of tools will accelerate your software development process. Contact us to learn more.Role DescriptionDesign and architect scalable,...
-
ETL Developer
3 days ago
Mumbai, Maharashtra, India Hatchtra Innotech Pvt. Ltd. Full time ₹ 1,32,000 - ₹ 1,50,000 per yearRole -: ETL/SSIS DeveloperExp YearsBudget - 11 LPAShift Time: 1 pm to 10 pmLocation - Mumbai / Pune / ChandigarhNotice Period - Immediate Joiner - LWD Till 31 Oct ONLYJob Summary:We are seeking a skilled ETL/SSIS Developer to join our data engineering team. The ideal candidate will have hands-on experience in designing, developing, and maintaining ETL...
-
ETL Developer
3 days ago
Mumbai, Maharashtra, India Trivoli Digital Pvt. Ltd. Full time ₹ 8,00,000 - ₹ 12,00,000 per yearHello,Greetings from Trivoli Digital Pvt LtdWe are hiring forJob Title: ETL Migration Specialist – SSIS to GoAnywhereLocation: RemoteAbout the RoleWe are looking for experienced ETL professionals with hands-on expertise in Microsoft SSIS and GoAnywhere to support a migration project. The role involves analyzing, migrating, testing, and optimizing existing...
-
Sr ETL DataStage Developer
6 days ago
Mumbai, Maharashtra, India translab Full time ₹ 9,00,000 - ₹ 12,00,000 per yearWe have an exciting opportunity for the position of ETL Developer with one of our reputed clients in Mumbai. Please find the job details below:Job Title: ETL DeveloperLocation: MumbaiExperience: 4+ YearsJob Description:We are looking for an experienced ETL Developer with strong expertise in Oracle SQL, Oracle, IBM DataStage, and other ETL tools. The...
-
Senior Python Developer
2 weeks ago
Mumbai, Maharashtra, India SS&C TECHNOLOGIES Full time ₹ 15,00,000 - ₹ 25,00,000 per yearSenior Python developer As a leading financial services and healthcare technology company based on revenue, SS&C is headquartered in Windsor, Connecticut, and has 27,000 employees in 35 countries. Some 20,000 financial services and healthcare organizations, from the world's largest companies to small and mid-market firms, rely on SS&C for expertise, scale,...
-
ETL Consultant
4 days ago
Mumbai, Maharashtra, India Global Talent Track Private Limited (Global Talent Track)(231) Full time ₹ 5,00,000 - ₹ 12,00,000 per yearAbout the Job : The ideal candidate will be responsible for designing, developing, and maintaining robust ETL processes. The role requires expertise in handling large-scale data integration, ensuring data accuracy, and delivering scalable solutions aligned with business requirements. Key Responsibilities : - Design, develop, and implement...
-
Sr ETL Data Stage Developer
7 days ago
Mumbai, Maharashtra, India Translab Full time ₹ 5,00,000 - ₹ 12,00,000 per yearRole & responsibilitiesGreetings from Translab Technologies Pvt. Ltd.We have an exciting opportunity for the position of ETL Developer with one of our reputed clients in Mumbai. Please find the job details below:Job Title: ETL DeveloperLocation: MumbaiExperience: 4+ YearsJob Description:We are looking for an experienced ETL Developer with strong...
-
PLSQL & ETL developer with Banking Domain
3 days ago
Mumbai, Maharashtra, India Bahwan Cybertek Group Full time ₹ 1,00,000 - ₹ 3,00,000 per yearBahwan CyberTek Group is looking for a talented PLSQL & ETL Developer with experience in the Banking domain to join our innovative team. You will leverage your expertise in PLSQL and ETL processes to design, develop, and implement high-quality solutions that meet the evolving needs of our banking and financial services clients. Your role will be critical in...