Spark/PySpark Developer

1 month ago

Navi Mumbai, India ATech Full time

Job Profile : Spark ( Pyspark ) Developer

Industry Type : IT Services

Job description :

- The developer must have sound knowledge in Apache Spark and Python programming.

- Deep experience in developing data processing tasks using pySpark such as reading data from external sources, merge data, perform data enrichment and load in to target data destinations.

- Experience in deployment and operationalizing the code is added advantage


- Have knowledge and skills in Devops/version control and containerization.


- Preferable having deployment knowledge.

- Create Spark jobs for data transformation and aggregation


- Produce unit tests for Spark transformations and helper methods

- Write Scaladoc-style documentation with all code

- Design data processing pipelines to perform batch and Real- time/stream analytics on structured and unstructured data

- Spark query tuning and performance optimization


- Good understanding of different file formats (ORC, Parquet, AVRO) to optimize queries/processing and compression techniques.

- SQL database integration (Microsoft, Oracle, Postgres, and/or MySQL)

- Experience working with (HDFS, S3, Cassandra, and/or DynamoDB)

- Deep understanding of distributed systems (e.g. CAP theorem, partitioning, replication, consistency, and consensus)

- Experience in building cloud scalable high-performance data lake solutions

- Hands on expertise in cloud services like AWS, and/or Microsoft Azure.

- As a Spark developer you will manage the development of scalable distributed Architecture defined by the Architect or tech Lead in our team.

- Analyse, assemble large data sets to designed for the functional and non-functional requirements.

- You will develop ETL scripts for big data sources.

- Identify, design optimise data processing automate for reports and dashboards.

- You will be responsible for workflow optimizations, data optimizations and ETL optimization as per the requirements elucidated by the team.

- Work with stakeholders such as Product managers, Technical Leads Service Layer engineers to ensure end-to-end requirements are addressed.

- Strong team player to adhere to Software Development Life cycle (SDLC) and documentations needed to represent every stage of SDLC.

- Hands on working experience on any of the data engineering analytics platform (Hortonworks Cloudera MapR AWS), AWS preferred

- Hands-on experience on Data Ingestion Apache Nifi, Apache Airflow, Sqoop, and Oozie

- Hands-on working experience of data processing at scale with event driven systems, message queues (Kafka Flink Spark Streaming)

- Hands on working Experience with AWS Services like EMR, Kinesis, S3, Cloud Formation, Glue, API Gateway, Lake Foundation

- Hands on working Experience with AWS Athena

- Data Warehouse exposure on Apache Nifi, Apache Airflow, Kylo

- Operationalization of ML models on AWS (e.g. deployment, scheduling, model monitoring etc.)

- Feature Engineering Data Processing to be used for Model development

- Experience gathering and processing raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries, etc.)

- Experience building data pipelines for structured unstructured, real-time batch, events synchronous asynchronous using MQ, Kafka, Steam processing

- Hands-on working experience in analysing source system data and data flows, working with structured and unstructured data

- Must be very strong in writing SQL queries

(ref:hirist.tech)

  • Spark/PySpark Developer

    4 weeks ago

    Bihar/Jharkhand/Maharashtra/Pondicherry/Coimbatore/Patna/Aurangabad/Ranchi/Mumbai/Navi Mumbai/Pune/N, IN ATech Full time

    Job Profile : Spark ( Pyspark ) DeveloperIndustry Type : IT Services Job description :- The developer must have sound knowledge in Apache Spark and Python programming.- Deep experience in developing data processing tasks using pySpark such as reading data from external sources, merge data, perform data enrichment and load in to target data destinations.-...

  • Spark/PySpark Developer

    1 month ago

    Bihar,Jharkhand,Maharashtra,Pondicherry,Coimbatore,Patna,Aurangabad,Ranchi,Mumbai,Navi Mumbai,Pune,N, India ATech Full time

    Job Profile : Spark ( Pyspark ) DeveloperIndustry Type : IT Services Job description :- The developer must have sound knowledge in Apache Spark and Python programming.- Deep experience in developing data processing tasks using pySpark such as reading data from external sources, merge data, perform data enrichment and load in to target data destinations.-...

  • Java + Spark

    4 weeks ago

    Mumbai, India L&T Technology Services Ltd. Full time

    Location Pune/ Mumbai/ Chennai/ Hyderabad Years of Experience 5-10 Years Any Project specific Prerequisite skills Java Spark, Detailed JD Strong experience in **ETL development with Java & Spark** Strong experience with **Redshift, AWS S3, SQL** Experience in developing **microservices** Proficiency with **Lambda** expressions, **Pyspark** Hands...

  • Senior Data Engineer

    4 weeks ago

    Mumbai/Chennai, IN Cyber Sphere LLC Full time

    Senior Data EngineerOnsite : Mumbai/ChennaiAbout the Role :- This role is more focused on Pyspark with Cloud developer.About the Responsibilities :- This position provides direct input to project plans, schedules, and follows software methodologies and best practices in the development of cross-functional software products under a micro-services styled...

  • Senior Data Engineer

    4 weeks ago

    Mumbai,Chennai, India Cyber Sphere LLC Full time

    Senior Data EngineerOnsite : Mumbai/ChennaiAbout the Role : - This role is more focused on Pyspark with Cloud developer.About the Responsibilities : - This position provides direct input to project plans, schedules, and follows software methodologies and best practices in the development of cross-functional software products under a micro-services styled...

  • Data Engineer

    1 month ago

    Mumbai,Bangalore, India Voyager Partners Full time

    Job Description : Role : Data Engineer. Experience : 4 Years 8 Years. Company : V2 Solutions. Must Have : - Data bricks. Python, Spark Or PySpark. AWS. - Glue ETL, Data Catalog. EMR. Redshift. S3. DBT. - Batch Processing. - SQL. - Experience with Streaming. Integrations & Data Interoperability. Nice to have : - Infra K8s, Docker,...

  • Senior Data Engineer

    4 weeks ago

    Bangalore/Hyderabad/Mumbai/Pune, IN MLOPS SOLUTIONS PRIVATE LIMITED Full time

    Job Description :Primary skillset :Experience working with distributed technology tools for developing Batch and Streaming pipelines using :- SQL, Spark, PySpark - Airflow - Spark with Scala .(optional)- Able to write code which is optimized for performance.- Experience in Cloud platform, e.g., AWS, GCP, Azure, etc.- Able to quickly pick up new programming...

  • Senior Data Engineer

    1 month ago

    Bangalore,Hyderabad,Mumbai,Pune, India MLOPS SOLUTIONS PRIVATE LIMITED Full time

    Job Description :Primary skillset :Experience working with distributed technology tools for developing Batch and Streaming pipelines using :- SQL, Spark, PySpark - Airflow - Spark with Scala .(optional)- Able to write code which is optimized for performance.- Experience in Cloud platform, e.g., AWS, GCP, Azure, etc.- Able to quickly pick up new programming...

  • Software Engineer III

    2 weeks ago

    Mumbai, India JPMorgan Chase & Co. Full time

    Join our dynamic team as a software developer, where you will have the opportunity to solve complex problems and contribute to our innovative projects. With us, you can enhance your skills in Python, PySpark, and cloud architecture, while working in an inclusive and respectful team environment. This role offers immense growth potential and a chance to work...

  • Spark, Scala Developer

    4 weeks ago

    Mumbai, India Robosoft Technologies Full time

    Technical/Functional SkillsMust have 4+ years of IT experienceMust have good experience in Spark and ScalaGood to have experience instreaming systems like Spark streaming and StormExperience with Spark Data processing, Performance Tuning, Memory Management, Fault Tolerance, ScalabilityGood knowledge of Hive, Sqoop, Spark, Data warehousing and information...

  • Software Engineer III

    2 weeks ago

    Mumbai, India JPMorgan Chase & Co. Full time

    Join our dynamic team as a software developer, where you will have the opportunity to solve complex problems and contribute to our innovative projects. With us, you can enhance your skills in Python, PySpark, and cloud architecture, while working in an inclusive and respectful team environment. This role offers immense growth potential and a chance to work...

  • Software Engineer III

    2 weeks ago

    mumbai, India JPMorgan Chase & Co. Full time

    Join our dynamic team as a software developer, where you will have the opportunity to solve complex problems and contribute to our innovative projects. With us, you can enhance your skills in Python, PySpark, and cloud architecture, while working in an inclusive and respectful team environment. This role offers immense growth potential and a chance to work...

  • Spark, Scala Developer

    1 month ago

    mumbai, India Robosoft Technologies Full time

    Technical/Functional Skills Must have 4+ years of IT experience Must have good experience in Spark and Scala Good to have experience instreaming systems like Spark streaming and Storm Experience with Spark Data processing, Performance Tuning, Memory Management, Fault Tolerance, Scalability Good knowledge of Hive, Sqoop, Spark, Data...

  • Spark, Scala Developer

    1 month ago

    Mumbai, India Robosoft Technologies Full time

    Technical/Functional Skills Must have 4+ years of IT experience Must have good experience in Spark and Scala Good to have experience instreaming systems like Spark streaming and Storm Experience with Spark Data processing, Performance Tuning, Memory Management, Fault Tolerance, Scalability Good knowledge of Hive, Sqoop, Spark, Data...

  • Big Data Engineer

    4 weeks ago

    Goa/Mumbai/Jammu & Kashmir/Jammu/Srinagar/Pondicherry/Jaipur/Lucknow/Varanasi/Banaras/Patna/Ranchi, IN ATech Full time

    Designation: BIG DATA ENGINEERJob Description:Your Role and Responsibilities:- Understand a data warehousing solution and able to work independently in such an environment- Responsible in Project development and delivery experience of a few good size projects- Design, build, optimize and support new and existing data models and ETL processes based on our...

  • Azure Data Engineer

    1 month ago

    mumbai, India ADS247365 Full time

    Location : Mumbai, Maharashtra Duration : Full Time Job Description : Roles & Responsibilities : - Must have 4 years of Experience on Python.- Must have 4 years' relevant experience on Apache Spark OR pyspark. - Must have 4 years of experience in SQL (Must be proficient in writing Advanced complex queries). - Must have 4 years' relevant experience on...

  • Big Data Engineer

    2 weeks ago

    Goa/Mumbai/Jammu & Kashmir/Jammu/Srinagar/Pondicherry/Jaipur/Lucknow/Varanasi/Banaras/Patna/Ranchi, India ATech Full time

    Designation: BIG DATA ENGINEERJob Description:Your Role and Responsibilities:- Understand a data warehousing solution and able to work independently in such an environment- Responsible in Project development and delivery experience of a few good size projects- Design, build, optimize and support new and existing data models and ETL processes based on our...

  • Azure Data Engineer

    2 weeks ago

    Mumbai, India ADS247365 Full time

    Location : Mumbai, Maharashtra Duration : Full TimeJob Description :Roles & Responsibilities :- Must have 4 years of Experience on Python.- Must have 4 years' relevant experience on Apache Spark OR pyspark. - Must have 4 years of experience in SQL [Must be proficient in writing Advanced complex queries]. - Must have 4 years' relevant experience on...

  • Azure Data Engineer

    4 weeks ago

    Mumbai, Maharashtra, India ADS247365 Full time

    Location : Mumbai, Maharashtra Duration : Full TimeJob Description :Roles & Responsibilities :- Must have 4 years of Experience on Python.- Must have 4 years' relevant experience on Apache Spark OR pyspark. - Must have 4 years of experience in SQL [Must be proficient in writing Advanced complex queries]. - Must have 4 years' relevant experience on...

  • Azure Data Engineer

    2 weeks ago

    Mumbai, India ADS247365 Full time

    Location : Mumbai, Maharashtra Duration : Full Time Job Description : Roles & Responsibilities : - Must have 4 years of Experience on Python.- Must have 4 years' relevant experience on Apache Spark OR pyspark. - Must have 4 years of experience in SQL (Must be proficient in writing Advanced complex queries). - Must have 4 years' relevant...