PySpark Architect

1 month ago

pune, India RapidBraiins Full time

Job Description


- As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.

- You will work closely with cross-functional teams to develop robust, scalable, and efficient data solutions.

- Your deep experience in big data technologies and hands-on expertise in PySpark will be critical in driving our data initiatives forward.

Key Responsibilities :

Architectural Leadership :

- Lead the architecture, design, and implementation of large-scale data processing applications using PySpark.

- Provide technical guidance and mentorship to development teams on best practices for data processing and PySpark development.

Solution Design and Development :

- Design and implement end-to-end data pipelines to ingest, process, and analyze large volumes of data.

- Develop scalable and efficient PySpark applications to handle complex data transformations and aggregations.

- Ensure the solutions are designed for high availability, scalability, and performance.

Optimization and Performance Tuning :

- Optimize PySpark applications for performance and efficiency.

- Perform code reviews and provide recommendations for improvements.

- Identify and resolve performance bottlenecks in data processing workflows.

Collaboration and Stakeholder Management :

- Work closely with data engineers, data scientists, and other stakeholders to understand their requirements and translate them into technical solutions.

- Collaborate with infrastructure teams to ensure the deployment and scaling of data applications.

Documentation and Standards :

- Maintain comprehensive documentation of the architecture, design, and implementation details.

- Establish and enforce coding standards, best practices, and development methodologies.

Continuous Improvement :

- Stay updated with the latest trends and advancements in big data technologies.

- Identify opportunities for continuous improvement in data processing and analytics solutions.

Required Qualifications:

- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.

- 12 to 15 years of experience in software development with at least 5 years of experience in big data technologies.

- Extensive hands-on experience with PySpark and the Spark ecosystem.

- Strong expertise in designing and implementing large-scale data processing systems.

- Proficiency in Python programming and related libraries for data processing.

- Solid understanding of distributed computing principles and big data architectures.

- Experience with cloud platforms such as AWS, Azure, or GCP.

- Strong knowledge of data warehousing and ETL processes.

- Excellent problem-solving and analytical skills.

- Strong communication and collaboration skills, with the ability to work effectively with cross-functional teams.

Preferred Qualifications:

- Experience with additional big data technologies such as Hadoop, Hive, or Kafka.

- Knowledge of containerization and orchestration tools such as Docker and Kubernetes.

- Familiarity with data visualization tools and techniques.

- Certification in cloud platforms or big data technologies.

Job Location: Chennai/Pune/Kolkata

(ref:hirist.tech)

  • Pyspark AWS Architect ATC

    2 weeks ago

    Pune, Maharashtra, India virtusa consulting services pvt. ltd. Full time

    Apply for Pyspark, AWS Architect ATC, Career Progress Consultants in Pune for Year of Experience on

  • Pyspark AWS Architect ATC

    2 weeks ago

    Pune, India virtusa consulting services pvt. ltd. Full time

    Apply for Pyspark, AWS Architect ATC, Career Progress Consultants in Pune for 10 - 12 Year of Experience on TimesJobs.com.

  • PySpark Architect

    4 weeks ago

    Chennai/Kolkata/Pune, IN RapidBraiins Full time

    Job Description- As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.- You will work closely with cross-functional teams to develop robust, scalable, and efficient data solutions.- Your deep experience in big data technologies and hands-on expertise in PySpark will be...

  • PySpark Architect

    3 weeks ago

    Chennai/Kolkata/Pune, India RapidBraiins Full time

    Job Description - As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.- You will work closely with cross-functional teams to develop robust, scalable, and efficient data solutions.- Your deep experience in big data technologies and hands-on expertise in PySpark will...

  • PySpark Architect

    1 month ago

    Chennai/Kolkata/Pune, India RapidBraiins Full time

    Job Description - As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.- You will work closely with cross-functional teams to develop robust, scalable, and efficient data solutions.- Your deep experience in big data technologies and hands-on expertise in PySpark will...

  • PySpark Architect

    3 weeks ago

    Chennai/Kolkata/Pune, IN RapidBraiins Full time

    Job Description- As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.- You will work closely with cross-functional teams to develop robust, scalable, and efficient data solutions.- Your deep experience in big data technologies and hands-on expertise in PySpark will be...

  • Pyspark, AWS Architect

    1 week ago

    Pune, Maharashtra, India Virtusa Full time

    Pyspark, AWS Architect (ATC) CREQ184787 Description 10+ years of relevant work experience showing growth as a Data Engineer.Hands On programming experienceImplementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation.Experience of performance optimization in Batch and Real time processing applicationsExpertise in Data Governance and Data...

  • Pyspark, AWS Architect

    1 month ago

    Pune, India Virtusa Full time

    Pyspark, AWS Architect (ATC) - CREQ184787 Description 10+ years of relevant work experience showing growth as a Data Engineer. Hands On programming experience Implementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation. Experience of performance optimization in Batch and Real time processing applications Expertise in Data Governance and...

  • Pyspark, AWS Architect

    3 weeks ago

    Pune, India Virtusa Full time

    Pyspark, AWS Architect (ATC) - CREQ184787 Description 10+ years of relevant work experience showing growth as a Data Engineer. Hands On programming experience Implementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation. Experience of performance optimization in Batch and Real time processing applications Expertise in Data Governance and...

  • Pyspark, AWS Architect

    1 month ago

    pune, India Virtusa Full time

    Pyspark, AWS Architect (ATC) - CREQ184787 Description 10+ years of relevant work experience showing growth as a Data Engineer.Hands On programming experienceImplementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation.Experience of performance optimization in Batch and Real time processing applicationsExpertise in Data Governance and Data...

  • Pyspark, AWS Architect

    3 weeks ago

    pune, India Virtusa Full time

    Pyspark, AWS Architect (ATC) - CREQ184787 Description 10+ years of relevant work experience showing growth as a Data Engineer.Hands On programming experienceImplementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation.Experience of performance optimization in Batch and Real time processing applicationsExpertise in Data Governance and Data...

  • Azure Data Architect

    3 weeks ago

    Pune, India ACZ Global Pvt Ltd Full time

    Mandatory Skills : Solution is mandatoryTotal Exp : 12 yrs , relevant exp-5yearsShift Timings: 1pm to 10 pmMode of work: Hybrid- 3 Days MandatoryAzure Data ArchitectMandatory Skills - Solution Architecture - Pyspark + Databricks + Adf + Synapse is mandatoryJob Description:We are seeking a highly skilled and experienced Azure Data Architect to join our team....

  • Azure Data Architect

    2 months ago

    Pune, India ACZ Global Pvt Ltd Full time

    Mandatory Skills : Solution is mandatoryTotal Exp : 12 yrs , relevant exp-5yearsShift Timings: 1pm to 10 pmMode of work: Hybrid- 3 Days MandatoryAzure Data ArchitectMandatory Skills - Solution Architecture - Pyspark + Databricks + Adf + Synapse is mandatoryJob Description:We are seeking a highly skilled and experienced Azure Data Architect to join our team....

  • Deloitte - Data Engineer - PySpark/Python

    1 month ago

    pune, India Deloitte India Full time

    Job Description : - Software design, Scala & Spark, PySpark & Python development, automated testing of new and existing components in an Agile, DevOps and dynamic environment Promoting development standards, code reviews, mentoring, knowledge sharing Production support & troubleshooting. - Implement the tools and processes, handling performance, scale,...

  • Deloitte - Data Engineer - PySpark/Python

    1 week ago

    Pune, Maharashtra, India Deloitte India Full time

    Job Description : - Software design, Scala & Spark, PySpark & Python development, automated testing of new and existing components in an Agile, DevOps and dynamic environment Promoting development standards, code reviews, mentoring, knowledge sharing Production support & troubleshooting. Implement the tools and processes, handling performance, scale,...

  • Deloitte - Data Engineer - PySpark/Python

    2 hours ago

    pune, India Deloitte India Full time

    Job Description : - Software design, Scala & Spark, PySpark & Python development, automated testing of new and existing components in an Agile, DevOps and dynamic environment Promoting development standards, code reviews, mentoring, knowledge sharing Production support & troubleshooting. - Implement the tools and processes, handling performance, scale,...

  • Solution Architect

    4 weeks ago

    Hinjawadi, Pune, Maharashtra, India BirlaSoft Full time

    Country/Region: IN- Requisition ID: 13410- Work Model: - Position Type: - Salary Range: - Location: INDIA - PUNE - BIRLASOFT OFFICE - HINJAWADI**Title**:Solution Architect**: - Description: - Azure Data Architect Experience: 14 to 16 Years, 3+ Years of experience in Azure Data Architect - Cloud Architect with experience in Azure ADF, Databricks, PySpark...

  • Solution Architect

    3 weeks ago

    Hinjawadi, Pune, Maharashtra, India BirlaSoft Full time

    Country/Region: IN- Requisition ID: 7975- Work Model: - Position Type: - Salary Range: - Location: INDIA - PUNE - BIRLASOFT OFFICE - HINJAWADI**Title**:Solution Architect**: - Description: - Azure Data Architect Experience: 14 to 16 Years, 3+ Years of experience in Azure Data Architect - Cloud Architect with experience in Azure ADF, Databricks, PySpark -...

  • Solution Architect

    1 day ago

    Hinjawadi, Pune, Maharashtra, India BirlaSoft Full time

    Country/Region: IN- Requisition ID: 13410- Work Model: - Position Type: - Salary Range: - Location: INDIA - PUNE - BIRLASOFT OFFICE - HINJAWADI**Title**:Solution Architect**: - Description: - Azure Data Architect Experience: 14 to 16 Years, 3+ Years of experience in Azure Data Architect - Cloud Architect with experience in Azure ADF, Databricks, PySpark...

  • Azure Data Architect

    2 months ago

    Pune, India ACZ Global Pvt Ltd Full time

    Role : Azure Data ArchitectMandatory Skills : Solution Architecture - Pyspark + Databricks + Adf + Synapse is mandatoryJob Description :We are seeking a highly skilled and experienced Azure Data Architect to join our team. As an Azure Data Architect, you will play a key role in designing and implementing data solutions on the Microsoft Azure platform. The...