PySpark Architect

4 weeks ago

ChennaiKolkataPune, India RapidBraiins Full time

Job Description


- As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.

- You will work closely with cross-functional teams to develop robust, scalable, and efficient data solutions.

- Your deep experience in big data technologies and hands-on expertise in PySpark will be critical in driving our data initiatives forward.

Key Responsibilities :

Architectural Leadership :

- Lead the architecture, design, and implementation of large-scale data processing applications using PySpark.

- Provide technical guidance and mentorship to development teams on best practices for data processing and PySpark development.

Solution Design and Development :

- Design and implement end-to-end data pipelines to ingest, process, and analyze large volumes of data.

- Develop scalable and efficient PySpark applications to handle complex data transformations and aggregations.

- Ensure the solutions are designed for high availability, scalability, and performance.

Optimization and Performance Tuning :

- Optimize PySpark applications for performance and efficiency.

- Perform code reviews and provide recommendations for improvements.

- Identify and resolve performance bottlenecks in data processing workflows.

Collaboration and Stakeholder Management :

- Work closely with data engineers, data scientists, and other stakeholders to understand their requirements and translate them into technical solutions.

- Collaborate with infrastructure teams to ensure the deployment and scaling of data applications.

Documentation and Standards :

- Maintain comprehensive documentation of the architecture, design, and implementation details.

- Establish and enforce coding standards, best practices, and development methodologies.

Continuous Improvement :

- Stay updated with the latest trends and advancements in big data technologies.

- Identify opportunities for continuous improvement in data processing and analytics solutions.

Required Qualifications:

- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.

- 12 to 15 years of experience in software development with at least 5 years of experience in big data technologies.

- Extensive hands-on experience with PySpark and the Spark ecosystem.

- Strong expertise in designing and implementing large-scale data processing systems.

- Proficiency in Python programming and related libraries for data processing.

- Solid understanding of distributed computing principles and big data architectures.

- Experience with cloud platforms such as AWS, Azure, or GCP.

- Strong knowledge of data warehousing and ETL processes.

- Excellent problem-solving and analytical skills.

- Strong communication and collaboration skills, with the ability to work effectively with cross-functional teams.

Preferred Qualifications:

- Experience with additional big data technologies such as Hadoop, Hive, or Kafka.

- Knowledge of containerization and orchestration tools such as Docker and Kubernetes.

- Familiarity with data visualization tools and techniques.

- Certification in cloud platforms or big data technologies.

Job Location: Chennai/Pune/Kolkata

(ref:hirist.tech)

  • PySpark Architect

    6 days ago

    Kolkata, West Bengal, India RapidBraiins Full time

    Job DescriptionAs a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark. You will work closely with crossfunctional teams to develop robust, scalable, and efficient data solutions. Your deep experience in big data technologies and handson expertise in PySpark will be...

  • PySpark Architect

    4 weeks ago

    Chennai/Kolkata/Pune, IN RapidBraiins Full time

    Job Description- As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.- You will work closely with cross-functional teams to develop robust, scalable, and efficient data solutions.- Your deep experience in big data technologies and hands-on expertise in PySpark will be...

  • PySpark Architect

    3 weeks ago

    Chennai/Kolkata/Pune, IN RapidBraiins Full time

    Job Description- As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.- You will work closely with cross-functional teams to develop robust, scalable, and efficient data solutions.- Your deep experience in big data technologies and hands-on expertise in PySpark will be...

  • Pyspark AWS Architect ATC

    2 weeks ago

    Pune, Maharashtra, India virtusa consulting services pvt. ltd. Full time

    Apply for Pyspark, AWS Architect ATC, Career Progress Consultants in Pune for Year of Experience on

  • Pyspark AWS Architect ATC

    2 weeks ago

    Pune, India virtusa consulting services pvt. ltd. Full time

    Apply for Pyspark, AWS Architect ATC, Career Progress Consultants in Pune for 10 - 12 Year of Experience on TimesJobs.com.

  • PySpark Architect

    4 weeks ago

    Chennai, India RapidBraiins Full time

    Job Description- As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.- You will work closely with cross-functional teams to develop robust, scalable, and efficient data solutions.- Your deep experience in big data technologies and hands-on expertise in PySpark will be...

  • PySpark Architect

    3 weeks ago

    Chennai, India RapidBraiins Full time

    Job Description- As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.- You will work closely with cross-functional teams to develop robust, scalable, and efficient data solutions.- Your deep experience in big data technologies and hands-on expertise in PySpark will be...

  • PySpark Architect

    4 weeks ago

    pune, India RapidBraiins Full time

    Job Description - As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.- You will work closely with cross-functional teams to develop robust, scalable, and efficient data solutions.- Your deep experience in big data technologies and hands-on expertise in PySpark will...

  • PySpark Architect

    1 week ago

    pune, India RapidBraiins Full time

    Job Description - As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.- You will work closely with cross-functional teams to develop robust, scalable, and efficient data solutions.- Your deep experience in big data technologies and hands-on expertise in PySpark will...

  • Pyspark, AWS Architect

    6 days ago

    Pune, Maharashtra, India Virtusa Full time

    Pyspark, AWS Architect (ATC) CREQ184787 Description 10+ years of relevant work experience showing growth as a Data Engineer.Hands On programming experienceImplementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation.Experience of performance optimization in Batch and Real time processing applicationsExpertise in Data Governance and Data...

  • Azure Data Architect

    1 month ago

    Chennai, India ACZ Global Private Limited Full time

    Azure Data ArchitectMandatory Skills : Solution Architecture - Pyspark + Databricks + Adf + Synapse is mandatoryJob Description :We are seeking a highly skilled and experienced Azure Data Architect to join our team. As an Azure Data Architect, you will play a key role in designing and implementing data solutions on the Microsoft Azure platform. The ideal...

  • Pyspark, AWS Architect

    1 month ago

    Pune, India Virtusa Full time

    Pyspark, AWS Architect (ATC) - CREQ184787 Description 10+ years of relevant work experience showing growth as a Data Engineer.Hands On programming experienceImplementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation.Experience of performance optimization in Batch and Real time processing applicationsExpertise in Data Governance and Data...

  • Pyspark, AWS Architect

    3 weeks ago

    Pune, India Virtusa Full time

    Pyspark, AWS Architect (ATC) - CREQ184787 Description 10+ years of relevant work experience showing growth as a Data Engineer. Hands On programming experience Implementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation. Experience of performance optimization in Batch and Real time processing applications Expertise in Data Governance and...

  • Pyspark, AWS Architect

    3 weeks ago

    pune, India Virtusa Full time

    Pyspark, AWS Architect (ATC) - CREQ184787 Description 10+ years of relevant work experience showing growth as a Data Engineer.Hands On programming experienceImplementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation.Experience of performance optimization in Batch and Real time processing applicationsExpertise in Data Governance and Data...

  • Pyspark, AWS Architect

    1 month ago

    Pune, India Virtusa Full time

    Pyspark, AWS Architect (ATC) - CREQ184787 Description 10+ years of relevant work experience showing growth as a Data Engineer. Hands On programming experience Implementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation. Experience of performance optimization in Batch and Real time processing applications Expertise in Data Governance and...

  • Pyspark, AWS Architect

    1 month ago

    pune, India Virtusa Full time

    Pyspark, AWS Architect (ATC) - CREQ184787 Description 10+ years of relevant work experience showing growth as a Data Engineer.Hands On programming experienceImplementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS LakeFormation.Experience of performance optimization in Batch and Real time processing applicationsExpertise in Data Governance and Data...

  • Azure Data Architect

    1 month ago

    Bangalore/Chennai/Hyderabad/Mumbai/Pune/Noida, IN ACZ Global Private Limited Full time

    Azure Data ArchitectMandatory Skills : Solution Architecture - Pyspark + Databricks + Adf + Synapse is mandatoryJob Description :We are seeking a highly skilled and experienced Azure Data Architect to join our team. As an Azure Data Architect, you will play a key role in designing and implementing data solutions on the Microsoft Azure platform. The ideal...

  • Azure Data Architect

    1 month ago

    Bangalore/Chennai/Hyderabad/Mumbai/Pune/Noida, India ACZ Global Private Limited Full time

    Azure Data ArchitectMandatory Skills : Solution Architecture - Pyspark + Databricks + Adf + Synapse is mandatoryJob Description :We are seeking a highly skilled and experienced Azure Data Architect to join our team. As an Azure Data Architect, you will play a key role in designing and implementing data solutions on the Microsoft Azure platform. The ideal...

  • Azure Data Bricks+pyspark

    6 days ago

    Kolkata, West Bengal, India Cognizant Full time

    Roles and responsibilities: Mandatory: Strong in Azure, ADF, Data Lake, Databricks, Pyspark Handsonexperience in developing data lake solutions using Azure (Azure data factory for ingestion, Data lake gen 2 and Azure SQL server for storage, Azure analysis service for transformations, Azure data bricks) Implement a robust data pipeline using Microsoft Stack....

  • Senior Data Engineer

    2 months ago

    Chennai, India Cyber Sphere LLC Full time

    Senior Data EngineerOnsite : Mumbai/ChennaiAbout the Role :- This role is more focused on Pyspark with Cloud developer.About the Responsibilities :- This position provides direct input to project plans, schedules, and follows software methodologies and best practices in the development of cross-functional software products under a micro-services styled...