PySpark Architect

1 week ago

Kolkata, West Bengal, India RapidBraiins Full time
Job Description

As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.

  • You will work closely with crossfunctional teams to develop robust, scalable, and efficient data solutions.
  • Your deep experience in big data technologies and handson expertise in PySpark will be critical in driving our data initiatives forward.

Key Responsibilities :

Architectural Leadership :

  • Lead the architecture, design, and implementation of largescale data processing applications using PySpark.
  • Provide technical guidance and mentorship to development teams on best practices for data processing and PySpark development.

Solution Design and Development :

  • Design and implement endtoend data pipelines to ingest, process, and analyze large volumes of data.
  • Develop scalable and efficient PySpark applications to handle complex data transformations and aggregations.
  • Ensure the solutions are designed for high availability, scalability, and performance.

Optimization and Performance Tuning :

  • Optimize PySpark applications for performance and efficiency.
  • Perform code reviews and provide recommendations for improvements.
  • Identify and resolve performance bottlenecks in data processing workflows.

Collaboration and Stakeholder Management :

  • Work closely with data engineers, data scientists, and other stakeholders to understand their requirements and translate them into technical solutions.
  • Collaborate with infrastructure teams to ensure the deployment and scaling of data applications.

Documentation and Standards :

  • Maintain comprehensive documentation of the architecture, design, and implementation details.
  • Establish and enforce coding standards, best practices, and development methodologies.

Continuous Improvement :

  • Stay updated with the latest trends and advancements in big data technologies.
  • Identify opportunities for continuous improvement in data processing and analytics solutions.

Required Qualifications:

  • Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
  • 12 to 15 years of experience in software development with at least 5 years of experience in big data technologies.
  • Extensive handson experience with PySpark and the Spark ecosystem.
  • Strong expertise in designing and implementing largescale data processing systems.
  • Proficiency in Python programming and related libraries for data processing.
  • Solid understanding of distributed computing principles and big data architectures.
  • Experience with cloud platforms such as AWS, Azure, or GCP.
  • Strong knowledge of data warehousing and ETL processes.
  • Excellent problemsolving and analytical skills.
  • Strong communication and collaboration skills, with the ability to work effectively with crossfunctional teams.

Preferred Qualifications:

  • Experience with additional big data technologies such as Hadoop, Hive, or Kafka.
  • Knowledge of containerization and orchestration tools such as Docker and Kubernetes.
  • Familiarity with data visualization tools and techniques.
  • Certification in cloud platforms or big data technologies.

Job Location:
Chennai/Pune/Kolkata

)

  • Azure Data Bricks+pyspark

    1 week ago

    Kolkata, West Bengal, India Cognizant Full time

    Roles and responsibilities: Mandatory: Strong in Azure, ADF, Data Lake, Databricks, Pyspark Handsonexperience in developing data lake solutions using Azure (Azure data factory for ingestion, Data lake gen 2 and Azure SQL server for storage, Azure analysis service for transformations, Azure data bricks) Implement a robust data pipeline using Microsoft Stack....

  • Architect in Data Engineering

    1 week ago

    Kolkata, West Bengal, India Techylla Full time

    Company Overview Techylla is a specialized IT consulting firm based in India and US. Our focus lies in collaborating with Life Sciences companies, offering Data-Driven Decision Making solutions that encompass a spectrum of IT services and technology-driven business solutions. Our vision is to become the preferred partner for analytics, bridging the gap...

  • PySpark Architect

    3 weeks ago

    Chennai/Kolkata/Pune, India RapidBraiins Full time

    Job Description - As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.- You will work closely with cross-functional teams to develop robust, scalable, and efficient data solutions.- Your deep experience in big data technologies and hands-on expertise in PySpark will...

  • PySpark Architect

    1 month ago

    Chennai/Kolkata/Pune, India RapidBraiins Full time

    Job Description - As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.- You will work closely with cross-functional teams to develop robust, scalable, and efficient data solutions.- Your deep experience in big data technologies and hands-on expertise in PySpark will...

  • PySpark Architect

    4 weeks ago

    Chennai/Kolkata/Pune, IN RapidBraiins Full time

    Job Description- As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.- You will work closely with cross-functional teams to develop robust, scalable, and efficient data solutions.- Your deep experience in big data technologies and hands-on expertise in PySpark will be...

  • PySpark Architect

    3 weeks ago

    Chennai/Kolkata/Pune, IN RapidBraiins Full time

    Job Description- As a PySpark Architect, you will be responsible for architecting, designing, and implementing large-scale data processing systems using PySpark.- You will work closely with cross-functional teams to develop robust, scalable, and efficient data solutions.- Your deep experience in big data technologies and hands-on expertise in PySpark will be...

  • Databricks Engineer

    3 weeks ago

    Bangalore/Chennai/Delhi NCR/Pune/Kolkata, India Futureleap search partners LLP Full time

    Main Skills - Azure Databricks, PySpark, SQL and Azure Cloud. Role : Senior Databricks Engineer / Databricks Technical Lead/ Data ArchitectExperience : 8-15 yearsLocation : Bangalore, Chennai, Delhi, PuneAbout Company : We focus on last-mile delivery of powerful insights into profitable actions by uniting its strengths in business analytics, data science...

  • Databricks Engineer

    1 month ago

    Bangalore/Chennai/Delhi NCR/Pune/Kolkata, IN Futureleap search partners LLP Full time

    Main Skills - Azure Databricks, PySpark, SQL and Azure Cloud.Role : Senior Databricks Engineer / Databricks Technical Lead/ Data ArchitectExperience : 8-15 yearsLocation : Bangalore, Chennai, Delhi, PuneAbout Company :We focus on last-mile delivery of powerful insights into profitable actions by uniting its strengths in business analytics, data science and...

  • Databricks Engineer

    1 month ago

    Bangalore/Chennai/Delhi NCR/Pune/Kolkata, India Futureleap search partners LLP Full time

    Main Skills - Azure Databricks, PySpark, SQL and Azure Cloud. Role : Senior Databricks Engineer / Databricks Technical Lead/ Data ArchitectExperience : 8-15 yearsLocation : Bangalore, Chennai, Delhi, PuneAbout Company : We focus on last-mile delivery of powerful insights into profitable actions by uniting its strengths in business analytics, data science...

  • Databricks Engineer

    3 weeks ago

    Bangalore/Chennai/Delhi NCR/Pune/Kolkata, IN Futureleap search partners LLP Full time

    Main Skills - Azure Databricks, PySpark, SQL and Azure Cloud.Role : Senior Databricks Engineer / Databricks Technical Lead/ Data ArchitectExperience : 8-15 yearsLocation : Bangalore, Chennai, Delhi, PuneAbout Company :We focus on last-mile delivery of powerful insights into profitable actions by uniting its strengths in business analytics, data science and...

  • Senior Principal Consultant

    2 months ago

    Kolkata, India Genpact Full time

    A Data Architectwho can design and implement data modernization solutions in the cloud is responsible for developing and implementing data architecture strategies for organizations transitioning their data systems to the cloud. They collaborate with stakeholders to understand business requirements and translate them into scalable, secure, and high-performing...

  • Solutions Architect

    2 days ago

    kolkata, India LTIMindtree Full time

    Overall 15+ years of experience in Data & Analytics Architecture & Design Solid experience leading data teams in developing data engineering platforms. Good working knowledge of one hyperscalar data services and associated data engineering tech stack: 1-Azure Data Factory (ADF), ADF metadata driven pipelines, MS SQL Server, Python and PySpark, MS Purview 2...

  • Solutions Architect

    13 hours ago

    Kolkata, India LTIMindtree Full time

    Overall 15+ years of experience in Data & Analytics Architecture & DesignSolid experience leading data teams in developing data engineering platforms.Good working knowledge of one hyperscalar data services and associated data engineering tech stack:1-Azure Data Factory (ADF), ADF metadata driven pipelines, MS SQL Server, Python and PySpark, MS Purview2...

  • Solutions Architect

    3 days ago

    Kolkata, India LTIMindtree Full time

    Overall 15+ years of experience in Data & Analytics Architecture & DesignSolid experience leading data teams in developing data engineering platforms.Good working knowledge of one hyperscalar data services and associated data engineering tech stack:1-Azure Data Factory (ADF), ADF metadata driven pipelines, MS SQL Server, Python and PySpark, MS Purview2...

  • Solutions Architect

    2 days ago

    Kolkata, India LTIMindtree Full time

    Overall 15+ years of experience in Data & Analytics Architecture & DesignSolid experience leading data teams in developing data engineering platforms.Good working knowledge of one hyperscalar data services and associated data engineering tech stack:1-Azure Data Factory (ADF), ADF metadata driven pipelines, MS SQL Server, Python and PySpark, MS Purview2...

  • Senior Consultant

    2 months ago

    Bangalore,Gurgaon,Gurugram,Pune,Chennai,Kolkata, India VIDPRO CONSULTANCY SERVICES Full time

    Primary Roles and Responsibilities : - Developing Modern Data Warehouse solutions using Databricks and Azure Stack - Ability to provide solutions that are forward-thinking in data engineering and analytics space- Collaborate with DW/BI leads to understand new ETL pipeline development requirements.- Triage issues to find gaps in existing pipelines and fix...

  • Azure Data Engineer

    2 months ago

    Bangalore/Pune/Kolkata/Chennai/Gurgaon/Gurugram, IN VIDPRO CONSULTANCY SERVICES Full time

    Primary Roles and Responsibilities:- Developing Modern Data Warehouse solutions using Databricks and Azure Stack - Ability to provide solutions that are forward-thinking in data engineering and analytics space- Collaborate with DW/BI leads to understand new ETL pipeline development requirements.- Triage issues to find gaps in existing pipelines and fix the...

  • Azure Data Engineer

    2 months ago

    Bangalore/Pune/Kolkata/Chennai/Gurgaon/Gurugram, India VIDPRO CONSULTANCY SERVICES Full time

    Primary Roles and Responsibilities: - Developing Modern Data Warehouse solutions using Databricks and Azure Stack - Ability to provide solutions that are forward-thinking in data engineering and analytics space- Collaborate with DW/BI leads to understand new ETL pipeline development requirements.- Triage issues to find gaps in existing pipelines and fix the...

  • Azure Data Engineer

    3 weeks ago

    Bangalore/Pune/Kolkata/Chennai/Gurgaon/Gurugram, IN VIDPRO CONSULTANCY SERVICES Full time

    Primary Roles and Responsibilities:- Developing Modern Data Warehouse solutions using Databricks and Azure Stack - Ability to provide solutions that are forward-thinking in data engineering and analytics space- Collaborate with DW/BI leads to understand new ETL pipeline development requirements.- Triage issues to find gaps in existing pipelines and fix the...

  • Senior Consultant

    3 weeks ago

    Bangalore/Gurgaon/Gurugram/Pune/Chennai/Kolkata, India VIDPRO CONSULTANCY SERVICES Full time

    Primary Roles and Responsibilities : - Developing Modern Data Warehouse solutions using Databricks and Azure Stack - Ability to provide solutions that are forward-thinking in data engineering and analytics space- Collaborate with DW/BI leads to understand new ETL pipeline development requirements.- Triage issues to find gaps in existing pipelines and fix...

  • Data Engineer

    3 weeks ago

    Bangalore/Pune/Kolkata/Gurgaon/Gurugram/Chennai, India VIDPRO CONSULTANCY SERVICES Full time

    Mandatory Skills : Snowflake/Azure Data Factory/ PySpark / DatabricksPrimary Roles and Responsibilities : - Developing Modern Data Warehouse solutions using Snowflake, Databricks and ADF.- Ability to provide solutions that are forward-thinking in data engineering and analytics space- Collaborate with DW/BI leads to understand new ETL pipeline development...

  • Data Engineer

    2 months ago

    Bangalore/Pune/Kolkata/Gurgaon/Gurugram/Chennai, India VIDPRO CONSULTANCY SERVICES Full time

    Mandatory Skills : Snowflake/Azure Data Factory/ PySpark / DatabricksPrimary Roles and Responsibilities : - Developing Modern Data Warehouse solutions using Snowflake, Databricks and ADF.- Ability to provide solutions that are forward-thinking in data engineering and analytics space- Collaborate with DW/BI leads to understand new ETL pipeline development...