PySpark Developer

2 weeks ago


Gurugram, India Waytogo Consultants Full time

Responsibilities :

  • Design and develop PySpark applications using Spark DataFrames and SQL to extract, transform, and load (ETL) data.
  • Read data from various sources such as relational databases, data lakes, and cloud storage using PySpark connectors.
  • Clean, transform, and enrich data using PySpark functions for data wrangling and manipulation.
  • Build and maintain robust data pipelines using PySpark for efficient data processing.
  • Optimize PySpark applications for performance and scalability to handle large datasets effectively.
  • Collaborate with engineers and data scientists to understand data requirements and develop data-driven solutions.
  • Write unit tests for PySpark applications to ensure code quality and maintainability.
  • Document PySpark code and applications clearly and concisely for future reference and knowledge sharing.

Qualifications :

  • Bachelor's degree in Computer Science, Information Technology, or a related field (or equivalent experience).
  • 2+ years of experience in developing big data applications using PySpark.
  • Strong proficiency in Python programming language and object-oriented programming concepts.
  • In-depth understanding of Apache Spark architecture, including Spark DataFrames, Spark SQL, and distributed processing.
  • Experience working with big data platforms such as Hadoop, YARN, and data lakes (e.g., AWS S3, Azure Data Lake Storage).
  • Experience with cloud platforms (AWS, Azure, GCP) is a plus.
(ref:hirist.tech)
  • PySpark Developer

    1 month ago


    gurugram, India Waytogo Consultants Full time

    Responsibilities : Design and develop PySpark applications using Spark DataFrames and SQL to extract, transform, and load (ETL) data. Read data from various sources such as relational databases, data lakes, and cloud storage using PySpark connectors. Clean, transform, and enrich data using PySpark functions for data wrangling and manipulation. Build and...

  • PySpark Developer

    2 weeks ago


    Gurgaon/Gurugram, India Waytogo Consultants Full time

    Responsibilities : Design and develop PySpark applications using Spark DataFrames and SQL to extract, transform, and load (ETL) data. Read data from various sources such as relational databases, data lakes, and cloud storage using PySpark connectors. Clean, transform, and enrich data using PySpark functions for data wrangling and manipulation. Build and...

  • PySpark Developer

    4 weeks ago


    Gurgaon/Gurugram, IN Waytogo Consultants Full time

    Responsibilities :Design and develop PySpark applications using Spark DataFrames and SQL to extract, transform, and load (ETL) data.Read data from various sources such as relational databases, data lakes, and cloud storage using PySpark connectors.Clean, transform, and enrich data using PySpark functions for data wrangling and manipulation.Build and maintain...

  • Data Engineer

    1 month ago


    gurugram, India Cognizant Full time

    Position Summary: The Data engineer is responsible for development & maintenance of the Data Lake within Cognizant. You should be able to collaborate with technology cross-commits and business teams to code, test and deploy scalable solution and adhering to required quality standards. You must be a self-starter, think outside the box and enjoy coming...

  • Data Engineer

    1 month ago


    Gurugram, India Cognizant Full time

    Position Summary: The Data engineer is responsible for development & maintenance of the Data Lake within Cognizant. You should be able to collaborate with technology cross-commits and business teams to code, test and deploy scalable solution and adhering to required quality standards. You must be a self-starter, think outside the box and enjoy coming up...

  • AWS Data Engineer

    1 month ago


    Gurugram, India True Tech Professionals Full time

    We are seeking a skilled Senior Data Engineer to join our dynamic team.- The ideal candidate should have a strong background in data engineering with expertise in Python, Pyspark, AWS Glue, Athena, Redshift, and SQL.- The successful candidate will play a key role in designing, developing, and maintaining our data infrastructure, ensuring optimal performance...


  • Gurugram, India COFORGE Full time

    Responsibilities :Model Deployment & Management :- Ensure seamless operation of deployed machine learning models within the RCP platform.- Monitor and troubleshoot model performance for accuracy and efficiency.- Collaborate with data scientists and software engineers to optimize model pipelines for production.Machine Learning Operations (MLOps) :- Utilize...

  • ETL Data Engineer

    1 month ago


    Gurgaon,Gurugram, India Coders Brain Pvt Ltd Full time

    Job Description: ETL - Data Design, Develop and maintain ETL pipelines using Pyspark in Azure Databricks using delta tables.- Create build from Github and release pipeline for Ingestion and Databricks using Azure Devops / Harness- Monitor Performance of ETL Jobs, resolve any issue that arose and improve the performance metrics as needed.- Diagnose system...

  • AWS Data Engineer

    1 month ago


    Gurgaon,Gurugram, India True Tech Professionals Full time

    We are seeking a skilled Senior Data Engineer to join our dynamic team.- The ideal candidate should have a strong background in data engineering with expertise in Python, Pyspark, AWS Glue, Athena, Redshift, and SQL.- The successful candidate will play a key role in designing, developing, and maintaining our data infrastructure, ensuring optimal performance...

  • AWS Data Engineer

    4 weeks ago


    Gurgaon/Gurugram, IN True Tech Professionals Full time

    We are seeking a skilled Senior Data Engineer to join our dynamic team.- The ideal candidate should have a strong background in data engineering with expertise in Python, Pyspark, AWS Glue, Athena, Redshift, and SQL.- The successful candidate will play a key role in designing, developing, and maintaining our data infrastructure, ensuring optimal performance...


  • Gurugram, India NatWest Digital X Full time

    Join us as a Software EngineerThis is an opportunity for a driven Software Engineer to design and engineer software with the customer or user experience as the primary objectiveWe’ll look to you to engineer and maintain innovative, customer centric, high performance, secure and robust solutionsIt’s a chance to hone your existing technical skills and...


  • Gurgaon/Gurugram, India COFORGE Full time

    Responsibilities :Model Deployment & Management :- Ensure seamless operation of deployed machine learning models within the RCP platform.- Monitor and troubleshoot model performance for accuracy and efficiency.- Collaborate with data scientists and software engineers to optimize model pipelines for production.Machine Learning Operations (MLOps) :- Utilize...


  • Gurgaon/Gurugram, IN COFORGE Full time

    Responsibilities :Model Deployment & Management :- Ensure seamless operation of deployed machine learning models within the RCP platform.- Monitor and troubleshoot model performance for accuracy and efficiency.- Collaborate with data scientists and software engineers to optimize model pipelines for production.Machine Learning Operations (MLOps) :- Utilize...

  • Business Analyst

    1 month ago


    Pune,Bangalore,Hyderabad,Gurgaon,Gurugram,Chennai,Mumbai, India Idyllic Services Pvt Ltd Full time

    Job Description : Mandatory Skills : Business Analysis and PySpark (Both Mandatory). - Understanding of Global Change Delivery Business Transformation Frameworks, methodologies and best practice techniques - Outstanding understanding of Client Group structures, processes and objectives. - Understanding of banking and understanding of how change drives...


  • Gurugram, India Huquo Full time

    Role :- Design and deploy machine learning systems- Research and implement appropriate ML algorithms and tools- Develop machine learning applications according to requirements- Select appropriate datasets and data representation methods- Perform statistical analysis and fine-tuning using test results- Train and retrain systems when necessary- Extend existing...


  • gurugram, India Huquo Full time

    Role :- Design and deploy machine learning systems- Research and implement appropriate ML algorithms and tools- Develop machine learning applications according to requirements- Select appropriate datasets and data representation methods- Perform statistical analysis and fine-tuning using test results- Train and retrain systems when necessary- Extend existing...

  • Data Engineer

    1 month ago


    Bangalore,Gurgaon,Gurugram, India Symphoni HR Full time

    Role : Data EngineerOrganization : A Leading Consulting firmExperience : 4 to 8 years / 8 to 12 yearsQualification : B.Tech / B.E.Major exposure required in SQL, Pyspark / Spark and Azure cloudResponsibilities :- Design, build, and optimize data pipelines and ETL processes to ensure efficient data ingestion and data integration.- Develop and maintain...

  • Data Engineer

    2 weeks ago


    Gurugram, India Intuitive Apps Full time

    Key Skills : Minimum 6 years Exp in Data Engineer , 3yrs in Azure Data Factory, Pyspark, DatabricksJob Description :- Create and maintain optimal data pipeline architecture- Knowledge on Data Modelling, Data warehouse, and Data Analysis is also required.- Design and develop Data Warehouse solutions- Assemble large, complex data sets that meet functional /...

  • Data Engineer

    1 month ago


    Bangalore,Gurgaon,Gurugram, India GlobeHR Services Full time

    Job Title : Data EngineerExperience : 4 - 7 yearsJob Location : Period : 15- 20 DaysKey Skills : Azure Data Factory, Pyspark, T/SQL, PL/SQL, Exposure on other ETL Tools like SSIS, DatabricksJob Description : - Create and maintain optimal data pipeline architecture- Design and develop Data Warehouse solutions- Assemble large, complex data sets that meet...


  • Bangalore/Chennai/Pune/Gurgaon/Gurugram/Kolkata, India Pylon Management Consulting Full time

    Primary Roles and Responsibilities:- Developing Modern Data Warehouse solutions using Databricks and Azure Stack- Ability to provide solutions that are forward-thinking in data engineering and analytics space- Triage issues to find gaps in existing pipelines and fix the issues- Work with business to understand the need in reporting layer and develop data...