PySpark Developer
1 month ago
Responsibilities :
- Design and develop PySpark applications using Spark DataFrames and SQL to extract, transform, and load (ETL) data.
- Read data from various sources such as relational databases, data lakes, and cloud storage using PySpark connectors.
- Clean, transform, and enrich data using PySpark functions for data wrangling and manipulation.
- Build and maintain robust data pipelines using PySpark for efficient data processing.
- Optimize PySpark applications for performance and scalability to handle large datasets effectively.
- Collaborate with engineers and data scientists to understand data requirements and develop data-driven solutions.
- Write unit tests for PySpark applications to ensure code quality and maintainability.
- Document PySpark code and applications clearly and concisely for future reference and knowledge sharing.
Qualifications :
- Bachelor's degree in Computer Science, Information Technology, or a related field (or equivalent experience).
- 2+ years of experience in developing big data applications using PySpark.
- Strong proficiency in Python programming language and object-oriented programming concepts.
- In-depth understanding of Apache Spark architecture, including Spark DataFrames, Spark SQL, and distributed processing.
- Experience working with big data platforms such as Hadoop, YARN, and data lakes (e.g., AWS S3, Azure Data Lake Storage).
- Experience with cloud platforms (AWS, Azure, GCP) is a plus.
-
PySpark Developer
1 month ago
gurugram, India Waytogo Consultants Full timeResponsibilities : Design and develop PySpark applications using Spark DataFrames and SQL to extract, transform, and load (ETL) data. Read data from various sources such as relational databases, data lakes, and cloud storage using PySpark connectors. Clean, transform, and enrich data using PySpark functions for data wrangling and manipulation. Build and...
-
PySpark Developer
4 weeks ago
Gurgaon/Gurugram, IN Waytogo Consultants Full timeResponsibilities :Design and develop PySpark applications using Spark DataFrames and SQL to extract, transform, and load (ETL) data.Read data from various sources such as relational databases, data lakes, and cloud storage using PySpark connectors.Clean, transform, and enrich data using PySpark functions for data wrangling and manipulation.Build and maintain...
-
PySpark Developer
2 weeks ago
Gurgaon/Gurugram, India Waytogo Consultants Full timeResponsibilities : Design and develop PySpark applications using Spark DataFrames and SQL to extract, transform, and load (ETL) data. Read data from various sources such as relational databases, data lakes, and cloud storage using PySpark connectors. Clean, transform, and enrich data using PySpark functions for data wrangling and manipulation. Build and...
-
Data Engineer
1 month ago
gurugram, India Cognizant Full timePosition Summary: The Data engineer is responsible for development & maintenance of the Data Lake within Cognizant. You should be able to collaborate with technology cross-commits and business teams to code, test and deploy scalable solution and adhering to required quality standards. You must be a self-starter, think outside the box and enjoy coming...
-
Data Engineer
1 month ago
Gurugram, India Cognizant Full timePosition Summary: The Data engineer is responsible for development & maintenance of the Data Lake within Cognizant. You should be able to collaborate with technology cross-commits and business teams to code, test and deploy scalable solution and adhering to required quality standards. You must be a self-starter, think outside the box and enjoy coming up...
-
AWS Data Engineer
1 month ago
Gurugram, India True Tech Professionals Full timeWe are seeking a skilled Senior Data Engineer to join our dynamic team.- The ideal candidate should have a strong background in data engineering with expertise in Python, Pyspark, AWS Glue, Athena, Redshift, and SQL.- The successful candidate will play a key role in designing, developing, and maintaining our data infrastructure, ensuring optimal performance...
-
Gurugram, India COFORGE Full timeResponsibilities :Model Deployment & Management :- Ensure seamless operation of deployed machine learning models within the RCP platform.- Monitor and troubleshoot model performance for accuracy and efficiency.- Collaborate with data scientists and software engineers to optimize model pipelines for production.Machine Learning Operations (MLOps) :- Utilize...
-
ETL Data Engineer
1 month ago
Gurgaon,Gurugram, India Coders Brain Pvt Ltd Full timeJob Description: ETL - Data Design, Develop and maintain ETL pipelines using Pyspark in Azure Databricks using delta tables.- Create build from Github and release pipeline for Ingestion and Databricks using Azure Devops / Harness- Monitor Performance of ETL Jobs, resolve any issue that arose and improve the performance metrics as needed.- Diagnose system...
-
AWS Data Engineer
4 weeks ago
Gurgaon/Gurugram, IN True Tech Professionals Full timeWe are seeking a skilled Senior Data Engineer to join our dynamic team.- The ideal candidate should have a strong background in data engineering with expertise in Python, Pyspark, AWS Glue, Athena, Redshift, and SQL.- The successful candidate will play a key role in designing, developing, and maintaining our data infrastructure, ensuring optimal performance...
-
AWS Data Engineer
1 month ago
Gurgaon,Gurugram, India True Tech Professionals Full timeWe are seeking a skilled Senior Data Engineer to join our dynamic team.- The ideal candidate should have a strong background in data engineering with expertise in Python, Pyspark, AWS Glue, Athena, Redshift, and SQL.- The successful candidate will play a key role in designing, developing, and maintaining our data infrastructure, ensuring optimal performance...
-
Software Engineer, PySpark
1 month ago
Gurugram, India NatWest Digital X Full timeJoin us as a Software EngineerThis is an opportunity for a driven Software Engineer to design and engineer software with the customer or user experience as the primary objectiveWe’ll look to you to engineer and maintain innovative, customer centric, high performance, secure and robust solutionsIt’s a chance to hone your existing technical skills and...
-
Gurgaon/Gurugram, IN COFORGE Full timeResponsibilities :Model Deployment & Management :- Ensure seamless operation of deployed machine learning models within the RCP platform.- Monitor and troubleshoot model performance for accuracy and efficiency.- Collaborate with data scientists and software engineers to optimize model pipelines for production.Machine Learning Operations (MLOps) :- Utilize...
-
Gurgaon/Gurugram, India COFORGE Full timeResponsibilities :Model Deployment & Management :- Ensure seamless operation of deployed machine learning models within the RCP platform.- Monitor and troubleshoot model performance for accuracy and efficiency.- Collaborate with data scientists and software engineers to optimize model pipelines for production.Machine Learning Operations (MLOps) :- Utilize...
-
Business Analyst
1 month ago
Pune,Bangalore,Hyderabad,Gurgaon,Gurugram,Chennai,Mumbai, India Idyllic Services Pvt Ltd Full timeJob Description : Mandatory Skills : Business Analysis and PySpark (Both Mandatory). - Understanding of Global Change Delivery Business Transformation Frameworks, methodologies and best practice techniques - Outstanding understanding of Client Group structures, processes and objectives. - Understanding of banking and understanding of how change drives...
-
Machine Learning Engineer/Architect
2 weeks ago
Gurugram, India Huquo Full timeRole :- Design and deploy machine learning systems- Research and implement appropriate ML algorithms and tools- Develop machine learning applications according to requirements- Select appropriate datasets and data representation methods- Perform statistical analysis and fine-tuning using test results- Train and retrain systems when necessary- Extend existing...
-
Machine Learning Engineer/Architect
1 month ago
gurugram, India Huquo Full timeRole :- Design and deploy machine learning systems- Research and implement appropriate ML algorithms and tools- Develop machine learning applications according to requirements- Select appropriate datasets and data representation methods- Perform statistical analysis and fine-tuning using test results- Train and retrain systems when necessary- Extend existing...
-
Data Engineer
1 month ago
Bangalore,Gurgaon,Gurugram, India Symphoni HR Full timeRole : Data EngineerOrganization : A Leading Consulting firmExperience : 4 to 8 years / 8 to 12 yearsQualification : B.Tech / B.E.Major exposure required in SQL, Pyspark / Spark and Azure cloudResponsibilities :- Design, build, and optimize data pipelines and ETL processes to ensure efficient data ingestion and data integration.- Develop and maintain...
-
Data Engineer
2 weeks ago
Gurugram, India Intuitive Apps Full timeKey Skills : Minimum 6 years Exp in Data Engineer , 3yrs in Azure Data Factory, Pyspark, DatabricksJob Description :- Create and maintain optimal data pipeline architecture- Knowledge on Data Modelling, Data warehouse, and Data Analysis is also required.- Design and develop Data Warehouse solutions- Assemble large, complex data sets that meet functional /...
-
Data Engineer
1 month ago
Bangalore,Gurgaon,Gurugram, India GlobeHR Services Full timeJob Title : Data EngineerExperience : 4 - 7 yearsJob Location : Period : 15- 20 DaysKey Skills : Azure Data Factory, Pyspark, T/SQL, PL/SQL, Exposure on other ETL Tools like SSIS, DatabricksJob Description : - Create and maintain optimal data pipeline architecture- Design and develop Data Warehouse solutions- Assemble large, complex data sets that meet...
-
Azure Databricks Engineer/Data Engineer
2 weeks ago
Bangalore/Chennai/Pune/Gurgaon/Gurugram/Kolkata, India Pylon Management Consulting Full timePrimary Roles and Responsibilities:- Developing Modern Data Warehouse solutions using Databricks and Azure Stack- Ability to provide solutions that are forward-thinking in data engineering and analytics space- Triage issues to find gaps in existing pipelines and fix the issues- Work with business to understand the need in reporting layer and develop data...