PySpark Data Engineer

1 week ago


Thiruvananthapuram, Kerala, India Edstem Technologies Full time

We are seeking a talented and experienced PySpark Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, a passion for big data technologies, and the ability to work in a fast-paced, collaborative environment.

Key Responsibilities:
  • Data Pipeline Development: Design, develop, and maintain scalable data pipelines using PySpark to process large volumes of data from various sources.
  • Data Integration: Integrate data from multiple data sources and formats, ensuring high data quality and reliability.
  • Optimization: Optimize and tune data processing jobs for performance and cost-efficiency.
  • Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.
  • ETL Processes: Develop and maintain ETL processes to extract, transform, and load data into data warehouses and data lakes.
  • Data Quality: Implement data validation and monitoring processes to ensure data accuracy and consistency.
  • Documentation: Document data engineering processes, workflows, and best practices.
  • Troubleshooting: Identify, troubleshoot, and resolve data-related issues promptly.
Required Qualifications:
  • Experience: 3+ years of experience in data engineering or a related field.
  • Education: Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
  • Technical Skills:
  • Proficiency in PySpark and Python.
  • Strong knowledge of big data technologies such as Hadoop, Hive, and Spark.
  • Experience with cloud platforms (e.g., AWS, Azure, GCP) and their data services.
  • Familiarity with data warehousing solutions (e.g., Amazon Redshift, Google BigQuery, Snowflake).
  • Knowledge of relational and NoSQL databases (e.g., MySQL, MongoDB, Cassandra).
  • Data Processing: Experience with ETL/ELT processes and data pipeline orchestration tools (e.g., Apache Airflow, Apache NiFi).
  • Problem-Solving: Strong analytical and problem-solving skills.
  • Communication: Excellent verbal and written communication skills, with the ability to explain complex technical concepts to non-technical stakeholders.

  • Azure with Pyspark

    1 week ago


    Thiruvananthapuram, Kerala, India UST Global Full time

    12 Years 4 Openings ThiruvananthapuramRole description:Responsibilities:Data Pipeline Development: Design, implement, and optimize endtoend data pipelines on Azure, focusing on scalability and performance. Develop and maintain ETL workflows for seamless data processing.Azure Cloud Expertise: Utilize Azure services such as Azure Data Factory, Azure SQL...


  • Thiruvananthapuram, Kerala, India Nissan Motor Full time

    Quality & After Sales team is integral part to Nissan's manufacturing process. While targeting high quality in our vehicles, we work to increase the quality of its sales and service to customers in the purchasing process. The goal is to exceed customer expectations at all contact points.We are looking for Hands-On lead data engineer to build next generation...


  • Thiruvananthapuram, Kerala, India Nissan Motor Full time

    Nissan is a pioneer in Innovation and Technology. With a focus on Mobility, Operational Excellence, Value to our Customers and Electrification of vehicles, you can expect to be part of a very exciting journey here at Nissan. Nissan is going after a massive Digital Transformation backed by leading technologies across the organization globally. We are...


  • Thiruvananthapuram, Kerala, India QBrainX Full time

    Job Title: Data Engineering Lead (With Architecture Experience)Experience Level: More than 10 yearsLocation: CoimbatorePosition Type: Full-TimeNo of Opening: 2Immediate Joiner (OR) Who can join us within 15 to 20 days is preferred.Key Responsibilities:Architect and Design Data Solutions:Collaborate with crossfunctional teams to understand business...

  • Lead Data Engineer

    1 week ago


    Thiruvananthapuram, Kerala, India CloudQ IT Services Full time

    Role Overview :The Lead Data Engineer will design, implement, and maintain data solutions on Microsoft Azure and SQL Server platforms. You will lead a team of data engineers, collaborate with stakeholders, and ensure efficient data processing, storage, and retrieval.Technical Responsibilities : Design and maintain scalable data pipelines. Create solutions...


  • Thiruvananthapuram, Kerala, India mindcurv Full time

    About Mindcurv We assist our clients in reimagining their digital business, interactions, and technology to navigate the evolving digital landscape. We achieve this by crafting sustainable and responsible solutions for individuals operating in a digital environment.Mindcurv comprehensively addresses the market's demand to digitize business operations and...


  • Thiruvananthapuram, Kerala, India Nissan Motor Full time

    :We are looking for passionate senior ETL engineers, with Talend experience, who can assist in the process of maintaining all the relevant sources of data and ensuring that it is being stored properly. He/she is involved in developing the ETL processes and is involved in designing and preparing the codes that meet the program...


  • Thiruvananthapuram, Kerala, India Nissan Motor Full time

    Description : We are looking for passionate senior ETL engineers, with Talend experience, who can assist in the process of maintaining all the relevant sources of data and ensuring that it is being stored properly. He/she is involved in developing the ETL processes and is involved in designing and preparing the codes that meet the program specifications....


  • Thiruvananthapuram, Kerala, India Quantiphi Full time

    Role : Senior Data EngineerExperience Level : 4 to 7 YearsWork location : Trivandrum (Hybrid Only)Required Skills :Bachelor's degree in Computer Science or similar field or equivalent work experience.3+ years of experience on Data Warehousing, Data Engineering or Data Integration projects.Expert with data warehousing concepts, strategies, and tools.Strong...

  • Data Scientist

    1 week ago


    Thiruvananthapuram, Kerala, India TEMPLETON AND PARTNERS (INDIA) PRIVATE LIMITED Full time

    Job Description :Bachelor's or Master's degree in Computer Science, Engineering, Mathematics, or a related field 6+ years of experience in a Data Science role Experience with data processing and feature extraction Experience with training (predictive/prescriptive) models and hyperparameter tuning. Expertise with the RAG (Retrieve, Adapt, Generate) approach....

  • Data Architect

    1 week ago


    Thiruvananthapuram, Kerala, India Nissan Full time

    Nissan is a pioneer in Innovation and Technology. With a focus on Mobility, Operational Excellence, Value to our Customers and Electrification of vehicles, you can expect to be part of a very exciting journey here at Nissan.Nissan is going after a massive Digital Transformation backed by leading technologies across the organization globally. We are committed...

  • Data Scientist

    1 week ago


    Thiruvananthapuram, Kerala, India CONNECTING 2 WORK Full time

    Responsibilities: • Utilize Python programming language to clean, transform, and manipulate large datasets for analysis. • Develop and implement statistical models, machine learning algorithms, and predictive analytics to solve complex business problems. • Apply advanced data mining techniques and statistical analyses to identify patterns,...

  • Data Scientist

    1 week ago


    Thiruvananthapuram, Kerala, India CONNECTING 2 WORK Full time

    Responsibilities: Utilize Python programming language to clean, transform, and manipulate large datasets for analysis. Develop and implement statistical models, machine learning algorithms, and predictive analytics to solve complex business problems. Apply advanced data mining techniques and statistical analyses to identify patterns, correlations, and trends...

  • Data Analytics

    1 week ago


    Thiruvananthapuram, Kerala, India SMEClabs Full time

    Job InformationJob Opening Status InprogressIndustry IT ServicesDate of Closing 06/23/2023Gender Male/FemaleJob Location TrivandrumState/Province KeralaCity TrivandrumZip/Postal Code Country India Excellent written verbal communication skills, good exposure to the web world Should be proactive and hard working Degree in Computer Science, Engineering, or a...

  • Data Architect I

    1 week ago


    Thiruvananthapuram, Kerala, India UST Full time

    This position is responsible for guiding leaders and teams to make good technology decisions that are innovative, fiscally responsible, and aligned with the overall Technology Strategy at Advantasure, evolving our offerings toward a cloud-native, SaaS-centric Platform.Work closely with Agile development teams offering guidance, proof of concepts, and...


  • Thiruvananthapuram, Kerala, India mindcurv Full time

    We Help Businesses Thrive!Strategy and AdvisoryCreative Services and Digital ProductsClient Engagement PlatformsDigital Experience and SolutionsData ServicesCloud Platforms and Managed ServicesOur team of experts collaborates to enhance digital experiences, assisting businesses in achieving sustainable growth.Your Role as a Data Visualization...

  • Data Scientist

    1 week ago


    Thiruvananthapuram, Kerala, India IBS Software Full time

    Job Title Data ScientistLocation (s) CochinYears of Experience 58 yearsSenior Data Scientist '" JDOpportunity to work in complex Machine learning use cases for a Leader in the Travel Domain. You will be working with a high performing team of Consultants, Engineers, Analysts and Data scientists. Opportunity to create patented IPs.Responsibilities:Â- Work...


  • Thiruvananthapuram, Kerala, India CONNECTING 2 WORK Full time

    Job Description Responsibilities Below are some of the responsibilities an artificial intelligence, machine learning engineer is expected to assume in their position: Design and develop AI models to meet project requirements Convert AI/ML models into APIs that other developers can use Perform statistical analysis on big data sets Manage...


  • Thiruvananthapuram, Kerala, India CONNECTING 2 WORK Full time

    Responsibilities:Team Leadership:Lead and mentor a team of data scientists, providing guidance and supportfor their professional development.Foster a collaborative and innovative team culture that encouragesknowledge sharing and continuous improvement.Project Management: Oversee the planning, execution, and delivery of data science projects tomeet...

  • Databse engineer

    1 week ago


    Thiruvananthapuram, Kerala, India CONNECTING 2 WORK Full time

    Job DescriptionResponsibilitiesMaintaining and enhancing the performance of existing database programs.Assisting database development teams in designing new database programs that meet the organization's data storage needs.Monitoring databases and related systems to ensure optimized performance.Writing new support programs and scripts to increase data...