PySpark Data Engineer

2 weeks ago


thiruvananthapuram, India Edstem Technologies Full time

We are seeking a talented and experienced PySpark Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, a passion for big data technologies, and the ability to work in a fast-paced, collaborative environment.

 

Key Responsibilities:


  • Data Pipeline Development: Design, develop, and maintain scalable data pipelines using PySpark to process large volumes of data from various sources.
  • Data Integration: Integrate data from multiple data sources and formats, ensuring high data quality and reliability.
  • Optimization: Optimize and tune data processing jobs for performance and cost-efficiency.
  • Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.
  • ETL Processes: Develop and maintain ETL processes to extract, transform, and load data into data warehouses and data lakes.
  • Data Quality: Implement data validation and monitoring processes to ensure data accuracy and consistency.
  • Documentation: Document data engineering processes, workflows, and best practices.
  • Troubleshooting: Identify, troubleshoot, and resolve data-related issues promptly.

Required Qualifications:


  • Experience: 3+ years of experience in data engineering or a related field.
  • Education: Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field.
  • Technical Skills:
  • Proficiency in PySpark and Python.
  • Strong knowledge of big data technologies such as Hadoop, Hive, and Spark.
  • Experience with cloud platforms (e.g., AWS, Azure, GCP) and their data services.
  • Familiarity with data warehousing solutions (e.g., Amazon Redshift, Google BigQuery, Snowflake).
  • Knowledge of relational and NoSQL databases (e.g., MySQL, MongoDB, Cassandra).
  • Data Processing: Experience with ETL/ELT processes and data pipeline orchestration tools (e.g., Apache Airflow, Apache NiFi).
  • Problem-Solving: Strong analytical and problem-solving skills.
  • Communication: Excellent verbal and written communication skills, with the ability to explain complex technical concepts to non-technical stakeholders.



  • PySpark Data Engineer

    2 weeks ago


    Thiruvananthapuram, India Edstem Technologies Full time

    We are seeking a talented and experienced PySpark Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, a passion for big data technologies, and the ability to work in a fast-paced, collaborative environment. Key Responsibilities:Data Pipeline Development: Design, develop, and maintain scalable data...

  • Big Data Engineer

    2 months ago


    Thiruvananthapuram, India Tata Elxsi Full time

    Job Description : - At least 4+ years of experience pyspark framework.- Good experience writing complex SQL and No SQL databases.- Excellent coding and design skills, particularly in Pyspark and Python.- Strong practical working experience with Unix scripting in at least one of Python, HDFS, SQL, PL/SQL, Shell (either bash or zsh).- Experience in AWS...

  • Azure with Pyspark

    3 days ago


    Thiruvananthapuram, Kerala, India UST Global Full time

    9 - 12 Years- 4 Openings- Thiruvananthapuram**Role description**: **Responsibilities**: **Data Pipeline Development**: - Design, implement, and optimize end-to-end data pipelines on Azure, focusing on scalability and performance. - Develop and maintain ETL workflows for seamless data processing. **Azure Cloud Expertise**: - Utilize Azure services such as...


  • Thiruvananthapuram, Kerala, India UST Global Full time

    9 - 12 Years- 1 Opening- Thiruvananthapuram**Role description**: **Responsibilities**: - AWS Data Architecture Design, implement, and maintain scalable and robust data architectures on AWS. - Utilize AWS services such as S3, Glue, and Redshift for data storage, processing, and analytics. - ETL Development with PySpark Develop and implement ETL processes...

  • Big Data Engineer

    2 months ago


    Bangalore/Chennai/Pune/Trivandrum/Thiruvananthapuram, India Tata Elxsi Full time

    Job Description : - At least 4+ years of experience pyspark framework.- Good experience writing complex SQL and No SQL databases.- Excellent coding and design skills, particularly in Pyspark and Python.- Strong practical working experience with Unix scripting in at least one of Python, HDFS, SQL, PL/SQL, Shell (either bash or zsh).- Experience in AWS...

  • Big Data Engineer

    2 months ago


    Bangalore/Chennai/Pune/Trivandrum/Thiruvananthapuram, IN Tata Elxsi Full time

    Job Description : - At least 4+ years of experience pyspark framework.- Good experience writing complex SQL and No SQL databases.- Excellent coding and design skills, particularly in Pyspark and Python.- Strong practical working experience with Unix scripting in at least one of Python, HDFS, SQL, PL/SQL, Shell (either bash or zsh).- Experience in AWS...

  • Tata Elxsi

    2 weeks ago


    Bangalore/Chennai/Pune/Trivandrum/Thiruvananthapuram, IN Tata Elxsi Full time

    Job Description : - At least 4+ years of experience pyspark framework.- Good experience writing complex SQL and No SQL databases.- Excellent coding and design skills, particularly in Pyspark and Python.- Strong practical working experience with Unix scripting in at least one of Python, HDFS, SQL, PL/SQL, Shell (either bash or zsh).- Experience in AWS...

  • Tata Elxsi

    2 weeks ago


    Bangalore/Chennai/Pune/Trivandrum/Thiruvananthapuram, India Tata Elxsi Full time

    Job Description : - At least 4+ years of experience pyspark framework.- Good experience writing complex SQL and No SQL databases.- Excellent coding and design skills, particularly in Pyspark and Python.- Strong practical working experience with Unix scripting in at least one of Python, HDFS, SQL, PL/SQL, Shell (either bash or zsh).- Experience in AWS...


  • thiruvananthapuram, India Arting Digital Private Limited Full time

    Job Description Position Title:     Data Engineer- AtScale / DataCTC:                     Upto 10 LPA Experience:         3+YearLocation:             Pune, Bangalore, Chennai, TrivandrumWorking Mode:  OnsitePrimary Skills:    AtScale, Python, SQL, ETL, SAP dataGood to have:     AWS/Azure/GCP,...


  • Thiruvananthapuram, India Arting Digital Private Limited Full time

    Job DescriptionPosition Title:    Data Engineer- AtScale / DataCTC:                     Upto 10 LPA Experience:        3+YearLocation:             Pune, Bangalore, Chennai, TrivandrumWorking Mode: OnsitePrimary Skills:    AtScale, Python, SQL, ETL, SAP dataGood to have:    AWS/Azure/GCP,...

  • Associate Ii

    3 weeks ago


    Thiruvananthapuram, India UST Full time

    Role Proficiency: Interprets data and analyses results using statistical techniques under supervision Outcomes: - Complies and assist in mining and acquiring data from primary and secondary sources; reorganizing the data in a format that can be easily read by either a machine or a person - Assist in identifying analyzing and interpreting trends or patterns...

  • Senior Data Engineer

    3 weeks ago


    thiruvananthapuram, India Nissan Motor Full time

    Nissan is a pioneer in Innovation and Technology. With a focus on Mobility, Operational Excellence, Value to our Customers and Electrification of vehicles, you can expect to be part of a very exciting journey here at Nissan. Nissan is going after a massive Digital Transformation backed by leading technologies across the organization globally. We are...

  • Senior Data Engineer

    2 weeks ago


    thiruvananthapuram, India Nissan Motor Full time

    Nissan is a pioneer in Innovation and Technology. With a focus on Mobility, Operational Excellence, Value to our Customers and Electrification of vehicles, you can expect to be part of a very exciting journey here at Nissan. Nissan is going after a massive Digital Transformation backed by leading technologies across the organization globally. We are...

  • Senior Data Engineer

    3 weeks ago


    Thiruvananthapuram, India Nissan Motor Full time

    Nissan is a pioneer in Innovation and Technology. With a focus on Mobility, Operational Excellence, Value to our Customers and Electrification of vehicles, you can expect to be part of a very exciting journey here at Nissan. Nissan is going after a massive Digital Transformation backed by leading technologies across the organization globally. We are...

  • Senior Data Engineer

    2 weeks ago


    Thiruvananthapuram, India Nissan Motor Full time

    Nissan is a pioneer in Innovation and Technology. With a focus on Mobility, Operational Excellence, Value to our Customers and Electrification of vehicles, you can expect to be part of a very exciting journey here at Nissan. Nissan is going after a massive Digital Transformation backed by leading technologies across the organization globally. We are...

  • Senior Data Engineer

    2 months ago


    Thiruvananthapuram, India Nissan Motor Full time

    Quality & After Sales team is integral part to Nissan’s manufacturing process. While targeting high quality in our vehicles, we work to increase the quality of its sales and service to customers in the purchasing process. The goal is to exceed customer expectations at all contact points. We are looking for Hands-On lead data engineer to build next...

  • Senior Data Engineer

    2 months ago


    thiruvananthapuram, India Nissan Motor Full time

    Quality & After Sales team is integral part to Nissan’s manufacturing process. While targeting high quality in our vehicles, we work to increase the quality of its sales and service to customers in the purchasing process. The goal is to exceed customer expectations at all contact points. We are looking for Hands-On lead data engineer to build next...

  • Data Engineer

    2 months ago


    Thiruvananthapuram, India ProPMO Services Private Limited Full time

    Job Description :We have a requirement for 3 Data Engineers who could join us on a short notice.3 to 8 years of professional software development/engineering experience.Specialist experience working in Python coding as a Data Engineer. Must have good experience working with databases PostgreSQL/MySQL/MS SQL.Must have good experience working in AWS/GCP...

  • Senior Data Engineer

    2 weeks ago


    thiruvananthapuram, India Quantiphi Full time

    Role : Senior Data EngineerExperience Level : 4 to 7 YearsWork location : Trivandrum (Hybrid Only)Required Skills :● Bachelor’s degree in Computer Science or similar field or equivalent work experience.● 3+ years of experience on Data Warehousing, Data Engineering or Data Integration projects.● Expert with data warehousing concepts, strategies, and...

  • Senior Data Engineer

    2 months ago


    Thiruvananthapuram, India Quantiphi Full time

    Role: Senior Data EngineerExperience Level: 4 to 7 YearsWork location: Trivandrum (Hybrid Only)Required Skills:● Bachelor’s degree in Computer Science or similar field or equivalent work experience.● 3+ years of experience on Data Warehousing, Data Engineering or Data Integration projects.● Expert with data warehousing concepts, strategies, and...