PySpark Data Engineer

2 weeks ago


Bengaluru, India Edstem Technologies Full time

We are seeking a talented and experienced PySpark Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, a passion for big data technologies, and the ability to work in a fast-paced, collaborative environment.

 

Willing to work in Location: Trivandrum, Bengaluru, Chennai


Key Responsibilities:


  • Data Pipeline Development: Design, develop, and maintain scalable data pipelines using PySpark to process large volumes of data from various sources.
  • Data Integration: Integrate data from multiple data sources and formats, ensuring high data quality and reliability.
  • Optimization: Optimize and tune data processing jobs for performance and cost-efficiency.
  • Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.
  • ETL Processes: Develop and maintain ETL processes to extract, transform, and load data into data warehouses and data lakes.
  • Data Quality: Implement data validation and monitoring processes to ensure data accuracy and consistency.
  • Documentation: Document data engineering processes, workflows, and best practices.
  • Troubleshooting: Identify, troubleshoot, and resolve data-related issues promptly.


Required Qualifications:


  • Experience: 3+ years of experience in data engineering or a related field.
  • Education: Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field.
  • Technical Skills:
  • Proficiency in PySpark and Python.
  • Strong knowledge of big data technologies such as Hadoop, Hive, and Spark.
  • Experience with cloud platforms (e.g., AWS, Azure, GCP) and their data services.
  • Familiarity with data warehousing solutions (e.g., Amazon Redshift, Google BigQuery, Snowflake).
  • Knowledge of relational and NoSQL databases (e.g., MySQL, MongoDB, Cassandra).
  • Data Processing: Experience with ETL/ELT processes and data pipeline orchestration tools (e.g., Apache Airflow, Apache NiFi).
  • Problem-Solving: Strong analytical and problem-solving skills.
  • Communication: Excellent verbal and written communication skills, with the ability to explain complex technical concepts to non-technical stakeholders.

  • PySpark Data Engineer

    2 weeks ago


    Bengaluru, India Edstem Technologies Full time

    We are seeking a talented and experienced PySpark Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, a passion for big data technologies, and the ability to work in a fast-paced, collaborative environment.Willing to work inLocation: Trivandrum, Bengaluru, ChennaiKey Responsibilities:Data Pipeline...

  • PySpark Data Engineer

    2 weeks ago


    Bengaluru, India Edstem Technologies Full time

    We are seeking a talented and experienced PySpark Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, a passion for big data technologies, and the ability to work in a fast-paced, collaborative environment.   Willing to work in Location: Trivandrum, Bengaluru, Chennai Key Responsibilities: Data...

  • PySpark Data Engineer

    2 weeks ago


    Bengaluru, Karnataka, India Edstem Technologies Full time

    We are seeking a talented and experienced PySpark Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, a passion for big data technologies, and the ability to work in a fast-paced, collaborative environment.Willing to work inLocation: Trivandrum, Bengaluru, ChennaiKey Responsibilities:Data Pipeline...

  • Data Engineer

    1 week ago


    Bengaluru, India RiskInsight Consulting Pvt Ltd Full time

    We are seeking a skilled Data Engineer with expertise in Python and PySpark to join our dynamic team. In this role, you will design, develop, and optimize data pipelines and workflows to support our data infrastructure and analytics needs. Your contributions will be crucial in ensuring the reliability, scalability, and performance of our data...

  • Data Engineer

    2 months ago


    Bengaluru, India WIZSTAFFING PRIVATE LIMITED Full time

    Job Description :We at Captain Fresh, are building smart supply chain to deliver the highest quality seafood and meat for the Indian consumer.Our innovations in process management and workforce orchestration along with strong industry credentials are enabling us to deliver the fastest harvest-to-fork in the industry. Our endeavor is to leverage experience...

  • Data Engineer

    7 days ago


    Bengaluru, India RiskInsight Consulting Pvt Ltd Full time

    We are seeking a skilled Data Engineer with expertise in Python and PySpark to join our dynamic team. In this role, you will design, develop, and optimize data pipelines and workflows to support our data infrastructure and analytics needs. Your contributions will be crucial in ensuring the reliability, scalability, and performance of our data...

  • Data Engineer

    4 weeks ago


    Bengaluru, India HNM Solutions Full time

    Note : Only immediate joinerRole : PysparkExperience : 8+ yearsLocation : Hyderabad, ChenniaInterview mode is Walkin driveRequirements :Mandatory skills : (8+ Years of experience in Data engineering with 5+ Years on Pyspark/NoSQL is Mandatory)1. Person should be strong in Pyspark2. Should have hands on in MWAA (Airflow) / AWS EMR(Hadoop, Hive) framework3....

  • Big Data Engineer

    3 weeks ago


    Bengaluru, India Techno Wise Full time

    Job Description : We are seeking a talented Big Data Engineer proficient in PySpark to join our dynamic team. The ideal candidate will play a pivotal role in designing, implementing, and maintaining scalable data solutions leveraging the Big Data technologies like PySpark. This role requires a strong understanding of big data technologies, data engineering...


  • Bengaluru, Karnataka, India NAM Info Inc Full time

    AWS Data Engineer with PySpark Domain: Clinical / pharma/ healthcare Location: Bangalore or Chennai / Kolkata / Hyderabad Preferred Skills: AWS and Python and or PySpark AWS Redshift, Aurora, AWS Glue , AWS Lambda, etc. Extensive experience with Data gathering and ingestion (from multiple sources), and manipulation, orchestration and optimization on...


  • Bengaluru, Karnataka, India NAM Info Inc Full time

    AWS Data Engineer with PySparkDomain: Clinical / pharma/ healthcareLocation: Bangalore or Chennai / Kolkata / Hyderabad Preferred Skills:AWS and Python and or PySparkAWS Redshift, Aurora, AWS Glue, AWS Lambda, etc.Extensive experience with Data gathering and ingestion (from multiple sources), and manipulation, orchestration and optimization on AWS CloudSpark...

  • Data Engineer

    4 weeks ago


    Bengaluru, India Xcelyst Full time

    Role : Data Engineer - NifiExperience : 5 Years - 8 Years Notice period : Immediate - 30 daysSkills : Experience in Nifi and Data Engineering (Pyspark, SQL, ETL). Secondary : Big Data/Hadoop, Hive, Spark4+ Years in Data Engineer/1-2 yrs exp in Nifi We are seeking a highly skilled Data Engineer with expertise in Apache NiFi to join our data engineering team....

  • Cloud Data Engineer

    3 weeks ago


    Bengaluru, India Yo HR Consultancy Full time

    Profile:- Cloud Data EngineerExperience : 6 to 8 yearsLocation : Bangalore(on-site)Salary : Up to 30 lpaNotice period : Max. 30 daysMust Have :- Databricks and PySpark- PySQL- SQL- Cloud Platforms- ELT ToolsRoles :- Design, develop, and maintain scalable data pipelines and ETL processes using Databricks and PySpark.- Implement data transformation and...


  • Bengaluru, India LTIMindtree Full time

    Job Description:Primary Skill Set: Lead PySpark DeveloperSeconday Skill Set: SAS toolsLead a team of data engineers in the design and implementation of big data solutions using PySpark.Collaborate with cross-functional teams to understand business needs and develop data-driven solutions.Design and implement scalable and robust data pipelines using...


  • Bengaluru, Karnataka, India LTIMindtree Full time

    Job Description: Primary Skill Set: Lead PySpark Developer Seconday Skill Set: SAS tools Lead a team of data engineers in the design and implementation of big data solutions using PySpark. Collaborate with cross-functional teams to understand business needs and develop data-driven solutions. Design and implement scalable and robust data pipelines using...


  • Bengaluru, India LTIMindtree Full time

    Job Description: Primary Skill Set: Lead PySpark Developer Seconday Skill Set: SAS tools Lead a team of data engineers in the design and implementation of big data solutions using PySpark. Collaborate with cross-functional teams to understand business needs and develop data-driven solutions. Design and implement scalable and robust data pipelines using...


  • Bengaluru, India LTIMindtree Full time

    Job Description: Primary Skill Set: Lead PySpark Developer Seconday Skill Set: SAS tools Lead a team of data engineers in the design and implementation of big data solutions using PySpark. Collaborate with cross-functional teams to understand business needs and develop data-driven solutions. Design and implement scalable and robust data pipelines using...

  • Bigdata + Pyspark

    2 weeks ago


    Bengaluru, Karnataka, India Impetus Technologies Full time

    BIgdata + Pyspark :Big Data; Hadoop / HDFS,Pyspark,PythonExperience with design and coding across one or more platforms and languages (e.g. , Python/Pyspark/SQL) as appropriateHands-on expertise with application design, software developmentProficient in Big Data technologies Designs, codes, tests, corrects and documents large and/or complex programs and...

  • Data Engineer

    2 months ago


    Bengaluru, India Akshaya IT Business Solutions Full time

    Job Description :- Design and develop real-time data ingestion pipelines using Databricks and Spark Streaming to enable timely processing of large volumes of data.- Implement complex data transformations and aggregations to extract actionable insights from streaming data sources.- Collaborate with data scientists, analysts, and business stakeholders to...

  • Bigdata + Pyspark

    2 weeks ago


    Bengaluru, Karnataka, India Impetus Technologies Full time

    BIgdata + Pyspark : Big Data; Hadoop / HDFS,Pyspark,Python Experience with design and coding across one or more platforms and languages (e.g. , Python/Pyspark/SQL) as appropriateHands-on expertise with application design, software development Proficient in Big Data technologies Designs, codes, tests, corrects and documents large and/or complex programs and...

  • Colan Infotech

    2 months ago


    Bengaluru, India Colan Infotech Pvt Ltd Full time

    Skill Set : Pyspark / Scala Spark, Data Factory, Databricks, Python, SQL.Job Description :Roles And Responsibilities :- Must have cloud knowledge in Azure- Should have programming skills with the ability to write optimized and reusable high-quality code.- Design, develop and maintain scalable data pipelines using Pyspark / Scala Spark, Databricks, Python,...