AWS Big data pyspark SQL Spark Airflow

15 hours ago

Pune, India ScaleneWorks Full time

Job Title: AWS Big Data Engineer
About the Role:
We are looking for a highly skilled AWS Big Data Engineer with extensive experience in PySpark, SQL, Spark, and Airflow. The ideal candidate will have a strong background in coding and a deep understanding of big data technologies. You will be responsible for designing, developing, and maintaining big data solutions on AWS, ensuring data is processed efficiently and effectively.
Key Responsibilities:
Design, develop, and maintain big data solutions using AWS services.
Write and optimize complex SQL queries.
Develop and manage data pipelines using PySpark and Spark.
Implement and manage workflows using Apache Airflow.
Collaborate with data scientists and analysts to understand data requirements.
Ensure data quality and integrity throughout the data lifecycle.
Troubleshoot and resolve issues related to big data processing.
Stay updated with the latest trends and best practices in big data technologies.
Key Skills and Qualifications:
Proven experience with AWS big data services.
Strong coding experience in PySpark and SQL.
Expertise in Spark and Airflow.
Excellent problem-solving and analytical skills.
Ability to work independently and as part of a team.
Strong communication and documentation skills.
Relevant certifications (e.g., AWS Certified Big Data - Specialty) are a plus.

  • Data Engineer

    4 days ago

    Pune, India Tata Consultancy Services Full time

    Job Title :- Data Engineer - Pyspark Experience: 5 to 8 Years Location: Pune/Hyderabad Job Description Required Skills: 5+ years of experience in Big data and pyspark Must-Have Good work experience on Big Data Platforms like Hadoop, Spark, Scala, Hive, Impala, SQL Good-to-Have Good Spark, Pyspark,Big Data experience Spark UI/Optimization/debugging...

  • Data Engineer

    2 days ago

    Pune, India Tata Consultancy Services Full time

    Job Title :- Data Engineer - PysparkExperience: 5 to 8 YearsLocation: Pune/HyderabadJob DescriptionRequired Skills:5+ years of experience in Big data and pysparkMust-HaveGood work experience on Big Data Platforms like Hadoop, Spark, Scala, Hive, Impala, SQLGood-to-HaveGood Spark, Pyspark,Big Data experienceSpark UI/Optimization/debugging techniquesGood...

  • AWS, Python, Pyspark, EMR,Apache Airflow- Data Platform Engineer

    16 hours ago

    Pune, India Rapsys Technologies Pte Ltd. Full time

    We're Hiring: Data Platform Engineer! We are seeking an experienced Data Platform Engineer to design, build, and maintain scalable data infrastructure using AWS cloud services. The ideal candidate will have expertise in Python, PySpark, EMR, and Apache Airflow to develop robust data pipelines and analytics solutions that drive business insights. ...

  • AWS, Python, Pyspark, EMR,Apache Airflow- Data Platform Engineer

    9 hours ago

    Pune, India Rapsys Technologies Pte Ltd. Full time

    We're Hiring: Data Platform Engineer! We are seeking an experienced Data Platform Engineer to design, build, and maintain scalable data infrastructure using AWS cloud services. The ideal candidate will have expertise in Python, PySpark, EMR, and Apache Airflow to develop robust data pipelines and analytics solutions that drive business insights. ...

  • Data Engineer

    3 days ago

    Pune, India Tata Consultancy Services Full time

    Job Title :- Data Engineer - Pyspark Experience: 5 to 8 Years Location: Pune/Hyderabad Job Description Required Skills: 5+ years of experience in Big data and pyspark Must-Have Good work experience on Big Data Platforms like Hadoop, Spark, Scala, Hive, Impala, SQL Good-to-Have Good Spark, Pyspark,Big Data experience Spark UI/Optimization/debugging...

  • Data Engineer

    1 day ago

    Pune, India Tata Consultancy Services Full time

    Job Title :- Data Engineer - Pyspark Experience: 5 to 8 Years Location: Pune/Hyderabad Job Description Required Skills: 5+ years of experience in Big data and pyspark Must-Have Good work experience on Big Data Platforms like Hadoop, Spark, Scala, Hive, Impala, SQL Good-to-Have Good Spark, Pyspark,Big Data experience Spark UI/Optimization/debugging...

  • Data Engineer

    5 days ago

    Pune, India Tata Consultancy Services Full time

    Job Title :- Data Engineer - PysparkExperience: 5 to 8 YearsLocation: Pune/HyderabadJob DescriptionRequired Skills:5+ years of experience in Big data and pysparkMust-HaveGood work experience on Big Data Platforms like Hadoop, Spark, Scala, Hive, Impala, SQLGood-to-HaveGood Spark, Pyspark,Big Data experienceSpark UI/Optimization/debugging techniquesGood...

  • Data Engineer

    4 days ago

    pune, India Tata Consultancy Services Full time

    Job Title :- Data Engineer - PysparkExperience: 5 to 8 YearsLocation: Pune/HyderabadJob DescriptionRequired Skills:5+ years of experience in Big data and pysparkMust-HaveGood work experience on Big Data Platforms like Hadoop, Spark, Scala, Hive, Impala, SQLGood-to-HaveGood Spark, Pyspark,Big Data experienceSpark UI/Optimization/debugging techniquesGood...

  • Data Engineer

    2 days ago

    Pune, India Tata Consultancy Services Full time

    Job Title :- Data Engineer - Pyspark Experience: 5 to 8 Years Location: Pune/Hyderabad Job Description Required Skills: 5+ years of experience in Big data and pyspark Must-Have Good work experience on Big Data Platforms like Hadoop, Spark, Scala, Hive, Impala, SQL Good-to-Have Good Spark, Pyspark,Big Data experience Spark UI/Optimization/debugging...

  • Data Streaming

    2 days ago

    Pune, India ACE Talent Consulting Full time

    Job Description: Spark/Scala/PySpark developer who knows how to fully exploit the potential of our Spark cluster. Should have ability to clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data. **Responsibilities**: Create Scala/Spark/Pyspark jobs for data transformation and aggregation Produce...