![Edstem Technologies](https://media.trabajo.org/img/noimg.jpg)
PySpark Data Engineer
2 weeks ago
We are seeking a talented and experienced PySpark Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, a passion for big data technologies, and the ability to work in a fast-paced, collaborative environment.
Willing to work in Location: Trivandrum, Bengaluru, Chennai
Key Responsibilities:
- Data Pipeline Development: Design, develop, and maintain scalable data pipelines using PySpark to process large volumes of data from various sources.
- Data Integration: Integrate data from multiple data sources and formats, ensuring high data quality and reliability.
- Optimization: Optimize and tune data processing jobs for performance and cost-efficiency.
- Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.
- ETL Processes: Develop and maintain ETL processes to extract, transform, and load data into data warehouses and data lakes.
- Data Quality: Implement data validation and monitoring processes to ensure data accuracy and consistency.
- Documentation: Document data engineering processes, workflows, and best practices.
- Troubleshooting: Identify, troubleshoot, and resolve data-related issues promptly.
Required Qualifications:
- Experience: 3+ years of experience in data engineering or a related field.
- Education: Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field.
- Technical Skills:
- Proficiency in PySpark and Python.
- Strong knowledge of big data technologies such as Hadoop, Hive, and Spark.
- Experience with cloud platforms (e.g., AWS, Azure, GCP) and their data services.
- Familiarity with data warehousing solutions (e.g., Amazon Redshift, Google BigQuery, Snowflake).
- Knowledge of relational and NoSQL databases (e.g., MySQL, MongoDB, Cassandra).
- Data Processing: Experience with ETL/ELT processes and data pipeline orchestration tools (e.g., Apache Airflow, Apache NiFi).
- Problem-Solving: Strong analytical and problem-solving skills.
- Communication: Excellent verbal and written communication skills, with the ability to explain complex technical concepts to non-technical stakeholders.
-
PySpark Data Engineer
2 weeks ago
Bengaluru, India Edstem Technologies Full timeWe are seeking a talented and experienced PySpark Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, a passion for big data technologies, and the ability to work in a fast-paced, collaborative environment.Willing to work inLocation: Trivandrum, Bengaluru, ChennaiKey Responsibilities:Data Pipeline...
-
PySpark Data Engineer
2 weeks ago
Bengaluru, India Edstem Technologies Full timeWe are seeking a talented and experienced PySpark Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, a passion for big data technologies, and the ability to work in a fast-paced, collaborative environment. Willing to work in Location: Trivandrum, Bengaluru, Chennai Key Responsibilities: Data...
-
PySpark Data Engineer
2 weeks ago
Bengaluru, Karnataka, India Edstem Technologies Full timeWe are seeking a talented and experienced PySpark Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, a passion for big data technologies, and the ability to work in a fast-paced, collaborative environment.Willing to work inLocation: Trivandrum, Bengaluru, ChennaiKey Responsibilities:Data Pipeline...
-
Data Engineer
1 week ago
Bengaluru, India RiskInsight Consulting Pvt Ltd Full timeWe are seeking a skilled Data Engineer with expertise in Python and PySpark to join our dynamic team. In this role, you will design, develop, and optimize data pipelines and workflows to support our data infrastructure and analytics needs. Your contributions will be crucial in ensuring the reliability, scalability, and performance of our data...
-
Data Engineer
2 months ago
Bengaluru, India WIZSTAFFING PRIVATE LIMITED Full timeJob Description :We at Captain Fresh, are building smart supply chain to deliver the highest quality seafood and meat for the Indian consumer.Our innovations in process management and workforce orchestration along with strong industry credentials are enabling us to deliver the fastest harvest-to-fork in the industry. Our endeavor is to leverage experience...
-
Data Engineer
7 days ago
Bengaluru, India RiskInsight Consulting Pvt Ltd Full timeWe are seeking a skilled Data Engineer with expertise in Python and PySpark to join our dynamic team. In this role, you will design, develop, and optimize data pipelines and workflows to support our data infrastructure and analytics needs. Your contributions will be crucial in ensuring the reliability, scalability, and performance of our data...
-
Data Engineer
4 weeks ago
Bengaluru, India HNM Solutions Full timeNote : Only immediate joinerRole : PysparkExperience : 8+ yearsLocation : Hyderabad, ChenniaInterview mode is Walkin driveRequirements :Mandatory skills : (8+ Years of experience in Data engineering with 5+ Years on Pyspark/NoSQL is Mandatory)1. Person should be strong in Pyspark2. Should have hands on in MWAA (Airflow) / AWS EMR(Hadoop, Hive) framework3....
-
Big Data Engineer
3 weeks ago
Bengaluru, India Techno Wise Full timeJob Description : We are seeking a talented Big Data Engineer proficient in PySpark to join our dynamic team. The ideal candidate will play a pivotal role in designing, implementing, and maintaining scalable data solutions leveraging the Big Data technologies like PySpark. This role requires a strong understanding of big data technologies, data engineering...
-
AWS Data Engineer with PySpark
2 weeks ago
Bengaluru, Karnataka, India NAM Info Inc Full timeAWS Data Engineer with PySpark Domain: Clinical / pharma/ healthcare Location: Bangalore or Chennai / Kolkata / Hyderabad Preferred Skills: AWS and Python and or PySpark AWS Redshift, Aurora, AWS Glue , AWS Lambda, etc. Extensive experience with Data gathering and ingestion (from multiple sources), and manipulation, orchestration and optimization on...
-
AWS Data Engineer with PySpark
2 weeks ago
Bengaluru, Karnataka, India NAM Info Inc Full timeAWS Data Engineer with PySparkDomain: Clinical / pharma/ healthcareLocation: Bangalore or Chennai / Kolkata / Hyderabad Preferred Skills:AWS and Python and or PySparkAWS Redshift, Aurora, AWS Glue, AWS Lambda, etc.Extensive experience with Data gathering and ingestion (from multiple sources), and manipulation, orchestration and optimization on AWS CloudSpark...
-
Data Engineer
4 weeks ago
Bengaluru, India Xcelyst Full timeRole : Data Engineer - NifiExperience : 5 Years - 8 Years Notice period : Immediate - 30 daysSkills : Experience in Nifi and Data Engineering (Pyspark, SQL, ETL). Secondary : Big Data/Hadoop, Hive, Spark4+ Years in Data Engineer/1-2 yrs exp in Nifi We are seeking a highly skilled Data Engineer with expertise in Apache NiFi to join our data engineering team....
-
Cloud Data Engineer
3 weeks ago
Bengaluru, India Yo HR Consultancy Full timeProfile:- Cloud Data EngineerExperience : 6 to 8 yearsLocation : Bangalore(on-site)Salary : Up to 30 lpaNotice period : Max. 30 daysMust Have :- Databricks and PySpark- PySQL- SQL- Cloud Platforms- ELT ToolsRoles :- Design, develop, and maintain scalable data pipelines and ETL processes using Databricks and PySpark.- Implement data transformation and...
-
Lead PySpark Developer
2 weeks ago
Bengaluru, India LTIMindtree Full timeJob Description:Primary Skill Set: Lead PySpark DeveloperSeconday Skill Set: SAS toolsLead a team of data engineers in the design and implementation of big data solutions using PySpark.Collaborate with cross-functional teams to understand business needs and develop data-driven solutions.Design and implement scalable and robust data pipelines using...
-
Lead PySpark Developer
2 weeks ago
Bengaluru, Karnataka, India LTIMindtree Full timeJob Description: Primary Skill Set: Lead PySpark Developer Seconday Skill Set: SAS tools Lead a team of data engineers in the design and implementation of big data solutions using PySpark. Collaborate with cross-functional teams to understand business needs and develop data-driven solutions. Design and implement scalable and robust data pipelines using...
-
Lead PySpark Developer
2 weeks ago
Bengaluru, India LTIMindtree Full timeJob Description: Primary Skill Set: Lead PySpark Developer Seconday Skill Set: SAS tools Lead a team of data engineers in the design and implementation of big data solutions using PySpark. Collaborate with cross-functional teams to understand business needs and develop data-driven solutions. Design and implement scalable and robust data pipelines using...
-
Lead PySpark Developer
2 weeks ago
Bengaluru, India LTIMindtree Full timeJob Description: Primary Skill Set: Lead PySpark Developer Seconday Skill Set: SAS tools Lead a team of data engineers in the design and implementation of big data solutions using PySpark. Collaborate with cross-functional teams to understand business needs and develop data-driven solutions. Design and implement scalable and robust data pipelines using...
-
Bigdata + Pyspark
2 weeks ago
Bengaluru, Karnataka, India Impetus Technologies Full timeBIgdata + Pyspark :Big Data; Hadoop / HDFS,Pyspark,PythonExperience with design and coding across one or more platforms and languages (e.g. , Python/Pyspark/SQL) as appropriateHands-on expertise with application design, software developmentProficient in Big Data technologies Designs, codes, tests, corrects and documents large and/or complex programs and...
-
Data Engineer
2 months ago
Bengaluru, India Akshaya IT Business Solutions Full timeJob Description :- Design and develop real-time data ingestion pipelines using Databricks and Spark Streaming to enable timely processing of large volumes of data.- Implement complex data transformations and aggregations to extract actionable insights from streaming data sources.- Collaborate with data scientists, analysts, and business stakeholders to...
-
Bigdata + Pyspark
2 weeks ago
Bengaluru, Karnataka, India Impetus Technologies Full timeBIgdata + Pyspark : Big Data; Hadoop / HDFS,Pyspark,Python Experience with design and coding across one or more platforms and languages (e.g. , Python/Pyspark/SQL) as appropriateHands-on expertise with application design, software development Proficient in Big Data technologies Designs, codes, tests, corrects and documents large and/or complex programs and...
-
Colan Infotech
2 months ago
Bengaluru, India Colan Infotech Pvt Ltd Full timeSkill Set : Pyspark / Scala Spark, Data Factory, Databricks, Python, SQL.Job Description :Roles And Responsibilities :- Must have cloud knowledge in Azure- Should have programming skills with the ability to write optimized and reusable high-quality code.- Design, develop and maintain scalable data pipelines using Pyspark / Scala Spark, Databricks, Python,...