Senior PySpark Data Engineer
1 week ago
About the Role:
We are seeking a highly skilled Senior PySpark Data Engineer to join our team at Tata Consultancy Services in Bangalore, India.
Job Description:
The ideal candidate will have a minimum of 6-10 years of experience in designing and developing scalable data solutions using PySpark. They should have hands-on experience with complex transformations, large dataset handling, and good command over Python programming.
Key Responsibilities:
- Design and develop efficient data pipelines using PySpark for big data processing.
- Collaborate with cross-functional teams to identify business requirements and develop solutions.
- Maintain and optimize existing data processing systems to ensure high performance and reliability.
Requirements:
- Minimum 5 years of PySpark Development experience, especially in Spark SQL and Complex Transformations.
- Minimum 3 years of Python development experience.
- Minimum 2 years of experience in Azure Data Factory.
- Strong design and data solutioning skills.
- PySpark hands-on experience with complex transformations and large dataset handling experience.
- Good command and hands-on experience in Python.
- Experience working with following concepts, packages, and tools: Object oriented and functional programming, NumPy, Pandas, Matplotlib, requests, pytest, Jupyter, PyCharm and IDLE, Conda and Virtual Environment.
- Working experience must with Hive, HBase or similar.
- Azure Skills, Must have working experience in Azure Data Lake, Azure Data Factory, Azure Databricks, Azure SQL Databases, Azure DevOps, Azure AD Integration, Service Principal, Pass-thru login etc., Networking – vnet, private links, service connections, etc., Integrations – Event grid, Service Bus etc.
- Database skills: Oracle, Postgres, SQL Server – any one database experience.
- Oracle PL/SQL or T-SQL experience.
- Data modelling.
Good-to-Have:
- Understanding and experience in Azure Synapse Analytics.
- Understanding of Hadoop ecosystem.
- NLTK, Scikit-learn, TensorFlow, Keras.
About Us:
Tata Consultancy Services is a leading global IT services company.
Salary Range:
₹1,200,000 - ₹1,800,000 per annum.
-
india Senior Data Integration Engineer Full timeMust Have Skills/Skill Requirement:Design and architect integration solutions to connect various enterprise applications, systems, and databases.Develop and implement integration workflows, APIs, and data pipelines to enable smooth communication and data exchange between different applications.Utilize Azure Integration Services such as Azure Logic Apps,...
-
india Senior Data Integration Engineer Full timeMust Have Skills/Skill Requirement: Design and architect integration solutions to connect various enterprise applications, systems, and databases. Develop and implement integration workflows, APIs, and data pipelines to enable smooth communication and data exchange between different applications. Utilize Azure Integration Services such as Azure Logic Apps,...
-
Senior Data Integration Engineer
2 weeks ago
India Senior Data Integration Engineer Full timeMust Have Skills/Skill Requirement: Design and architect integration solutions to connect various enterprise applications, systems, and databases. Develop and implement integration workflows, APIs, and data pipelines to enable smooth communication and data exchange between different applications. Utilize Azure Integration Services such as Azure Logic...
-
ITI Data | PySpark Data Engineer
16 hours ago
india ITI Data Full timeLocation : IndiaType : Full-timeExperience : 10 – 13 years Functions : Consulting, Finance, Information Technology, Big Data Engineering Industries : Capital Markets, Investment Banking, Alternative Investments, Financial Services, Management Consulting, Information Technology and Services, HealthcareJob DescriptionWe are looking for a PySpark solutions...
-
Cloud Data Engineering Lead
7 days ago
India ITI Data Full timeITI Data is seeking a highly skilled Cloud Data Engineering Lead to spearhead the design and development of scalable data pipelines on AWS. With a strong background in PySpark, Python, and AWS services, you will be responsible for building robust data integration solutions that enable data-driven insights across various internal and external sources.This is...
-
Lead Data Engineer
4 weeks ago
India Wavicle Data Solutions Full timeJob Description:We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering.As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions.Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success.Key Responsibilities:Lead...
-
Lead Data Engineer
4 weeks ago
India Wavicle Data Solutions Full timeJob Description: We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering. As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions. Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success. Key...
-
Senior Data Engineer
2 days ago
India Recro Full timeWE ARE HIRING !! Designation - Senior Data Engineer Location - Bangalore / Kochi. ( Hybrid) Experienced - 6+ Data Engineering project which needs to source data from multiple sources(Batch sources primarily, streaming sources is something that is good to have) to transform load into Data Lake. The client wants to used open stack technologies...
-
india Wavicle Data Solutions Full timeJob Description: We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering. As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions. Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success. Key Responsibilities:...
-
india Wavicle Data Solutions Full timeJob Description:We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering.As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions.Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success.Key Responsibilities:Lead...
-
india Wavicle Data Solutions Full timeJob Description: We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering. As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions. Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success. Key Responsibilities:...
-
AWS Data Engineer
3 months ago
India ITI Data Full timeJob Description We are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources....
-
AWS Data Engineer
4 months ago
India ITI Data Full timeJob DescriptionWe are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources. There...
-
AWS Data Engineer
4 months ago
India ITI Data Full timeJob Description We are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources....
-
AWS Data Engineer
1 month ago
india ITI Data Full timeJob Description We are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources....
-
AWS Data Engineer
4 weeks ago
India ITI Data Full timeJob Description We are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources....
-
Senior Data Engineer
1 day ago
India Tech Mahindra Full timeSenior Data Engineer Location; Pan India 1. ADF Minimum 5 year's experience) 2. PySpark ( 5+ years' experience ) 3. SQL (10 + years' experience).
-
Senior Data Engineer
1 month ago
India iO Associates - UKEU Full timeJob Title: Senior Data Engineer Location: Mohali Salary: up to 30 LPA iO Associates is looking for a Senior Data Engineer in Databricks, Python, Scala, Azure Synapse, and Azure Data Factory to join our dynamic team. Responsibilities include developing and troubleshooting complex data pipelines using Databricks, PySpark, Synapse, ADF, ADLS...
-
Lead Data Engineer: Strategic Data Architect
4 weeks ago
India Wavicle Data Solutions Full timeWavicle Data Solutions is seeking an experienced Lead Data Engineer to play a pivotal role in architecting and implementing data solutions.About the RoleThis position offers a competitive salary of $180,000 - $220,000 per annum, commensurate with experience. Key ResponsibilitiesDesign and develop scalable and efficient data pipelines using Python and...
-
india Data Warehouse Engineer Full timeExperience : 2- 5 YearsPrimary Skills : Strong SQL knowledge, Data warehouse concepts and ETL hands on experience, Azure services like Data Factory, Azure Data Lake, and Good analysis skills.Good to have skills: Power BI, Databricks, Python.
