Cynosure Corporate Solutions | Lead Data Engineer
4 days ago
Roles and Responsibilities:
- Working with clients to understand their data.
- Based on the understanding you will be building the data structures and pipelines.
- You will be working on the application from end to end collaborating with UI and other development teams.
- You will be working with various cloud providers such as Azure & AWS.
- You will be engineering data using the Hadoop/Spark ecosystem.
- You will be responsible for designing, building, optimizing and supporting new and existing data pipelines.
- Orchestrating jobs using various tools such Oozie, Airflow, etc.
- Developing programs for cleaning and processing data.
- You will be responsible for building the data pipelines to migrate and load the data into the HDFS either on-prem or in the cloud.
- Developing Data ingestion/process/integration pipelines effectively.
- Creating Hive data structures, metadata and loading the data into data lakes / Bigdata warehouse environments.
- Optimized (Performance tuning) many data pipelines effectively to minimize cost.
- Code versioning control and git repository is up to date.
- You should be able to explain the data pipeline to internal and external stakeholders.
- You will be responsible for building and maintaining CI/CD of the data pipelines.
Preferred Qualifications:
- Bachelor’s degree in computer science or related field.
- Minimum of 5+ years working experience with Spark, Hadoop eco systems.
- Minimum of 4+ years working experience on designing data streaming pipelines
- Minimum experience of 3+ years on NoSQL and Spark Streaming.
- Proven experience with big data ecosystem tools such as Sqoop, Spark, SQL, API, Hive, Oozie, Airflow, etc..
- Solid experience in all phases of SDLC with 10+ years of experience (plan, design, develop, test, release, maintain and support)
- Hands-on experience using Azure’s data engineering stack.
- Should have implemented projects using programming languages such as Scala or Python.
- Working experience on SQL complex data merging techniques such as windowing functions etc..
- Hands-on experience with on-prem distribution tools such as Cloudera/Horton Works/MapR.
-
india Cynosure Corporate Solutions Full timeRoles and Responsibilities:Working with clients to understand their data.Based on the understanding you will be building the data structures and pipelines.You will be working on the application from end to end collaborating with UI and other development teams.You will be working with various cloud providers such as Azure & AWS.You will be engineering data...
-
India Cynosure Corporate Solutions Full timeCynosure Corporate Solutions is seeking a highly skilled Data Engineer to lead our big data solutions team. This role requires expertise in designing, building, and optimizing complex data pipelines using Hadoop/Spark ecosystem.About the Role:We are looking for an experienced professional who can work collaboratively with cross-functional teams to deliver...
-
india Wavicle Data Solutions Full timeJob Description: We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering. As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions. Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success. Key Responsibilities:...
-
india Wavicle Data Solutions Full timeJob Description:We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering.As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions.Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success.Key Responsibilities:Lead...
-
india Wavicle Data Solutions Full timeJob Description: We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering. As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions. Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success. Key Responsibilities:...
-
Lead data engineer
3 weeks ago
India Wavicle Data Solutions Full timeJob Description: We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering. As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions. Your proficiency in Python, Py Spark, AWS, Databricks, SQL, and leadership skills will be crucial for success. Key...
-
Lead Data Engineer
4 weeks ago
India Wavicle Data Solutions Full timeJob Description:We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering.As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions.Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success.Key Responsibilities:Lead...
-
Lead Data Engineer
4 weeks ago
India Wavicle Data Solutions Full timeJob Description: We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering. As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions. Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success. Key...
-
Lead Data Engineer: Strategic Data Architect
4 weeks ago
India Wavicle Data Solutions Full timeWavicle Data Solutions is seeking an experienced Lead Data Engineer to play a pivotal role in architecting and implementing data solutions.About the RoleThis position offers a competitive salary of $180,000 - $220,000 per annum, commensurate with experience. Key ResponsibilitiesDesign and develop scalable and efficient data pipelines using Python and...
-
india Data Warehouse Engineer Full timeExperience : 2- 5 YearsPrimary Skills : Strong SQL knowledge, Data warehouse concepts and ETL hands on experience, Azure services like Data Factory, Azure Data Lake, and Good analysis skills.Good to have skills: Power BI, Databricks, Python.
-
Data Warehouse Engineer
2 weeks ago
India Data Warehouse Engineer Full timeExperience : 2- 5 Years Primary Skills : Strong SQL knowledge, Data warehouse concepts and ETL hands on experience, Azure services like Data Factory, Azure Data Lake, and Good analysis skills. Good to have skills: Power BI, Databricks, Python.
-
Data Warehouse Expert for Analytics Projects
2 weeks ago
India Data Warehouse Engineer Full timeAbout the RoleWe are seeking an experienced Data Warehouse Engineer to join our team. In this role, you will design, develop, and maintain data warehouses that support business intelligence and analytics initiatives.
-
india Senior Data Integration Engineer Full timeMust Have Skills/Skill Requirement:Design and architect integration solutions to connect various enterprise applications, systems, and databases.Develop and implement integration workflows, APIs, and data pipelines to enable smooth communication and data exchange between different applications.Utilize Azure Integration Services such as Azure Logic Apps,...
-
india Senior Data Integration Engineer Full timeMust Have Skills/Skill Requirement: Design and architect integration solutions to connect various enterprise applications, systems, and databases. Develop and implement integration workflows, APIs, and data pipelines to enable smooth communication and data exchange between different applications. Utilize Azure Integration Services such as Azure Logic Apps,...
-
Senior Data Integration Engineer
2 weeks ago
India Senior Data Integration Engineer Full timeMust Have Skills/Skill Requirement: Design and architect integration solutions to connect various enterprise applications, systems, and databases. Develop and implement integration workflows, APIs, and data pipelines to enable smooth communication and data exchange between different applications. Utilize Azure Integration Services such as Azure Logic...
-
Cloud Data Engineering Lead
2 weeks ago
India ITI Data Full timeITI Data is seeking a highly skilled Cloud Data Engineering Lead to spearhead the design and development of scalable data pipelines on AWS. With a strong background in PySpark, Python, and AWS services, you will be responsible for building robust data integration solutions that enable data-driven insights across various internal and external sources.This is...
-
india IRIS-Corporate Solutions Full timeCompany Profile IRIS Corporate Solutions Private Limited is a new-age HR solutions company focused on preparing Millennials, Gen Z, and Gen Alpha for the future. With three decades of expertise in HR services, we deliver quality interventions to create a skilled workforce through recognized HR delivery models. Our technology-driven approach offers...
-
IGT Solutions | Lead Data Engineer | india
6 days ago
india IGT Solutions Full timeRole: Lead Data Engineer Primary Skills: Python, Pyspark, AWS Glue, EMR, Lambda, Snowflake, ETL, PLSQL Experience: - 7+ Years Location: - Remote Notice period: - Immediate Joiner Job Overview : We are seeking a skilled and experienced Lead Data Engineer to join our data engineering team. The ideal candidate will have over 8 years of hands-on experience...
-
Lead Data Engineer
6 days ago
India IGT Solutions Full timeRole: Lead Data Engineer Primary Skills: Python, Pyspark, AWS Glue, EMR, Lambda, Snowflake, ETL, PLSQL Experience: - 7+ Years Location: - Remote Notice period: - Immediate Joiner Job Overview : We are seeking a skilled and experienced Lead Data Engineer to join our data engineering team. The ideal candidate will have over 8 years of hands-on...
-
India ITI Data Full timeWe are seeking an experienced PySpark Data Engineer to join our team at ITI Data. As a key member of our team, you will design and build scalable data solutions for one of our Fortune 500 Client programs.About the Role:The ideal candidate will have extensive experience in working with Hadoop, NoSQL databases, and Spark framework on Python. You will be...