Senior Big Data Engineer

As a seasoned professional in the field of data engineering, you will play a pivotal role in designing and implementing cutting-edge data solutions that drive business growth and innovation.

Key responsibilities include architecting and defining data flows for big data/data lake use cases, with excellent knowledge on implementing full life cycle of data management principles such as Data Governance, Architecture, Modelling, Storage, Security, Master data, and Quality.

You will work closely with cross-functional teams to improve data models that feed BI tools, increasing data accessibility, and fostering data-driven decision making across the organization.

Collaborative team environment where you will be expected to act as a coach and provide consultancy services and advice to data engineers by offering technical guidance, ensuring architecture principles, design standards and operational requirements are met.

Responsible for estimating the cluster size, core size, monitoring, and troubleshooting of the data bricks cluster and analysis server to produce optimal capacity for computing data ingestion.

Deliver master data cleansing and improvement efforts; including automated and cost-effective solutions for processing, cleansing, and verifying the integrity of data used for analysis.

Experience working with distributed storage and processing massive data into data lake using Scala or Python, with proficiency in Hadoop and Spark eco systems like HDFS, Hive, Sqoop, Oozie, Spark core, streaming.

Proven experience in pulling data through REST API, ODATA, XML, Web services, with ability to work with ETL tools with strong knowledge on ETL concepts.

Strong focus on delivering outcomes, with expertise in securing the big data environment including encryption, tunnelling, access control, secure isolation.

• \
• Bachelor's degree (masters' preferred) in Computer Science, Engineering, or any other technology related field\
• 10+ years of experience in data analytics platform and hands-on experience on ETL and ELT transformations with strong SQL programming knowledge.\
• 5+ years of hands-on experience on big data engineering, distributed storage and processing massive data into data lake using Scala or Python.\
• Proficient knowledge on Hadoop and Spark eco systems like HDFS, Hive, Sqoop, Oozie, Spark core, streaming.\
• Experience with programming languages such as Scala, Java, Python and Shell scripting\
• Proven Experience in pulling data through REST API, ODATA, XML,Web services.\
• Experience with Azure product offerings and data platform.\
• Experience in data modelling (data marts, snowflake/Star, Normalization, SCD2).\
• Architect and defining the data flows and building highly efficient, scalable data pipelines.\
• To work in tandem with the Enterprise and Domain Architects to understand the business goals and vision, and to contribute to the Enterprise Roadmaps.\
• Strong troubleshooting skills, problem solving skills of any issues stopping business progress.\
• Coordinate with multiple business stake holders to understand the requirement and deliver.\
• Conducting a continuous audit of data management system performance, refine whenever required, and report immediately any breach or loopholes to the stakeholders.\
• Allocate task to various team members, track the status and provide the report on activities to management.\
• Understand the physical and logic plan of execution and optimize the performance of data pipelines.\
• Extensive background in data mining and statistical analysis.\
• Able to understand various data structures and common methods in data transformation.\
• Ability to work with ETL tools with strong knowledge on ETL concepts.\
• Strong focus on delivering outcomes.\
• Data management: modelling, normalization, cleaning, and maintenance\
• Understand Data architectures, Data warehousing principles and be able to participate in the design and development of conventional data warehouse solutions.

The selected candidate will enjoy a competitive salary and benefits package, along with opportunities for career growth and professional development.

This is an exciting opportunity to join our dynamic team and contribute to the success of our organisation.