Senior BigData Engineer

Job Summary

We are seeking an experienced and highly skilled Senior Data Engineer to lead the design, development, and optimization of scalable data engineering solutions across our Enterprise Data Lake (EDL) and hybrid data platforms. This role involves technical leadership in data architecture, pipeline orchestration, metadata governance, and data product development, with a strong emphasis on aligning engineering practices with business goals. The ideal candidate will demonstrate a deep understanding of modern data technologies, data modelling, and data governance frameworks.

Primary Objectives

• Provide technical leadership, guidance, and act as a reference point for data engineering and analytical solutions, especially in Data Products.

• Develop and implement modern Data Engineering solutions aligned with industry best practices.

• Drive DataOps/MLOps knowledge-sharing and collaboration across business and IT teams.

• Translate conceptual and logical data models into efficient and maintainable physical data implementations.

Key Responsibilities

• Design and evolve the overall data architecture, ensuring scalability, flexibility, and compliance with enterprise standards.

• Build efficient, secure, and reliable data pipelines using the Bronze-Silver-Gold architecture within EDL.

• Develop and orchestrate scheduled jobs in the EDL environment to support continuous ingestion and transformation.

• Implement Apache Iceberg for data versioning, governance, and optimization.

• Leverage the Medallion framework to standardize data product maturity and delivery.

• Govern metadata, data lineage, and business glossary using tools like Apache Atlas.

• Ensure data security, privacy, and regulatory compliance across all data processes.

• Support Data Mesh principles by collaborating with domain teams to design and implement reusable Data Products.

• Integrate data across structured, semi-structured, and unstructured sources from enterprise systems such as ODS and CRM systems.

• Drive adoption of DataOps/MLOps best practices and mentor peers across units.

• Generate and manage large-scale batch files using Spark and Hive for high-volume data processing.

• Design and implement document-based data models and transform relational models into NoSQL document-oriented structures (eg NoSQL Database or similar system).

Required Skills & Qualifications

• Bachelor's, Master's, or PhD in Computer Science, Data Engineering, or a related discipline.

• 5–7 years of experience in data engineering and distributed data systems.

• Strong hands-on experience with Apache Hive, HBase, Kafka, Solr, Elasticsearch.

• Proficient in data architecture, data modelling, and pipeline scheduling/orchestration.

• Operational experience with Data Mesh, Data Product development, and hybrid cloud data platforms.

• Familiarity with CRM systems, including CRM system, and data sourcing/mapping strategies.

• Proficient in managing metadata, glossary, and lineage tools like Apache Atlas.

• Proven experience in generating large-scale batch files using Spark and Hive.

• Strong understanding of document-based data models and the transformation of relational schemas into document-oriented structures.

Additional Technical & Business Competencies

• Expertise in data administration, modelling, mapping, collection, and distribution.

• Strong understanding of business workflows to support metadata governance.

• Hands-on experience with analytics and DWH tools (e.g., SAS, Oracle, MS SQL, Python, R Programming).

• Familiarity with data modelling tools (e.g., ERWIN), and enterprise databases (Oracle, IBM DB2, MS SQL, Hadoop, Object Store).

• Experience working across hybrid cloud environments (e.g., AWS, Azure Data Factory).

• In-depth knowledge of ETL/ELT processes and automation frameworks.

• Analytical thinker with strong problem-solving and communication skills.

• Able to collaborate effectively across technical and business teams.

• Proven ability to deliver high-quality outcomes within