Senior Data Engineer

Senior Data Engineer Position Overview We are seeking an experienced Senior Data Engineer with 8-12 years of hands-on experience to join our team on-site. This role requires a self-starter who can architect and implement robust data foundation layers while providing technical leadership and guidance to the engineering team. The ideal candidate will have deep expertise in modern data platforms, particularly Databricks/Delta Lake, and a strong understanding of how to build scalable, reliable data infrastructure that powers enterprise analytics and AI initiatives. Key Responsibilities Data Foundation & Architecture - Design, build, and maintain scalable data foundation layers that serve as the backbone for analytics, reporting, and AI/ML initiatives - Architect end-to-end data solutions including storage, integration, pipelines, and modeling components - Establish best practices and standards for data architecture across the organization - Evaluate and recommend new technologies and tools to enhance the data platform Data Storage & Management - Design and implement data storage solutions using modern cloud-based platforms - Build and optimize data warehouses, data lakes, and lakehouse architectures - Implement data partitioning, indexing, and optimization strategies for performance at scale - Ensure data availability, reliability, and disaster recovery capabilities Data Pipelines & Integration - Develop robust, scalable ETL/ELT pipelines for batch and real-time data processing - Build data integration frameworks connecting diverse data sources (APIs, databases, files, streaming) - Implement data quality checks and monitoring throughout the pipeline - Optimize pipeline performance and troubleshoot production issues - Automate data workflows and orchestration processes Data Modeling - Design conceptual, logical, and physical data models aligned with business requirements - Implement dimensional modeling (star/snowflake schemas) for analytics use cases - Create and maintain data dictionaries and documentation - Ensure data models support both operational and analytical workloads Team Leadership & Collaboration - Mentor and guide junior and mid-level data engineers on technical best practices - Conduct code reviews and ensure high-quality, maintainable code standards - Lead technical design discussions and architectural decisions - Collaborate with data scientists, analysts, and business stakeholders to understand requirements - Drive continuous improvement initiatives within the data engineering team Required Qualifications Experience & Education - 8-12 years of experience in data engineering or related roles - Bachelor's degree in Computer Science, Engineering, Information Systems, or related field (or equivalent experience) - Proven track record of building enterprise-scale data platforms - Experience in building intelligent AI agents and RAG-based applications along with good exposure to Prompt engineering - Expertise in implementing and optimizations with Databases such as SQL Server, Postgres SQL, Clickhouse etc., Core Technical Skills Must Have: - Data Foundation Layer Expertise: Deep understanding and practical implementation of: o Data storage architectures (data lakes, warehouses, lakehouse) o Data pipeline development and orchestration o Data integration patterns and frameworks o Data modeling techniques and best practices - Programming: Strong proficiency in Python and SQL - Cloud Platforms: Experience with Azure (or comparable) - Big Data Technologies: Spark, Kafka, or similar distributed processing frameworks AI & Modern Tools - Working knowledge of AI tools and their integration with data platforms - Experience supporting ML/AI workflows and feature engineering pipelines Soft Skills - Self-Starter: Ability to work independently, take initiative, and drive projects to completion - Leadership: Proven ability to guide and mentor team members - Communication: Excellent verbal and written communication skills to collaborate with technical and non-technical stakeholders - Problem-Solving: Strong analytical and troubleshooting abilities - Adaptability: Comfortable working in fast-paced environments with evolving requirements