Sr. Data Engineer

We are an Australian Data and AI company and are expanding our Databricks Capability in India and are seeking a Senior Databricks Data Engineer to build scalable, production-grade data platforms that enable AI and advanced analytics for our global clients. We are open, transparent, no-nonsense, innovative problem solvers and eternal learners as a team where individual passion and brilliance is combined with a great team spirit to nurture everyone’s journey to be their best and achieve what they aspire. Role Overview: As a Senior Databricks Data Engineer, you will architect, design, and develop data pipelines and modeling layers that power analytical and AI-driven workloads. You are likely to work on end-to-end Databricks projects—from ingestion to curation to enablement for ML/AI—while collaborating with consulting and client delivery teams across regions. Responsibilities: - Solution Ownership: Take end-to-end responsibility for project delivery, including planning, prioritization, and ensuring data solutions meet quality and performance expectations. - Stakeholder Collaboration: Engage with business and product teams to understand data needs, translate requirements, and align engineering solutions with strategic goals. - Cross-Functional Communication: Present complex data concepts in a clear, actionable manner to non-technical stakeholders and leadership. - Continuous Improvement: Identify process inefficiencies and lead initiatives to improve data workflows, scalability, and team productivity. - Design and build scalable, reliable data pipelines using Databricks Auto Loader, Delta Live Tables (DLT), and Workflows. - Develop and maintain curated Medallion architecture layers (Bronze, Silver, Gold) using Delta Lake and Databricks SQL. - Orchestrate and deploy pipelines using Databricks Asset Bundles (DABs) and integrate with CI/CD tools (GitHub Actions, Azure DevOps). - Implement data ingestion using Lakeflow Connect and integrate with external sources such as Kafka, ADLS, and REST APIs. - Develop real-time and batch processing jobs with PySpark, SQL, and Delta Lake. - Ensure data quality, lineage, and governance using Unity Catalog and DLT Expectations. - Collaborate with data scientists and ML engineers to prepare feature-ready datasets for MLflow and AI workloads. - Optimise Databricks clusters and jobs for cost, performance, and reliability. - Participate in client workshops, provide architectural input, and communicate technical outcomes effectively to business stakeholders. - Take ownership of projects ensuring exceptional customer outcomes - Contribute to a data-driven culture that fosters innovation and agility. Required Skills & Competencies 3 - 5 years of hands-on experience with Databricks, including proficiency in: - Demonstrated experience in client-facing delivery, including conducting online workshops, eliciting data requirements, and presenting technical findings to business and technical stakeholders. - Databricks Certification (Data Engineer Associate or Professional) required. - PySpark, SQL, and data modeling (Dimensional, ELT). - Medallion architecture and Delta Lake optimization (Z-Ordering, Optimize, Compaction). - Unity Catalog, Workflows, and Feature Store. - Azure Data Services (ADLS, ADF, Synapse) or equivalent on AWS/GCP. - Auto Loader for incremental ingestion - Delta Live Tables (DLT) for declarative ETL pipelines - Lakeflow Connect for source integration - Databricks SQL for analytics and transformation - Ability to thrive in a fast-paced, entrepreneurial, team environment Preferred (Nice to have) - Prior experience in Retail, CPG, or Transportation & Logistics domains. - CI/CD skills for Databricks using GitHub Actions or Azure DevOps and Databricks Asset Bundles (DABs) for deployment automation - Experience implementing data quality monitoring and governance frameworks. What we offer At our core, we are building a youthful, forward-looking Data & AI brand — where engineers, consultants, and innovators thrive on solving meaningful problems with cutting-edge technologies. We offer an environment that blends technical depth, consulting exposure, and personal growth in equal measure. - Flexible Work Options – Work 100% remotely or in a hybrid mode from our Chennai base — we trust our people to deliver outcomes, not fill timesheets. - Learning & Mentorship Culture – Work directly with senior data and AI leaders who invest in your career growth, leadership development, and technical mastery. - High-Calibre Clients & Projects – Collaborate with mature global clients pursuing ambitious Data and AI transformation goals — exposure to real enterprise-grade challenges.