Lead Data Engineer

Role Overview:The Lead Data Engineer is a senior technical leader responsible for architecting end -to -end data flow solutions and guiding the engineering team to deliver complex data projects. In this role, you will design data flow architectures that solve complex business problems, ensuring that all components (ingestion, processing, storage, and access) work together seamlessly to meet client needs. The Lead engineer not only brings expert -level technical skills in data engineering (including mastery of tools like Spark, Trino, Iceberg, etc.), but also excels in systems design and strategic planning. This role involves close collaboration with clients and stakeholders at a strategic level – translating high -level business objectives into a scalable data architecture and roadmap. A formal, consultative approach and strong leadership abilities are essential, as the Lead Data Platform Engineer is expected to oversee project delivery, mentor the team, and ensure the data platform aligns with business goals. Responsibilities: Data Architecture Design: Design scalable and maintainable data architectures for the organization's needs. This includes architecting data pipelines and workflows (batch, streaming, and event -driven) that can handle large data volumes and complex transformations. The lead engineer will make high -level decisions on choosing the right technologies (e.g., deciding when to use Spark vs. streaming frameworks, how to leverage table formats like Iceberg) to create a robust architecture that supports diverse analytical and reporting requirements.Technical Leadership and Mentoring: Provide technical leadership to the data engineering team. Oversee the work of engineers on various projects, conduct design and code reviews, and ensure best practices are followed in coding, testing, and deployment. Mentor junior and senior engineers alike, fostering growth in technical skills and architectural thinking.Client Engagement and Stakeholder Management: Act as the primary technical liaison for clients and business stakeholders. Engage with clients regularly (often daily) to understand their evolving business challenges, gather requirements, and propose data -driven solutions. Translate complex business problems into technical designs, effectively communicating how the proposed data solutions will address client needs. Manage stakeholder expectations and ensure transparency on project progress and challenges.Strategic Planning and Innovation: Collaborate with organizational leadership to align the data platform strategy with broader business objectives. Identify opportunities to improve or extend the data platform with new technologies or methodologies (for example, evaluating new cloud services, or adopting data mesh principles for better domain -oriented architecture). Lead the evaluation and integration of new tools (such as emerging data processing frameworks or analytics platforms) to keep the data infrastructure modern and efficient.Project Oversight and Delivery: Oversee end -to -end delivery of data platform projects, ensuring they are completed on time and meet quality standards. Coordinate across cross -functional teams – including data science, analytics, and IT operations – to integrate their requirements into the data platform. Troubleshoot high -priority issues and guide the team in resolving bottlenecks in data processing or pipeline performance.Governance, Security, and Best Practices: Establish and enforce best practices in data governance, security, and compliance across all data engineering activities. This includes setting standards for data quality, access control, documentation, and recovery procedures. The lead engineer ensures that all data solutions not only function well but also adhere to corporate and regulatory policies for data management.Requirements10+ Years of Data Engineering Experience: Extensive experience (typically around 8 -10 years or more) in data engineering or related fields, with a track record of designing and delivering large -scale data systems. Experience should include building complex data pipelines and architecting data solutions that have been successfully deployed in production.Architecture and Systems Design Expertise: Demonstrated ability to design data flow architectures for real -time and batch processing that are scalable, fault -tolerant, and meet business requirements. Strong knowledge of architecture patterns (such as data lake vs. data warehouse approaches, streaming architectures, and data integration patterns) and the trade -offs involved in technology choices.Mastery of Data Technologies: Deep expertise in SQL and programming (Python or equivalent) is assumed. In addition, mastery of big data and analytics technologies is required – for example, hands -on experience with Apache Spark, Trino, and Apache Iceberg in large -scale environments. Familiarity with cloud -based data platforms (AWS/GCP/Azure services for data processing and storage) and infrastructure -as -code for deploying data systems is expected.Leadership and Communication: Exceptional leadership skills to guide an engineering team and drive technical projects to completion. Ability to mentor and develop talent on the team. Excellent communication and interpersonal skills are essential, as this role interfaces with senior client stakeholders and internal executives. Experience in a client -facing leadership role (such as technical lead or solutions architect in consulting engagements) is highly desirable. The candidate should be adept at communicating complex technical concepts in a clear, professional manner to non -technical audiences.Strategic Thinking: Ability to see the "big picture" and align technical decisions with business strategy. Capable of roadmap planning for the data platform's evolution and making decisions that balance short -term needs with long -term scalability and maintainability.Problem -Solving and Adaptability: Proven problem -solving skills, especially in diagnosing and resolving performance, scalability, or data quality issues in complex systems. Comfortable working in a fast -paced environment and adapting to new technologies or methodologies as needed. A track record of innovation or process improvement in previous roles is a plus.Education: Bachelor's degree in Computer Science, Information Systems, or related field required; Master's degree preferred. A strong foundation in software engineering and system design is required, and any additional certifications or coursework in data architecture or cloud architecture would be beneficial. Benefits Working at Modern  First and foremost, our value system is sacred to us – HEAT: Humility, Empathy, Accountability and Transparency. Beyond this, we are fond of individuals who are curious, love solving problems and can see the larger picture. We love to take a leap of faith on potential. If you believe you haven't had the chance to do your life's best work, Modern is the place for you.   Modern embraces competition for great talent. We have been able to get great talent onboard owing to the attractive compensation and benefits we provide, in addition to the upside we share with all our employees in the form of ESOPs. Moreover, our ESOP policies are highly employee -friendly replicating ethos of some of the best Silicon Valley tech startups. We are committed to making sure our employees create significant value for themselves.