Lead Data Engineer

As a Lead Data Engineer, you will be playing a crucial role in leading a dynamic team of data engineers and being responsible for technical architecture, design, implementation, optimizing and maintenance of data engineering solution components for our customers. Your will be required to have strong hands-on experience in implementing and leading technical architecture, design and delivery of data engineering solutions on the Microsoft Azure platform, with a good understanding of data warehouse and data lake architecture patterns.Key ResponsibilitiesTechnical Architecture & Design Lead technical architecture and design efforts for data engineering solution components for delivering business solutionsDesign and implement scalable, secure and resilient data engineering solutions on Microsoft Azure platform, with a good understanding of data architecture patternsEvaluate and select appropriate technology stack for data engineering solution components – cloud native services, open-source tools, data storage, data processing frameworks etc.Data Engineering DevelopmentLead, mentor and manage a team of dynamic data engineers for development, testing and implementation of data pipelines ensuing quality, performance and on-time delivery Responsible for detailed technical design and data mapping specificationEnsure adherence to industry best practices, coding standards and guidelines driving continuous improvement in code quality and delivery excellenceOptimize data solutions for performance, cost-efficiency and scalability Ensure data quality and integrity for downstream systems and applicationsTechnical Project ManagementPlan, prioritize and manage data engineering projects, ensuring timely delivery with quality Identify and mitigate risks, troubleshoot issues, handle customer escalationsEnsure alignment with business requirements and priorities, working closely with stakeholders – both internal and customerCollaboration and DocumentationMaintain comprehensive technical documentation, including architecture diagrams, high level / low level data integration design, data mapping specifications, database design etc.Working closely with peers, BI engineers and data scientists to enable advanced analytics, machine learning models and reporting solutionsRequirements6+ years of hands-on experience in delivering data engineering solutions Experience leading a team of 5+ team members, responsible for technical project management including stakeholder management, design review, code reviews and mentoringStrong hands-on experience in Microsoft Azure platform, with experience in data engineering related services and technologies including Azure Data Factory, Azure Functions, Azure Databricks, Azure Synapse Analytics, Azure Event Hub, Azure Stream Analytics and Azure Data Lake StorageExpertise in database concepts, SQL and relational modellingGood understanding and experience working with NoSQL databasesStrong hands-on experience in implementing data pipelines using Python and PySparkGood understanding of data lake and data warehouse concepts, modern data architecture patterns and industry trends in data engineering spaceHands-on experience in data modelling, with a good understanding of both ER modelling and dimensional modellingGood understanding and experience implementing data quality management solutionsGood understanding of metadata management for data engineering solutions and data platformsGood understanding of data governance concepts – including data stewardship, data security and data privacy managementExperience with version control tools like Git for managing version control for source codeExperience with DevOps processes, including experience implementing CI/CD pipelines for data engineering solution componentsStrong analytical and problem-solving skills.Excellent communication and collaboration skillsGood to Have:Certifications on Azure platform, preferably related to Data EngineeringExperience working with Azure Logic Apps and API ManagementGood understand and experience working with BI/visualization tools like PowerBILocation: Chennai, BangaloreEmployment Type: Full-TimeExperience: 6+ years Email: careers@neoware.ai / vidya@neoware.aiWebsite: https://neoware.ai/careers/ About Neoware Neoware is a specialist Data to Intelligence solutions provider. We combine consulting, data engineering, AI/ML, Gen AI and BI expertise with core values - Care, Pursuit of Excellence and Outcome Focus. We help customers solve problems and find answers to challenges in financial Services, healthcare and pharma, manufacturing, media and publishing, sports, technology, energy and utilities and retail.  At Neoware, we partner with technology leaders like AWS, Snowflake, Google Cloud, Databricks and Mage Data in data, intelligence and security. They are our extended team, and our collaboration goes beyond capability building to develop solution accelerators for faster implementation.  Headquartered in New Jersey, USA, with key operations in India (Chennai and Bangalore), Neoware is committed to drive innovation and excellence in the Data and AI landscape.