InvestCloud - Lead Data Engineer

Description : About The Team : As a Lead Data Engineer, you will be joining the newly formed AI, Data & Analytics team, whose mission is to drive increased value from the data InvestCloud captures to enable a smarter financial future for our clients, focused on enhanced intelligence. Ensuring we have fit-for-purpose modern capabilities is a key goal for the team. This is a unique opportunity to shape the architecture and the tech execution patterns of a green field ecosystem to create a next-generation advisor and client experience. We're building for scale. As such, much of what we design and implement today will be the technology/infrastructure which will serve thousands of clients and petabyte-level volumes of data. The core stack we use and are building is : - AWS as our cloud provider - Oracle as our legacy data warehouse - Snowflake as our next-gen data warehouse - Mage AI for data ingestion and processing - Kafka as our message bus - Terraform for building infrastructureKey Responsibilities : - Ask the right questions to deeply understand stakeholder needs, ensuring solutions are aligned with real problems. - Drive the technical strategy of the team, and its execution through prioritization, and delivery management. - Lead a team to build reliable and scalable pipelines and capabilities across the platform, as well as monitor and support the capabilities we offer. - Inspire, teach and guide your team members; own design sessions, upskill the team where needed, maintain the teams cohesion. - Own complex architectures tying multiple services, SaaS tooling and third-party data together, leveraging a strong understanding of a cloud-based stack. - Uphold high-quality standards across documentation, testing, resiliency, monitoring, and code quality. - Drive towards efficiency, lower our cloud spend, tackle our tech debt and look for ways to simplify code, infrastructure and data models across the platform. - Promote data quality, governance and security as a first-class citizen of the platform, complying with relevant regulationsRequired Skills : - You have at least 10 years of relevant professional experience in Data Engineering - You have shaped the architecture of a mature cloud data platform (AWS, GCP, Azure), designed for different consumer types and providing quantifiable business value. - You have served as a leader in technology in the past; youve built a roadmap, planned data/code migrations and led execution, made mistakes and learned from them. - You have interacted with a large community of stakeholders before; you understand the business use cases and can tailor your communication to ICs and Senior Management. - You have designed and implemented performant, reusable, and scalable data models in a cloud data warehouse (dbt, BigQuery, Snowflake) and in a legacy ecosystem (Oracle, Postgres) - You have hands-on experience in building resilient batch (Airflow, Fivetran, Mage AI, Airbyte) and streaming (Kafka, Kinesis, Flink, Spark) data pipelines at scale (> 1 TB/day) - You can build and maintain your own infrastructure through IaC (Terraform, OpenTofu, Ansible), containerization (Docker) and CI/CD (Jenkins, Github Actions). - You preach operational procedures from data and infrastructure observability (Monte Carlo, Datadog, Prometheus), alerting and incident management (PagerDuty, ) - You bring deep expertise in SQL and Python, and use them to lead by example in technical execution - Youre AI-proficient or AI-curious, eager to explore how emerging technologies can enhance team productivity. - You have experimented with Machine Learning frameworks (TensorFlow, PyTorch, Scikit-learn) and LLM frameworks (e.g. Langchain)Apply without meeting all requirements statement : If you dont meet every requirement but believe youd thrive in this role, wed still love to hear from you. We're always keen to speak to people who connect with our mission and values.Location and Travel : The ideal candidate will be expected to work from the office on a regular basis (3 days minimum per week). Occasional travel may be required.Compensation : The salary range will be determined based on experience, skills, and geographic location.Equal Opportunity Employer : InvestCloud is committed to fostering an inclusive workplace and welcomes applicants from all backgrounds