Cognite - Principal Engineer - Data Transfromation

Description :About Cognite :Embark on a transformative journey with Cognite, a global SaaS forerunner in leveraging AI and data to unravel complex business challenges through our cutting-edge offerings including Cognite Atlas AI, an industrial agent workbench, and the Cognite Data Fusion (CDF) platform.We were awarded the 2022 Technology Innovation Leader for Global Digital Industrial Platforms & Cognite was recognized as 2024 Microsoft Energy and Resources Partner of the Year.In the realm of industrial digital transformation, we stand at the forefront, reshaping the future of Oil & Gas, Chemicals, Pharma and other Manufacturing and Energy sectors.Join us in this venture where AI and data meet ingenuity, and together, we forge the path to a smarter, more connected industrial future.Our values :Impact : Cogniters strive to make an impact in all that they do.We are result-oriented, always asking ourselves.Ownership : Cogniters embrace a culture of ownership.We go beyond our comfort zones to contribute to the greater good, fostering inclusivity and sharing responsibilities for challenges and success.Relentless : Cogniters are relentless in their pursuit of innovation.We are determined and deliverable (never ruthless or reckless), facing challenges head-on and viewing setbacks as opportunities for growth.About Cognite & This Role :Cognite is revolutionizing industrial data management through our flagship product, Cognite Data Fusion a state-of-the-art SaaS platform that transforms how industrial companies leverage their data.We're seeking an exceptional.Staff Data Platform Engineer who will drive technical strategy, architectural excellence, and cross-organizational impact.As a Principal Software Engineer, you'll be a technical leader who shapes platform direction, mentors engineering teams, and tackles the most complex data engineering challenges in industrial technology.This role combines deep technical expertise with strategic thinking and leadership influence.Strategic Impact & Leadership :Technical Strategy & Architecture :- Define platform architecture for next-generation industrial data processing capabilities serving 100M+ daily requests.- Drive technology roadmap decisions that impact multiple engineering teams and product areas.- Lead architectural reviews and establish technical standards across the data platform organization.- Champion innovation initiatives that differentiate Cognite's technical capabilities in the industrial data market.Platform Engineering Excellence :- Architect foundational systems that serve as building blocks for multiple product teams and use cases.- Design advanced abstractions and frameworks that accelerate development velocity across the organization.- Own end-to-end platform reliability including SLAs, disaster recovery, and operational excellence.- Drive platform observability strategy with comprehensive metrics, alerting, and distributed tracing.Advanced Technical Leadership :- Lead Apache Spark ecosystem contributions modify core components, contribute to open-source projects, influence roadmaps.- Architect multi-petabyte streaming systems using Flink, Kafka, and custom processing engines for industrial IoT data.- Design custom query engines and DSLs optimized for time-series analytics and operational data patterns.- Pioneer advanced optimization techniques for columnar storage, query planning, and distributed execution.Organizational Impact :- Mentor Staff and Senior engineers across multiple teams, developing technical leadership capabilities.- Drive cross-functional initiatives involving ML platform, product engineering, and infrastructure teams.- Establish engineering culture around technical excellence, innovation, and customer obsession.- Represent Cognite externally through conference speaking, opensource leadership, and technical thought leadership.Exceptional Candidate Profile :Technical Leadership Mastery (8+ years) :- Distributed Systems Architecture Advanced Apache Spark expertise contributed to Spark core, Catalyst optimizer, or execution engine with merged commits - Large-scale streaming architecture designed and operated Flink/Kafka systems processing 1M+ events/second with complex state management - Custom execution engine development built domain-specific query engines or processing frameworks from scratch - Performance engineering excellence consistently delivered 10x+ performance improvements through algorithmic and architectural innovations.Platform Engineering Leadership : - Multi-tenant platform design architected platforms serving 50+ engineering teams with sophisticated resource isolation and management - Advanced data lake architecture -designed exabyte-scale storage systems with intelligent tiering, compaction, and query optimization - Service mesh and infrastructure implemented sophisticated microservices architectures with advanced networking, security, and observability.Data Engineering Innovation : - OLAP system expertise deep experience with ClickHouse, Pinot, Druid internals including cluster management and query optimization - Advanced stream processing implemented complex event processing, windowing, and stateful computations for industrial use cases - Data modeling excellence designed sophisticated schemas for dimensional modeling, event sourcing, and real-time analytics.Technical Leadership Excellence :Strategic Technical Decision Making - Technology evaluation leadership led organization-wide technology adoption decisions with comprehensive POCs and risk analysis Architecture governance established and maintained architectural principles, standards, and review processes - Technical debt management strategically balanced feature velocity, system reliability, and long-term maintainability.Open Source & Community Leadership :- Open-source contributions to major Apache projects in the data space (e Apache Spark or Kafka) is a big plus Significant open source contributions maintainer or core contributor to major Apache projects (Spark, Flink, Kafka, Airflow) - Technical thought leadership regular speaker at major conferences (Strata, Spark Summit, QCon) with recognized expertise - Community building -organized engineering communities, contributed to standards bodies, or led technical advisory roles Cross-Functional Collaboration - Product partnership excellence translated ambiguous business requirements into robust technical architectures - Engineering leadership mentored and developed multiple Senior and Staff engineers with measurable career progression - Executive communication effectively communicated complex technical concepts to C-level stakeholders and board members.Startup Excellence & Scale :- High-Impact Delivery Rapid scaling experience led technical initiatives during hypergrowth phases (10x team/infrastructure scaling) - Customerfacing technical leadership directly engaged with enterprise customers on complex technical integrations - Business impact ownership drove technical initiatives that resulted in measurable business outcomes (cost reduction, revenue enablement).- Innovation & Ambiguity Navigation Greenfield platform development architected major platform components from concept to production at scale - Technical risk management successfully navigated high-risk technical decisions with incomplete information - Competitive differentiation -developed technical capabilities that created sustainable competitive advantages.Technical Environment & Challenges :Advanced Technical Stack :- Core Languages : Kotlin, Scala, Java (advanced JVM optimization).- Big Data Ecosystem : Apache Spark (internals), Apache Flink, Apache Kafka, Apache Airflow.- Advanced Storage : ClickHouse, PostgreSQL, Elasticsearch, S3-compatible systems with custom optimization.- Infrastructure : Kubernetes (operator development), Terraform,Advanced observability stack.Cutting-Edge Technologies You'll Drive :- Table Format Innovation : Apache Iceberg, Delta Lake internals andoptimization.- Query Engine Development : Trino/Presto, Apache Pinot, custom engine development.- Advanced Streaming : Complex event processing, exactly-once semantics, advanced windowing.- ML/AI Integration : Feature stores, model serving infrastructure, MLOps platform integration.Unique Technical Challenges :- Industrial IoT at Scale : Processing sensor data from millions of industrial assets in real-time.- Complex Time-Series Analytics : Advanced temporal queries, anomaly detection, predictive analytics.- Multi-Cloud Optimization : Cost and performance optimization across AWS, Azure, GCP.- Enterprise Integration : Secure, scalable integration with legacy industrial systems (ref:hirist.tech)