Senior Data Architect

About the Role

Our organization is developing a next-generation data intelligence platform using a microservices architecture. We're seeking a senior technical leader to spearhead the development of our Entity Resolution and Network Generation services. This role will involve hands-on technical leadership, where you'll architect and implement distributed graph computing solutions processing billions of entities and relationships.

The Platform

We leverage:

• Microservices architecture with Kubernetes orchestration
• Apollo for distributed processing
• Elasticsearch for real-time search and fuzzy matching
• Scala as the primary development language
• Data mesh principles with API-first design

Entity Resolution Service

You'll design and implement distributed entity resolution algorithms capable of processing billions of records. Key responsibilities include:

• Building blocking strategies (e.g., LSH, canopy clustering) optimized for Spark at scale
• Developing fuzzy matching algorithms leveraging Elasticsearch's capabilities
• Creating ML-enhanced matching with explainable AI for match decisions
• Implementing incremental resolution supporting real-time and batch modes
• Designing APIs for entity lookup with sub-100ms latency requirements

Network Generation Service

You'll architect distributed graph generation pipelines using GraphX/GraphFrames. Key responsibilities include:

• Implementing graph analytics algorithms (PageRank, community detection, centrality measures)
• Designing storage strategies for multi-billion edge graphs in Parquet/distributed file systems
• Building temporal graph support for time-evolving networks
• Creating high-performance graph serving APIs with complex query capabilities
• Optimizing graph partitioning to minimize shuffle and maximize locality

AI Model Development

You'll build Graph Neural Networks (GNNs): Develop GNN models (e.g., GraphSAGE, GATv2) using PyTorch Geometric or DGL to analyze corporate and transaction networks, detecting fraud rings and risk patterns. Additionally, you'll implement Entity Resolution: Design algorithms for fuzzy matching, semantic matching (Sentence-BERT), and clustering to unify entities across heterogeneous data sources (e.g., CSVs, APIs, PDFs).

Explainable AI (XAI)

You'll champion transparency by integrating SHAP, LIME, and GNNExplainer to provide clear, interpretable explanations for model predictions, meeting regulatory and ethical standards. Ensure fairness by auditing models for bias and embedding ethical principles into every stage of development.

Cross-Service Responsibilities

You'll ensure seamless integration between entity resolution and network generation. Design data lineage tracking across both services and implement comprehensive monitoring and observability. Contribute to API design and service contracts. Optimize for 10x scale growth.

Required Qualifications

Technical Expertise:

• 7+ years of experience in distributed computing and big data systems
• 5+ years specifically in entity resolution and graph analytics at scale
• Expert-level Scala programming skills
• Deep experience with Apache Spark, including custom optimizations
• Production experience with Elasticsearch for search and matching

Domain Knowledge:

• Strong understanding of blocking algorithms and their trade-offs
• Experience with probabilistic record linkage and similarity measures
• Expertise in graph algorithms and their distributed implementations
• Knowledge of graph storage formats and query optimization
• Understanding of ML applications in entity resolution

Systems Design

Experience designing microservices architectures. Track record of building fault-tolerant, scalable systems. API design experience with GraphQL or REST. Performance optimization and capacity planning expertise.

Benefits

We offer competitive compensation and flexible remote work arrangements. You'll have access to the latest hardware and cloud resources for development and participate in our Long term Incentive plan.

],

---