Data Architect for Entity Resolution and Graph Analytics

About the Role

We are seeking a technical leader to head up our Entity Resolution and Network Generation services.

The Platform

Our cloud-native platform leverages:

• Microservices architecture with Kubernetes orchestration
• Apache Spark for distributed processing
• Elasticsearch for real-time search and fuzzy matching
• Scala as the primary development language
• Data mesh principles with API-first design

Core Responsibilities

Entity Resolution Service

• Design and implement distributed entity resolution algorithms capable of processing billions of records
• Build blocking strategies (e.g. LSH, canopy clustering) optimized for Spark at scale
• Develop fuzzy matching algorithms leveraging Elasticsearch's capabilities
• Create ML-enhanced matching with explainable AI for match decisions
• Implement incremental resolution supporting real-time and batch modes
• Design APIs for entity lookup with sub-100ms latency requirements

Network Generation Service

• Architect distributed graph generation pipelines using GraphX/GraphFrames
• Implement graph analytics algorithms (PageRank, community detection, centrality measures)
• Design storage strategies for multi-billion edge graphs in Parquet/distributed file systems
• Build temporal graph support for time-evolving networks
• Create high-performance graph serving APIs with complex query capabilities
• Optimize graph partitioning to minimize shuffle and maximize locality

AI Model Development

• Build Graph Neural Networks (GNNs): Develop GNN models (e.g. GraphSAGE, GATv2) using PyTorch Geometric or DGL to analyze corporate and transaction networks, detecting fraud rings and risk patterns.
• Implement Entity Resolution: Design algorithms for fuzzy matching, semantic matching (Sentence-BERT), and clustering to unify entities across heterogeneous data sources (e.g. CSVs, APIs, PDFs).
• Create Risk Scoring Models: Combine rule-based, supervised (XGBoost), and unsupervised (Isolation Forest) methods to generate composite risk scores, optimized for real-time and large data processing in trillions.
• Advance Composite AI: Leverage ContexQ's proprietary approach, integrating symbolic AI, vector embeddings, and graph AI for robust entity resolution and network analytics.

Explainable AI (XAI)

• Champion Transparency: Integrate SHAP, LIME, and GNNExplainer to provide clear, interpretable explanations for model predictions, meeting regulatory and ethical standards.
• Ensure Fairness: Audit models for bias and fairness, embedding ethical principles into every stage of development.

Cross-Service Responsibilities

• Ensure seamless integration between entity resolution and network generation
• Design data lineage tracking across both services
• Implement comprehensive monitoring and observability
• Contribute to API design and service contracts
• Optimize for 10x scale growth

Required Skills and Qualifications

Technical Expertise

• 7+ years of experience in distributed computing and big data systems
• 5+ years specifically in entity resolution and graph analytics at scale
• Expert-level Scala programming skills
• Deep experience with Apache Spark, including custom optimizations
• Production experience with Elasticsearch for search and matching
• Proven track record building systems processing billions of entities/edges

Domain Knowledge

• Strong understanding of blocking algorithms and their trade-offs
• Experience with probabilistic record linkage and similarity measures
• Expertise in graph algorithms and their distributed implementations
• Knowledge of graph storage formats and query optimization
• Understanding of ML applications in entity resolution
• BASIC experience of Banking compliances - FinCrime, Fraud

Systems Design

• Experience designing microservices architectures
• Track record of building fault-tolerant, scalable systems
• API design experience with GraphQL or REST
• Performance optimization and capacity planning expertise

Preferred Qualifications

PhD in Computer Science or related field with focus on graphs/entity resolution

• Contributions to open-source projects (especially Spark, GraphX, Elasticsearch)
• Experience with graph databases (Neo4j, Neptune, JanusGraph) or equivalent
• Publications or conference talks on entity resolution or graph analytics
• Experience with real-time stream processing (Kafka, Spark Streaming)
• Knowledge of graph neural networks and embedding techniques

Technical Environment

• Languages: Scala (primary), Python, Java
• Big Data: Apache Spark 3.x, Hadoop ecosystem
• Search: Elasticsearch 8.x
• Orchestration: Kubernetes, Docker
• Storage: HDFS/S3/GCS, Parquet
• Monitoring: Prometheus, Grafana, Jaeger
• CI/CD: Modern DevOps practices

We're looking for someone who thinks in distributed systems and can optimize for both latency and throughput, a technical leader who can make architectural decisions and implement them, a strong communicator who can explain complex graph concepts to stakeholders, a self-directed engineer who can own large technical initiatives end-to-end, and a performance-obsessed developer who benchmarks everything.

Impact You'll Make

• Define the architecture for entity resolution serving multiple business domains
• Build the graph intelligence layer powering advanced analytics and ML
• Create systems that will process billions of entities with millisecond latencies
• Establish best practices for graph computing in our organization
• Mentor other engineers on distributed graph algorithms

---