Senior Data Engineer

We're building a high-performance data intelligence platform using a scalable architecture. We need a senior expert to lead the development of our Entity Resolution and Network Generation services.

This is a technical leadership role where you'll design and implement distributed graph computing solutions for billions of entities and relationships.

Our cloud-native platform leverages:

• Microservices architecture with orchestration
• Apache Spark for distributed processing
• Elasticsearch for real-time search and fuzzy matching
• Scala as the primary development language

Data mesh principles with API-first design ensure seamless integration between entity resolution and network generation.

1. Entity Resolution Service

• Design and implement distributed entity resolution algorithms capable of processing billions of records
• Build blocking strategies (e.g. LSH, canopy clustering) optimized for Spark at scale
• Develop fuzzy matching algorithms leveraging Elasticsearch's capabilities
• Create ML-enhanced matching with explainable AI for match decisions
• Implement incremental resolution supporting real-time and batch modes
• Design APIs for entity lookup with sub-100ms latency requirements

• Architect distributed graph generation pipelines using GraphX/GraphFrames
• Implement graph analytics algorithms (PageRank, community detection, centrality measures)
• Design storage strategies for multi-billion edge graphs in Parquet/distributed file systems
• Build temporal graph support for time-evolving networks
• Create high-performance graph serving APIs with complex query capabilities
• Optimize graph partitioning to minimize shuffle and maximize locality

• Build Graph Neural Networks (GNNs): Develop GNN models (e.g., GraphSAGE, GATv2) using PyTorch Geometric or DGL to analyze corporate and transaction networks
• Implement Entity Resolution: Design algorithms for fuzzy matching, semantic matching (Sentence-BERT), and clustering to unify entities across heterogeneous data sources
• Create Risk Scoring Models: Combine rule-based, supervised (XGBoost), and unsupervised (Isolation Forest) methods to generate composite risk scores
• Advance Composite AI: Leverage ContexQ's proprietary approach, integrating symbolic AI, vector embeddings, and graph AI for robust entity resolution and network analytics

• Champion Transparency: Integrate SHAP, LIME, and GNNExplainer to provide clear, interpretable explanations for model predictions
• Ensure Fairness: Audit models for bias and fairness, embedding ethical principles into every stage of development

• Technical Expertise:
  • 7+ years of experience in distributed computing and big data systems
  • 5+ years specifically in entity resolution and graph analytics at scale
  • Expert-level Scala programming skills
  • Deep experience with Apache Spark, including custom optimizations
• Domain Knowledge:
  • Strong understanding of blocking algorithms and their trade-offs
  • Experience with probabilistic record linkage and similarity measures
  • Expertise in graph algorithms and their distributed implementations
• Systems Design:
  • Experience designing microservices architectures
  • Track record of building fault-tolerant, scalable systems

Competitive compensation package with flexible remote work arrangements, latest hardware and cloud resources for development, LTIP - Long term Incentive plan, 75% of base as Bonus payment at the end of 4th year in service, Equity potential of up to USD 150K every year.

Technical screen focusing on distributed systems and graph algorithms, system design session on entity resolution at scale, coding session implementing a graph algorithm in Scala, architecture discussion with the team, final round with leadership.