Senior Graph Computing Engineer

Unlock your potential as a seasoned expert in AI and Machine Learning.

We're on the hunt for a top-notch Senior Engineer to spearhead the development of our Entity Resolution and Network Generation services. This is an opportunity to take charge, drive innovation, and lead the way in distributed graph computing solutions.

About the Role

This role requires a deep understanding of complex systems, with a focus on building scalable, fault-tolerant architectures. You'll be working with cutting-edge technologies like Apache Spark, Elasticsearch, and GraphX/GraphFrames to create high-performance graph serving APIs.

Responsibilities

• Design and implement distributed entity resolution algorithms capable of processing billions of records.
• Build blocking strategies (e.g. LSH, canopy clustering) optimized for Spark at scale.
• Develop fuzzy matching algorithms leveraging Elasticsearch's capabilities.
• Create ML-enhanced matching with explainable AI for match decisions.
• Implement incremental resolution supporting real-time and batch modes.
• Design APIs for entity lookup with sub-100ms latency requirements.
• Architect distributed graph generation pipelines using GraphX/GraphFrames.
• Implement graph analytics algorithms (PageRank, community detection, centrality measures).
• Design storage strategies for multi-billion edge graphs in Parquet/distributed file systems.
• Build temporal graph support for time-evolving networks.
• Optimize graph partitioning to minimize shuffle and maximize locality.
• Lead the development of Graph Neural Networks (GNNs): Develop GNN models (e.g., GraphSAGE, GATv2) using PyTorch Geometric or DGL to analyze corporate and transaction networks, detecting fraud rings and risk patterns.
• Implement Entity Resolution: Design algorithms for fuzzy matching, semantic matching (Sentence-BERT), and clustering to unify entities across heterogeneous data sources (e.g., CSVs, APIs, PDFs).
• Create Risk Scoring Models: Combine rule-based, supervised (XGBoost), and unsupervised (Isolation Forest) methods to generate composite risk scores, optimized for real-time and large data processing in trillions.
• Advance Composite Ai: Leverage ContexQ's proprietary approach, integrating symbolic Ai, vector embeddings, and graph Ai for robust entity resolution and network analytics.
• Champion Transparency: Integrate SHAP, LIME, and GNNExplainer to provide clear, interpretable explanations for model predictions, meeting regulatory and ethical standards.
• Ensure Fairness: Audit models for bias and fairness, embedding ethical principles into every stage of development.
• Cross-service Responsibilities:
  • Ensure seamless integration between entity resolution and network generation.
  • Design data lineage tracking across both services.
  • Implement comprehensive monitoring and observability.
  • Contribute to Api design and service contracts.
  • Optimize for 10x scale growth.

Required Qualifications

Technical Expertise

• 7+ years of experience in distributed computing and big data systems.
• 5+ years specifically in entity resolution and graph analytics at scale.
• Expert-level Scala programming skills.
• Deep experience with Apache Spark, including custom optimizations.
• Production experience with Elasticsearch for search and matching.
• Proven track record building systems processing billions of entities/edges.

Domain Knowledge

• Strong understanding of blocking algorithms and their trade-offs.
• Experience with probabilistic record linkage and similarity measures.
• Expertise in graph algorithms and their distributed implementations.
• Knowledge of graph storage formats and query optimization.
• Understanding of ML applications in entity resolution.
• Brief experience of Banking compliances - FinCrime, Fraud.

Systems Design

• Experience designing microservices architectures.
• Track record of building fault-tolerant, scalable systems.
• API design experience with GraphQL or REST.
• Performance optimization and capacity planning expertise.

Preferred Qualifications

PhD in Computer Science or related field with focus on graphs/entity resolution.

Contributions to open-source projects (especially Spark, GraphX, Elasticsearch).

Experience with graph databases (Neo4j, Neptune, JanusGraph) or equivalent.

Publications or conference talks on entity resolution or graph analytics.

Experience with real-time stream processing (Kafka, Spark Streaming).

Knowledge of graph neural networks and embedding techniques.

Technical Environment

• Languages: Scala (primary), Python, Java.
• Big Data: Apache Spark 3.x, Hadoop ecosystem.
• Search: Elasticsearch 8.x.
• Orchestration: Kubernetes, Docker.
• Storage: HDFS/S3/GCS, Parquet.
• Monitoring: Prometheus, Grafana, Jaeger.
• CI/CD: Modern DevOps practices.

What We're Looking For

A technical leader who can make architectural decisions and implement them.

Someone who thinks in distributed systems and can optimize for both latency and throughput.

A strong communicator who can explain complex graph concepts to stakeholders.

A self-directed engineer who can own large technical initiatives end-to-end.

A performance-obsessed developer who benchmarks everything.

Impact You'll Make

Define the architecture for entity resolution serving multiple business domains.

Build the graph intelligence layer powering advanced analytics and ML.

Create systems that will process billions of entities with millisecond latencies.

Establish best practices for graph computing in our organization.

Mentor other engineers on distributed graph algorithms.

Compensation & Benefits

Competitive senior/staff-level compensation.

Flexible remote work arrangements.

Latest hardware and cloud resources for development.

LTIP - Long term Incentive plan.

75% of base as Bonus payment at the end of 4th year in service.

Equity potential of up to in excess of USD 150K every year.

Interview Process

Technical screen focusing on distributed systems and graph algorithms.

System design session on entity resolution at scale.

Coding session implementing a graph algorithm in Scala.

Architecture discussion with the team.

Final round with leadership.

To Apply

Please include:

Links to relevant open-source contributions.

Brief description of the largest graph system you've built (nodes/edges scale).

Your approach to a specific entity resolution challenge you've solved.

Any publications or talks on graph computing or entity resolution.