Senior ai/ml engineer

**About the Role**We're building a next-generation data intelligence platform using a microservices architecture, and we need a senior expert to lead the development of our Entity Resolution and Network Generation services. This is a hands-on technical leadership role where you'll architect and implement distributed graph computing solutions processing billions of entities and relationships.The PlatformOur cloud-native platform leverages:Microservices architecture with Kubernetes orchestrationApache Spark for distributed processingElasticsearch for real-time search and fuzzy matchingScala as the primary development languageData mesh principles with API-first designCore Responsibilities**Entity Resolution Service**Design and implement distributed entity resolution algorithms capable of processing billions of recordsBuild blocking strategies (e.g. LSH, canopy clustering) optimized for Spark at scaleDevelop fuzzy matching algorithms leveraging Elasticsearch's capabilitiesCreate ML-enhanced matching with explainable AI for match decisionsImplement incremental resolution supporting real-time and batch modesDesign APIs for entity lookup with sub-100ms latency requirements**Network Generation Service**Architect distributed graph generation pipelines using Graph X/Graph FramesImplement graph analytics algorithms (Page Rank, community detection, centrality measures)Design storage strategies for multi-billion edge graphs in Parquet/distributed file systemsBuild temporal graph support for time-evolving networksCreate high-performance graph serving APIs with complex query capabilitiesOptimize graph partitioning to minimize shuffle and maximize locality**AI Model Development**Build Graph Neural Networks (GNNs): Develop GNN models (e.g., Graph SAGE, GATv2) using Py Torch Geometric or DGL to analyze corporate and transaction networks, detecting fraud rings and risk patterns.Implement Entity Resolution: Design algorithms for fuzzy matching, semantic matching (Sentence-BERT), and clustering to unify entities across heterogeneous data sources (e.g., CSVs, APIs, PDFs).Create Risk Scoring Models: Combine rule-based, supervised (XGBoost), and unsupervised (Isolation Forest) methods to generate composite risk scores, optimized for real-time and large data processing in trillions.Advance Composite AI: Leverage Contex Q's proprietary approach, integrating symbolic AI, vector embeddings, and graph AI for robust entity resolution and network analytics.**Explainable AI (XAI)**Champion Transparency: Integrate SHAP, LIME, and GNNExplainer to provide clear, interpretable explanations for model predictions, meeting regulatory and ethical standards.Ensure Fairness: Audit models for bias and fairness, embedding ethical principles into every stage of development.**Cross-Service Responsibilities**Ensure seamless integration between entity resolution and network generationDesign data lineage tracking across both servicesImplement comprehensive monitoring and observabilityContribute to API design and service contractsOptimize for 10x scale growthRequired QualificationsTechnical Expertise7+ years of experience in distributed computing and big data systems5+ years specifically in entity resolution and graph analytics at scaleExpert-level Scala programming skillsDeep experience with Apache Spark, including custom optimizationsProduction experience with Elasticsearch for search and matchingProven track record building systems processing billions of entities/edges**Domain Knowledge**Strong understanding of blocking algorithms and their trade-offsExperience with probabilistic record linkage and similarity measuresExpertise in graph algorithms and their distributed implementationsKnowledge of graph storage formats and query optimizationUnderstanding of ML applications in entity resolutionBasic experience of Banking compliances - Fin Crime, Fraud**Systems Design**Experience designing microservices architecturesTrack record of building fault-tolerant, scalable systemsAPI design experience with Graph QL or RESTPerformance optimization and capacity planning expertise**Preferred Qualifications**Ph D in Computer Science or related field with focus on graphs/entity resolutionContributions to open-source projects (especially Spark, Graph X, Elasticsearch)Experience with graph databases (Neo4j, Neptune, Janus Graph) or equivalentPublications or conference talks on entity resolution or graph analyticsExperience with real-time stream processing (Kafka, Spark Streaming)Knowledge of graph neural networks and embedding techniques**Technical Environment**Languages: Scala (primary), Python, JavaBig Data: Apache Spark 3.x, Hadoop ecosystemSearch: Elasticsearch 8.xOrchestration: Kubernetes, DockerStorage: HDFS/S3/GCS, ParquetMonitoring: Prometheus, Grafana, JaegerCI/CD: Modern Dev Ops practicesWhat We're Looking ForSomeone who thinks in distributed systems and can optimize for both latency and throughputA technical leader who can make architectural decisions and implement themStrong communicator who can explain complex graph concepts to stakeholdersSelf-directed engineer who can own large technical initiatives end-to-endPerformance-obsessed developer who benchmarks everythingImpact You'll MakeDefine the architecture for entity resolution serving multiple business domainsBuild the graph intelligence layer powering advanced analytics and MLCreate systems that will process billions of entities with millisecond latenciesEstablish best practices for graph computing in our organizationMentor other engineers on distributed graph algorithms**Compensation & Benefits**Competitive senior/staff-level compensationFlexible remote work arrangementsLatest hardware and cloud resources for developmentLTIP - Long term Incentive plan.**75% of base as Bonus