Graph explainability answers the question: which parts of the input graph caused this prediction? When a GNN classifies an account as fraudulent, the explanation is not a list of feature importances. It is a subgraph: the specific neighbor nodes, the specific edges, and the specific features that drove the score. This subgraph explanation is what an analyst needs to investigate the flag, what a regulator needs for compliance, and what a model developer needs for debugging.
What makes graph explainability different
Tabular explainability (SHAP, LIME) answers: “which input features mattered?” Graph explainability must answer three questions simultaneously:
- Which features? What attributes of the target node mattered (account age, balance)?
- Which edges? Which connections influenced the prediction (shared device with flagged account)?
- Which subgraph structure? Which neighborhood patterns drove the score (dense cluster of interconnected accounts)?
The explanation is a subgraph with highlighted edges and features, not a flat list of feature importances. This is both more informative and more challenging to compute and present.
GNNExplainer
GNNExplainer (Ying et al., 2019) is the foundational method. For a specific prediction (node v classified as fraudulent), it learns two soft masks:
- Edge mask: a weight [0, 1] for each edge in node v's computation graph. Higher weight = more important for this prediction.
- Feature mask: a weight [0, 1] for each input feature dimension. Higher weight = more important.
The masks are optimized to maximize mutual information between the masked subgraph and the original prediction. The result: the minimal set of edges and features that, if kept, produce the same prediction.
Attention-based explanations
Graph Attention Networks (GAT) compute learned attention weights for each edge. These weights are naturally interpretable: a high attention weight on edge (j → i) means neighbor j's message was weighted heavily when computing node i's embedding.
Advantages of attention-based explanations:
- Free: no additional computation beyond the forward pass
- Per-layer: attention weights at each layer show how information flows through the graph
- Intuitive: “the model paid attention to these neighbors”
Limitations:
- Attention is not always faithful: high attention does not always mean high causal influence on the prediction
- Attention only explains which neighbors were weighted, not which features were important
- Only available for attention-based architectures (GAT, TransformerConv)
Other explanation methods
- PGExplainer: learns a global explanation model (not per-instance like GNNExplainer). Faster at inference: one forward pass generates explanations for any node.
- SubgraphX: uses Monte Carlo tree search to find the most important connected subgraph. Produces explanations that are connected subgraphs (more interpretable than disconnected edge sets).
- GraphMask: learns which edges can be removed without changing the prediction. Edges that cannot be removed are the explanation.
- Gradient-based: compute gradients of the prediction with respect to node features and edge weights. Higher gradient = more important. Fast but noisy.
Enterprise requirements
In regulated industries, explainability is not optional:
- Financial services: regulators require explanations for fraud flags, credit decisions, and AML alerts. “The model said so” is not acceptable.
- Healthcare: clinical decision support must explain why a patient is flagged as high-risk.
- Insurance: claim denial must be justified with specific factors.
Graph explanations are especially valuable because they identify not just what features matter but which relationships matter: “this claim is flagged because the claimant shares a phone number with 3 other claimants who filed similar claims within 2 weeks.”