Foundation Models: Pre-trained Models That Generalize Across Tasks and Datasets

A foundation model is a large neural network pre-trained on diverse data that generalizes across multiple tasks and datasets without task-specific training. In the graph domain, this means a model trained on many relational databases that can make predictions on entirely new databases it has never seen. The model has learned general patterns about how entities relate, how behavior evolves over time, and how graph structure predicts outcomes.

What makes a model a foundation model

Three properties distinguish foundation models from standard trained models:

1. Pre-trained on diverse data: Not one dataset but many. A graph foundation model trains on e-commerce databases, financial transaction logs, social networks, and healthcare records. This diversity is what enables generalization.
2. Self-supervised pre-training: The model learns without human labels. Masked token prediction, contrastive learning, or next-event prediction provide the training signal from the data itself. This enables training on massive unlabeled datasets.
3. Transfer to new tasks: The pre-trained model makes useful predictions on tasks and data it was not explicitly trained for. Zero-shot (no fine-tuning) or few-shot (minimal fine-tuning) performance exceeds models trained from scratch.

How graph foundation models work

The training pipeline has three stages:

Stage 1: Pre-training

The model processes diverse relational databases, each converted to a heterogeneous graph. Using masked token prediction, it learns to reconstruct hidden cell values from relational context. This teaches general patterns: customers with declining purchase frequency tend to churn, accounts with circular transaction patterns are suspicious, products bought together are in the same category.

Stage 2: Zero-shot inference

Given a new database and a prediction task (“Which customers will churn in the next 30 days?”), the foundation model:

1. Converts the database to a heterogeneous temporal graph
2. Encodes all entities using its pre-trained graph transformer
3. Applies a general prediction head to the target node representations

No training occurs. The model relies entirely on patterns learned during pre-training.

Stage 3: Fine-tuning (optional)

For maximum accuracy, the model is fine-tuned on labeled data from the target task. Because the encoder already produces rich representations, fine-tuning converges quickly (minutes, not hours) and requires little labeled data.

Why graphs are natural for foundation models

Relational databases share structural patterns that transfer well:

• Universal relationship types: Customer → order → product appears in e-commerce, retail, subscription, and marketplace databases. The relational pattern is the same.
• Common temporal dynamics: Engagement decay, seasonal patterns, and lifecycle stages appear across every customer-centric database.
• Structural invariants: High-degree nodes are hubs. Dense clusters indicate communities. Bipartite structure indicates user-item interactions. These patterns are universal.

Benchmark results

On the RelBench benchmark (7 databases, 30 prediction tasks, 103 million rows), foundation models demonstrate clear advantages:

• Flat-table LightGBM (task-specific, trained on target): 62.44 AUROC
• Task-specific GNN (trained from scratch on target): 75.83 AUROC
• KumoRFM zero-shot (no training on target): 76.71 AUROC
• KumoRFM fine-tuned (minutes of fine-tuning): 81.14 AUROC

The zero-shot foundation model outperforms both task-specific approaches that had full access to the training data. This demonstrates genuine transfer of relational patterns across databases.

Limitations and open questions

• Domain specificity: A foundation model trained on enterprise relational data may not transfer well to molecular graphs or social networks. Domain-specific pre-training still matters.
• Schema adaptation: Different databases have different schemas. The model needs a mechanism to handle arbitrary table structures and column types at inference time.
• Compute cost: Pre-training is expensive (days to weeks on GPU clusters). The cost is amortized across tasks but still substantial upfront.