> ## Documentation Index
> Fetch the complete documentation index at: https://kumo.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Online Serving

> End-to-end guide: train, export, and deploy your Kumo model for real-time inference

Kumo's online serving runs predictions at request time — suitable for live recommendations, real-time fraud scoring, and any use case where you need a score in milliseconds rather than a scheduled job. Unlike [batch prediction](/fine-tuning/training), which scores your entire dataset at once, online serving keeps a live endpoint running that you query with a single entity at a time.

<Info>
  The deployment steps below mirror the [interactive SDK example notebook](https://github.com/kumo-ai/kumo-serving/blob/master/interactive_sdk/example_notebook.ipynb), which you can download and run end to end.
</Info>

## How it works

Online serving in Kumo combines two models:

* A **base model** trained on your full graph that produces rich entity embeddings capturing long-term patterns.
* A **distilled model** that runs at request time, combining those stored embeddings with the latest signals to produce a fast, accurate prediction.

The end-to-end flow is:

1. Train a base model on your graph.
2. Train a distilled model using the base model's embeddings.
3. Generate embeddings from the base model and export the serving bundle to S3.
4. Deploy the bundle to a live inference endpoint.
5. Query the endpoint from your application.

Steps 1–3 use the fine-tuning SDK. Steps 4–5 use the `kumoai.online` deployment SDK.

## Train the base model

Train your base model as usual — this is the same workflow covered in [Training & Predictions](/fine-tuning/training). Save the job ID; you'll need it in the next step.

```python theme={null}
import kumoai as kumo

train_table = pq.generate_training_table()
model_plan  = pq.suggest_model_plan()

trainer    = kumo.Trainer(model_plan)
result     = trainer.fit(graph, train_table, non_blocking=False)
base_job_id = result.job_id
```

## Train the distilled model

The distilled model is a smaller, faster model trained to predict at request time. It takes the base model's embeddings as inputs — pass `base_model_id` to link the two.

```python theme={null}
from kumoai.trainer import DistillationTrainer

train_table_serving = pq_serving.generate_training_table()

distilled_plan   = pq_serving.suggest_distilled_model_plan(base_model_id=base_job_id)
distiller        = DistillationTrainer(distilled_plan, base_job_id)
dist_result      = distiller.fit(graph, train_table_serving, non_blocking=False)
distilled_job_id = dist_result.job_id
```

`pq_serving` is a `PredictiveQuery` on the same graph as `pq`, targeting the entity you want to score at request time — for example, a transaction or a recommendation candidate.

## Generate embeddings and export

Run batch prediction on the base model to produce entity embeddings, then export both the distilled model and the embeddings to S3 as a ready-to-deploy bundle.

```python theme={null}
from kumoai.artifact_export.config import OutputConfig
from kumoai.trainer import ModelOutputConfig, export_model

base_trainer = kumo.Trainer.load(base_job_id)
pred_table   = pq.generate_prediction_table(non_blocking=True)

bp_result = base_trainer.predict(
    graph=graph,
    prediction_table=pred_table,
    output_config=OutputConfig(
        output_types={"predictions", "embeddings"},
        output_connector=connector,
        output_table_name="embeddings_for_export",
    ),
    training_job_id=base_job_id,
    non_blocking=False,
)

export_config = ModelOutputConfig(
    training_job_id=distilled_job_id,
    batch_prediction_job_id=bp_result.job_id,
    output_path="s3://your-bucket/path/to/serving-bundle/",
)
export_model(export_config, non_blocking=False)
```

<Info>
  Export targets S3 URIs (`s3://…`). Contact your Kumo team if you need to export to a different storage provider.
</Info>

## Connect to the deployment control plane

With your bundle in S3, switch to the `kumoai.online` SDK to deploy and manage the live service.

Install the SDK:

```bash theme={null}
pip install kumoai
```

Your Kumo team will provide these values when they provision your tenant:

| Variable        | What it is                 |
| --------------- | -------------------------- |
| `BASE_URL`      | Your control-plane API URL |
| `CLIENT_ID`     | Cognito app client ID      |
| `CLIENT_SECRET` | Cognito app client secret  |
| `TOKEN_URL`     | Cognito token endpoint     |

```python theme={null}
import os
import kumoai.online as kumo_online

BASE_URL      = os.environ.get("BASE_URL",      "<your-control-plane-url>")
CLIENT_ID     = os.environ.get("CLIENT_ID",     "<your-cognito-client-id>")
CLIENT_SECRET = os.environ.get("CLIENT_SECRET", "<your-cognito-client-secret>")
TOKEN_URL     = os.environ.get("TOKEN_URL",     "<your-cognito-token-url>")
SCOPE         = os.environ.get("SCOPE",         "https://inference.kumo.ai/invoke")

client = kumo_online.init(
    url=BASE_URL,
    client_id=CLIENT_ID,
    client_secret=CLIENT_SECRET,
    token_url=TOKEN_URL,
    scope=SCOPE,
)

print("healthy:", client.health())
```

## Register your model

Point the SDK at the S3 bundle from Step 3. The last path segment becomes the model name.

```python theme={null}
MODEL_ID     = "my-model-v1"
S3_MODEL_URI = "s3://your-bucket/path/to/serving-bundle/two_stage_gnn/"

client.register_model(model_id=MODEL_ID, s3_model_uri=S3_MODEL_URI)
```

## Deploy an inference service

Choose a GPU instance type and create the service:

```python theme={null}
import time

instance_types = client.list_instance_types()
INSTANCE_TYPE  = "g6.4xlarge"
SERVICE_NAME   = "my-service"

svc = client.create_inference_service(
    name=SERVICE_NAME,
    model_id=MODEL_ID,
    instance_type=INSTANCE_TYPE,
)
```

`g6.4xlarge` is a good starting point for most models. The service takes 1–5 minutes to start — poll until it's ready:

```python theme={null}
def wait_until_ready(name: str, timeout: int = 600, poll: int = 10) -> None:
    deadline = time.time() + timeout
    while time.time() < deadline:
        statuses = client.get_status()
        s = statuses.get(name)
        ready = bool(s and s.ready)
        print(f"{name}: ready={ready}  {s.message if s else ''}")
        if ready:
            return
        time.sleep(poll)
    raise TimeoutError(f"{name} did not become ready within {timeout}s")

wait_until_ready(SERVICE_NAME)
```

## Run inference

Send a request to your live endpoint. Input names and shapes come from your exported model's `config.pbtxt`.

```python theme={null}
inputs = [
    {
        "name": "anchor_time",
        "datatype": "INT64",
        "shape": [1, 1],
        "data": [1352371793912000000],
    },
]

response = svc.infer(inputs=inputs)
print(response)
```

<Tip>
  You can also load inputs from a JSON file:

  ```python theme={null}
  import json
  from pathlib import Path

  body   = json.loads(Path("request.json").read_text())
  inputs = body["inputs"] if isinstance(body, dict) else body
  print(svc.infer(inputs=inputs))
  ```
</Tip>

## Clean up

Delete the service and registered model when you're done to avoid ongoing costs:

```python theme={null}
svc.delete()
client.delete_registered_model(MODEL_ID)
```

## Next Steps

The [example notebook](https://github.com/kumo-ai/kumo-serving/blob/master/interactive_sdk/example_notebook.ipynb) covers two additional features once you're comfortable with the basics.

**Autoscaling** — automatically scale replicas based on CPU usage:

```python theme={null}
from kumoai.online import ScaleMetric

svc.create_autoscaling(
    min_capacity=1,
    max_capacity=5,
    scale_metric=ScaleMetric.cpu,
    scale_target=70,
)
```

**Canary rollouts** — gradually shift traffic to a new model version while the existing one keeps serving the rest:

```python theme={null}
client.register_model(model_id="my-model-v2", s3_model_uri=S3_MODEL_URI_V2)
svc.start_canary(canary_model_id="my-model-v2", canary_traffic_percent=10)
```

Once you're satisfied, call `svc.promote()` to make the canary the new stable version. If something goes wrong, call `svc.rollback()` to route all traffic back to the original.

## See also

* [Introduction](/fine-tuning/introduction) — set up connectors, tables, graphs, and predictive queries.
* [Training & Predictions](/fine-tuning/training) — train models and generate batch predictions.