Search Recommendations

​

Solution Background and Business Value

Traditional search engines in e-commerce rely on ranking functions like Okapi BM25 or large language models (LLMs) to measure the semantic similarity between queries and product descriptions. While effective, these methods only consider textual relevance and do not leverage historical user behavior to optimize for purchases.

A data-driven approach can enhance search rankings by integrating historical business data. This ensures that the most relevant and high-converting products appear at the top, benefiting both the user experience and business revenue. Personalized search considers factors such as:

• User preferences and past interactions to tailor results.

• Conversion likelihood, ranking items higher if they are more likely to be purchased.

• Business objectives, ensuring high-value items or trending products gain visibility.

​

Data Requirements and Schema

To build an optimized search recommendation model, we need structured historical data. Kumo AI enables us to extend the model by incorporating additional signals over time.

Core Tables

1. Queries Table

   • Stores user search queries.

   • Key attributes:

     • query_id: Unique identifier for the search query.

     • query: Text of the search.

     • Other optional attributes: query category, external LLM embeddings.

2. Items Table

   • Stores product information.

   • Key attributes:

     • item_id: Unique identifier for each item.

     • start_timestamp / end_timestamp: Availability window.

     • Other optional attributes: description, color, category, LLM embeddings, and vision embeddings.

3. Users Table

   • Stores customer information.

   • Key attributes:

     • user_id: Unique identifier for each user.

     • Other optional features: age, location, join_timestamp.

4. Add to Cart Table

   • Captures search interactions leading to purchases.

   • Key attributes:

     • timestamp: When the event happened.

     • query_id, item_id, user_id: Links interactions to queries, items, and users.

     • Other optional attributes: purchase likelihood, engagement time.

Entity Relationship Diagram (ERD)

​

Predictive Queries

To recommend the most relevant products for a given search query while also personalizing results, we need two models:

Model 1: Query-Item Relevance

• Purpose: Identifies the most relevant items for a search query.

• Training Data: Search queries and add-to-cart interactions.

• Predictive Query:

PREDICT
    LIST_DISTINCT(
        add_to_cart.item_id, 0, N, days
    ) RANK TOP X
FOR EACH
    queries.query_id

Model 2: User-Item Personalization

• Purpose: Re-ranks the query results based on user preferences.

• Training Data: User-item interactions from add-to-cart events.

• Predictive Query:

PREDICT
    LIST_DISTINCT(
        add_to_cart.item_id, 0, N, days
    ) RANK TOP X
FOR EACH
    users.user_id

​

Deployment Strategy

1. Batch Processing for Query Recommendations:

   • Use Model 1 to generate top product recommendations for each query.

   • Refresh these predictions daily to reflect product availability and buying trends.

2. User and Item Embeddings:

   • Use Model 2 to generate user and item embeddings.

   • Refresh embeddings daily to maintain up-to-date personalization.

3. Real-Time Query Handling:

   • When a user submits a search query, retrieve the precomputed recommendations.

   • Re-rank results using dot product similarity between user and item embeddings.

   • Use ElasticSearch or similar tools for efficient real-time retrieval and ranking.

Handling Anonymous Users

• If a user is not logged in, use the query recommendations without personalization.

• The ranking from Model 1 provides a strong default order.

Integrating with Existing Search Engines

• If using Okapi BM25 or another retrieval algorithm, apply Model 2 to re-rank results.

​

Building models in Kumo SDK

1. Initialize the Kumo SDK

import kumoai as kumo

kumo.init(url="https://<customer_id>.kumoai.cloud/api", api_key=API_KEY)

2. Connect data

connector = kumo.S3Connector("s3://your-dataset-location/")

3. Select tables

queries = kumo.Table.from_source_table(
    source_table=connector.table('queries'),
    primary_key='query_id',
).infer_metadata()

items = kumo.Table.from_source_table(
    source_table=connector.table('items'),
    primary_key='item_id',
).infer_metadata()

users = kumo.Table.from_source_table(
    source_table=connector.table('users'),
    primary_key='user_id',
).infer_metadata()

add_to_cart = kumo.Table.from_source_table(
    source_table=connector.table('add_to_cart'),
    time_column='timestamp',
).infer_metadata()

4. Create graph schema

graph = kumo.Graph(
    tables={
        'queries': queries,
        'items': items,
        'users': users,
        'add_to_cart': add_to_cart,
    },
    edges=[
        dict(src_table='add_to_cart', fkey='query_id', dst_table='queries'),
        dict(src_table='add_to_cart', fkey='item_id', dst_table='items'),
        dict(src_table='add_to_cart', fkey='user_id', dst_table='users'),
    ],
)

graph.validate(verbose=True)

5. Train the model

pquery_query_item = kumo.PredictiveQuery(
    graph=graph,
    query="PREDICT LIST_DISTINCT(add_to_cart.item_id, 0, X, days) RANK TOP 50 FOR EACH queries.query_id"
)
pquery_user_item = kumo.PredictiveQuery(
    graph=graph,
    query="PREDICT LIST_DISTINCT(add_to_cart.item_id, 0, X, days) RANK TOP 50 FOR EACH users.user_id"
)

pquery_query_item.validate(verbose=True)
model_plan = pquery_query_item.suggest_model_plan()
trainer = kumo.Trainer(model_plan)
training_job = trainer.fit(
    graph=graph,
    train_table=pquery.generate_training_table(non_blocking=True),
    non_blocking=False,
)
print(f"Training metrics: {training_job.metrics()}")