Feedback

Any data, ground truth or otherwise, that helps assess model performance.

Suggest Edits

Feedback overview

Feedback is any data that can be gathered from the real world that provides information about model performance. This could be ground truth for supervised learning, or customer feedback for generative use-cases. Both cases can also take advantage of implicit feedback like session length.

Feedback lives in the row representing a prediction, as illustrated below:

778

📘

The following applies to Gantry SDK 0.5 and newer

Users of Gantry SDK prior to 0.5:
The Gantry SDK prior to 0.5 uses a different system for joining feedback to its corresponding inputs, outputs, and predictions. Please contact Gantry support if you need help migrating to this system.

You can determine which version of the Gantry SDK you are using with the command pip show gantry.

Feedback can occur long after the model made its original prediction. Feedback can be delayed for up to six months, or up to a desired data retention interval, whichever is shorter. Gantry takes care of tying feedback to the prediction it corresponds to via the join_key parameter.

Code samples

Logging a prediction event with join_key:

gantry.log_record(
  "my-awesome-app",
  inputs=inputs,
  outputs=outputs,
  join_key=some_prediction_id,
)

Logging feedback with join_key:

gantry.log_record(
  "my-awesome-app",
  feedback=feedback,
  join_key=some_prediction_id,
)

Auto-populating join_key:

join_key = gantry.log_record(
  "my-awesome-app",
  inputs=inputs,
  outputs=outputs,
)

Building join_key with intelligent values based on input:

inputs = {
  "college_degree": True,
  "loan_amount": 1000.0,
  "job_title": "Software Engineer",
  "loan_id": uuid.uuid4(),
}

gantry.log_record(
  "my-awesome-app",
  inputs=inputs,
  outputs=outputs,
  join_key=f"loan_id_{inputs['loan_id']}"
)

Gantry provides a helper method to create the join_key based on a dictionary:

from gantry import JoinKey

inputs = {
  "college_degree": True,
  "loan_amount": 1000.0,
  "job_title": "Software Engineer",
  "loan_id": uuid.uuid4(),
}

gantry.log_record(
  "my-awesome-app",
  inputs=inputs,
  outputs=outputs,
  join_key=JoinKey.from_dict(inputs)
)

📘

Delayed Feedback Ingestion

Delayed feedback is processed in five minutes intervals. It might take up to five minutes plus the time it takes to process the data. This will likely be less than 15 total minutes, but might be longer depending on the volume of data ingested.

Example in Flask

from flask import Flask, jsonify, request
import gantry

from loan_utils import preprocess_features

GANTRY_API_KEY = "YOUR_API_KEY"  # see Getting Your API Key
GANTRY_APPLICATION_NAME = "my-awesome-app"  # name your application
GANTRY_APPLICATION_VERSION = "1.0"  # name your version

app = Flask(__name__)

gantry.init(
    api_key=GANTRY_API_KEY,
    environment="production",
)


# Every time this function is called, log inputs & predictions under "loan_pred"
def _predict(
    loan_id: str,  # id of loan, important metadata but not used in predicting
    college_degree: bool,  # if applicant has college degree
    loan_amount: float,  # in $1,000s
):
    preprocessed_features = preprocess_features(
        loan_id, college_degree, loan_amount, X_mean=X_train_mean, X_std=X_train_std
    )
    prediction = bool(model.predict(preprocessed_features.reshape(1, -1))[0])
    inputs = {"college_degree": college_degree, "loan_amount": loan_amount}
    outputs = {"loan_repaid_pred": prediction}
    gantry.log_record(
        GANTRY_APPLICATION_NAME,
        version=GANTRY_APPLICATION_VERSION,
        inputs=inputs,
        outputs=outputs,
        join_key=loan_id,
    )
    return prediction


@app.route("/loans/<loan_id>/predict", methods=["POST"])
def predict(loan_id):
    """Call this endpoint to make a prediction on whether a given loan
    will be repaid.
    """
    data = request.get_json(force=True)
    return jsonify({"result": _predict(loan_id, **data)})
  
@app.route("/loans/<loan_id>/repay", methods=["POST"])
def repay_status(loan_id):
    """Call this endpoint to update the actual loan repay status"""
    data = request.get_json(force=True)
    
    gantry.log_record(
      GANTRY_APPLICATION_NAME,
      feedback={"loan_repaid": data["repaid"]},
      join_key=loan_id,
    )
    return {"response": "ok"}

Updated 18 days ago