Feedback
Any data, ground truth or otherwise, that helps assess model performance.
Feedback overview
Feedback is any data that can be gathered from the real world that provides information about model performance. This could be ground truth for supervised learning, or customer feedback for generative use-cases. Both cases can also take advantage of implicit feedback like session length.
Feedback lives in the row representing a prediction, as illustrated below:

The following applies to Gantry SDK 0.5 and newer
Users of Gantry SDK prior to 0.5:
The Gantry SDK prior to 0.5 uses a different system for joining feedback to its corresponding inputs, outputs, and predictions. Please contact Gantry support if you need help migrating to this system.You can determine which version of the Gantry SDK you are using with the command
pip show gantry
.
Feedback can occur long after the model made its original prediction. Feedback can be delayed for up to six months, or up to a desired data retention interval, whichever is shorter. Gantry takes care of tying feedback to the prediction it corresponds to via the join_key
parameter.
Code samples
Logging a prediction event with join_key
:
gantry.log_record(
"my-awesome-app",
inputs=inputs,
outputs=outputs,
join_key=some_prediction_id,
)
Logging feedback with join_key
:
gantry.log_record(
"my-awesome-app",
feedback=feedback,
join_key=some_prediction_id,
)
Auto-populating join_key
:
join_key = gantry.log_record(
"my-awesome-app",
inputs=inputs,
outputs=outputs,
)
Building join_key
with intelligent values based on input:
inputs = {
"college_degree": True,
"loan_amount": 1000.0,
"job_title": "Software Engineer",
"loan_id": uuid.uuid4(),
}
gantry.log_record(
"my-awesome-app",
inputs=inputs,
outputs=outputs,
join_key=f"loan_id_{inputs['loan_id']}"
)
Gantry provides a helper method to create the join_key
based on a dictionary:
from gantry import JoinKey
inputs = {
"college_degree": True,
"loan_amount": 1000.0,
"job_title": "Software Engineer",
"loan_id": uuid.uuid4(),
}
gantry.log_record(
"my-awesome-app",
inputs=inputs,
outputs=outputs,
join_key=JoinKey.from_dict(inputs)
)
Delayed Feedback Ingestion
Delayed feedback is processed in five minutes intervals. It might take up to five minutes plus the time it takes to process the data. This will likely be less than 15 total minutes, but might be longer depending on the volume of data ingested.
Example in Flask
from flask import Flask, jsonify, request
import gantry
from loan_utils import preprocess_features
GANTRY_API_KEY = "YOUR_API_KEY" # see Getting Your API Key
GANTRY_APPLICATION_NAME = "my-awesome-app" # name your application
GANTRY_APPLICATION_VERSION = "1.0" # name your version
app = Flask(__name__)
gantry.init(
api_key=GANTRY_API_KEY,
environment="production",
)
# Every time this function is called, log inputs & predictions under "loan_pred"
def _predict(
loan_id: str, # id of loan, important metadata but not used in predicting
college_degree: bool, # if applicant has college degree
loan_amount: float, # in $1,000s
):
preprocessed_features = preprocess_features(
loan_id, college_degree, loan_amount, X_mean=X_train_mean, X_std=X_train_std
)
prediction = bool(model.predict(preprocessed_features.reshape(1, -1))[0])
inputs = {"college_degree": college_degree, "loan_amount": loan_amount}
outputs = {"loan_repaid_pred": prediction}
gantry.log_record(
GANTRY_APPLICATION_NAME,
version=GANTRY_APPLICATION_VERSION,
inputs=inputs,
outputs=outputs,
join_key=loan_id,
)
return prediction
@app.route("/loans/<loan_id>/predict", methods=["POST"])
def predict(loan_id):
"""Call this endpoint to make a prediction on whether a given loan
will be repaid.
"""
data = request.get_json(force=True)
return jsonify({"result": _predict(loan_id, **data)})
@app.route("/loans/<loan_id>/repay", methods=["POST"])
def repay_status(loan_id):
"""Call this endpoint to update the actual loan repay status"""
data = request.get_json(force=True)
gantry.log_record(
GANTRY_APPLICATION_NAME,
feedback={"loan_repaid": data["repaid"]},
join_key=loan_id,
)
return {"response": "ok"}
Updated 18 days ago