Feedback
Any data, ground truth or otherwise, that helps assess model performance.
Feedback overview
Feedback is any data that can be gathered from the real world that provides information about model performance. This could be ground truth for supervised learning, or customer feedback for generative use-cases. Both cases can also take advantage of implicit feedback like session length.
Feedback lives in the row representing a prediction, as illustrated below:
Logging it to Gantry allows you to perform analysis on it in workspaces or in the SDK:

% positive feedback over time
Feedback can occur long after the model made its original prediction. Feedback can be delayed for up to 90 days, or up to a desired data retention interval, whichever is shorter. Gantry takes care of tying feedback to the prediction it corresponds to via the join_key
parameter.
Code samples for Custom
applications
Custom
applicationsWorking with join_key
Logging a prediction event with join_key
:
import gantry
gantry.init(api_key=GANTRY_API_KEY)
application = gantry.get_application(GANTRY_APP_NAME)
inputs = {
"prompt": "I read the news today oh boy",
}
outputs = {
"generation": "About a lucky man who made the grade"
}
application.log(
inputs=[inputs],
outputs=[outputs],
join_keys=[some_prediction_id],
)
Auto-populating join_key
:
import gantry
gantry.init(api_key=GANTRY_API_KEY)
application = gantry.get_application(GANTRY_APP_NAME)
inputs = {
"prompt": "I read the news today oh boy",
}
outputs = {
"generation": "About a lucky man who made the grade"
}
batch_id, join_key = application.log(
inputs=[inputs],
outputs=[outputs],
)
Building join_key
with intelligent values based on input:
import gantry
import uuid
gantry.init(api_key=GANTRY_API_KEY)
application = gantry.get_application(GANTRY_APP_NAME)
inputs = {
"college_degree": True,
"loan_amount": 1000.0,
"job_title": "Software Engineer",
"loan_id": uuid.uuid4(),
}
outputs = {
"generation": "About a lucky man who made the grade"
}
application.log(
inputs=[inputs],
outputs=[outputs],
join_keys=[f"loan_id_{inputs['loan_id']}"]
)
Gantry provides a helper method to create the join_key
based on a dictionary:
from gantry import JoinKey
inputs = {
"college_degree": True,
"loan_amount": 1000.0,
"job_title": "Software Engineer",
"loan_id": uuid.uuid4(),
}
outputs = {
"generation": "About a lucky man who made the grade"
}
application.log(
inputs=[inputs],
outputs=[outputs],
join_keys=[JoinKey.from_dict(inputs)]
)
Joining delayed feedback with the original prediction
Logging feedback with join_key
:
import gantry
gantry.init(api_key=GANTRY_API_KEY)
application = gantry.get_application(GANTRY_APP_NAME)
feedback = {
"correction_accepted": True,
}
application.log(
feedbacks=[feedback],
join_keys=[some_prediction_id],
)
Delayed Feedback Ingestion
Delayed feedback is processed in five minutes intervals. It might take up to five minutes plus the time it takes to process the data. This will likely be less than 15 total minutes, but might be longer depending on the volume of data ingested.
Code samples for Completion applications
Gantry offers support for logging user feedback on a per-message/per-OpenAI-API-call basis. Set the feedback
field when logging the OpenAI API call:
import gantry
from gantry.applications.llm_utils import fill_prompt
import openai
from openai.util import convert_to_dict
gantry.init()
my_llm_app = gantry.get_application(GANTRY_APP_NAME)
version = my_llm_app.get_version("test")
config = version.config
prompt = config['prompt']
def generate(values):
filled_in_prompt = fill_prompt(prompt, values)
request = {
"model": "text-davinci-002",
"prompt": filled_in_prompt,
}
results = openai.Completion.create(**request)
my_llm_app.log_completion(
open_ai_api_request=request,
open_ai_api_response=convert_to_dict(results),
request_attributes={"prompt_values": values},
feedback={"thumbs_up": True},
version=version.version_number,
)
return results
Joining delayed feedback with the original completion
To log feedback asynchronously, keep track of the id
field returned in the OpenAI API response. This will be used as your join_key
:
gantry.init()
application = gantry.get_application(GANTRY_APP_NAME)
application.log(
feedbacks=[feedback],
join_keys=[id_returned_by_openai],
)
Updated 2 months ago