Feedback overview

Feedback is any data that can be gathered from the real world that provides information about model performance. This could be ground truth for supervised learning, or customer feedback for generative use-cases. Both cases can also take advantage of implicit feedback like session length.

Feedback lives in the row representing a prediction, as illustrated below:

Logging it to Gantry allows you to perform analysis on it in workspaces or in the SDK:

Feedback can occur long after the model made its original prediction. Feedback can be delayed for up to 90 days, or up to a desired data retention interval, whichever is shorter. Gantry takes care of tying feedback to the prediction it corresponds to via the join_key parameter.

Code samples for Custom applications

Working with join_key

Logging a prediction event with join_key:

import gantry
gantry.init(api_key=GANTRY_API_KEY)
application = gantry.get_application(GANTRY_APP_NAME)

inputs = {
  "prompt": "I read the news today oh boy",
}

outputs = {
  "generation": "About a lucky man who made the grade"
}

application.log(
  inputs=[inputs],
  outputs=[outputs],
  join_keys=[some_prediction_id],
)

Auto-populating join_key:

import gantry
gantry.init(api_key=GANTRY_API_KEY)
application = gantry.get_application(GANTRY_APP_NAME)

inputs = {
  "prompt": "I read the news today oh boy",
}

outputs = {
  "generation": "About a lucky man who made the grade"
}


batch_id, join_key = application.log(
  inputs=[inputs],
  outputs=[outputs],
)

Building join_key with intelligent values based on input:

import gantry
import uuid
gantry.init(api_key=GANTRY_API_KEY)
application = gantry.get_application(GANTRY_APP_NAME)

inputs = {
  "college_degree": True,
  "loan_amount": 1000.0,
  "job_title": "Software Engineer",
  "loan_id": uuid.uuid4(),
}

outputs = {
  "generation": "About a lucky man who made the grade"
}

application.log(
  inputs=[inputs],
  outputs=[outputs],
  join_keys=[f"loan_id_{inputs['loan_id']}"]
)

Gantry provides a helper method to create the join_key based on a dictionary:

from gantry import JoinKey

inputs = {
  "college_degree": True,
  "loan_amount": 1000.0,
  "job_title": "Software Engineer",
  "loan_id": uuid.uuid4(),
}

outputs = {
  "generation": "About a lucky man who made the grade"
}

application.log(
  inputs=[inputs],
  outputs=[outputs],
  join_keys=[JoinKey.from_dict(inputs)]
)

Joining delayed feedback with the original prediction

Logging feedback with join_key:

import gantry
gantry.init(api_key=GANTRY_API_KEY)
application = gantry.get_application(GANTRY_APP_NAME)

feedback = {
  "correction_accepted": True,
}

application.log(
  feedbacks=[feedback],
  join_keys=[some_prediction_id],
)

📘

Delayed Feedback Ingestion

Delayed feedback is processed in five minutes intervals. It might take up to five minutes plus the time it takes to process the data. This will likely be less than 15 total minutes, but might be longer depending on the volume of data ingested.

Code samples for Completion applications

Gantry offers support for logging user feedback on a per-message/per-OpenAI-API-call basis. Set the feedback field when logging the OpenAI API call:

import gantry
from gantry.applications.llm_utils import fill_prompt
import openai
from openai.util import convert_to_dict

gantry.init()
my_llm_app = gantry.get_application(GANTRY_APP_NAME)

version = my_llm_app.get_version("test")
config = version.config
prompt = config['prompt']

def generate(values):
    filled_in_prompt = fill_prompt(prompt, values)
    request = {
        "model": "text-davinci-002",
        "prompt": filled_in_prompt,
    }
    results = openai.Completion.create(**request)

    my_llm_app.log_completion(
        open_ai_api_request=request,
        open_ai_api_response=convert_to_dict(results),
        request_attributes={"prompt_values": values},
      	feedback={"thumbs_up": True},
        version=version.version_number,
    )

    return results

Joining delayed feedback with the original completion

To log feedback asynchronously, keep track of the id field returned in the OpenAI API response. This will be used as your join_key:

gantry.init()
application = gantry.get_application(GANTRY_APP_NAME)

application.log(
  feedbacks=[feedback],
  join_keys=[id_returned_by_openai],
)