The first step to getting value from Gantry is to start gathering data about your model's behavior. Gantry works by accumulating records corresponding to predictions consisting of inputs and outputs. At any subsequent time you can send "feedback," either ground truth any other data, that helps the quality of a prediction. These predictions can also be enriched with tags and projections (derived values) to provide a clearer picture of model behavior.
We will dive into all of those terms in detail in the sections that follow, but the mental model you should for a Gantry record is illustrated below:
In this example we have a text generation model with a single input and output. Let's review each of the columns along with the role it plays:
|A unique and stable identifier for this prediction, can be used to apply feedback at any point in time.|
|The name of this application within Gantry. Roughly speaking, each application has a corresponding "infinite dataframe" consisting of rows like this.|
|An example of how to use our tagging system to indicate that this record was captured in production.|
|Another example of how we can enrich a prediction with data that might not be an input, but helps us understand how the model impacts users.|
|The prompt provided by the user, the model input.|
|The output produced by the text generation.|
|This is an example of feedback that is not "ground truth" but merely the opinion of the user, and well suited to assessing how well our users are receiving the model's predictions.|
|An example of using projections to "project" a higher dimensional input, raw text, into a scalar that can help us understand the model's behavior more systematically.|
Next we will look at each the different types of columns in more detail, starting with predictions.
Updated 7 days ago