Frequently Asked Questions#

What’s the impact to my service when I instrument my model with Gantry?#

Minimal!

The broad strokes are that we log your data to a holding place and perform computation later, off the critical path of your application code.

The specifics of how this is implemented depend on your particular deployment. For information about our deployment that’s designed for scale, see our docs on Deploy Gantry on AWS.

Oops! I sent Gantry an integer when I should’ve sent a boolean#

It’s ok if you send the wrong data type to Gantry, and it’s easy to recover!

Gantry will auto-infer the data type of features, outputs, and feedback from the first piece of data into the system. So, if the first piece of data sent for output loan_approved is a integer, when you meant to send a boolean, Gantry will think that loan_approved should always be an integer.

Every time the version is bumped, Gantry will re-infer all data types. So, right now, if you send the wrong data type (or if you just want to experiment with a new data type) then the solution is to bump the version when you send the next piece of data.

First, go to your dashboard to find the latest version. Then, manually invoke gantry.log_prediction_event() or gantry.log_feedback_event() with your new data types, and a new version. The version can be either a number or a string - whichever helps you keep track of your versions. Gantry will take care of mapping version to our internal version number on the backend.

gantry.log_prediction_event(
   "loan_pred",
   inputs={ ... },
   outputs=...,
   version=new_version_number_or_string,
    ...
)

What metrics do you support?#

See Metrics for a detailed list of performance metrics that we compute.

We also compute a handful of distance and distribution metrics for drift detection:

  • Kullback-Leibler divergence

  • D1 Distance

  • \(D_{\infty}\) Distance

  • Kolmogorov-Smirnov statistic, along with its p-value

  • Chi-square, along with its p-value

Can I get my data back out of Gantry?#

Yes! You can use the Gantry Python SDK to programmatically access your data, without going through a web UI.

See more on our Using the Gantry SDK docs page.

What is feedback?#

A prediction is the inputs to and outputs of your model. Feedback is a label, a ground truth value, immediate user feedback, or any other value that you use to determine “how good was my prediction”?

Some feedback values, like the item a user clicks on in a search, come almost instantanously after a prediction is made. Others, like whether or not a user purchased an item or defaulted on a loan, are delayed by days, or even months. Luckily, Gantry supports both of these cases!

See more at Logging feedback.