Working with Images: Distracted Driving

This tutorial walks through integrating a model that performs predictions on images with Gantry.

By the time you complete this tutorial, you will understand how to:

• Send images to Gantry
• Log feedback separately from logging predictions
• Build, deploy, and debug a custom projection

If you're looking for a more detailed tutorial on navigating through the Gantry dashboard and exploring Gantry integrations with Huggingface, check out the quickstart instead.

Setting the scene

Focus is a ML system that analyzes drivers' behavior through dash cam images and detects whether the drivers are distracted. We have deployed a model that takes an image of a driver and returns to us either NORMAL_DRIVING or the type of distraction in which the driver is engaged. We'd like to know how well this model is performing and under which situations it might underperform.

Before we jump in, let's set up our imports. This tutorial assumes you have a Gantry API key and have downloaded the SDK.

You'll also need to clone the sample code:

git clone https://github.com/gantry-ml/gantry-demos.git \
    && cd gantry-demos/distracted-driving

We recommend creating a virtual environment to make sure this tutorial does not interfere with your native or default Python environment:

python -m venv venv
source venv/bin/activate

Now, install the Python dependencies:

pip install -r requirements.txt

Lastly, copy the .env.sample file into .env and replace the API key placeholder with your own.

When you're working with your own personal private bucket, you'll have to register it with Gantry. Since this tutorial uses a public bucket, no registering is required.

Logging data and predictions

Our tutorial begins in a jupyter notebook where we will explore some data and log it to Gantry. Launch the notebook and navigate to tutorial.ipynb.

jupyter lab

We can monitor our data ingestion into Gantry via the jobs tab on the lefthand side of the dashboard. When all the data is ingested, we can click on the explore tab to view it.

Build a custom projection

It'll be easier to understand overall performance if we have a boolean indicator regarding whether driver is distracted. The many outputs of distracted types make it hard to understand the data in aggregate. Let's set up a custom projection called is_distracted. This is done via a combination of a python function and a YAML file. These are already defined for us in the custom_projections folder in the demo directory. Let's open those files and take a look.

def is_distracted(label):
    return label != "NORMAL_DRIVING"

version: 1
function_definition:
  projection_name: is_distracted
  entrypoint: custom_projections.py
  function_name: is_distracted
  output:
    type: Boolean
  inputs:
    - type: Text

With the function and configuration file defined, we can provision the projection to Gantry and use it like any other projection. Provisioning is done via the CLI. In the distracted-driving directory, run the following command:

GANTRY_API_KEY=<your-api-key> gantry-cli projection create --projection-dir custom_projection

Now, lets head back over to the UI to set up our custom projection.

If we go to add projection, we can see that all the fields are disabled. This tells us there might be wrong with our projection input type.

We can debug this by going to the schema tab. Here, we can see that outputs.labels is of type Categorical.

In our YAML file, we defined it as Text. That's the issue. Let's change it and re-provision using the same command from earlier:

version: 1
function_definition:
  projection_name: is_distracted
  entrypoint: custom_projections.py
  function_name: is_distracted
  output:
    type: Boolean
  inputs:
    - type: Categorical

GANTRY_API_KEY=<your-api-key> gantry-cli projection create --projection-dir custom_projection

Now if we go to add the projection, we can do that successfully. Make sure to click the 'Backfill all projections` button so that Gantry applies this projection to the data we've already ingested.

Let's explore this new projection on the explore tab in the UI. To debug error cases, let's look at the data when filtering ground_truth to false, but is_distracted output from the model to true.

If we look at the light level, we can see that all these error cases are on low light images. This indicates that we potentially need to retrain the model on darker images.

What's next?

Gantry makes it easy to extract just low light images via Curators and Datasets.
For a tutorial walking through these two concepts, check out our quickstart.