Schema and Data Types

Data types in Gantry encode the way data should be interpreted, regardless of the format it is in when it is logged.

How are data types used in Gantry?

The data type for each field determines how that field can be used in Gantry. For example:

• How that data is displayed in the data table
• What alerts, metrics, and projections can be computed on the field

Available data types

The following types are available in Gantry:

Array types

Gantry has built in support for one-dimensional Array types, which are logged as Python lists of native Python types: str, float, int, bool. Arrays are inherently high-dimensional, like Text, so Gantry often uses Projections to better understand their content. For example, length is a property that can be monitored for all arrays using the vector.length Projection. For numerical arrays, vector.<min, max. norm> Projections can be computed to in order to observe other scalar scalar properties such as statistics (mean), extrema (min, max), and norms (L1, L2). Filtering on Arrays is also possible using the Array contains filter, which will return any record whose Array field contains an exact or partial match for at least one element. For example, an “Array contains” filter for the substring "is" would return any Record with an Array<str> field with any element containing "is", such as ["this is", "an array", "of strings"]. This can be useful for filtering multi-label lists or tag attributes that might be part of an application.

Schema inference

Gantry infers the schema for an application based on the first batch of data that logged. As more data is logged, new fields that appear are automatically added to the schema. To change the type of a field, the schema needs to be edited.

Supported types and inferencing logic

The following table details supported types, and the logic used to infer that type. Note that for primitive & array primitive types, Gantry relies on type checking logic built into pyarrow

Mixed columns

If a column contains mixed primitives or mixed array primitives, then the type of the column will be set to Unknown. For more complex string types, such as Image or Audio, the column will only be inferred as that type if the majority of the values are that type. Otherwise, the column will be set to Unknown.

Nested schema

Gantry supports nested schema inference. . is reserved to represent a nested field. Any . characters in the field name will be converted to _.

Here is an example of data with nested fields:

{
  "integer": {
    "feature.1": 1,
    "feature.2": 2,
    "feature_3": 3
  },
  "categorical": {
    "feature_1": "category.1",
    "feature.2": "category.2"
  },
  "text.feature.1": "Hello, World"
}

Gantry infers the above schema as follows:

Handling bad data

Gantry does data cleaning to handle bad data for primitives. Future support is planned for cleaning more complex types.

Editing the schema

🚧

When the type of a feature is changed, all the old data will be deleted and new data will be logged with the new type. There is planned support for type changes between compatible types without data loss in the future.

Sometimes the schema will be inferred incorrectly, or the data type of a field will change. To edit a schema manually:

1. Use the sidebar to navigate to the ingestion page:

2. Click the three dots on the right side of the page to change the data type:

Example: data types and schemas

gantry.log_record(
    application="my_app",
    inputs={
        "numerical_feature_1": 1.1,
        "numerical_feature_2": 87,
        "categorical_feature_1": "cat_A",
        "text_feature_1": "cat_A is one of our categories",
        "image_feature_1": "s3://bucket/image.jpg",
        "audio_feature_1": "https://storage.googleapis.com/example-presigned-audio.mp3",
    },
    ...
)

In this example, there are a few ambiguities in how the data will be handled:

• Both of the inputs numerical_feature_* are numbers, but they each have different Python data types.
• The inputs categorical_feature_1 and text_feature_1 are both represented as strings in Python, but the former is meant to be interpreted as a category and the latter as text.
• All of text_feature_1, image_feature_1 , audio_feature_1 are strings, but the first one is just text, whereas the latter two should be specially treated as Image and Audio.

Data types are stored in an application’s Schema. The Schema maps fields to Gantry data types. The following is an example schema for the example application above: