Testing¶

Note

Also see the numpydantic.testing API docs and the Writing an Interface guide

Numpydantic exposes a system for combinatoric testing across dtypes, shapes, and interfaces in the numpydantic.testing module.

These helper classes and functions are included in the distributed package so they can be used for downstream development of independent interfaces (though we always welcome contributions!)

Validation Cases¶

Each test case is parameterized by a ValidationCase.

The case is intended to be able to be partially filled in so that multiple validation cases can be merged together, but also used independently by falling back on default values.

There are three major parts to a validation case:

Annotation specification: annotation_dtype and annotation_shape specifies how the NDArray ValidationCase.annotation that is used to test against is generated
Array specification: dtype and shape specify that array that will be generated to test against the annotation
Interface specification: An InterfaceCase that refers to an Interface, and provides array generation and other auxilary logic.

Typically, one specifies a dtype along with an annotation dtype or a shape along with an annotation shape (or implicitly against the defaults for either), along with a value for passes that indicates if that combination is valid.

from numpydantic.testing import ValidationCase

dtype_case = ValidationCase(
    id="int_int", 
    dtype=int, 
    annotation_dtype=int, 
    passes=True
)
shape_case = ValidationCase(
    id="cool_shape", 
    shape=(1,2,3), 
    annotation_shape=(1,"*","2-4"), 
    passes=True
)

merged = dtype_case.merge(shape_case)
console.print(merged.model_dump(exclude={'annotation', 'model'}, exclude_unset=True))

{
    'id': 'int_int-cool_shape',
    'annotation_shape': (1, '*', '2-4'),
    'annotation_dtype': <class 'int'>,
    'shape': (1, 2, 3),
    'dtype': <class 'int'>,
    'passes': True,
    'marks': set()
}

When merging validation cases, the merged case only passes if all the original cases do.

from numpydantic.testing import ValidationCase

dtype_case = ValidationCase(
    id="int_int", 
    dtype=int, 
    annotation_dtype=int, 
    passes=True
)
shape_case = ValidationCase(
    id="uncool_shape", 
    shape=(1,2,3), 
    annotation_shape=(9,8,7), 
    passes=False
)

merged = dtype_case.merge(shape_case)
console.print(merged.model_dump(exclude={'annotation', 'model'}, exclude_unset=True))

{
    'id': 'int_int-uncool_shape',
    'annotation_shape': (9, 8, 7),
    'annotation_dtype': <class 'int'>,
    'shape': (1, 2, 3),
    'dtype': <class 'int'>,
    'passes': False,
    'marks': set()
}

We provide a convenience function merged_product() for creating a merged product of multiple sets of test cases.

For example, you may want to create a set of dtype and shape cases and validate against all combinations

from numpydantic.testing.helpers import merged_product

dtype_cases = [
    ValidationCase(dtype=int, annotation_dtype=int, passes=True),
    ValidationCase(dtype=int, annotation_dtype=float, passes=False)
]
shape_cases = [
    ValidationCase(shape=(1,2,3), annotation_shape=(1,2,3), passes=True),
    ValidationCase(shape=(4,5,6), annotation_shape=(1,2,3), passes=False)
]

iterator = merged_product(dtype_cases, shape_cases)

console.print([i.model_dump(exclude_unset=True, exclude={'model', 'annotation'}) for i in iterator])

[
    {
        'id': '',
        'annotation_shape': (1, 2, 3),
        'annotation_dtype': <class 'int'>,
        'shape': (1, 2, 3),
        'dtype': <class 'int'>,
        'passes': True,
        'marks': set()
    },
    {
        'id': '',
        'annotation_shape': (1, 2, 3),
        'annotation_dtype': <class 'int'>,
        'shape': (4, 5, 6),
        'dtype': <class 'int'>,
        'passes': False,
        'marks': set()
    },
    {
        'id': '',
        'annotation_shape': (1, 2, 3),
        'annotation_dtype': <class 'float'>,
        'shape': (1, 2, 3),
        'dtype': <class 'int'>,
        'passes': False,
        'marks': set()
    },
    {
        'id': '',
        'annotation_shape': (1, 2, 3),
        'annotation_dtype': <class 'float'>,
        'shape': (4, 5, 6),
        'dtype': <class 'int'>,
        'passes': False,
        'marks': set()
    }
]

You can pass constraints to the merged_product() iterator to filter cases that match some value, for example to get only the cases that pass:

iterator = merged_product(dtype_cases, shape_cases, conditions={"passes": True})
console.print([i.model_dump(exclude_unset=True, exclude={'model', 'annotation'}) for i in iterator])

[
    {
        'id': '',
        'annotation_shape': (1, 2, 3),
        'annotation_dtype': <class 'int'>,
        'shape': (1, 2, 3),
        'dtype': <class 'int'>,
        'passes': True,
        'marks': set()
    }
]

Interface Cases¶

Validation cases can be paired with interface cases that handle generating arrays for the given interface from the specification in the validation case.

Since some array interfaces like Zarr have multiple possible forms of an array (in memory, on disk, in a zip file, etc.) an interface may have multiple cases that are important to test against.

The InterfaceCase.make_array() method does what you’d expect it to, creating an array, and returning the appropriate input type for the interface:

from numpydantic.testing.interfaces import NumpyCase, ZarrNestedCase

NumpyCase.make_array(shape=(1,2,3), dtype=float)

array([[[0., 0., 0.],
        [0., 0., 0.]]])

ZarrNestedCase.make_array(shape=(1,2,3), dtype=float, path=Path("__tmp__/zarr_dir"))

ZarrArrayPath(file='__tmp__/zarr_dir/nested.zarr', path='a/b/c')

Interface cases also define when an interface should skip a given test parameterization. For example, some array formats can’t support arbitrary object serialization, and the video class can only support 8-bit arrays of a specific shape

from numpydantic.testing.interfaces import VideoCase

VideoCase.skip(shape=(1,1), dtype=float)

True

This, and the array generation methods are propagated up into a ValidationCase that contains them

case = ValidationCase(shape=(1,2,3), dtype=float, interface=VideoCase)
case.skip()

True

The merged_product() iterator automatically excludes any combinations of interfaces and test parameterizations that should be skipped.

Making Fixtures¶

Pytest fixtures are a useful way to re-use validation case products. To keep things tidy, you may want to use marks and ids when creating them so that you can run tests against specific interfaces or conditions with the pytest -m mark system.

import pytest

@pytest.fixture(
    params=(
        pytest.param(
            p, 
            id=p.id, 
            marks=getattr(pytest.mark, p.interface.interface.name)
        )
        for p in iterator
    )
)
def my_cases(request):
    return request.param