TODO¶
v2¶
We will be moving away from using nptyping in v2.0.0.
It was written for an older era in python before the dramatic changes in the Python type system and is no longer actively maintained. We will be reimplementing a syntax that extends its array specification syntax to include things like ranges and extensible dtypes with varying precision (and is much less finnicky to deal with).
Type Checker Integration¶
The .pyi stubfile generation (numpydantic.meta) works for
keeping type checkers from complaining about various array formats
not literally being NDArray objects, but it doesn’t do the kind of
validation we would want to be able to use NDArray objects as full-fledged
python types, including validation propagation through scopes and
IDE type checking for invalid literals.
We want to hook into the type checking process to satisfy these type checkers:
mypy - has hooks, can be done with an extension
pyright - unclear if has hooks, might nee to monkeypatch
pycharm - unlikely this is possible, extensions need to be in Java and installed separately
Validation¶
Todo
Support pydantic value/range constraints - less than, greater than, etc.
Todo
Support different precision modes - eg. exact precision, or minimum precision where specifying a float32 would also accept a float64, etc.
Metadata¶
Todo
Use names in nptyping annotations in generated JSON schema metadata
All TODOs¶
Todo
Fix shape/writing for grayscale videos
Todo
jonny write the code of conduct
Todo
Jonny write the interface contrib docs
Todo
Jonny write the contribution docs
Todo
To better support static type hinting and inspection (ie. so the type checker
is not only aware of the union of all return_types, but the specific array
type that was passed on model instantiation, as well as potentially
do shape and dtype checks during type checking (eg. so a wrongly shaped or dtyped
array assignment will be highlighted as wrong), we will be exploring adding
mypy/pylance/pyright hooks for dynamic type evaluation.
Todo
Document dev environment
Really it’s very simple, you just clone a fork and install
the dev environment like pip install '.[dev]'
Todo
Document pull requests if we ever receive one
Todo
This section is under construction as of 1.6.1
Much of the details of dtypes are covered in syntax
and in numpydantic.dtype , but this section will specifically
address how dtypes are handled both generically and by interfaces
as we expand custom dtype handling <3.
For details of support and implementation until the docs have time for some love, please see the tests, which are the source of truth for the functionality of the library for now and forever.
Todo
Document limitations for complex numbers and strategies for serialization/validation
Todo
Datetimes are supported by every interface except :class:.VideoInterface ,
with the caveat that HDF5 loses timezone information, and thus all timestamps should
be re-encoded to UTC before saving/loading.
More generic datetime support is TODO.
Todo
Generic objects are supported by all interfaces except
:class:.VideoInterface , :class;.HDF5Interface , and :class:.ZarrInterface .
this might be expected, but there is also hope, TODO fill in serialization plans.
Todo
Strings are supported by all interfaces except :class:.VideoInterface .
TODO is fill in the subtleties of how this works
Todo
For now these are handled as tuples of dtypes, see the source of numpydantic.dtype.Float . They should either be handled as Unions or as a more prescribed meta-type.
For now, use int and float to refer to the general concepts of
“any int” or “any float” even if this is a bit mismatched from the numpy usage.
Todo
Compound dtypes are currently unsupported,
though the HDF5 interface supports indexing into compound dtypes
as separable dimensions/arrays using the third “field” parameter in
hdf5.H5ArrayPath .
Todo
Sorry this is unreadable, need to recall how to change the theme for generated mermaid diagrams but it is very late and i want to push this.
Todo
Currently, the version of the package the interface is from (usually numpydantic)
will be stored, but there is no means of resolving it on the fly.
If there is a mismatch between the marked interface description and the interface
that was matched on revalidation, a warning is emitted, but validation
attempts to proceed as normal.
This feature is for extra-verbose provenance, rather than airtight serialization and deserialization, but PRs welcome if you would like to make it be that way.
Todo
We will also add a separate mark_version parameter for marking
the specific version of the relevant interface package, like zarr, or numpy,
patience.
Todo
Future versions will support interfaces providing type maps for declaring equality between dtypes that may be specific to that library but should be considered equivalent to numpy or other library’s dtypes.
Todo
Future versions will also support declaring minimum or maximum precisions, so one might say “at least a 16-bit float” and also accept a 32-bit float.
Todo
numpydantic currently does not support structured dtypes or numpy.recarray
specifications like nptyping does. It will in future versions.
Todo
numpydantic also does not support the variable shape definition form like
Shape['Dim, Dim']
where there are two dimensions of any shape as long as they are equal
because at the moment it appears impossible to express dynamic constraints
(ie. minItems/maxItems that depend on the shape of another array)
in JSON Schema. A future minor version will allow them by generating a JSON
schema with a warning that the equal shape constraint will not be represented.
See: https://github.com/orgs/json-schema-org/discussions/730
Todo
Support pydantic value/range constraints - less than, greater than, etc.
Todo
Support different precision modes - eg. exact precision, or minimum precision where specifying a float32 would also accept a float64, etc.
Todo
Use names in nptyping annotations in generated JSON schema metadata