Reasons to love Python: Validating Dataclasses (Part 1)
I love Python. Sure it’s not the fastest language in the world, and the package management can sometimes be painful, but I still love it.
I often reach for Python when developing the scripts that wrap systems together. Read a configuration file, process that data and do something with it.
I find myself reaching for a Pydantic BaseModel but I don’t want to create a virtualenv just for the script (uv script mode nonwithstanding), and so I just make it a dataclass and spend the next 20 minutes fighting type errors in my data.
What if there was a better way? What if I could re-implement just enough of a BaseModel in little enough code that I can just paste it into the script?
Python is marvelous language. It is full of features that feel like they must be intrinsics of the compiler, but if you peek under the hood, the machinery is there, in all it’s sometimes hacky glory, ready to be inspected.
Dataclasses are one of them. Add a few annotations, and hey-presto! The magic is performed and your bare class now has an __init__, a __str__, even __eq__ if you ask for it!
Unpacking YAML into a dataclass
Let’s start with a simple example: we want to define a model, and populate it with data from a .yaml file.
from __future__ import annotations
from dataclasses import dataclass
from typing import TextIO
import yaml
@dataclass(frozen=True)
class Person:
name: str
age: int
@classmethod
def load(cls, file: TextIO) -> Person:
return cls(**yaml.safe_load(file))
if __name__ == "__main__":
with open("person.yaml") as f:
person = Person.load(f)
print(person)
So far so good. The script opens the data file, loads it into a dict and uses ** to splat it into Person.__init__, the constructor generated by the dataclass decorator.
$ python load-dataclass.py
Person(name='Thibaud', age=35)
Un fortunately, this only checks that the data file contains the right fields, not that the fields have the correct types. If we add an extra field, Person.__init__ receives an extra argument and we get an exception:
>>> Person(name="Thibaud")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Person.__init__() missing 1 required positional argument: 'age'
And if we are missing a field:
>>> Person(name="Thibaud", age=35, nationality="FR")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Person.__init__() got an unexpected keyword argument 'nationality'
And if we specify all the arguments, but one of them is the wrong type we do not get an error.
>>> Person(name=12, age="None of your business!")
Person(name=12, age='None of your business!')
Validating Scalar Fields
To ensure that the value that we assign to each field has the correct type, we can introspect the dataclass to read out the type hint for each field, and then check that the value has correct type (or can be casted to the correct type).
Python gives us a very convenient get_type_hints function for this that takes care of the nitty gritty details, hands us a dictionary of names, and the associated annotations.
from __future__ import annotations
from dataclasses import fields, is_dataclass
from typing import get_type_hints, Any, Callable, Type, cast
def _coerce[T](
field_type: Type[T], value: Any, field_name: str
) -> T:
# If the value has the correct type, move on
if isinstance(value, field_type):
return value
# Otherwise, try to cast it to the correct type
try:
# Tell the checker the class accepts the raw value.
return cast(Callable[[Any], T], field_type)(value)
except (ValueError, TypeError):
err = (
f"Invalid value {value!r} for '{field_name}'. "
f"Expected '{field_type.__name__}'."
)
raise ValueError(err) from None
def hydrate_dataclass[T](cls: type[T], data: dict[str, Any]) -> T:
assert is_dataclass(cls)
result = data.copy()
hints = get_type_hints(cls)
for f in fields(cls):
# Skip fields that are not present in the source dictionary.
# When constructing the dataclass, these will become missing
# arguments.
if f.name not in result:
continue
result[f.name] = _coerce(
hints[f.name], result[f.name], f.name
)
# Look for fields that are not defined in the dataclass.
# Allows us to make a better error message than the TypeError.
unknown = set(data) - {f.name for f in fields(cls)}
if unknown:
raise ValueError(f"Unknown fields: {sorted(unknown)}")
# Build the dataclass from the dictionary and return it
return cls(**result)
If we run this script with a data file that has all correct types, the dataclass is built as before. If the data file has the wrong types though, we get nice error messages:
>>> hydrate_dataclass(Person, {"name": "Thibaud", "age": "Not saying!"})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "hydrate_dataclass.py", line 29, in hydrate_dataclass
raise ValueError(err) from None
ValueError: Invalid value 'Not saying!' for 'age'. Expected 'int'.
Converting Enums on the Fly
With what we have so far, we can validate scalar types, but what about enumerations? Can we magically convert values from our input files into the correct enumeration members?
class TeamRole(StrEnum):
Principal = "Team Principal"
Driver = "Driver"
PitCrew = "PitCrew"
@dataclass(frozen=True)
class Person:
role: TeamRole
name: str
In python, enumerations are classes derived from the Enum class. A simple is_subclass(x, Enum) check is enough to detect them.
Once we find an enumeration, we look up it’s members, and return the correct enumerator.
def _coerce[T](
field_type: Type[T], value: Any, field_name: str
) -> T:
# If the value has the correct type, move on
if isinstance(value, field_type):
return value
if isinstance(field_type, type) and issubclass(field_type, Enum):
# The enumerator in the data should be a string,
# and it should be an enumerator on the Enum.
if (
isinstance(value, str)
and value in field_type.__members__
):
return field_type[value]
# Enumerator not found. Raise an error.
err = (
f"Invalid value {value!r} for '{field_name}'. "
f"Expected one of : {', '.join(field_type.__members__)}"
)
raise ValueError(err) from None
# Otherwise, try to cast it to the correct type
try:
# Tell the checker the class accepts the raw value.
return cast(Callable[[Any], T], field_type)(value)
except (ValueError, TypeError):
err = (
f"Invalid value {value!r} for '{field_name}'. "
f"Expected '{field_type.__name__}'."
)
raise ValueError(err) from None
That’s all it took, one is_subclass check and we’re spitting out enumerations!
>>> hydrate_dataclass(Person, {"name": "Thibaud", "role": "PitCrew"})
Person(role=<TeamRole.PitCrew: 'PitCrew'>, name='Thibaud')
If we try to pass in a value that is not a valid enumerator for the enumeration, we get a descriptive error.
>>> hydrate_dataclass(Person, {"name": "Marcus", "role": "Boss"})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "hydrate_dataclass.py", line 37, in hydrate_dataclass
raise ValueError(err) from None
ValueError: Invalid 'Boss' for 'role'. Expected one of: Principal, Driver, PitCrew
Loading Nested Dataclasses
Validating one dataclass is a nice start, but sometimes our data is nested, and better represented as a series of nested dataclasses.
@dataclass(frozen=True)
class Person:
name: str
@dataclass(frozen=True)
class Volunteer:
identity: Person
email: str
What happens if we try to load this with our hydrate_dataclass function?
>>> hydrate_dataclass(Volunteer, {"identity": {"name": "Thibaud"}})
Volunteer(
identity=Person(name={'name': 'Thibaud'}),
email='hello@example.com'
)
That looks good at first sight. But wait, why do we have name={‘name’: ‘Thibaud’} instead of just name=’Thibaud’?
Our code is smart enough to detect enumerations, and special case them, but it doesn’t (yet) know anything about dataclasses. Since isinstance(value, field_type) returns false, the value is converted by
result[f.name] = field_type(value)
Which is functionally equivalent to running
result[f.name] = Person(name={'name': 'Thibaud'})
When what we need to do is
result[f.name] = Person(**{'name': 'Thibaud'})
It’s an easy fix, we just need to special case dataclasses. Once again, Python gives us exactly what we need: is_dataclass.
def _coerce[T](
field_type: Type[T], value: Any, field_name: str
) -> T: # If the value has the correct type, move on
if isinstance(value, field_type):
return value
if isinstance(field_type, type) and issubclass(field_type, Enum):
# The enumerator in the data should be a string,
# and it should be an enumerator on the Enum.
if (
isinstance(value, str)
and value in field_type.__members__
):
return field_type[value]
# Enumerator not found. Raise an error.
err = (
f"Invalid value {value!r} for '{field_name}'. "
f"Expected one of : {', '.join(field_type.__members__)}"
)
raise ValueError(err) from None
if is_dataclass(field_type):
if not isinstance(value, dict):
err = f"Invalid {value!r} for '{field_name}'. Expected dict."
raise ValueError(err)
return hydrate_dataclass(field_type, value)
# Otherwise, try to cast it to the correct type
try:
# Tell the checker the class accepts the raw value.
return cast(Callable[[Any], T], field_type)(value)
except (ValueError, TypeError):
err = (
f"Invalid value {value!r} for '{field_name}'. "
f"Expected '{field_type.__name__}'."
)
raise ValueError(err) from None
And with that, we can load nested dataclasses:
>>> hydrate_dataclass(Volunteer, {"identity": {"name": "Thibaud"}})
Volunteer(
identity=Person(name='Thibaud'),
email='hello@example.com'
)
The Full Script
In about 50 lines of code, we can validate a fairly useful set of dataclasses. We can’t handle lists, dictionaries, optional values, and unions yet, but they are surprisingly easy to implement.
That will be for a future post.
In the meantime, you can find the full script that we have worked through in this post here: