- Notifications
You must be signed in to change notification settings - Fork216
CONTRIBUTIONS ONLY: Voluptuous, despite the name, is a Python data validation library.
License
alecthomas/voluptuous
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
What does this mean? I do not have time to fix issues myself. The only way fixes or new features will be added is by people submitting PRs.
Current status: Voluptuous is largely feature stable. There hasn't been a need to add new features in a while, but there are some bugs that should be fixed.
Why? I no longer use Voluptuous personally (in fact I no longer regularly write Python code). Rather than leave the project in a limbo of people filing issues and wondering why they're not being worked on, I believe this notice will more clearly set expectations.
Voluptuous,despite the name, is a Python data validation library. Itis primarily intended for validating data coming into Python as JSON,YAML, etc.
It has three goals:
- Simplicity.
- Support for complex data structures.
- Provide useful error messages.
Voluptuous now has a mailing list! Send a mail tovoluptuous@librelist.com to subscribe. Instructionswill follow.
You can also contact me directly viaemail orTwitter.
To file a bug, create anew issue on GitHub with a short example of how to replicate the issue.
The documentation is providedhere.
Documentation is built usingSphinx. You can install it by
pip install -r requirements.txtFor buildingsphinx-apidoc from scratch you need to set PYTHONPATH tovoluptuous/voluptuous repository.
The documentation is providedhere.
SeeCHANGELOG.md.
Validators are simple callables:No need to subclass anything, just use a function.
Errors are simple exceptions:A validator can justraise Invalid(msg) and expect the user to getuseful messages.
Schemas are basic Python data structures:Should your data be a dictionary of integer keys to strings?{int: str} does what you expect. List of integers, floats orstrings?[int, float, str].
Designed from the ground up for validating more than just forms:Nested data structures are treated in the same way as any othertype. Need a list of dictionaries?[{}]
Consistency:Types in the schema are checked as types. Values are compared asvalues. Callables are called to validate. Simple.
Twitter'suser search API acceptsquery URLs like:
$ curl'https://api.twitter.com/1.1/users/search.json?q=python&per_page=20&page=1'To validate this we might use a schema like:
>>>from voluptuousimport Schema>>> schema= Schema({...'q':str,...'per_page':int,...'page':int,... })
This schema very succinctly and roughly describes the data required bythe API, and will work fine. But it has a few problems. Firstly, itdoesn't fully express the constraints of the API. According to the API,per_page should be restricted to at most 20, defaulting to 5, forexample. To describe the semantics of the API more accurately, ourschema will need to be more thoroughly defined:
>>>from voluptuousimport Required, All, Length, Range>>> schema= Schema({... Required('q'): All(str, Length(min=1)),... Required('per_page',default=5): All(int, Range(min=1,max=20)),...'page': All(int, Range(min=0)),... })
This schema fully enforces the interface defined in Twitter'sdocumentation, and goes a little further for completeness.
"q" is required:
>>>from voluptuousimport MultipleInvalid, Invalid>>>try:... schema({})...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="required key not provided @ data['q']"True
...must be a string:
>>>try:... schema({'q':123})...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="expected str for dictionary value @ data['q']"True
...and must be at least one character in length:
>>>try:... schema({'q':''})...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="length of value must be at least 1 for dictionary value @ data['q']"True>>> schema({'q':'#topic'})== {'q':'#topic','per_page':5}True
"per_page" is a positive integer no greater than 20:
>>>try:... schema({'q':'#topic','per_page':900})...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="value must be at most 20 for dictionary value @ data['per_page']"True>>>try:... schema({'q':'#topic','per_page':-10})...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="value must be at least 1 for dictionary value @ data['per_page']"True
"page" is an integer >= 0:
>>>try:... schema({'q':'#topic','per_page':'one'})...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)"expected int for dictionary value @ data['per_page']">>> schema({'q':'#topic','page':1})== {'q':'#topic','page':1,'per_page':5}True
Schemas are nested data structures consisting of dictionaries, lists,scalars andvalidators. Each node in the input schema is patternmatched against corresponding nodes in the input data.
Literals in the schema are matched using normal equality checks:
>>> schema= Schema(1)>>> schema(1)1>>> schema= Schema('a string')>>> schema('a string')'a string'
Types in the schema are matched by checking if the corresponding valueis an instance of the type:
>>> schema= Schema(int)>>> schema(1)1>>>try:... schema('one')...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="expected int"True
URLs in the schema are matched by usingurlparse library.
>>>from voluptuousimport Url>>> schema= Schema(Url())>>> schema('http://w3.org')'http://w3.org'>>>try:... schema('one')...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="expected a URL"True
Lists in the schema are treated as a set of valid values. Each elementin the schema list is compared to each value in the input data:
>>> schema= Schema([1,'a','string'])>>> schema([1])[1]>>> schema([1,1,1])[1, 1, 1]>>> schema(['a',1,'string',1,'string'])['a', 1, 'string', 1, 'string']
However, an empty list ([]) is treated as is. If you want to specify a list that cancontain anything, specify it aslist:
>>> schema= Schema([])>>>try:... schema([1])...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="not a valid value @ data[1]"True>>> schema([])[]>>> schema= Schema(list)>>> schema([])[]>>> schema([1,2])[1, 2]
Sets and frozensets are treated as a set of valid values. Each elementin the schema set is compared to each value in the input data:
>>> schema= Schema({42})>>> schema({42})== {42}True>>>try:... schema({43})...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="invalid value in set"True>>> schema= Schema({int})>>> schema({1,2,3})== {1,2,3}True>>> schema= Schema({int,str})>>> schema({1,2,'abc'})== {1,2,'abc'}True>>> schema= Schema(frozenset([int]))>>>try:... schema({3})...raiseAssertionError('Invalid not raised')...except Invalidas e:... exc= e>>>str(exc)=='expected a frozenset'True
However, an empty set (set()) is treated as is. If you want to specify a setthat can contain anything, specify it asset:
>>> schema= Schema(set())>>>try:... schema({1})...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="invalid value in set"True>>> schema(set())==set()True>>> schema= Schema(set)>>> schema({1,2})== {1,2}True
Validators are simple callables that raise anInvalid exception whenthey encounter invalid data. The criteria for determining validity isentirely up to the implementation; it may check that a value is a validusername withpwd.getpwnam(), it may check that a value is of aspecific type, and so on.
The simplest kind of validator is a Python function that raisesValueError when its argument is invalid. Conveniently, many builtinPython functions have this property. Here's an example of a datevalidator:
>>>from datetimeimport datetime>>>defDate(fmt='%Y-%m-%d'):...returnlambdav: datetime.strptime(v, fmt)
>>> schema= Schema(Date())>>> schema('2013-03-03')datetime.datetime(2013, 3, 3, 0, 0)>>>try:... schema('2013-03')...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="not a valid value"True
In addition to simply determining if a value is valid, validators maymutate the value into a valid form. An example of this is theCoerce(type) function, which returns a function that coerces itsargument to the given type:
defCoerce(type,msg=None):"""Coerce a value to a type. If the type constructor throws a ValueError, the value will be marked as Invalid. """deff(v):try:returntype(v)exceptValueError:raiseInvalid(msgor ('expected %s'%type.__name__))returnf
This example also shows a common idiom where an optional human-readablemessage can be provided. This can vastly improve the usefulness of theresulting error messages.
Each key-value pair in a schema dictionary is validated against eachkey-value pair in the corresponding data dictionary:
>>> schema= Schema({1:'one',2:'two'})>>> schema({1:'one'}){1: 'one'}
By default any additional keys in the data, not in the schema willtrigger exceptions:
>>> schema= Schema({2:3})>>>try:... schema({1:2,2:3})...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="extra keys not allowed @ data[1]"True
This behaviour can be altered on a per-schema basis. To allowadditional keys useSchema(..., extra=ALLOW_EXTRA):
>>>from voluptuousimportALLOW_EXTRA>>> schema= Schema({2:3},extra=ALLOW_EXTRA)>>> schema({1:2,2:3}){1: 2, 2: 3}
To remove additional keys useSchema(..., extra=REMOVE_EXTRA):
>>>from voluptuousimportREMOVE_EXTRA>>> schema= Schema({2:3},extra=REMOVE_EXTRA)>>> schema({1:2,2:3}){2: 3}
It can also be overridden per-dictionary by using the catch-all markertokenextra as a key:
>>>from voluptuousimport Extra>>> schema= Schema({1: {Extra:object}})>>> schema({1: {'foo':'bar'}}){1: {'foo': 'bar'}}
By default, keys in the schema are not required to be in the data:
>>> schema= Schema({1:2,3:4})>>> schema({3:4}){3: 4}
Similarly to how extra_ keys work, this behaviour can be overriddenper-schema:
>>> schema= Schema({1:2,3:4},required=True)>>>try:... schema({3:4})...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="required key not provided @ data[1]"True
And per-key, with the marker tokenRequired(key):
>>> schema= Schema({Required(1):2,3:4})>>>try:... schema({3:4})...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="required key not provided @ data[1]"True>>> schema({1:2}){1: 2}
If a schema hasrequired=True, keys may be individually marked asoptional using the marker tokenOptional(key):
>>>from voluptuousimport Optional>>> schema= Schema({1:2, Optional(3):4},required=True)>>>try:... schema({})...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="required key not provided @ data[1]"True>>> schema({1:2}){1: 2}>>>try:... schema({1:2,4:5})...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="extra keys not allowed @ data[4]"True
>>> schema({1:2,3:4}){1: 2, 3: 4}You can usevoluptuous.Self to define a nested schema:
>>>from voluptuousimport Schema, Self>>> recursive= Schema({"more": Self,"value":int})>>> recursive({"more": {"value":42},"value":41})== {'more': {'value':42},'value':41}True
Often it comes handy to have a baseSchema that is extended with morerequirements. In that case you can useSchema.extend to create a newSchema:
>>>from voluptuousimport Schema>>> person= Schema({'name':str})>>> person_with_age= person.extend({'age':int})>>>sorted(list(person_with_age.schema.keys()))['age', 'name']
The originalSchema remains unchanged.
Each key-value pair in a schema dictionary is validated against eachattribute-value pair in the corresponding object:
>>>from voluptuousimport Object>>>classStructure(object):...def__init__(self,q=None):...self.q= q...def__repr__(self):...return'<Structure(q={0.q!r})>'.format(self)...>>> schema= Schema(Object({'q':'one'},cls=Structure))>>> schema(Structure(q='one'))<Structure(q='one')>
To allow value to be None as well, use Any:
>>>from voluptuousimport Any>>> schema= Schema(Any(None,int))>>> schema(None)>>> schema(5)5
Validators must throw anInvalid exception if invalid data is passedto them. All other exceptions are treated as errors in the validator andwill not be caught.
EachInvalid exception has an associatedpath attribute representingthe path in the data structure to our currently validating value, as wellas anerror_message attribute that contains the message of the originalexception. This is especially useful when you want to catchInvalidexceptions and give some feedback to the user, for instance in the context ofan HTTP API.
>>>defvalidate_email(email):..."""Validate email."""...ifnot"@"in email:...raise Invalid("This email is invalid.")...return email>>> schema= Schema({"email": validate_email})>>> exc=None>>>try:... schema({"email":"whatever"})...except MultipleInvalidas e:... exc= e>>>str(exc)"This email is invalid. for dictionary value @ data['email']">>> exc.path['email']>>> exc.msg'This email is invalid.'>>> exc.error_message'This email is invalid.'
Thepath attribute is used during error reporting, but also during matchingto determine whether an error should be reported to the user or if the nextmatch should be attempted. This is determined by comparing the depth of thepath where the check is, to the depth of the path where the error occurred. Ifthe error is more than one level deeper, it is reported.
The upshot of this is thatmatching is depth-first and fail-fast.
To illustrate this, here is an example schema:
>>> schema= Schema([[2,3],6])
Each value in the top-level list is matched depth-first in-order. Giveninput data of[[6]], the inner list will match the first element ofthe schema, but the literal6 will not match any of the elements ofthat list. This error will be reported back to the user immediately. Nobacktracking is attempted:
>>>try:... schema([[6]])...raiseAssertionError('MultipleInvalid not raised')...except MultipleInvalidas e:... exc= e>>>str(exc)=="not a valid value @ data[0][0]"True
If we pass the data[6], the6 is not a list type and so will notrecurse into the first element of the schema. Matching will continue onto the second element in the schema, and succeed:
>>> schema([6])[6]Validation rules that involve multiple fields can be implemented ascustom validators. It's recommended to useAll() to do a two-passvalidation - the first pass checking the basic structure of the data,and only after that, the second pass applying your cross-fieldvalidator:
defpasswords_must_match(passwords):ifpasswords['password']!=passwords['password_again']:raiseInvalid('passwords must match')returnpasswordsschema=Schema(All(# First "pass" for field types {'password':str,'password_again':str},# Follow up the first "pass" with your multi-field rulespasswords_must_match))# validschema({'password':'123','password_again':'123'})# raises MultipleInvalid: passwords must matchschema({'password':'123','password_again':'and now for something completely different'})
With this structure, your multi-field validator will run withpre-validated data from the first "pass" and so will not have to doits own type checking on its inputs.
The flipside is that if the first "pass" of validation fails, yourcross-field validator will not run:
# raises Invalid because password_again is not a string# passwords_must_match() will not run because first-pass validation already failedschema({'password':'123','password_again':1337})
Voluptuous is usingpytest:
$ pip install pytest$ pytest
To also include a coverage report:
$ pip install pytest pytest-cov coverage>=3.0$ pytest --cov=voluptuous voluptuous/tests/Voluptuous is heavily inspired byValidino, and to a lesser extent,jsonvalidator andjson_schema.
pytest-voluptuous is apytest plugin that helps inusing voluptuous validators inasserts.
I greatly prefer the light-weight style promoted by these libraries tothe complexity of libraries like FormEncode.
About
CONTRIBUTIONS ONLY: Voluptuous, despite the name, is a Python data validation library.