How to create custom model fields¶
Introduction¶
Themodel reference documentation explains how touse Django’s standard field classes –CharField,DateField, etc. For many purposes, those classes areall you’ll need. Sometimes, though, the Django version won’t meet your preciserequirements, or you’ll want to use a field that is entirely different fromthose shipped with Django.
Django’s built-in field types don’t cover every possible database column type– only the common types, such asVARCHAR andINTEGER. For more obscurecolumn types, such as geographic polygons or even user-created types such asPostgreSQL custom types, you can define your own DjangoFieldsubclasses.
Alternatively, you may have a complex Python object that can somehow beserialized to fit into a standard database column type. This is another casewhere aField subclass will help you use your object with your models.
Our example object¶
Creating custom fields requires a bit of attention to detail. To make thingseasier to follow, we’ll use a consistent example throughout this document:wrapping a Python object representing the deal of cards in a hand ofBridge.Don’t worry, you don’t have to know how to play Bridge to follow this example.You only need to know that 52 cards are dealt out equally to four players, whoare traditionally callednorth,east,south andwest. Our class lookssomething like this:
classHand:"""A hand of cards (bridge style)"""def__init__(self,north,east,south,west):# Input parameters are lists of cards ('Ah', '9s', etc.)self.north=northself.east=eastself.south=southself.west=west# ... (other possibly useful methods omitted) ...
This is an ordinary Python class, with nothing Django-specific about it.We’d like to be able to do things like this in our models (we assume thehand attribute on the model is an instance ofHand):
example=MyModel.objects.get(pk=1)print(example.hand.north)new_hand=Hand(north,east,south,west)example.hand=new_handexample.save()
We assign to and retrieve from thehand attribute in our model just likeany other Python class. The trick is to tell Django how to handle saving andloading such an object.
In order to use theHand class in our models, wedo not have to changethis class at all. This is ideal, because it means you can easily writemodel support for existing classes where you cannot change the source code.
Note
You might only be wanting to take advantage of custom database columntypes and deal with the data as standard Python types in your models;strings, or floats, for example. This case is similar to ourHandexample and we’ll note any differences as we go along.
Background theory¶
Database storage¶
Let’s start with model fields. If you break it down, a model field provides away to take a normal Python object – string, boolean,datetime, orsomething more complex likeHand – and convert it to and from a formatthat is useful when dealing with the database. (Such a format is also usefulfor serialization, but as we’ll see later, that is easier once you have thedatabase side under control).
Fields in a model must somehow be converted to fit into an existing databasecolumn type. Different databases provide different sets of valid column types,but the rule is still the same: those are the only types you have to workwith. Anything you want to store in the database must fit into one ofthose types.
Normally, you’re either writing a Django field to match a particular databasecolumn type, or you will need a way to convert your data to, say, a string.
For ourHand example, we could convert the card data to a string of 104characters by concatenating all the cards together in a predetermined order –say, all thenorth cards first, then theeast,south andwest cards. SoHand objects can be saved to text or character columns in the database.
What does a field class do?¶
All of Django’s fields (and when we sayfields in this document, we alwaysmean model fields and notform fields) aresubclasses ofdjango.db.models.Field. Most of the information thatDjango records about a field is common to all fields – name, help text,uniqueness and so forth. Storing all that information is handled byField.We’ll get into the precise details of whatField can do later on; for now,suffice it to say that everything descends fromField and then customizeskey pieces of the class behavior.
It’s important to realize that a Django field class is not what is stored inyour model attributes. The model attributes contain normal Python objects. Thefield classes you define in a model are actually stored in theMeta classwhen the model class is created (the precise details of how this is done areunimportant here). This is because the field classes aren’t necessary whenyou’re just creating and modifying attributes. Instead, they provide themachinery for converting between the attribute value and what is stored in thedatabase or sent to theserializer.
Keep this in mind when creating your own custom fields. The DjangoFieldsubclass you write provides the machinery for converting between your Pythoninstances and the database/serializer values in various ways (there aredifferences between storing a value and using a value for lookups, forexample). If this sounds a bit tricky, don’t worry – it will become clearer inthe examples below. Just remember that you will often end up creating twoclasses when you want a custom field:
The first class is the Python object that your users will manipulate.They will assign it to the model attribute, they will read from it fordisplaying purposes, things like that. This is the
Handclass in ourexample.The second class is the
Fieldsubclass. This is the class that knowshow to convert your first class back and forth between its permanentstorage form and the Python form.
Writing a field subclass¶
When planning yourField subclass, first give somethought to which existingField class your new fieldis most similar to. Can you subclass an existing Django field and save yourselfsome work? If not, you should subclass theFieldclass, from which everything is descended.
Initializing your new field is a matter of separating out any arguments thatare specific to your case from the common arguments and passing the latter tothe__init__() method ofField (or your parentclass).
In our example, we’ll call our fieldHandField. (It’s a good idea to callyourField subclass<Something>Field, so it’seasily identifiable as aField subclass.) It doesn’tbehave like any existing field, so we’ll subclass directly fromField:
fromdjango.dbimportmodelsclassHandField(models.Field):description="A hand of cards (bridge style)"def__init__(self,*args,**kwargs):kwargs["max_length"]=104super().__init__(*args,**kwargs)
OurHandField accepts most of the standard field options (see the listbelow), but we ensure it has a fixed length, since it only needs to hold 52card values plus their suits; 104 characters in total.
Note
Many of Django’s model fields accept options that they don’t do anythingwith. For example, you can pass botheditable andauto_now to adjango.db.models.DateField and it will ignore theeditable parameter(auto_now being set implieseditable=False). No error is raised in this case.
This behavior simplifies the field classes, because they don’t need tocheck for options that aren’t necessary. They pass all the options tothe parent class and then don’t use them later on. It’s up to you whetheryou want your fields to be more strict about the options they select, or touse the more permissive behavior of the current fields.
TheField.__init__() method takes the following parameters:
namerel: Used for related fields (likeForeignKey). For advanceduse only.serialize: IfFalse, the field will not be serialized when the modelis passed to Django’sserializers. Defaults toTrue.db_tablespace: Only for index creation, ifthe backend supportstablespaces. You canusually ignore this option.auto_created:Trueif the field wasautomatically created, as for theOneToOneFieldused by model inheritance. For advanced use only.
All of the options without an explanation in the above list have the samemeaning they do for normal Django fields. See thefield documentation for examples and details.
Field deconstruction¶
The counterpoint to writing your__init__() method is writing thedeconstruct() method. It’s used duringmodel migrations to tell Django how to take an instance of your new fieldand reduce it to a serialized form - in particular, what arguments to pass to__init__() to recreate it.
If you haven’t added any extra options on top of the field you inherited from,then there’s no need to write a newdeconstruct() method. If, however,you’re changing the arguments passed in__init__() (like we are inHandField), you’ll need to supplement the values being passed.
deconstruct() returns a tuple of four items: the field’s attribute name,the full import path of the field class, the positional arguments (as a list),and the keyword arguments (as a dict). Note this is different from thedeconstruct() methodfor custom classeswhich returns a tuple of three things.
As a custom field author, you don’t need to care about the first two values;the baseField class has all the code to work out the field’s attributename and import path. You do, however, have to care about the positionaland keyword arguments, as these are likely the things you are changing.
For example, in ourHandField class we’re always forcibly settingmax_length in__init__(). Thedeconstruct() method on the baseField class will see this and try to return it in the keyword arguments;thus, we can drop it from the keyword arguments for readability:
fromdjango.dbimportmodelsclassHandField(models.Field):def__init__(self,*args,**kwargs):kwargs["max_length"]=104super().__init__(*args,**kwargs)defdeconstruct(self):name,path,args,kwargs=super().deconstruct()delkwargs["max_length"]returnname,path,args,kwargs
If you add a new keyword argument, you need to write code indeconstruct()that puts its value intokwargs yourself. You should also omit the valuefromkwargs when it isn’t necessary to reconstruct the state of the field,such as when the default value is being used:
fromdjango.dbimportmodelsclassCommaSepField(models.Field):"Implements comma-separated storage of lists"def__init__(self,separator=",",*args,**kwargs):self.separator=separatorsuper().__init__(*args,**kwargs)defdeconstruct(self):name,path,args,kwargs=super().deconstruct()# Only include kwarg if it's not the defaultifself.separator!=",":kwargs["separator"]=self.separatorreturnname,path,args,kwargs
More complex examples are beyond the scope of this document, but remember -for any configuration of your Field instance,deconstruct() must returnarguments that you can pass to__init__ to reconstruct that state.
Pay extra attention if you set new default values for arguments in theField superclass; you want to make sure they’re always included, ratherthan disappearing if they take on the old default value.
In addition, try to avoid returning values as positional arguments; wherepossible, return values as keyword arguments for maximum future compatibility.If you change the names of things more often than their position in theconstructor’s argument list, you might prefer positional, but bear in mind thatpeople will be reconstructing your field from the serialized version for quitea while (possibly years), depending how long your migrations live for.
You can see the results of deconstruction by looking in migrations that includethe field, and you can test deconstruction in unit tests by deconstructing andreconstructing the field:
name,path,args,kwargs=my_field_instance.deconstruct()new_instance=MyField(*args,**kwargs)self.assertEqual(my_field_instance.some_attribute,new_instance.some_attribute)
Field attributes not affecting database column definition¶
You can overrideField.non_db_attrs to customize attributes of a field thatdon’t affect a column definition. It’s used during model migrations to detectno-opAlterField operations.
For example:
classCommaSepField(models.Field):@propertydefnon_db_attrs(self):returnsuper().non_db_attrs+("separator",)
Changing a custom field’s base class¶
You can’t change the base class of a custom field because Django won’t detectthe change and make a migration for it. For example, if you start with:
classCustomCharField(models.CharField):...
and then decide that you want to useTextField instead, you can’t changethe subclass like this:
classCustomCharField(models.TextField):...
Instead, you must create a new custom field class and update your models toreference it:
classCustomCharField(models.CharField):...classCustomTextField(models.TextField):...
As discussed inremoving fields, youmust retain the originalCustomCharField class as long as you havemigrations that reference it.
Documenting your custom field¶
As always, you should document your field type, so users will know what it is.In addition to providing a docstring for it, which is useful for developers,you can also allow users of the admin app to see a short description of thefield type via thedjango.contrib.admindocs application. To do this provide descriptivetext in adescription class attribute of your custom field. Inthe above example, the description displayed by theadmindocs applicationfor aHandField will be ‘A hand of cards (bridge style)’.
In thedjango.contrib.admindocs display, the field description isinterpolated withfield.__dict__ which allows the description toincorporate arguments of the field. For example, the description forCharField is:
description=_("String (up to%(max_length)s)")
Useful methods¶
Once you’ve created yourField subclass, you mightconsider overriding a few standard methods, depending on your field’s behavior.The list of methods below is in approximately decreasing order of importance,so start from the top.
Custom database types¶
Say you’ve created a PostgreSQL custom type calledmytype. You cansubclassField and implement thedb_type() method, like so:
fromdjango.dbimportmodelsclassMytypeField(models.Field):defdb_type(self,connection):return"mytype"
Once you haveMytypeField, you can use it in any model, just like any otherField type:
classPerson(models.Model):name=models.CharField(max_length=80)something_else=MytypeField()
If you aim to build a database-agnostic application, you should account fordifferences in database column types. For example, the date/time column typein PostgreSQL is calledtimestamp, while the same column in MySQL is calleddatetime. You can handle this in adb_type() method bychecking theconnection.vendor attribute. Current built-in vendor namesare:sqlite,postgresql,mysql, andoracle.
For example:
classMyDateField(models.Field):defdb_type(self,connection):ifconnection.vendor=="mysql":return"datetime"else:return"timestamp"
Thedb_type() andrel_db_type() methods are called byDjango when the framework constructs theCREATETABLE statements for yourapplication – that is, when you first create your tables. The methods are alsocalled when constructing aWHERE clause that includes the model field –that is, when you retrieve data using QuerySet methods likeget(),filter(), andexclude() and have the model field as an argument.
Some database column types accept parameters, such asCHAR(25), where theparameter25 represents the maximum column length. In cases like these,it’s more flexible if the parameter is specified in the model rather than beinghardcoded in thedb_type() method. For example, it wouldn’t make much senseto have aCharMaxlength25Field, shown here:
# This is a silly example of hardcoded parameters.classCharMaxlength25Field(models.Field):defdb_type(self,connection):return"char(25)"# In the model:classMyModel(models.Model):# ...my_field=CharMaxlength25Field()
The better way of doing this would be to make the parameter specifiable at runtime – i.e., when the class is instantiated. To do that, implementField.__init__(), like so:
# This is a much more flexible example.classBetterCharField(models.Field):def__init__(self,max_length,*args,**kwargs):self.max_length=max_lengthsuper().__init__(*args,**kwargs)defdb_type(self,connection):return"char(%s)"%self.max_length# In the model:classMyModel(models.Model):# ...my_field=BetterCharField(25)
Finally, if your column requires truly complex SQL setup, returnNone fromdb_type(). This will cause Django’s SQL creation code to skipover this field. You are then responsible for creating the column in the righttable in some other way, but this gives you a way to tell Django to get out ofthe way.
Therel_db_type() method is called by fields such asForeignKey andOneToOneField that point to another field to determinetheir database column data types. For example, if you have anUnsignedAutoField, you also need the foreign keys that point to that fieldto use the same data type:
# MySQL unsigned integer (range 0 to 4294967295).classUnsignedAutoField(models.AutoField):defdb_type(self,connection):return"integer UNSIGNED AUTO_INCREMENT"defrel_db_type(self,connection):return"integer UNSIGNED"
Converting values to Python objects¶
If your customField class deals with data structures that are morecomplex than strings, dates, integers, or floats, then you may need to overridefrom_db_value() andto_python().
If present for the field subclass,from_db_value() will be called in allcircumstances when the data is loaded from the database, including inaggregates andvalues() calls.
to_python() is called by deserialization and during theclean() method used from forms.
As a general rule,to_python() should deal gracefully with any of thefollowing arguments:
An instance of the correct type (e.g.,
Handin our ongoing example).A string
None(if the field allowsnull=True)
In ourHandField class, we’re storing the data as aVARCHAR field inthe database, so we need to be able to process strings andNone in thefrom_db_value(). Into_python(), we need to also handleHandinstances:
importrefromdjango.core.exceptionsimportValidationErrorfromdjango.dbimportmodelsfromdjango.utils.translationimportgettext_lazyas_defparse_hand(hand_string):"""Takes a string of cards and splits into a full hand."""p1=re.compile(".{26}")p2=re.compile("..")args=[p2.findall(x)forxinp1.findall(hand_string)]iflen(args)!=4:raiseValidationError(_("Invalid input for a Hand instance"))returnHand(*args)classHandField(models.Field):# ...deffrom_db_value(self,value,expression,connection):ifvalueisNone:returnvaluereturnparse_hand(value)defto_python(self,value):ifisinstance(value,Hand):returnvalueifvalueisNone:returnvaluereturnparse_hand(value)
Notice that we always return aHand instance from these methods. That’s thePython object type we want to store in the model’s attribute.
Forto_python(), if anything goes wrong during value conversion, you shouldraise aValidationError exception.
Converting Python objects to query values¶
Since using a database requires conversion in both ways, if you overridefrom_db_value() you also have to overrideget_prep_value() to convert Python objects back to query values.
For example:
classHandField(models.Field):# ...defget_prep_value(self,value):return"".join(["".join(l)forlin(value.north,value.east,value.south,value.west)])
Warning
If your custom field uses theCHAR,VARCHAR orTEXTtypes for MySQL, you must make sure thatget_prep_value()always returns a string type. MySQL performs flexible and unexpectedmatching when a query is performed on these types and the providedvalue is an integer, which can cause queries to include unexpectedobjects in their results. This problem cannot occur if you alwaysreturn a string type fromget_prep_value().
Converting query values to database values¶
Some data types (for example, dates) need to be in a specific formatbefore they can be used by a database backend.get_db_prep_value() is the method where those conversions shouldbe made. The specific connection that will be used for the query ispassed as theconnection parameter. This allows you to usebackend-specific conversion logic if it is required.
For example, Django uses the following method for itsBinaryField:
defget_db_prep_value(self,value,connection,prepared=False):value=super().get_db_prep_value(value,connection,prepared)ifvalueisnotNone:returnconnection.Database.Binary(value)returnvalue
In case your custom field needs a special conversion when being saved that isnot the same as the conversion used for normal query parameters, you canoverrideget_db_prep_save().
Preprocessing values before saving¶
If you want to preprocess the value just before saving, you can usepre_save(). For example, Django’sDateTimeField uses this method to set the attributecorrectly in the case ofauto_now orauto_now_add.
If you do override this method, you must return the value of the attribute atthe end. You should also update the model’s attribute if you make any changesto the value so that code holding references to the model will always see thecorrect value.
Specifying the form field for a model field¶
To customize the form field used byModelForm, you canoverrideformfield().
The form field class can be specified via theform_class andchoices_form_class arguments; the latter is used if the field has choicesspecified, the former otherwise. If these arguments are not provided,CharField orTypedChoiceFieldwill be used.
All of thekwargs dictionary is passed directly to the form field’s__init__() method. Normally, all you need to do is set up a good defaultfor theform_class (and maybechoices_form_class) argument and thendelegate further handling to the parent class. This might require you to writea custom form field (and even a form widget). See theforms documentation for information about this.
If you wish to exclude the field from theModelForm, youcan override theformfield() method to returnNone.
Continuing our ongoing example, we can write theformfield()method as:
classHandField(models.Field):# ...defformfield(self,**kwargs):# Exclude the field from the ModelForm when some condition is met.some_condition=kwargs.get("some_condition",False)ifsome_condition:returnNone# Set up some defaults while letting the caller override them.defaults={"form_class":MyFormField}defaults.update(kwargs)returnsuper().formfield(**defaults)
This assumes we’ve imported aMyFormField field class (which has its owndefault widget). This document doesn’t cover the details of writing custom formfields.
Emulating built-in field types¶
If you have created adb_type() method, you don’t need to worry aboutget_internal_type() – it won’t be used much. Sometimes, though, yourdatabase storage is similar in type to some other field, so you can use thatother field’s logic to create the right column.
For example:
classHandField(models.Field):# ...defget_internal_type(self):return"CharField"
No matter which database backend we are using, this will mean thatmigrate and other SQL commands create the right column type forstoring a string.
Ifget_internal_type() returns a string that is not known to Django forthe database backend you are using – that is, it doesn’t appear indjango.db.backends.<db_name>.base.DatabaseWrapper.data_types – the stringwill still be used by the serializer, but the defaultdb_type()method will returnNone. See the documentation ofdb_type()for reasons why this might be useful. Putting a descriptive string in as thetype of the field for the serializer is a useful idea if you’re ever going tobe using the serializer output in some other place, outside of Django.
Converting field data for serialization¶
To customize how the values are serialized by a serializer, you can overridevalue_to_string(). Usingvalue_from_object() is thebest way to get the field’s value prior to serialization. For example, sinceHandField uses strings for its data storage anyway, we can reuse someexisting conversion code:
classHandField(models.Field):# ...defvalue_to_string(self,obj):value=self.value_from_object(obj)returnself.get_prep_value(value)
Some general advice¶
Writing a custom field can be a tricky process, particularly if you’re doingcomplex conversions between your Python types and your database andserialization formats. Here are a couple of tips to make things go moresmoothly:
Look at the existing Django fields (indjango/db/models/fields/__init__.py) for inspiration. Try to finda field that’s similar to what you want and extend it a little bit,instead of creating an entirely new field from scratch.
Put a
__str__()method on the class you’re wrapping up as a field. Thereare a lot of places where the default behavior of the field code is to callstr()on the value. (In our examples in this document,valuewouldbe aHandinstance, not aHandField). So if your__str__()method automatically converts to the string form of your Python object, youcan save yourself a lot of work.
Writing aFileField subclass¶
In addition to the above methods, fields that deal with files have a few otherspecial requirements which must be taken into account. The majority of themechanics provided byFileField, such as controlling database storage andretrieval, can remain unchanged, leaving subclasses to deal with the challengeof supporting a particular type of file.
Django provides aFile class, which is used as a proxy to the file’scontents and operations. This can be subclassed to customize how the file isaccessed, and what methods are available. It lives atdjango.db.models.fields.files, and its default behavior is explained in thefile documentation.
Once a subclass ofFile is created, the newFileField subclass must betold to use it. To do so, assign the newFile subclass to the specialattr_class attribute of theFileField subclass.
A few suggestions¶
In addition to the above details, there are a few guidelines which can greatlyimprove the efficiency and readability of the field’s code.
The source for Django’s own
ImageField(indjango/db/models/fields/files.py) is a great example of how tosubclassFileFieldto support a particular type of file, as itincorporates all of the techniques described above.Cache file attributes wherever possible. Since files may be stored inremote storage systems, retrieving them may cost extra time, or evenmoney, that isn’t always necessary. Once a file is retrieved to obtainsome data about its content, cache as much of that data as possible toreduce the number of times the file must be retrieved on subsequentcalls for that information.
