Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Override model serializer globally, big integer field#9722

Discussion options

First of all thanks for this project, i can't use Django without drf 😀

Im looking for a way to override serializers.ModelSerializer globally

Little bit of context:
Our application is grown big and we want to switch django backend from BigAutoField to SnowflakeAutoField.

My problem:

  • ts/js (My frontend) do not support numbers over 2^52,
  • I don't want to use Bigint in ts
  • i don't want to touch MAX_SAFE_INTEGER

So the best solution in my opinion is to serialize bigint as string.

classSnowflakeField(serializers.Field):defto_representation(self,value):returnstr(value)defto_internal_value(self,data):try:returnint(data)exceptValueError:raiseserializers.ValidationError("Invalid integer value")classSnowflakeModelSerializer(serializers.ModelSerializer):"""    Snowflake is a bigint so convert to string to avoid frontend memory handling problems    """defbuild_standard_field(self,field_name,model_field):field_class,field_kwargs=super().build_standard_field(field_name,model_field)# Use BigIntAsStringField per BigIntegerFieldifisinstance(model_field,models.BigIntegerField):field_class=SnowflakeFieldreturnfield_class,field_kwargs

Anyway, since django uses bigint as id, serializing it as sting could be a better option in any case since the id can grow big

Possible ways

  1. use my custom serializer around the code (not so clean)
classMyModelSerializer(SnowflakeModelSerializer):classMeta:model=MyModelfields='__all__'
  1. Write a Custom JsonRender and JsonParser

and do something like this with a lot of assumptions, spoiler bad programming

classSnowflakeJSONRenderer(JSONRenderer):def_convert_big_integers(self,obj):ifisinstance(obj,dict):return {key:self._convert_big_integers(value)forkey,valueinobj.items()}elifisinstance(obj,list):return [self._convert_big_integers(item)foriteminobj]elifisinstance(obj,int)andobj>2**53:returnstr(obj)returnobj

What im looking for

As far as i know there is no option to do this

REST_FRAMEWORK = {
'DEFAULT_MODEL_SERIALIZER_CLASS': 'path.to.utils.serializers.CustomModelSerializer',
}

Am i missing something? In your opinion what could be the best solution? I'm open to any suggestion

You must be logged in to vote

I'd recommend going with the first option. It doesn't look pleasant at first, but it's easily maintainable if you add aDjango system check to validate that the correct class is used.

Do you have a specific reason to use snowflake btw? If you are open to alternatives, I'd recommend UUID7 as a solid one.

Replies: 2 comments 6 replies

Comment options

I'd recommend going with the first option. It doesn't look pleasant at first, but it's easily maintainable if you add aDjango system check to validate that the correct class is used.

Do you have a specific reason to use snowflake btw? If you are open to alternatives, I'd recommend UUID7 as a solid one.

You must be logged in to vote
0 replies
Answer selected bybrowniebroke
Comment options

Thx for the replay@ulgens

Actually we are also considering uuid, for sure it's a simpler implementation and probably a better compromise in our situation

I'm try to evaluate snowflake option for this reasons:

  • url shorter so more easy to copy and read (and espose incremental ids is not the best option)
  • smaller and faster in db than uuid
  • distributed generation
  • time based sorting, but in later releases
  • i don't want to change again ids 😂

Probably it's overkill but i'm not committed to the switch yet

I've made some tests with this code and it's working, it's still a fast implementation.
I still need to consider:

  • snowflake instance id sync, prob with redis that we alread have as cache
  • time sync, in our case azure already do this
fromdjango.dbimportmodelsfromdjango.utils.encodingimportsmart_strfromrest_frameworkimportserializersclassSnowflakeField(serializers.Field):"""Convert snowflake data to string"""defto_representation(self,value):returnsmart_str(value)defto_internal_value(self,data):try:returnint(data)exceptValueError:raiseserializers.ValidationError("Invalid integer value")classSnowflakePrimaryKeyRelatedField(serializers.PrimaryKeyRelatedField,SnowflakeField):"""Handle related primary keys as snowflake"""defto_representation(self,value):ifself.pk_fieldisnotNone:returnself.pk_field.to_representation(value.pk)returnstr(value.pk)defto_internal_value(self,data):data=super(SnowflakeField).to_representation(data)returnsuper(serializers.PrimaryKeyRelatedField).to_internal_value(data)classSnowflakeSlugRelatedField(serializers.SlugRelatedField):"""Implement if used"""classSnowflakeModelSerializer(serializers.ModelSerializer):"""    Snowflake is a bigint so convert to string to avoid frontend memory handling problems    """serializer_field_mapping= {**serializers.ModelSerializer.serializer_field_mapping,models.BigAutoField:SnowflakeField,models.BigIntegerField:SnowflakeField    }serializer_related_field=SnowflakePrimaryKeyRelatedFieldserializer_related_to_field=SnowflakeSlugRelatedField

My pk model

fromdjango.dbimportmodelsclassSnowflakeBigAutoField(models.BigAutoField):defpre_save(self,model_instance,add):ifaddandnotgetattr(model_instance,self.attname,None):value=self.get_next_value()setattr(model_instance,self.attname,value)returnsuper().pre_save(model_instance,add)@staticmethoddefget_next_value():returnnext(sf)# settings.pyDEFAULT_AUTO_FIELD='core.snowflake.SnowflakeBigAutoField'

If we deoloy with snowfake i like to share the production code here, because i didn't find any useful resource about snowflake id with django

DEFAULT_MODEL_SERIALIZER_CLASS

In any case, is it possible in your opinion try to introduce this setting? I can submit a pull request.
I looked at the code and it looks possible, am I missing something?

You must be logged in to vote
6 replies
@HoodyH
Comment options

then some other unique identifier that's is shorter for URLs

We are doing this right now almost everywhere except on some dashboard elements, but it's kinda a pain 🥲

At one point we also introduced the ability for the user to customize the slug to reach certain resources, it was more confusing than useful, so we switched back to autogeneration.

I like something like discord only one id in and out, no names no personalization no problems, it's ok even if uuid it's a bit longer. I don't see problems in this aproach, am i wrong?

Not having this dict configurable without customizing the class seems problematic to me

Next week i try to implemente this solution and let you know

My understanding is that you have a "Snowflake ID to JSON" conversion problem

There is actually a kind of bug: if you take a BigInt in DB, for example2^64 and represent as int, JavaScript/TypeScript Number can only safely represent integers up to2^53−1. JSON follows the same limitation, so although no error occurs, it is still technically incorrect.

It is possible to find a workaround on client side, but it is not guaranteed to work on all browsers.

@ulgens
Comment options

There is actually a kind of bug: if you take a BigInt in DB, for example 2^64 and represent as int, JavaScript/TypeScript Number can only safely represent integers up to 2^53−1. JSON follows the same limitation, so although no error occurs, it is still technically incorrect.

Sorry if I missed it but what's the result of this behaviour? Does it fail? Does it produce an incorrect response? If a Django native datatype can't be converted to a proper json by DRF, we may consider to add a warning and update the docs about it.

@HoodyH
Comment options

Sorry for the late reply, in the meantime i've made some research and extensive tests.

JSON does not have a native BigInt type — it only supports double-precision floating-point numbers (IEEE 754), that are safely precise for integers only up to 2⁵³ − 1 (i.e. 9,007,199,254,740,991). Beyond this threshold, numbers lose precision and may be silently truncated or rounded when parsed in JavaScript or other languages.

It depends on the json library implementation, this particular issue doesn’t happen in Python but it’s not consistent across all languages JSON libraries

The twitter example:

Twitter introduced via api id both numeric and id_str as string to avoid this exact problem.

Why DRF test frontend works?

Because it uses FormData, basically it is all string data, and since the server side render is handled by python no bigint errors occour

Resources:

Since the majority of frontends that will call api are javascript, this is an issue that have to be addressed in my opinion, i know that the bigint problem with ids will happen only at very large scale and probably a lot of people will never encounter this, but if we find a solution it would be wonderfull.

COERCE_DECIMAL_TO_STRING
I suggest doing like this flag also for bigint, that is the exact same problem only for floating point that js also does not support well :')

@HoodyH
Comment options

@ulgens i've opened a issue about this.9733

@ulgens
Comment options

👏🏻 I think this approach has better focus on the actual issue, and precisely defines what needs to be handled. Thanks for all the details. I'll be following people's reactions under#9733, and I can help with the PR when it's ready.

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Labels
None yet
2 participants
@HoodyH@ulgens
Converted from issue

This discussion was converted from issue #9721 on June 20, 2025 07:19.


[8]ページ先頭

©2009-2025 Movatter.jp