- Notifications
You must be signed in to change notification settings - Fork1
A large text field, stored compressed in the database, for Django and MySQL.
License
techsmartkids/django-mysql-compressed-fields
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
This package providesCompressedTextField, a MySQL-specificDjango model field similar toTextField orCharField that stores itsvalue in compressed form viazlib.
In particular you can replace a TextField or CharField like:
fromdjango.dbimportmodelsclassProjectTextFile(models.Model):content=models.TextField(blank=True)
with:
fromdjango.dbimportmodelsfrommysql_compressed_fieldsimportCompressedTextFieldclassProjectTextFile(models.Model):content=CompressedTextField(blank=True)
such that the text value of the field is actually compressed in the database.
String-based lookups are supported:
html_files=ProjectTextFile.objects.filter(content__contains='<html')html_files=ProjectTextFile.objects.filter(content__startswith='<!DOCTYPE')html_files=ProjectTextFile.objects.filter(content__endswith='</html>')empty_html_files=ProjectTextFile.objects.filter(content__in=['','<html></html>'])
Advanced manipulations with MySQL'sCOMPRESS(),UNCOMPRESS(), andUNCOMPRESSED_LENGTH() functions are also supported:
fromdjango.db.modelsimportFfrommysql_compressed_fieldsimportUncompressedLengthfiles=ProjectTextFile.objects.only('id').annotate(content_length=UncompressedLength(F('content')))
To migrate an existing TextField or CharField to be a CompressedTextField:
- Install this package:
pip3 install django-mysql-compressed-fields
- Find an existing Django model with an uncompressedTextField orCharFieldthat you want to compress. For example:
fromdjango.dbimportmodelsclassProjectTextFile(models.Model):content=models.TextField(blank=True)
- Add a
*_compressedsibling field that will be used to hold the compressedversion of the original field. Mark it asdefault=''. Include an explicitdb_column=...value:
fromdjango.dbimportmodelsfrommysql_compressed_fieldsimportCompressedTextFieldclassProjectTextFile(models.Model):content=models.TextField(blank=True)content_compressed=CompressedTextField(blank=True,default='',# needed by Django when adding a fielddb_column='content_compressed',# pin column name )
- Generate a migration to add the compressed field:
python3 manage.py makemigrations
- Generate a new empty migration in the same app where the field is defined,which we will use to populate the compressed field:
python3 manage.py makemigrations --empty __APP_NAME__
- Open the empty migration file. It should look something like:
fromdjango.dbimportmigrationsclassMigration(migrations.Migration):dependencies= [ ('ide','0002_projecttextfile_content_compressed'), ]operations= [ ]
- Edit the
operationsfield to use a RunPython step to populatethe compressed field from the uncompressed field:
fromdjango.dbimportmigrationsfromdjango.db.modelsimportFfrommysql_compressed_fieldsimportCompressdef_populate_content_compressed(apps,schema_editor):ProjectTextFile=apps.get_model('ide','ProjectTextFile')# NOTE: Assumes "content" field is already UTF-8 encoded,# because CompressedTextField assumes UTF-8 encoding.ProjectTextFile.objects.update(content_compressed=Compress(F('content')))classMigration(migrations.Migration):dependencies= [ ('ide','0002_projecttextfile_content_compressed'), ]operations= [migrations.RunPython(code=_populate_content_compressed,reverse_code=migrations.RunPython.noop,atomic=False, ) ]
- Remove the original uncompressed field from the model,leaving only the compressed field remaining:
fromdjango.dbimportmodelsfrommysql_compressed_fieldsimportCompressedTextFieldclassProjectTextFile(models.Model):content_compressed=CompressedTextField(blank=True,default='',# needed by Django when adding a fielddb_column='content_compressed',# pin column name )
- Generate a migration to remove the uncompressed field:
python3 manage.py makemigrations
- Rename the compressed field without the
*_compressedsuffixso that it now has the name of the original uncompressed field:
fromdjango.dbimportmodelsfrommysql_compressed_fieldsimportCompressedTextFieldclassProjectTextFile(models.Model):content=CompressedTextField(blank=True,default='',# needed by Django when adding a fielddb_column='content_compressed',# pin column name )
- Generate a migration to rename the field:
python3 manage.py makemigrations- When prompted whether the field was renamed, answer
y(for yes).
- You now have a compressed version of the original field. All done! 🎉
This project is brought to you byTechSmart, which seeks to inspire thenext generation of K-12 teachers and students to learn coding and createamazing things with computers. We use Django heavily.
All classes and functions below should be imported directly frommysql_compressed_fields. For example:
frommysql_compressed_fieldsimportCompressedTextField
A large text field, stored compressed in the database.
Generally behaves likeTextField. Stores values in the database using thesame database column type asBinaryField. The value is compressed in thesame format that MySQL'sCOMPRESS() function uses. Compression anddecompression is performed by Django and not the database.
encode_errors controls how encoding errors are handled when saving the field.decode_errors controls how decoding errors are handled when loading the field. If'strict' (the default), a UnicodeError exception is raised. Other possible values are'ignore','replace', and any other name registered viacodecs.register_error(). SeeError Handlers for details.
If you specify amax_length attribute, it will be reflected in theTextarea widget of the auto-generated form field. However it is notenforced at the model or database level. Themax_length applies to thelength of the uncompressed text rather than the compressed text.
String-based lookups can be used with this field type.Such lookups will transparently decompress the field on the database server.
html_files=ProjectTextFile.objects.filter(content__contains='<html')html_files=ProjectTextFile.objects.filter(content__startswith='<!DOCTYPE')html_files=ProjectTextFile.objects.filter(content__endswith='</html>')empty_html_files=ProjectTextFile.objects.filter(content__in=['','<html></html>'])
Note that F-expressions that reference this field type will always refer tothe compressed value rather than the uncompressed value. So you may need touse the Compress() and Uncompress() database functions manually when workingwith F-expressions.
# Copy a TextField value (in utf8 collation) to a CompressedTextFieldProjectTextFile.objects.filter(...).update(content=Compress(F('name')))# Copy a CompressedTextField value to a TextField (in utf8 collation)ProjectTextFile.objects.filter(...).update(name=Uncompress(F('content')))# Copy a CompressedTextField value to a CompressedTextFieldProjectTextFile.objects.filter(...).update(content=F('content'))
The default form widget for this field is adjango.contrib.admin.widgets.AdminTextareaWidget (a kind ofTextInput).
The MySQLCOMPRESS() function, usable inF() expressions.
The MySQLUNCOMPRESS() function, usable inF() expressions.
The MySQLUNCOMPRESSED_LENGTH() function, usable inF() expressions.
defcompress(uncompressed_bytes:bytes)->bytes:
The MySQLCOMPRESS() function.
defuncompress(compressed_bytes:bytes)->bytes:
The MySQLUNCOMPRESS() function.
defuncompressed_length(compressed_bytes:bytes)->int:
The MySQLUNCOMPRESSED_LENGTH() function.
defcompressed_length(uncompressed_bytes:bytes,*,chunk_size:int=64*1000,stop_if_greater_than:Optional[int]=None)->int:
Returns the length of COMPRESS(uncompressed_bytes).
Ifstop_if_greater_than is specified and a result greater thanstop_if_greater_than is returned then the compressed length isno less than the returned result.
- InstallDocker.
- Install MySQL CLI tools:
- If macOS, install using brew:
brew install mysql-client@8.0 - Otherwise install from source:https://downloads.mysql.com/archives/community/
- If macOS, install using brew:
- Add MySQL CLI tools to path:
export PATH="/usr/local/opt/mysql-client@8.0/bin:$PATH"
- Start MySQL server:
docker run --name ide_db_server -e MYSQL_DATABASE=ide_db -e MYSQL_ROOT_PASSWORD=root -p 127.0.0.1:8889:3306 -d mysql:8.0
- Run tests:
cd tests/test_data/mysitepoetry installpoetry run python3 manage.py test
- Add the
encode_errorsanddecode_errorsoptions toCompressedTextField.
- Fix to support Django 4.1.
- Add logo.
- Initial release.
About
A large text field, stored compressed in the database, for Django and MySQL.
Resources
License
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Packages0
Uh oh!
There was an error while loading.Please reload this page.