Repository files navigation

Charamel is a pure Python universal character encoding library that supportsall ofPython character encodings.The library is based on machine learning and trained to handle more than 60 languages.All that with no external dependencies. Ain't it sweet? 🍭

$ pip install charamel

• 🌈 Powered by machine learning
• 📦 No dependencies
• ⚡ Faster than other pure Python libraries
• 🐍 Supports all98 Python encodings
• 🌍 Works on 60+ languages
• 🔎 97% accuracy

API is centered aroundDetector class, withdetect method being responsible for basic encoding detection:

>>>fromcharamelimportDetector>>>detector=Detector()>>>content=b'El espa\xf1ol o castellano del lat\xedn hablado'>>>detector.detect(content)<Encoding.ISO_8859_14:'iso8859_14'>

This returns the most likely encoding that can decode the byte string. Let's try it out:

>>>fromcharamelimportEncoding>>>content.decode(Encoding.ISO_8859_14)'El español o castellano del latín hablado'

To get multiple likely encodings along with confidences in range[0, 1], useprobe method:

>>>detector.probe(content,top=3)[(<Encoding.ISO_8859_14:'iso8859_14'>,0.9964286725192874), (<Encoding.CP_1258:'cp1258'>,0.9919203166700203), (<Encoding.ISO_8859_3:'iso8859_3'>,0.9915028923264849)]

Detector can be configured to use a subset of encodings. Less possible encodings lead to faster detection:

>>>detector=Detector(encodings=[Encoding.UTF_8,Encoding.BIG_5])

Another usefulDetector parameter ismin_confidence. Basically, this parameter regulates how conservative theDetector will be.Confidence for encodings that are returned bydetect andprobe methods must be greater thatmin_confidence:

>>>detector=Detector(min_confidence=0.5)

If no encoding confidences exceedmin_confidence,detect will returnNone andprobe will return an empty list.

Below is the comparison betweenCharamel and other available Python encoding detection libraries:

How to run this benchmark (requires Python 3.6+):

$ git clone git@github.com:chomechome/charamel.git$cd charamel$ pip install poetry>=1.0.5$ make benchmark

It also produces a detailed breakdown for all represented encodings:

* - not officially support for detector