Repository files navigation

Abacus let you count item frequencies in big datasets with a fixed amount of memory.

Unlike a regular counter it trades off accuracy for memory.This is useful for particular tasks, for example in NLP/ML related tasks you might want to count millions of itemshowever approximate counts are good enough.

Example:

counter:=abacus.New(maxMemoryMB=10)// abacus will use max 10MB to store your countscounter.Update([]string{"item1","item2","item2"})counter.Counts("item1")// 1 , counts for "item1"counter.Total()// 3 ,Total number of counts (sum of counts of all elements)counter.Cardinality()// 2 , How many different items are there?

Abacus lets you define how much memory you want to use and you go from there counting items.Of course there are some limitations, and if you set the memory threshold too low, you might get innacurate counts.

• Counting bigrams (words) fromWiki corpus.
• Compared memory and accuracy ofAbacus vs using amap[string]int

Corpus Data Structure Used Memory Accuracy

Note: This is me playing with Golang again, heavily based onBounter

Used to count item frequencies.

Used to calculate the cardinality

Icon made byfree-icon