Repository files navigation

Malaysian-Dataset, We gather Malaysian dataset!

We are trying our best to make better documentation, all data pushed tohttps://huggingface.co/mesolitica andhttps://huggingface.co/malaysia-ai

Proper documentation is available athttps://malaysian-dataset.readthedocs.io

Contributors heavily crawled Malaysian websites, you can check out the full list of crawled websites athttps://github.com/users/huseinzol05/projects/1

1. We catch most of live data from Twitter, Facebook and Instagram usingcrawlers, So we just search using Elasticsearch query.

1. We use Google Translate.
2. We use LLM, including ChatGPT3.5, ChatGPT4, Mixtral, LLama3 70B.
3. We use Malaya translation,https://huggingface.co/mesolitica/translation-t5-small-standard-bahasa-cased-v2

1. Supervised small samples and then trained a base model.
2. Trained base model predict larger samples, retrain next studentmodels on high confident labelled data.
3. Repeat.

1. Generate using ChatGPT3.5, ChatGPT4, Mixtral, LLama3 70B.

1. Any missingmp.py, get it athttps://gist.github.com/huseinzol05/98974ae8c6c7a65d4bc0af9f5003786a
2. Any missing python scripts, please contact me ASAP or create an issue.
3. Please at least email us first before distributing these data. Remember all these hard workings we want to give it for free.
4. What do you see just the data, but nobody can see how much we spent our cost to make it public.

1. Feel free to contact me to request new dataset.
2. Feel free to open an issue if the link to dataset is forbidden, sometime I forgot to make it open to public.

A lot of data here semisupervised / translated / tagged / decoded usingthird party software, example, Google Translate, Google Speech, so toavoid any future complication, it is better not use this data forcommercial purposes but allow for certain research purposes.

Thanks toIm Big,LigBlou,Mesolitica andKeyReply for sponsoring AWS Google andprivate cloud to deploy distributed crawlers.