data-processing

Star

Here are 1,542 public repositories matching this topic...

Language:All

Filter by language

All1,542 Python546 Jupyter Notebook345 JavaScript63 Java60 R55 HTML38 Go37 C++35 TypeScript33 PHP28

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

pathwaycom /pathway

Star28.8k

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

python rust streaming real-time kafka etl machine-learning-algorithms stream-processing data-analytics dataflow data-processing data-pipelines batch-processing pathway iot-analytics etl-framework time-series-analysis

UpdatedJul 18, 2025
Python

onceupon /Bash-Oneliner

Star10.5k

A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.

linux shell bash terminal system hardware grep data-processing variables xargs xwindow one-liners linux-administration oneliner-commands shell-oneliner

UpdatedApr 3, 2025

johnkerl /miller

Star9.4k

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

tsv devops json statistics csv command-line json-data tabular-data data-reduction unix-toolkit statistical-analysis csv-format devops-tools data-regression data-processing command-line-tools data-cleaning streaming-algorithms streaming-data miller

UpdatedJul 11, 2025
Go

TomWright /dasel

Sponsor

Star7.5k

Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.

config go cli golang yaml toml parser json query xml configuration update selector data-structures data-wrangling devops-tools data-processing yaml-processor json-processing

UpdatedJul 10, 2025
Go

NVIDIA /DALI

Star5.5k

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

python machine-learning deep-learning neural-network mxnet gpu image-processing pytorch gpu-tensorflow data-processing data-augmentation audio-processing paddle image-augmentation fast-data-pipeline

UpdatedJul 14, 2025
C++

modelscope /data-juicer

Star4.8k

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

data-science data data-visualization data-analysis data-processing multi-modal data-pipeline synthetic-data pre-training foundation-models large-language-models llm llms instruction-tuning

UpdatedJul 18, 2025
Python

deepseek-ai /smallpond

Star4.7k

A lightweight data processing framework built on DuckDB and 3FS.

data-processing duckdb

UpdatedMar 5, 2025
Python

unionai-oss /pandera

Star3.9k

A light-weight, flexible, and expressive statistical data testing library

testing schema validation data-validation pandas-dataframe assertions pandas testing-tools data-processing dataframes data-cleaning hypothesis-testing data-verification pandas-validation data-check data-assertions dataframe-schema pandas-validator

UpdatedJul 16, 2025
Python

dashbitco /broadway

Star2.5k

Concurrent and multi-stage data ingestion and data processing with Elixir

elixir broadway concurrent data-processing genstage data-ingestion

UpdatedJun 6, 2025
Elixir

microsoft /DialoGPT

Star2.4k

Large-scale pretraining for dialogue

machine-learning dialogue text-generation pytorch transformer data-processing text-data gpt-2 dialogpt

UpdatedOct 17, 2022
Python

asyml /texar

Star2.4k

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project:http://casl-project.ai/

python machine-learning natural-language-processing deep-learning tensorflow machine-translation text-generation data-processing bert text-data dialog-systems gpt-2 texar xlnet casl-project