Movatterモバイル変換

Skip to content

#

data-processing

Here are 1,976 public repositories matching this topic...

Language:All

Filter by language

All1,976 Python711 Jupyter Notebook373 JavaScript89 Java77 HTML62 R57 Go44 TypeScript42 C++40 PHP32

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

pathway

pathwaycom /pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

python rust streaming real-time kafka etl machine-learning-algorithms stream-processing data-analytics dataflow data-processing data-pipelines batch-processing pathway iot-analytics etl-framework time-series-analysis

UpdatedDec 17, 2025
Python

Bash-Oneliner

onceupon /Bash-Oneliner

A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.

linux shell bash terminal system hardware grep data-processing variables xargs xwindow one-liners linux-administration oneliner-commands shell-oneliner

UpdatedApr 3, 2025

miller

johnkerl /miller

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

tsv devops json statistics csv command-line json-data tabular-data data-reduction unix-toolkit statistical-analysis csv-format devops-tools data-regression data-processing command-line-tools data-cleaning streaming-algorithms streaming-data miller

UpdatedDec 17, 2025
Go

TomWright /dasel

Select, put and delete data from JSON, TOML, YAML, XML, INI, HCL and CSV files with a single tool. Also available as a go mod.

config go cli golang yaml toml parser json query xml configuration update selector data-structures data-wrangling devops-tools data-processing yaml-processor json-processing

UpdatedDec 17, 2025
Go

datajuicer /data-juicer

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

data-science data data-visualization data-analysis data-processing multi-modal data-pipeline synthetic-data pre-training foundation-models large-language-models llm llms instruction-tuning

UpdatedDec 17, 2025
Python

DALI

NVIDIA /DALI

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

python machine-learning deep-learning neural-network mxnet gpu image-processing pytorch gpu-tensorflow data-processing data-augmentation audio-processing paddle image-augmentation fast-data-pipeline

UpdatedDec 17, 2025
C++

deepseek-ai /smallpond

A lightweight data processing framework built on DuckDB and 3FS.

data-processing duckdb

UpdatedMar 5, 2025
Python

unionai-oss /pandera

A light-weight, flexible, and expressive statistical data testing library

testing schema validation data-validation pandas-dataframe assertions pandas testing-tools data-processing dataframes data-cleaning hypothesis-testing data-verification pandas-validation data-check data-assertions dataframe-schema pandas-validator

UpdatedDec 12, 2025
Python

cocoindex-io /cocoindex

Data transformation framework for AI. Ultra performant, with incremental processing. 🌟 Star if you like it!

python rust data real-time ai pipeline etl indexing data-engineering knowledge-graph help-wanted data-processing semantic-search hacktoberfest change-data-capture data-infrastructure data-indexing rag llm context-engineering

UpdatedDec 17, 2025
Rust

broadway

dashbitco /broadway

Concurrent and multi-stage data ingestion and data processing with Elixir

elixir broadway concurrent data-processing genstage data-ingestion

UpdatedDec 13, 2025
Elixir

numaproj /numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs

kubernetes pipeline stream-processing map-reduce k8s data-processing hacktoberfest

UpdatedDec 17, 2025
Rust

microsoft /DialoGPT

Large-scale pretraining for dialogue

machine-learning dialogue text-generation pytorch transformer data-processing text-data gpt-2 dialogpt

UpdatedOct 17, 2022
Python

asyml /texar

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project:http://casl-project.ai/

python machine-learning natural-language-processing deep-learning tensorflow machine-translation text-generation data-processing bert text-data dialog-systems gpt-2 texar xlnet casl-project

UpdatedAug 26, 2021
Python

bytewax /bytewax

Python Stream Processing

python rust data-science machine-learning stream-processing data-engineering dataflow data-processing streaming-data

UpdatedMar 27, 2025
Python

python-bonobo /bonobo

Extract Transform Load for Python 3.5+

automation parallelization python3 data-processing bonobo extract-transform-load

UpdatedMay 12, 2023
Python

OpenDCAI /DataFlow

Easy Data Preparation with latest LLMs-based Operators and Pipelines.

data-science data operators data-processing data-pipelines data-cleaning data-synthesis gradio-interface llms data-agent vllm-backend sglang-bankend quick-data-processing

UpdatedDec 16, 2025
Python

pyper-dev /pyper

Concurrent Python made simple

python data multiprocessing concurrency parallel-computing data-engineering asyncio threading data-collection data-processing data-pipelines

UpdatedFeb 4, 2025
Python

GoogleCloudPlatform /data-science-on-gcp

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

data-science machine-learning data-visualization data-engineering cloud-computing data-analysis data-processing data-pipeline

UpdatedOct 24, 2025
Jupyter Notebook

dolma

allenai /dolma

Data and tools for generating and inspecting OLMo pre-training data.

nlp data-processing machile-learning large-language-models llm

UpdatedNov 5, 2025
Python

NVIDIA-NeMo /Curator

Scalable data pre processing and curation toolkit for LLMs

python data data-processing data-preparation deduplication data-quality data-curation data-prep fine-tuning fast-data-processing data-processing-pipelines datacuration large-language-models llm llmapps large-scale-data-processing datarecipes semantic-deduplication llm-data-quality

UpdatedDec 17, 2025
Python

Improve this page

Add a description, image, and links to thedata-processing topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with thedata-processing topic, visit your repo's landing page and select "manage topics."

[8]ページ先頭

©2009-2025 Movatter.jp