- Notifications
You must be signed in to change notification settings - Fork7
Sedpack - Scalable and efficient data packing
License
google/sedpack
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
Mainly refactored from theSCAAML project.
See the documentation website:https://google.github.io/sedpack/.
To use this library you need to have a working version ofTensorFlow2.x.
Development dependencies:
- python-dev and gcc forxxhash
- Clone the repository:
git clone https://github.com/google/sedpack
- Install dependencies:
python3 -m pip install --require-hashes -r requirements.txt
- Install the package in development mode:
python3 -m pip install --editable .
(shortpip install -e .
or legacypython setup.py develop
)
- Activate your Python virtual environment
- Install Rust
- Run
maturin develop --release
- Run
python -m pytest
from the project root directory -- no tests shouldbe skipped
Make sure to have:sudo apt install python3 python3-pip python3-venv
andactivated the virtual environment.
Install requirements:pip install --require-hashes -r base-tooling-requirements.txt
Update:pip-compile pyproject.toml --generate-hashes --upgrade
and commit requirements.txt.
pip install sedpack
A tutorial and documentation is available athttps://google.github.io/sedpack/.
Code for the tutorials is available in thedocs/tutorials
directory. For a"hello world" seehttps://google.github.io/sedpack/tutorials/mnist/.
This is not an official Google product.
About
Sedpack - Scalable and efficient data packing