- Notifications
You must be signed in to change notification settings - Fork20
seaweedfs implemented in pure Rust
License
Apache-2.0, MIT licenses found
Licenses found
helyim/helyim
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
- Can choose no replication or different replication levels, rack and data center aware.
- Automatic compaction to reclaim disk space after deletion or update.
- Automatic master servers fail over - no single point of failure (SPOF).
- Erasure Coding for warm storage Rack-Aware 10.4 erasure coding reduces storage cost.
cargo +nightly install helyim
By default, the master node runs on port 9333, and the volume nodes run on port 8080. Let's start one master node, and one volume node on port 8080. Ideally, they should be started from different machines. We'll use localhost as an example.
Helyim uses HTTP REST operations to read, write, and delete. The responses are in JSON or JSONP format.
cargo run --release --bin helyim master
cargo run --release --bin helyim volume --port 8080 --folders ./target
To upload a file: first, send a HTTP POST, PUT, or GET request to/dir/assign to get anfid and a volume server URL:
curl http://localhost:9333/dir/assign{"fid":"6,16b7578a5","url":"127.0.0.1:8080","public_url":"127.0.0.1:8080","count":1,"error":""}Second, to store the file content, send a HTTP multipart POST request tourl + '/' + fid from the response:
curl -F file=@./sun.jpg http://127.0.0.1:8080/6,16b7578a5{"name":"sun.jpg","size":1675569,"error":""}To update, send another POST request with updated file content.
For deletion, send an HTTP DELETE request to the sameurl + '/' + fid URL:
curl -X DELETE http://127.0.0.1:8080/6,16b7578a5
cargo run --release --bin helyim filer
To upload a file without assigning file id
curl -F file=@./sun.jpg http://127.0.0.1:8888/sun.jpg
When initiating a Raft cluster, it is necessary to specify the same node sequence when starting the Leader and Follower instances.
You can view the cluster status by accessinghttp://localhost:9333/cluster/status.
# start master1cargo run --bin helyim -- master --ip 127.0.0.1 --port 9333 \ --peers 127.0.0.1:9333 \ --peers 127.0.0.1:9335 \ --peers 127.0.0.1:9337# start master2cargo run --bin helyim -- master --ip 127.0.0.1 --port 9335 \ --peers 127.0.0.1:9333 \ --peers 127.0.0.1:9335 \ --peers 127.0.0.1:9337# start master3cargo run --bin helyim -- master --ip 127.0.0.1 --port 9337 \ --peers 127.0.0.1:9333 \ --peers 127.0.0.1:9335 \ --peers 127.0.0.1:9337
My laptop results on Lenovo IdeaPad Pro 16 (2023) with SSD, CPU: 14 Intel Core i9 5.4GHz.
It seems to be slower thanseaweedfs, especially in terms of reading.
➜./weed benchmark -server=localhost:9333This is SeaweedFS version 0.76 linux amd64------------ Writing Benchmark ----------Completed 15199 of 1048576 requests, 1.4% 15198.1/s 15.3MB/sCompleted 31887 of 1048576 requests, 3.0% 16687.9/s 16.8MB/sCompleted 48439 of 1048576 requests, 4.6% 16551.6/s 16.7MB/s...Completed 994044 of 1048576 requests, 94.8% 16645.2/s 16.8MB/sCompleted 1010800 of 1048576 requests, 96.4% 16755.8/s 16.9MB/sCompleted 1027412 of 1048576 requests, 98.0% 16612.2/s 16.7MB/sCompleted 1044319 of 1048576 requests, 99.6% 16907.0/s 17.0MB/sConcurrency Level: 16Time taken for tests: 63.249 secondsComplete requests: 1048576Failed requests: 0Total transferred: 1106759553 bytesRequests per second: 16578.50 [#/sec]Transfer rate: 17088.29 [Kbytes/sec]Connection Times (ms) min avg max stdTotal: 0.1 0.9 29.8 0.4Percentage of the requests served within a certain time (ms) 50% 0.9 ms 66% 1.0 ms 75% 1.1 ms 90% 1.3 ms 95% 1.5 ms 98% 1.7 ms 99% 1.8 ms 100% 29.8 ms------------ Randomly Reading Benchmark ----------Completed 89963 of 1048576 requests, 8.6% 89957.6/s 90.5MB/sCompleted 187560 of 1048576 requests, 17.9% 97597.1/s 98.2MB/sCompleted 283486 of 1048576 requests, 27.0% 95925.8/s 96.6MB/sCompleted 382035 of 1048576 requests, 36.4% 98549.4/s 99.2MB/sCompleted 480649 of 1048576 requests, 45.8% 98613.9/s 99.3MB/sCompleted 583585 of 1048576 requests, 55.7% 102933.7/s 103.6MB/sCompleted 683954 of 1048576 requests, 65.2% 100370.9/s 101.0MB/sCompleted 782522 of 1048576 requests, 74.6% 98567.9/s 99.2MB/sCompleted 883504 of 1048576 requests, 84.3% 100982.7/s 101.7MB/sCompleted 987320 of 1048576 requests, 94.2% 103814.3/s 104.5MB/sConcurrency Level: 16Time taken for tests: 10.600 secondsComplete requests: 1048576Failed requests: 0Total transferred: 1106777459 bytesRequests per second: 98925.73 [#/sec]Transfer rate: 101969.36 [Kbytes/sec]Connection Times (ms) min avg max stdTotal: 0.0 0.1 2.3 0.1Percentage of the requests served within a certain time (ms) 50% 0.1 ms 95% 0.2 ms 98% 0.4 ms 100% 2.3 ms
- seaweedfs - SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.
- zergling - seaweedFS re-implemented in Rust.
About
seaweedfs implemented in pure Rust