Repository files navigation

Anabstract-leveldown compliant implementation ofLevelDOWN that usesAmazon S3 as a backing store. S3 is actually a giant key-value store on the cloud, even though it is marketed as a file store. Use this database with theLevelUP API.

To use this optimally, please read "Performance considerations" and "Warning about concurrency" sections below.

You could also use this as an alternative API to read/write S3. The API simpler to use when compared to the AWS SDK!

Installs3leveldown and peer dependencieslevelup and@aws-sdk/client-s3 withyarn ornpm.

$ npm install s3leveldown @aws-sdk/client-s3 levelup

See theLevelUP API for high level usage.

Constructor ofs3leveldown backing store. Use withlevelup.

Arguments:

• location name of the S3 bucket with optional sub-folder. Examplemybucket ormybucket/folder.
• s3 OptionalS3 client fromaws-sdk. A default client will be used if not specified.

Please refer to theAWS SDK docs to set up your API credentials before using.

constlevelup=require('levelup');constS3LevelDown=require('s3leveldown');(async()=>{// create DBconstdb=levelup(newS3LevelDown('mybucket'));// put itemsawaitdb.batch().put('name','Pikachu').put('dob','February 27, 1996').put('occupation','Pokemon').write();// read itemsawaitdb.createReadStream().on('data',data=>{console.log('data',`${data.key.toString()}=${data.value.toString()}`);}).on('close',()=>{console.log('done!')});})();

constlevelup=require('levelup');constS3LevelDown=require('s3leveldown');constdb=levelup(newS3LevelDown('my_bucket'));db.batch().put('name','Pikachu').put('dob','February 27, 1996').put('occupation','Pokemon').write(function(){db.readStream().on('data',console.log).on('close',function(){console.log('Pika pi!')})});

You could also use s3leveldown with S3 compatible servers such asMinIO.

constlevelup=require('levelup');constS3LevelDown=require('s3leveldown');constAWS=require('aws-sdk');consts3=newAWS.S3({apiVersion:'2006-03-01',accessKeyId:'YOUR-ACCESSKEYID',secretAccessKey:'YOUR-SECRETACCESSKEY',endpoint:'http://127.0.0.1:9000',s3ForcePathStyle:true,signatureVersion:'v4'});constdb=levelup(newS3LevelDown('my_bucket',s3));

You can create your Level DB in a sub-folder in your S3 bucket, just usemy_bucket/sub_folder when passing the location.

There are a few performance caveats due to the limited API provided by the AWS S3 API:

• When iterating, getting values is expensive. A seperate S3 API call is made to get the value of each key. If you don't need the value, pass{ values: false } in the options. Each S3 API call can return 1000 keys, so if there are 3000 results, 3 calls are made to list the keys, and if getting values as well, another 3000 API calls are made.

• Avoid iterating large datasets when passing{ reverse: true }. Since the S3 API call do not allow retrieving keys in reverse order, the entire result set needs to be stored in memory and reversed. If your database is large ( >5k keys ), be sure to provide start (gt,gte) and end (lt,lte), or the entire database will need to be fetched.

• By default when iterating, 1000 keys will be returned. If you only want 10 keys for example, set{ limit: 10 } and the S3 API call will only request 10 keys. Note that if you have{ reverse: true }, this optimisation does not apply as we need to fetch everything from start to end and reverse it in memory. To override the default number of keys to return in a single API call, you can set the s3ListObjectMaxKeys option when creating the iterator. The maximum accepted by the S3 API is 1000.

• Specify the AWS region of the bucket to improve performance, by callingAWS.config.update({ region: 'ap-southeast-2' }); replaceap-southeast-2 with your region.

Individual operations (putgetdel) are atomic as guaranteed by S3, but the implementation ofbatch is not atomic. Two concurrent batch calls will have their operations interwoven. Don't use any plugins which require this to be atomic or you will end up with your database corrupted! However, if you can guarantee that only one process will write the S3 bucket at a time, then this should not be an issue. Ideally, you want to avoid race conditions where two processes are writing to the same key at the same time. In those cases the last write wins.

Iterator snapshots are not supported. When iterating through a list of keys and values, you may get the changes, similar to dirty reads.

S3LevelDown usesdebug. To see debug message set the environment variableDEBUG=S3LevelDown.

To run the test suite, you need to set a S3 bucket to the environment variableS3_TEST_BUCKET. Also be sure toset your AWS credentials

$ S3_TEST_BUCKET=my-test-bucket npm runtest

MIT