Repository files navigation

Minimal, idiomatic, stream-based Scala interface for key/value store implementations.It provides abstractions for S3-like key/value store backed by different persistencemechanisms (i.e. S3, FileSystem, sftp, etc).

fs2-blobstore is deployed to maven central, add to build.sbt:

libraryDependencies ++= Seq(  "com.lendup.fs2-blobstore" %% "core" % "0.6.+",  "com.lendup.fs2-blobstore" %% "sftp" % "0.6.+",  "com.lendup.fs2-blobstore" %% "s3" % "0.6.+")

core module has minimal dependencies and only providesFileStore implementation.sftp module providesSftpStore and depends onJsch client.s3 module providesS3Store and depends onAWS S3 SDK

The goal of theStore interface is tohave a common representation of key/value functionality (get, put, list, etc) asstreams that can be composed, transformed and piped just like any otherfs2.Streamorfs2.Sink regardless of the underlying storage mechanism.

This is especially useful for unit testing if you are building a S3 or SFTP backedsystem as you can provide a filesystem based implementation for tests that isguaranteed to work the same way as you production environment.

The three main activities in a key/value store are modeled like:

deflist(path:Path): fs2.Stream[F,Path]defget(path:Path,chunkSize:Int): fs2.Stream[F,Byte]defput(path:Path,contentLength:Long): fs2.Sink[F,Byte]

Note thatlist andget are modeled as streams since they are reading(potentially) very large amounts of data from storage, whileput isrepresented as a sink of byte so that any stream of bytes can by pipedinto it to upload data to storage.

import blobstore.implicits._

StoreOps andPathOps provide functionality onbothStore andPath for commonly performed tasks (i.e. upload/download afile from/to local filesystem, collect all returned paths when listing, composingpaths or extracting filename of the path).

Most of these common tasks encapsulate stream manipulation and provide a simplerinterface that return the corresponding effect monad. These are also very goodexamples of how to use blobstore streams and sink in different scenarios.

All store implementations must support and pass the suite of tests inAbstractStoreTest.It is expected that each store implementation (like s3, sftp, file) shouldcontain theStore implementation and at least one test suite that inheritsfromAbstractStoreTest and overrides store and root attributes:

classMyStoreImplTestextends blobstore.AbstractStoreTest {overridevalstore: blobstore.Store[cats.effect.IO]=MyStoreImpl( ... )overridevalroot:String="my_store_impl_tests"}

This test suite will guarantee that basic operations are supported properly andconsistent with all otherStore implementations.

Running Tests:

Tests are set up to run via docker-compose:

docker-compose run --rm sbt"testOnly * -- -l blobstore.IntegrationTest"

This will start aminio (Amazon S3 compatibleobject storage server) and SFTP containers and run all tests not annotated as@IntegrationTest.

Yes, we understandSftpStoreTest andS3StoreTest are alsointegration testsbecause they connect to external services, but we don't mark them as such becausewe found these containers that allow to run them along unit tests and we want toexercise as much of the store code as possible.

Currently, tests forSftpStore andBoxStore are annotated with@IntegrationTestbecause: (1) SFTP tests fail to run against sftp container in travis, and (2) wehave not found a box docker image. To runBoxStore integration tests locallyyou need to provide env vars forBOX_TEST_BOX_DEV_TOKEN andBOX_TEST_ROOT_FOLDER_ID.

Run box/sftp tests with:

sbt box/testdocker-compose run --rm sbt sftp/test

Note: this will exerciseAbstractStoreTest tests against your box.com account.

blobstore.Path is the representation ofkey in the key/value store. The keyrepresentation is based on S3 that has aroot (or bucket) and akey string.

When functions in theStore interface that receive aPath should assume that onlyroot and key values are set, there is no guarantee that the other attributes ofPathwould be filled: size, isDir, lastModified. On the other hand, when aStore implementsthe list function, it is expected that all 3 fields will be present in the response.

By importing implicitPathOps into the scope you can make use of path composition/andfilename function that returns the substring of the path's key after the last pathseparator.

NOTE: a good improvement to the path abstraction would be to handle OS specificseparators when referring to filesystem paths.

• FileStore backed by localFileSystem. FileStore is provided as part of core module because it doesn'tinclude any additional dependencies and it is used as the default source storein TransferOps tests. It only requires root path in the local file system:

  importblobstore.Store,blobstore.fs.FileStoreimportjava.nio.file.Pathsimportcats.effect.IOvalstore:Store[IO]=FileStore[IO](Paths.get("tmp/"))

• S3Store backed byAWS S3.It requires an authenticatedAmazonS3 client:

  importblobstore.Store,blobstore.s3.S3Storeimportcom.amazonaws.services.s3.transfer.TransferManagerBuilderimportcats.effect.IOvalstore:Store[IO]=S3Store[IO](TransferManagerBuilder.standard().build())

• SftpStore backed bySFTP server withJsch client. It requires aconnectedChannelSftp:

  importblobstore.Store,blobstore.sftp.SftpStoreimportcom.jcraft.jsch.{ChannelSftp,JSch}importcats.effect.IOvaljsch=newJSch()valsession= jsch.getSession("sftp.domain.com")session.connect()valchannel= session.openChannel("sftp").asInstanceOf[ChannelSftp]channel.connect(5000)valstore:Store[IO]=SftpStore("root/server/path", channel)

• BoxStore backed byaBoxAPIConnection,which has multiple options for authentication. This requires that you have a Box app set up already.SeeBox SDK documentation for more details:

  importblobstore.Store,blobstore.box.BoxStoreimportcom.box.sdk.BoxAPIConnectionvalapi=newBoxAPIConnection("myDeveloperToken")valstore:Store[IO]=BoxStore[IO](api,"rootFolderId")