Repository files navigation

Universal data access layer for web applications.

Typically on the server, you call your API or database directly to fetch some data. However, on the client, you cannot always call your services in the same way (i.e, cross domain policies). Instead, XHR/fetch requests need to be made to the server which get forwarded to your service.

Having to write code differently for both environments is duplicative and error prone. Fetchr provides an abstraction layer over your data service calls so that you can fetch data using the same API on the server and client side.

npm install fetchr --save

Important: when on browser,Fetchr relies fully onFetch API. If you need to support old browsers, you will need to install a polyfill as well (eg.https://github.com/github/fetch).

Follow the steps below to setup Fetchr properly. This assumes you are using theExpress framework.

On the server side, add the Fetchr middleware into your express app at a custom API endpoint.

Fetchr middleware expects that you're using thebody-parser middleware (or an alternative middleware that populatesreq.body) before you use Fetchr middleware.

importexpressfrom'express';importFetcherfrom'fetchr';importbodyParserfrom'body-parser';constapp=express();// you need to use body-parser middleware before fetcher middlewareapp.use(bodyParser.json());app.use('/myCustomAPIEndpoint',Fetcher.middleware());

On the client side, it is necessary for thexhrPath option to match the path where the middleware was mounted in the previous step

xhrPath is an optional config property that allows you to customize the endpoint to your services, defaults to/api.

importFetcherfrom'fetchr';constfetcher=newFetcher({xhrPath:'/myCustomAPIEndpoint',});

You will need to register any data services that you wish to use inyour application. The interface for your service will be an objectthat must define aresource property and at least oneCRUDoperation. Theresource property will be used when you call one of theCRUD operations.

// app.jsimportFetcherfrom'fetchr';importmyDataServicefrom'./dataService';Fetcher.registerService(myDataService);

// dataService.jsexportdefault{// resource is requiredresource:'data_service',// at least one of the CRUD methods is requiredread:asyncfunction({ req, resource, params, config}){return{data:'foo'};},// other methods// create: async function({ req, resource, params, body, config }) {},// update: async function({ req, resource, params, body, config }) {},// delete: async function({ req, resource, params, config }) {}};

Data services might need access to each individual request, for example, to get the current logged in user's session.For this reason, Fetcher will have to be instantiated once per request.

On the serverside, this requires fetcher to be instantiated per request, in express middleware.On the clientside, this only needs to happen on page load.

// app.js - serverimportexpressfrom'express';importFetcherfrom'fetchr';importmyDataServicefrom'./dataService';constapp=express();// register the serviceFetcher.registerService(myDataService);// register the middlewareapp.use('/myCustomAPIEndpoint',Fetcher.middleware());app.use(function(req,res,next){// instantiated fetcher with access to req objectconstfetcher=newFetcher({xhrPath:'/myCustomAPIEndpoint',// xhrPath will be ignored on the serverside fetcher instantiationreq:req,});// perform read call to get datafetcher.read('data_service').params({id:42}).then(({ data, meta})=>{// handle data returned from data fetcher in this callback}).catch((err)=>{// handle error});});

// app.js - clientimportFetcherfrom'fetchr';constfetcher=newFetcher({xhrPath:'/myCustomAPIEndpoint',// xhrPath is REQUIRED on the clientside fetcher instantiation});fetcher.read('data_api_fetcher').params({id:42}).then(({ data, meta})=>{// handle data returned from data fetcher in this callback}).catch((err)=>{// handle errors});// for create you can use the body() method to pass datafetcher.create('data_api_create').body({some:'data'}).then(({ data, meta})=>{// handle data returned from data fetcher in this callback}).catch((err)=>{// handle errors});

See thesimple example.

Service calls on the client transparently become fetch requests.It is a good idea to set cache headers on common fetch calls.You can do so by providing a third parameter in your service's callback.If you want to look at what headers were set by the service you just called,simply inspect the third parameter in the callback.

Note: If you're using promises, the metadata will be available on themetaproperty of the resolved value.

// dataService.jsexportdefault{resource:'data_service',read:asyncfunction({ req, resource, params, config}){return{data:'response',// business logicmeta:{headers:{'cache-control':'public, max-age=3600',},statusCode:200,// You can even provide a custom statusCode for the fetch response},};},};

fetcher.read('data_service').params({id: ###}).then(({ data, meta}){// data will be 'response'// meta will have the header and statusCode from above});

There is a convenience method calledfetcher.getServiceMeta on the fetchr instance.This method will return the metadata for all the calls that have happened so farin an array format.In the server, this will include all service calls for the current request.In the client, this will include all service calls for the current session.

Usually you instantiate fetcher with some default options for the entire browser session,but there might be cases where you want to update these options later in the same session.

You can do that with theupdateOptions method:

// Startconstfetcher=newFetcher({xhrPath:'/myCustomAPIEndpoint',xhrTimeout:2000,});// Later, you may want to update the xhrTimeoutfetcher.updateOptions({xhrTimeout:4000,});

When an error occurs in your Fetchr CRUD method, you should throw an error object. The error object should contain astatusCode (default 500) andoutput property that contains a JSON serializable object which will be sent to the client.

exportdefault{resource:'FooService',read:asyncfunctioncreate(req,resource,params,configs){consterr=newError('it failed');err.statusCode=404;err.output={message:'Not found',more:'meta data'};err.meta={foo:'bar'};throwerr;},};

And in your service call:

fetcher.read('someData').params({id:'42'}).catch((err)=>{// err instanceof FetchrError -> true// err.message -> "Not found"// err.meta -> { foo: 'bar' }// err.name = 'FetchrError'// err.output -> { message: "Not found", more: "meta data" }// err.rawRequest -> { headers: {}, method: 'GET', url: '/api/someData' }// err.reason -> BAD_HTTP_STATUS | BAD_JSON | TIMEOUT | ABORT | UNKNOWN// err.statusCode -> 404// err.timeout -> 3000// err.url -> '/api/someData'});

An object with anabort method is returned when creating fetchr requests on client.This is useful if you want to abort a request before it is completed.

constreq=fetcher.read('someData').params({id:42}).catch((err)=>{// err.reason will be ABORT});req.abort();

xhrTimeout is an optional config property that allows you to set timeout (in ms) for all clientside requests, defaults to3000.On the clientside, xhrPath and xhrTimeout will be used for all requests.On the serverside, xhrPath and xhrTimeout are not needed and are ignored.

importFetcherfrom'fetchr';constfetcher=newFetcher({xhrPath:'/myCustomAPIEndpoint',xhrTimeout:4000,});

If you want to set a timeout per request you can callclientConfig with atimeout property:

fetcher.read('someData').params({id:42}).clientConfig({timeout:5000})// wait 5 seconds for this request before timing out.catch((err)=>{// err.reason will be TIMEOUT});

For some applications, there may be a situation where you need to process the service params passed in the request before they are sent to the actual service. Typically, you would process them in the service itself. However, if you need to perform processing across many services (i.e. sanitization for security), then you can use theparamsProcessor option.

paramsProcessor is a function that is passed into theFetcher.middleware method. It is passed three arguments, the request object, the serviceInfo object, and the service params object. TheparamsProcessor function can then modify the service params if needed.

Here is an example:

/**    Using the app.js from above, you can modify the Fetcher.middleware    method to pass in the paramsProcessor function. */app.use('/myCustomAPIEndpoint',Fetcher.middleware({paramsProcessor:function(req,serviceInfo,params){console.log(serviceInfo.resource,serviceInfo.operation);returnObject.assign({foo:'fillDefaultValueForFoo'},params);},}),);

For some applications, there may be a situation where you need to modify the response before it is passed to the client. Typically, you would apply your modifications in the service itself. However, if you need to modify the responses across many services (i.e. add debug information), then you can use theresponseFormatter option.

responseFormatter is a function that is passed into theFetcher.middleware method. It is passed three arguments, the request object, response object and the service response object (i.e. the data returned from your service). TheresponseFormatter function can then modify the service response to add additional information.

Take a look at the example below:

/**    Using the app.js from above, you can modify the Fetcher.middleware    method to pass in the responseFormatter function. */app.use('/myCustomAPIEndpoint',Fetcher.middleware({responseFormatter:function(req,res,data){data.debug='some debug information';returndata;},}),);

Now when an request is performed, your response will contain thedebug property added above.

Fetchr providesCORS support by allowing you to pass the full origin host intocorsPath option.

For example:

importFetcherfrom'fetchr';constfetcher=newFetcher({corsPath:'http://www.foo.com',xhrPath:'/fooProxy',});fetcher.read('service').params({foo:1}).clientConfig({cors:true});

Additionally, you can also customize how the GET URL is constructed by passing in theconstructGetUri property when you execute yourread call:

importqsfrom'qs';functioncustomConstructGetUri(uri,resource,params,config){// this refers to the Fetcher object itself that this function is invoked with.if(config.cors){returnuri+'/'+resource+'?'+qs.stringify(this.context);}// Return `falsy` value will result in `fetcher` using its internal path construction instead.}importFetcherfrom'fetchr';constfetcher=newFetcher({corsPath:'http://www.foo.com',xhrPath:'/fooProxy',});fetcher.read('service').params({foo:1}).clientConfig({cors:true,constructGetUri:customConstructGetUri,});

You can protect your Fetchr middleware paths from CSRF attacks by adding a middleware in front of it:

app.use('/myCustomAPIEndpoint', csrf(), Fetcher.middleware());

You could usehttps://github.com/expressjs/csurf for this as an example.

Next you need to make sure that the CSRF token is being sent with our requests so that they can be validated. To do this, pass the token in as a key in theoptions.context object on the client:

constfetcher=newFetcher({xhrPath:'/myCustomAPIEndpoint',// xhrPath is REQUIRED on the clientside fetcher instantiationcontext:{// These context values are persisted with client calls as query params_csrf:'Ax89D94j',},});

This_csrf will be sent in all client requests as a query parameter so that it can be validated on the server.

When calling a Fetcher service you can pass an optional config object.

When this call is made from the client, the config object is used to set some request options and can be used to override default options:

//app.js - clientconstconfig={timeout:6000,// Timeout (in ms) for each requestunsafeAllowRetry:false,// for POST requests, whether to allow retrying this post};fetcher.read('service').params({id:1}).clientConfig(config);

For requests from the server, the config object is simply passed into the service being called.

You can set Fetchr to automatically retry failed requests by specifying aretry configuration in the global or in the request configuration:

// Globallyconstfetchr=newFetchr({retry:{maxRetries:2},});// Per requestfetchr.read('service').clientConfig({retry:{maxRetries:1},});

With the above configuration, Fetchr will retry twice all requeststhat fail but only once when callingread('service').

You can further customize how the retry mechanism works. These are allsettings and their default values:

constfetchr=newFetchr({retry:{maxRetries:2,// amount of retries after the first failed requestinterval:200,// maximum interval between each request in ms (see note below)statusCodes:[0,408],// response status code that triggers a retry (see note below)},unsafeAllowRetry:false,// allow unsafe operations to be retried (see note below)}

interval

The interval between each request respects the following formula, based on the exponential backoff and full jitter strategy published inthis AWS architecture blog post:

Math.random()*Math.pow(2,attempt)*interval;

attempt is the number of the current retry attempt startingfrom 0. By defaultinterval corresponds to 200ms.

statusCodes

For historical reasons, fetchr only retries 408 responses and noresponses at all (for example, a network error, indicated by a statuscode 0). However, you might find useful to also retry on other codesas well (502, 503, 504 can be good candidates for an automaticretries).

unsafeAllowRetry

By default, Fetchr only retriesread requests. This is done forsafety reasons: reading twice an entry from a database is not as badas creating an entry twice. But if your application or resourcedoesn't need this kind of protection, you can allow retries by settingunsafeAllowRetry totrue and fetchr will retry all operations.

By Default, fetchr appends all context values to the request url as query params.contextPicker allows you to have greater control over which context variables get sent as query params depending on the request method (GET orPOST). This is useful when you want to limit the number of variables in aGET url in order not to accidentallycache bust.

contextPicker follows the same format as thepredicate parameter inlodash/pickBy with two arguments:(value, key).

constfetcher=newFetcher({context:{// These context values are persisted with client calls as query params_csrf:'Ax89D94j',device:'desktop',},contextPicker:{GET:function(value,key){// for example, if you don't enable CSRF protection for GET, you are able to ignore it with the urlif(key==='_csrf'){returnfalse;}returntrue;},// for other method e.g., POST, if you don't define the picker, it will pick the entire context object},});constfetcher=newFetcher({context:{// These context values are persisted with client calls as query params_csrf:'Ax89D94j',device:'desktop',},contextPicker:{GET:['device'],// predicate can be an array of strings},});

When calling a Fetcher service you can add custom request headers.

A request contains custom headers when you addheaders option to 'clientConfig'.

constconfig={headers:{'X-VERSION':'1.0.0',},};fetcher.read('service').params({id:1}).clientConfig(config);

All requests contain custom headers when you addheaders option to constructor arguments of 'Fetcher'.

importFetcherfrom'fetchr';constfetcher=newFetcher({headers:{'X-VERSION':'1.0.0',},});

To collect fetcher service's success/failure/latency stats, you can configurestatsCollector forFetchr. ThestatsCollector function will be invoked with one argumment:stats. Thestats object will contain the following fields:

• resource: The name of the resource for the request
• operation: The name of the operation,create|read|update|delete
• params: The params object for the resource
• statusCode: The status code of the response
• err: The error object of failed request; null if request was successful
• time: The time spent for this request, in milliseconds

importFetcherfrom'fetchr';constfetcher=newFetcher({xhrPath:'/myCustomAPIEndpoint',statsCollector:function(stats){// just console logging as a naive example.  there is a lot more you can do here,// like aggregating stats or filtering out stats you don't want to monitorconsole.log('Request for resource',stats.resource,'with',stats.operation,'returned statusCode:',stats.statusCode,' within',stats.time,'ms',);},});

app.use('/myCustomAPIEndpoint',Fetcher.middleware({statsCollector:function(stats){// just console logging as a naive example.  there is a lot more you can do here,// like aggregating stats or filtering out stats you don't want to monitorconsole.log('Request for resource',stats.resource,'with',stats.operation,'returned statusCode:',stats.statusCode,' within',stats.time,'ms',);},}),);

• Fetchr

This software is free to use under the Yahoo! Inc. BSD license.See theLICENSE file for license text and copyright information.