• Version: v10.15.0
• Platform: Ubuntu 16, Win 10. Haven't tested macOS
• Subsystem: fs

I'm seeing a 7.6-13.5x drop in read throughput between 8.x and 10.x in boththe readfile benchmark and our real-world benchmarks that heavily exercisefs.readFile. Based on my troubleshooting, I think it's from#17054.

The readfile benchmark (Ubuntu 16):

From what I can extract from the comments in#17054, eitherno degradation ora 3.6-4.8x degradation was expected for the len=16M cases.

As for why I think it's because of#17054, the benchmark below comparesfs.readFile against an approximation of howfs.readFile used to work (one-shot read), measuring time to read the same 16 MB file 50 times.

// npm i asyncconstfs=require("fs");constasync=require("async");functionchunked(filename,cb){fs.readFile(filename,cb);}functiononeshot(filename,cb){fs.open(filename,"r",0o666,(err,fd)=>{if(err)returncb(err);constonerr=err=>fs.close(fd,()=>cb(err));fs.fstat(fd,(err,stats)=>{if(err)returnonerr(err);constdata=Buffer.allocUnsafe(stats.size);fs.read(fd,data,0,stats.size,0,(err,bytesRead)=>{if(err)returnonerr(err);if(bytesRead!==stats.size){consterr=newError("Read fewer bytes than requested");returnonerr(err);}fs.close(fd,err=>cb(err,data));});});});}fs.writeFileSync("./test.dat",Buffer.alloc(16e6,'x'));functionbm(method,name,cb){conststart=Date.now();async.timesSeries(50,(n,next)=>{method("./test.dat",next);},err=>{if(err)returncb(err);constdiff=Date.now()-start;console.log(name,diff);cb();});}async.series([cb=>bm(chunked,"fs.readFile()",cb),cb=>bm(oneshot,"oneshot",cb)])

We've switched tofs.fstat() and thenfs.read() as a work-around (above; note how many lines of code it is), but I wouldn't be surprised if this has negatively impacted other apps/tools. As far as the original justification: I'm not convinced that was an appropriate fix or a problem that needed fixing. Web servers (where DoS attacks are relevant) should generally be usingfs.createReadStream.pipe(response) for serving files (ignoring other use cases for now). Other sorts of apps like build tools, compilers and test frameworks (DoS attacks irrelevant) are the ones that more often need to read an entire file as fast as possible. In some senses, the fix made the DoS situation worse by increasing overhead (reducing number of useful CPU cycles) and by infinitely partitioning reads (each new request delaysall existing requests -- the requests are waiting for their own ticks plus some number of ticks from all of the other requests). We built our web app to load-shed while maintaining quality of service for as many clients as possible, so interleaving is not what we want. -- Happy to discuss that more on- or off-line.

Is anyone else able to verify that this degradation exists and/or was expected?

cc:@davisjam