I agree, you can make App.static using parameter route and sendFile

yeah should probably use a stream like in the examplehere, but seem to work for my use case (just a smaller js files) for testing locally (do you know the size limit?), but i agree you should probably use nginx or similar in production.

if you have small files and they are immutable, then when the server starts, scan the folder with the files, save all the files in the local JS cache, and always give the respond from the JS cache, you don’t even need promises

good idea - the "local JS cache" is just a map in memory you define i guess

i added something like thathere :)

forfileCache useMap anddata.toString() no need to do this, save binary buffer and compress it with gzip if it is text files

still you can always have cloudflare proxy, and setup it in 10 minutes from scratch. Though you wouldn't need to change you approach on server.
My aim is to have this option for those who need it. I'm lazy and I'm trying to have nice monolith to maintain everything on one server. GDPR and cookie files etc. why should I setup S3 for files or other http serving place for this or for some static html.

uWS will probably get a cache at some point. Maybe as a paid feature (along with other features). With a cache, all file serving routes could be shitty but still fast in hot path

too soon to be paid. You need to get big players first.

I'm kinda surprised this is still going.

I don't want to have to make a wildcard and do file lookups. I just want a simple thing like express, where I just do .static(route)

Ok. Then use Express. It really is that simple.

uWS does not have any plan to add app.static unless a strong motivation comes along. And that motivation is not going to be you, writing that you are surprised nothing happened.

I already know you guys aren't going to do that. I'm just amused that this is still going on after you let us know that you guys aren't doing it. More like the thread should just be closed and we add an note in an FAQ about it.

It could be done at some point, it's a valid idea.

But it isn't exactly simple to do, if you want it well done. uWS does not add half-assed solutions. You need sendfile and kTLS to make it anywhere near the best. kTLS is a major feature.

I hear you say; "but I don't care about the best solution I just want it to be simple", ok, well again then use Express. Express definitely is simpler than uWS.

Oops, that was a stupid question, I was able to cause backpressure, the problem was in my proxy server.

Add this to your code:

Memory tracking in GC environments is quite difficult. Maybe it's not uWS but a nodejs error, or maybe something is happening in v8, GC or something else. You probably need some special solutions for analyzing this, but what’s the point of all this, fuck it.

@Msfrhdsa That (your variant) function is not correct and will send corrupt data to end users. You haven't tested it enough to stress the case where tryEnd consumed part of the chunk.

Have you considered just asking ChatGPT if you don't understand? I asked it, and it clearly understands it:

ThepipeStreamOverResponse function is a helper function designed to pipe data from a readable stream (readStream) over an HTTP response (res). This function is typically used in web servers to send large files or data streams to the client efficiently. Here's a breakdown of the function:

Parameters

• res: The HTTP response object, which is used to send data back to the client.
• readStream: A Node.jsReadableStream from which data is read.
• totalSize: The total size of the data being streamed. This is used to indicate the progress of the streaming process.

Process

1. Handling Data Events:

   • The function listens fordata events on thereadStream. When data chunks are available, they are processed one by one.
   • Each chunk of data is converted to an ArrayBuffer using thetoArrayBuffer(chunk) function (not defined in the provided code, but presumably converts the chunk to a format suitable for sending over the response).
   • The current write offset in the response is stored usingres.getWriteOffset().

2. Sending Data:

   • ThetryEnd method on theres object attempts to send the data chunk to the client. It returns two values:
     • ok: Indicates whether the chunk was successfully sent.
     • done: Indicates whether this chunk was the last one to send.
   • Ifdone istrue, the function callsonAbortedOrFinishedResponse to handle the end of the response (this function is not defined in the provided code but typically handles cleanup and completion).

3. Handling Flow Control:

   • If a chunk cannot be sent immediately (ok isfalse), thereadStream is paused to prevent reading more data.
   • The chunk and its offset are saved in theres object for later use.
   • Theres.onWritable event is used to listen for when the response can accept more data. When triggered, the function attempts to send the remaining data starting from the last unsent position.
   • TheonWritable callback must returntrue orfalse to indicate whether the data was successfully sent or not.

4. Error Handling:

   • If an error occurs in thereadStream, a simple error message is logged. The function suggests that error handling should be implemented here, such as closing the response or other cleanup actions.

5. Handling Aborted Responses:

   • The function sets up anonAborted handler on theres object. This handler will be called if the response is aborted (e.g., if the client disconnects), ensuring proper cleanup via theonAbortedOrFinishedResponse function.

Key Considerations

• Flow Control: This function carefully manages the flow of data to avoid overwhelming the client or the network. It uses mechanisms like pausing the read stream and handling backpressure throughonWritable.
• Error and Abort Handling: Although error handling is not fully implemented, there are placeholders indicating where such logic should be added.
• Performance Considerations: The function is designed to handle streaming efficiently, even with large data sizes, by sending data in chunks and managing the connection state.

Overall, the function is a robust solution for streaming data over HTTP in a Node.js environment, ensuring efficient data transfer and handling potential issues like backpressure and client disconnects.

why there is an offset in onWritable in VideoStreamer.js. It is always equal to res.getWriteOffset() and res.abOffset

No, it isn't. They certainly can be different, and will be different in production stress.

ChatGPT telling you why (you should really ask ChatGPT instead of having confident outbursts here)

Why Offsets Are Used

1. Handling Partial Writes:

   • When a chunk of data is sent usingres.tryEnd(ab, totalSize), the function might not be able to send the entire chunk in one go due to network conditions or limitations on the client's side. This can result in partial writes.
   • Theoffset in theonWritable callback represents the position in the current chunk where the next write should begin. This is essential to ensure that any partially sent data is not resent, which would lead to duplication and errors.

2. Backpressure Management:

   • Backpressure occurs when the rate of incoming data (from the readable stream) exceeds the rate at which the data can be sent out (through the response). To manage this, thereadStream is paused when thetryEnd method returnsok asfalse.
   • Theres.getWriteOffset() method returns the current write offset, which indicates how much of the response has been sent. This helps in determining how much data has been successfully written so far.
   • Theres.abOffset stores the offset at which the current chunk (res.ab) was first attempted to be written. Together withres.getWriteOffset(), it helps in calculating the correct slice of the data to resend.

3. Correct Data Handling:

   • InonWritable, the function checks the difference betweenres.getWriteOffset() andres.abOffset. This difference tells us how much of the previously stored buffer has been sent.
   • The data chunk is then sliced starting from this offset and passed again totryEnd. This ensures that only the unsent portion of the data is retried, maintaining data integrity.

Why It Might Seem Redundant

It might seem that using bothres.getWriteOffset() andres.abOffset is redundant since they are typically expected to represent the same point in the stream if everything is working perfectly. However, these offsets are crucial for the following reasons:

• Edge Cases: There could be situations where the response might not be able to send all the data in one go, especially under high load or network fluctuations. The offsets ensure that any data that couldn't be sent due to such conditions is correctly handled.
• Stream Pauses and Resumes: When thereadStream is paused, theonWritable event provides a mechanism to resume the stream once the response can accept more data. Offsets help in ensuring that the data flow resumes correctly from the last unsent byte.
• Data Integrity: By keeping track of bothres.getWriteOffset() andres.abOffset, the function ensures that it neither skips any data nor sends any data twice.

In summary, the use of offsets in theonWritable callback is a safeguard to handle partial writes, backpressure, and network reliability issues, ensuring smooth and correct data streaming to the client.