since this is a cleaning code, what about doing something like that instead?
$encoding = mb_detect_encoding($filename, null, true) ?: '8bit';

This would indeed help with generating the filenameFallback no matter how weird the $filename is. But consideringResponseHeaderbag::makeDisposition the$filename parameter is described and expected to beA unicode string and placed in the header field parameterfilename* defined as utf-8 encoded without further processing. If passing e.g. an ISO-8859-1 encoded $filename with german umlauts and not throwing this exception would lead to a bad or invalid Content-Disposition response header. MaybesetContentDisposition should actually ensure the $filename to be utf-8 encoded in any case. But this could be a BC break, although I think everything but utf-8 seems to be a misusage. What do you think?

I think we have no need to be stricter now than we were already. If people put non-utf8 string, that's going to be sent and displayed in some way by browsers, so there is already a "reporting" effect.
throwing an exception would just break code that work "fine" now. So, still on the "8bit" side :)

Should we use this error message instead?

thrownew \InvalidArgumentException(sprintf('The "%s" filename encoding cannot be detected.',$filename));

When$filename is not a "clean" utf-8 string, but have a broken encoding, this could break the output wherever the exception message will be used. Actually we had an exception from the mongodb related code not showing up in the first place, cause its message included the broken non utf-8 encoded filename which in turn caused an Exception in the ExceptionListener, while trying to generate a JSON response with an invalid utf-8 string. So I suggest not to include the filename here.