Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork33.7k
Description
Bug report
Bug description:
TL;DR
_SSLProtocolTransport.is_closing should match its inner_SelectorTransport.is_closing, indicating to the user that the transport is actually closed instead of silently logging an error.
Description
I've been using the aio-libs libraryaiohttp in production together with its WebSocket client implementation, and found an interesting issue that sometimes occured on certain devices (Specifically M-series macbooks).
The logs i've been seeing looks something like this:
( Indication that we are sending messages over websocket successfully )...[2024-05-07 09:34:13,403] WARNING asyncio.write: socket.send() raised exception....( Sucessfully sent a message over websocket ) ...[2024-05-07 09:34:13,553] WARNING asyncio.write: socket.send() raised exception.( No more indication that we're sending or recieving messages over websocket )Digging deeper the issue occurs when the connection has been lost due to an exception when invokingsocket.send, this normally will result in the Transportsis_closing() function returningTrue.
The issue occurs when using TLS, which now uses the transport_SSLProtocolTransport which implements its ownis_closing logic.
When_SocketSelectorTransport.write gets an OSError such asBroken Pipe (which is the issue i've experienced in the wild) it sets its inner transport state as closed but when a library such as aiohttp checks its transportis_closing it returnsFalse leading to it silently assuming that it is still connected.
I've been able to recreate the flow by raising a different exception (by manually closing the socket) but the error source and flow is the same in both cases as far as i can tell.
Full example (out of the box + SSL cert generation)
importasyncioimportcontextlibimportloggingimportsocketimportsslimportsubprocessimporttempfile@contextlib.contextmanagerdefserver_ssl_context(host):withtempfile.NamedTemporaryFile()askeyfile,tempfile.NamedTemporaryFile()ascertfile:subprocess.run(['openssl','req','-new','-newkey','ec','-pkeyopt','ec_paramgen_curve:prime256v1','-keyout',keyfile.name,'-nodes','-x509','-days','365','-subj',f'/CN={host}','-out',certfile.name, ],check=True,shell=False)context=ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)context.load_cert_chain(certfile.name,keyfile.name)yieldcontext@contextlib.contextmanagerdefclient_ssl_context():try:context=ssl.create_default_context(ssl.Purpose.SERVER_AUTH)context.check_hostname=Falsecontext.verify_mode=ssl.CERT_NONEyieldcontextfinally:passasyncdefclient_handle(reader,writer): ...asyncdefmain(host,port):withserver_ssl_context(host)asserver_context,client_ssl_context()asclient_context:awaitasyncio.start_server(client_handle,host,port,ssl=server_context)reader,writer=awaitasyncio.open_connection(host,port,ssl=client_context)transport=writer._transportfromasyncio.sslprotoimport_SSLProtocolTransportassertisinstance(transport,_SSLProtocolTransport)inner_transport=transport._ssl_protocol._transportfromasyncio.selector_eventsimport_SelectorSocketTransportassertisinstance(inner_transport,_SelectorSocketTransport)sock:socket.socket=inner_transport._sockassertisinstance(sock,socket.socket)# Simulate a broken pipe, this invokes the OS error "Bad file descriptor"# but triggers the same flow as "Broken Pipe"sock.close()# Invoke write so socket returns an OSErrorprint('[Client] Sending x6: %r'%'Hello, world!')# Increment _conn_lost to more than 5 to trigger the logging.# This silently fails, but the logging is triggered.foriinrange(6):writer.write('Hello, world!'.encode())awaitwriter.drain()print(f"{inner_transport._conn_lost=},{transport.is_closing()=},{inner_transport.is_closing()=}")if__name__=='__main__':logging.basicConfig(level=logging.DEBUG)asyncio.run(main('localhost',8443),debug=True)
CPython versions tested on:
3.12
Operating systems tested on:
Linux, macOS
Linked PRs
- gh-118950: Fix SSLProtocol.connection_lost not being called when OSError is thrown on asyncio.write. #118960
- [3.13] gh-118950: Fix SSLProtocol.connection_lost not being called when OSError is thrown (GH-118960) #125931
- [3.12] gh-118950: Fix SSLProtocol.connection_lost not being called when OSError is thrown (GH-118960) #125932
Metadata
Metadata
Assignees
Projects
Status