Basic HTTPServer stuck in “socket.readinto : return self._sock.recv_into(b)”
I
have a very basic implementation of http.server.HTTPServer that I’m starting in Python 3.6. I’m using it to authenticate OAuth2 against the Google Analytics API. By standard, I created a simple handle to receive the OAuth2 authorization key and used the serve_forever() method. However, the thread used to create the service hangs forever, and it seems to be stuck deep in the Python 3.6 implementation.
Interestingly, in this case, I don’t think my code matters at all. See the call stack reported by the listener I set up for debugging:
service_action started
service_action complete
service_action started
service_action complete
service_action started
service_action complete
service_action started
service_action complete
"socketserver. BaseServer.serve_forever : self._handle_request_noblock()"
"socketserver. BaseServer._handle_request_noblock : self.get_request()"
"socketserver. BaseServer._handle_noblock_request : self.process_request()"
"socketserver. BaseServer.process_request : self.finish_request()"
"socketserver. BaseServer.finish_request : self. RequestHandlerClass(request, client_address, self) # request == <socket.socket fd=1828, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1' , 8080), raddr=('127.0.0.1', 57504)>, client_address == ('127.0.0.1', 57504), self == <MDGCoreLib.Utilities.HttpServer.Server.Server object at 0x0000026304E3FC88>"
"socketserver. BaseRequestHandler.__init__ : set self.client_address"
"socketserver. BaseRequestHandler.__init__ : set self.server"
"socketserver. BaseRequestHandler.__init__ : setup()"
"socketserver. StreamRequestHandler.setup : self.connection = self.request"
"socketserver. BaseRequestHandler.__init__ : self.handle()"
"server. BaseHttpRequestHandler.handle : self.close_connection = True"
"server. BaseHTTPServer.handle_one_request : self.raw_requestline = self.rfile.readline(65537)"
"socket.readinto : self._checkClosed()"
"socket.readinto : return self._sock.recv_into(b) #b == <memory at 0x0000026304E38C48>"
"server. BaseHTTPHandler.handle_one_request : if len(self.raw_requestline) > 65536"
"server. BaseHTTPHandler.handle_one_request : if not self.raw_requestline"
"server. BaseHTTPHandler.handle_one_request : if not self.parse_request()"
"server. BaseHTTPHandler.handle_one_request : if not hassattr(self, mname)"
#My handler runs
*"OAuthGrantRequestHandler : self.send_response"
127.0.0.1 - - [27/Aug/2018 14:35:16] "GET /? (REDACTED) HTTP/1.1" 200 -
Request handler completed*
#My handler finishes
Parent thread : joining Thread 2
"server. BaseHTTPHandler.handle_one_request : flushing wfile to socket finishing request"
"socketserver. BaseRequestHandler.__init__ : self.finish()"
"socketserver. StreamRequestHandler.finish : "
"socketserver. BaseServer.process_request : self.shutdown_request"
service_action started
service_action complete
"socketserver. BaseServer.serve_forever : self._handle_request_noblock()"
"socketserver. BaseServer._handle_request_noblock : self.get_request()"
"socketserver. BaseServer._handle_noblock_request : self.process_request()"
"socketserver. BaseServer.process_request : self.finish_request()"
"socketserver. BaseServer.finish_request : self. RequestHandlerClass(request, client_address, self) # request == <socket.socket fd=1840, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1' , 8080), raddr=('127.0.0.1', 57505)>, client_address == ('127.0.0.1', 57505), self == <MDGCoreLib.Utilities.HttpServer.Server.Server object at 0x0000026304E3FC88>"
"socketserver. BaseRequestHandler.__init__ : set self.client_address"
"socketserver. BaseRequestHandler.__init__ : set self.server"
"socketserver. BaseRequestHandler.__init__ : setup()"
"socketserver. StreamRequestHandler.setup : self.connection = self.request"
"socketserver. BaseRequestHandler.__init__ : self.handle()"
"server. BaseHttpRequestHandler.handle : self.close_connection = True"
"server. BaseHTTPServer.handle_one_request : self.raw_requestline = self.rfile.readline(65537)"
"socket.readinto : self._checkClosed()"
"socket.readinto : return self._sock.recv_into(b) #b == <memory at 0x0000026304E38C48>"
"server. BaseHTTPHandler.handle_one_request : if len(self.raw_requestline) > 65536"
"server. BaseHTTPHandler.handle_one_request : if not self.raw_requestline"
"server. BaseHTTPHandler.handle_one_request : if not self.parse_request()"
"server. BaseHTTPHandler.handle_one_request : if not hassattr(self, mname)"
#My Handler Runs
"OAuthGrantRequestHandler : self.send_response"
127.0.0.1 - - [27/Aug/2018 14:35:18] "GET /favicon.ico HTTP/1.1" 200 -
Request handler completed
#My Handler Finishes
"server. BaseHTTPHandler.handle_one_request : flushing wfile to socket finishing request"
"socketserver. BaseRequestHandler.__init__ : self.finish()"
"socketserver. StreamRequestHandler.finish : "
"socketserver. BaseServer.process_request : self.shutdown_request"
service_action started
service_action complete
"socketserver. BaseServer.serve_forever : self._handle_request_noblock()"
"socketserver. BaseServer._handle_request_noblock : self.get_request()"
"socketserver. BaseServer._handle_noblock_request : self.process_request()"
"socketserver. BaseServer.process_request : self.finish_request()"
"socketserver. BaseServer.finish_request : self. RequestHandlerClass(request, client_address, self) # request == <socket.socket fd=2008, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1' , 8080), raddr=('127.0.0.1', 57506)>, client_address == ('127.0.0.1', 57506), self == <MDGCoreLib.Utilities.HttpServer.Server.Server object at 0x0000026304E3FC88>"
"socketserver. BaseRequestHandler.__init__ : set self.client_address"
"socketserver. BaseRequestHandler.__init__ : set self.server"
"socketserver. BaseRequestHandler.__init__ : setup()"
"socketserver. StreamRequestHandler.setup : self.connection = self.request"
"socketserver. BaseRequestHandler.__init__ : self.handle()"
"server. BaseHttpRequestHandler.handle : self.close_connection = True"
"server. BaseHTTPServer.handle_one_request : self.raw_requestline = self.rfile.readline(65537)"
"socket.readinto : self._checkClosed()"
"socket.readinto : return self._sock.recv_into(b) #b == <memory at 0x0000026304E38C48>"
The program '[54321] Python @ tcp://localhost:55098/?legacyUnitTest' has exited with code -1 (0xffffffff).
Above is all the contents of the thread running the “serve_forever()” method, you can see that the parent thread is waiting to join, where the output is Parent thread : joining Thread 2. When the output is “OAuthGrantRequestHandler: self.send_response" and
Request handler completed: But internally the python library is never called again until socket.readin: return self._sock.recv_into(b) while attempting to process the last request.
Anyone has had this Python hang up issue before, is there any problem around it? I’m a bit overwhelmed because it’s stuck between server_forever() loops that encounter my handler/code.
Update 1
So, comparing the fiddler trace to the request in HTTPServer, I found that there were only two HTTPRequests, but HTTPServer.serve_forever() tried to read the third request, which is where it hangs. I also verified (you can see in the output above) that the “shutdown_request()” method and the underlying method are being called to close the socket. So why does the serve_forever loop continue to pass the check below and call self._handle_request_noblock(): when no new request is received
ready = selector.select(poll_interval)
if ready:
self._handle_request_noblock()
Solution
There are pretty much the same problems, so in the deep dive into socketserver. After the StreamRequestHandler implementation, I discovered how to set a timeout so that those bad read attempts would eventually unblock.
class MyRequestHandler(BaseHTTPRequestHandler):
timeout = 2 # This is what makes the difference (seconds)
def do_GET(self):
...