Cannot close SSL/TLS connection of "Aggressive" keep-alive?

[okay1204]

Hi, I'm trying to fetch this specific webpage programmatically: https://health.usnews.com/best-nursing-homes/search?sort=name-asc&page=1

The webserver is forcing an SSL/ TLS connection, so its response never finishes. Here's my code:

import aiohttp
import asyncio

MAX_PAGE = 1541
US_HEALTH_URL = 'https://health.usnews.com/best-nursing-homes/search?sort=name-asc&page='
HEADERS = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36',
    'cookie': '',
    'connection': 'close'
}

async def handleUSHealthPage(session: aiohttp.ClientSession, url: str):
    print(url)
    async with session.get(url, headers = HEADERS) as response:
        print(await response.text())

async def main():
    async with aiohttp.ClientSession() as session:
        await handleUSHealthPage(session, US_HEALTH_URL + '1')


asyncio.get_event_loop().run_until_complete(main())

The issue is that the program is halted indefinitely at this line: async with session.get(url, headers = HEADERS) as response: I've done some additional test requests to this site using the requests module, and that actually seemed to work, returning the webpage HTML. How do I go about fetching this in aiohttp?

[Dreamsorcerer]

I'm not sure why you think this has something to do TLS, but I get the same result with both http and https. The server appears to not be sending anything or closing the connection. If I use curl, I get a 403 forbidden page, whereas it works fine in my browser. So, this server is obviously doing something dodgy.

I don't know what the server is doing, but maybe you need to change a bunch of different headers in order to match what the browser (or requests) is sending, before the server will send a response correctly?

[okay1204]

If you include this header: user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36 it does not return a 403 forbidden page, but rather initiates a TLS connection. You can check this using the -v argument. Here's the output:

curl -v https://health.usnews.com/best-nursing-homes/search?sort=name-asc&page=1 -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36'
*   Trying 104.100.85.232...
* TCP_NODELAY set
* Connected to health.usnews.com (104.100.85.232) port 443 (#0)
* schannel: SSL/TLS connection with health.usnews.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 188 bytes...
* schannel: sent initial handshake data: sent 188 bytes
* schannel: SSL/TLS connection with health.usnews.com port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with health.usnews.com port 443 (step 2/3)
* schannel: encrypted data got 4096
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: encrypted data length: 4022
* schannel: encrypted data buffer: offset 4022 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with health.usnews.com port 443 (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 5046 length 5046
* schannel: encrypted data length: 93
* schannel: encrypted data buffer: offset 93 length 5046
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with health.usnews.com port 443 (step 2/3)
* schannel: encrypted data got 738
* schannel: encrypted data buffer: offset 831 length 5046
* schannel: sending next handshake data: sending 126 bytes...
* schannel: SSL/TLS connection with health.usnews.com port 443 (step 2/3)
* schannel: encrypted data got 258
* schannel: encrypted data buffer: offset 258 length 5046
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with health.usnews.com port 443 (step 3/3)
* schannel: stored credential handle in session cache
> GET /best-nursing-homes/search?sort=name-asc HTTP/1.1
> Host: health.usnews.com
> User-Agent: curl/7.55.1
> Accept: */*

I'm confused on how requests.get actually does work with the same headers.

[okay1204]

It turns out the SSL/TLS was just the protocol for HTTPS and requests.get was sending some other headers by default:

{
    'Accept-Encoding': 'gzip, deflate',
    'Accept': '*/*',
    'Connection': 'keep-alive'
}

I can fetch data from the website now- sorry for wasting your time.