AttributeError: Response content isn't text in "response_is_ban"

[iamumairayub]

I have created this policy

class MyBanPolicy(BanDetectionPolicy):
    def response_is_ban(self, request, response):
        # use default rules, but also consider HTTP 200 responses
        ban = super(MyBanPolicy, self).response_is_ban(request, response)

        if b'tests' in response.body:
            logging.info('test found')
            ban = True

        return ban

But my check doesnt work because response is I think gzipped

I tried response.css("my selector").get() but I get AttributeError: Response content isn't text error

How can I check existence of an element using response.css() in response_is_ban method?

Do I need to first unzip the response each time? that does not look suitable

PS: I am not sending gzip deflate header to target website at all

Python 3.6.9
Scrapy 2.3.0

[fschn90]

have you found a solution or workaround for this?

im trying something similar:

class BanPolicy(BanDetectionPolicy):
    def response_is_ban(self, request, response):
        port = response.meta.get('proxy')[-4:]

but am also getting an error:

return self.request.meta
AttributeError: 'NoneType' object has no attribute 'meta'

AttributeError: Response.meta not available, this response is not tied to any request

thanks in advance!
Best, Flo

Python 3.10.6
Scrapy 2.8.0