Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Response and other context is not passed to nested loader #34

Open
migr1 opened this issue Mar 8, 2018 · 0 comments
Open

Response and other context is not passed to nested loader #34

migr1 opened this issue Mar 8, 2018 · 0 comments
Labels
enhancement New feature or request

Comments

@migr1
Copy link

migr1 commented Mar 8, 2018

When passing the response to an item loader:

loader = ItemLoader(item=Product(), response=response)

The response can be used in an input processor via the loader_context param:

def make_absolute_url(url, loader_context):
    return loader_context['response'].urljoin(url)

However, when using a nested loader:

loader = ItemLoader(item=Product(), response=response)
nested_loader = loader.nested_xpath('...')

The input processor fails with the exception:

  File "C:\migr\crawlers\crawlers\items.py", line 16, in make_absolute_url
    return loader_context['response'].urljoin(url)
AttributeError: 'NoneType' object has no attribute 'urljoin'

response is not passed to the input processor and I believe the reason is https://github.com/scrapy/scrapy/blob/acd2b8d43b5ebec7ffd364b6f335427041a0b98d/scrapy/loader/__init__.py#L55
where a new context is created without reusing any of the current context.

To me, this is unexpected behavior. Since the loader is nested, I presumed that all context except the selector would be preserved unless explicitly overwritten in the call to nested_xpath. But maybe there is an explanation for not passing response and/or other context on to the nested loader. I can file a pull request if you agree that the behavior should be changed.

@Gallaecio Gallaecio transferred this issue from scrapy/scrapy Oct 30, 2020
@Gallaecio Gallaecio added the enhancement New feature or request label Feb 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants