Duplicate URLs in search results

[iansinnott]

It's possible for URLs to return multiple times. each exact URL should only show up once. if a URL is matched on the URL itself as well as page body those should be unified.

see #7

[nfcampos]

Another consideration here is what to do with websites whose content changes over time, eg the homepage of a blog, I think it's fine to just let the latest content replace the earlier, as if you were interested in it at the time you probably clicked into a specific link. But other folks may have other opinions

[iansinnott]

Another consideration here is what to do with websites whose content changes over time

Yeah, my thinking here is that overwriting is fine. For historic versions i'd like to add a plugin that will automatically submit URLs you've visited to the internet archive so that you can still access historical versions of the full page if desired. Likely opt-in, since it's a third party service neither the user nor BP control.