From 82959e1fd39af4c3ae6dd264078724e93f4f2ad8 Mon Sep 17 00:00:00 2001 From: PadishahIII <83501709+PadishahIII@users.noreply.github.com> Date: Tue, 30 Apr 2024 18:22:55 +0800 Subject: [PATCH] Update README.md --- README.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index af23f43..fd72bf1 100644 --- a/README.md +++ b/README.md @@ -107,6 +107,11 @@ Options: ``` ### Advanced Usage +#### Validate the Status of Links +Use `--validate` option to check the status of found links, this helps reduce invalid links in the result. +```bash +secretscraper -u https://scrapeme.live/shop/ --validate --max-page=10 +``` #### Thorough Crawl @@ -226,8 +231,8 @@ rules: --- # Change Log -## 2024.4.29 Version 1.3.9 -- Add `--validate` option +## 2024.4.30 Version 1.3.9 +- Add `--validate` option: Validate urls after the crawler finish, which helps reduce useless links - Optimize url collector - Optimize built-in regex ## 2024.4.29 Version 1.3.8