Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/firecrawl data source #5232

Merged
merged 50 commits into from
Jun 14, 2024
Merged

Feat/firecrawl data source #5232

merged 50 commits into from
Jun 14, 2024

Conversation

JohnJyong
Copy link
Contributor

@JohnJyong JohnJyong commented Jun 14, 2024

Description

support firecrawl to crawl website as knowledge

Fixes # (issue)

Type of Change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update, included: Dify Document
  • Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
  • Dependency upgrade

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • TODO

Suggested Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
  • optional I have made corresponding changes to the documentation
  • optional I have added tests that prove my fix is effective or that my feature works
  • optional New and existing unit tests pass locally with my changes

@JohnJyong JohnJyong requested a review from takatost June 14, 2024 15:27
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. 📚 feat:datasource Data sources like web, Notion, Logseq, Lark, Docs labels Jun 14, 2024
# Conflicts:
#	api/controllers/console/datasets/error.py
@takatost takatost mentioned this pull request Jun 14, 2024
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Jun 14, 2024
@takatost takatost merged commit ba5f8af into main Jun 14, 2024
11 checks passed
@takatost takatost deleted the feat/firecrawl-data-source branch June 14, 2024 18:46
dengpeng pushed a commit to dengpeng/dify that referenced this pull request Jun 16, 2024
Co-authored-by: Nicolas <[email protected]>
Co-authored-by: chenhe <[email protected]>
Co-authored-by: takatost <[email protected]>
Scorpion1221 added a commit to yybht155/dify that referenced this pull request Jun 26, 2024
* commit '12c815c597b121357151c798aae6580304416937': (97 commits)
  fix: ExtractSetting optional value missing None as default val (langgenius#5238)
  version to 0.6.11 (langgenius#5224)
  Feat/firecrawl data source (langgenius#5232)
  update tooltip (langgenius#5235)
  fix: note editor italic (langgenius#5230)
  fix: z-index (langgenius#5229)
  Update README.md (langgenius#5228)
  fix: allow the name and icon of the web app to be set independently of that of the bot itself (langgenius#5225)
  fix: initialize site with customized icon and icon_background (langgenius#5227)
  feat: support firecrawl frontend code (langgenius#5226)
  feat(Tools): Add Feishu multi-dimensional table operation function (langgenius#5213)
  chore: development script for syncing Poetry lockfile (langgenius#5170)
  fix: workspace member's last_active should be last_active_time, but not last_login_time (langgenius#4906)
  fix: number variable cause type error in openai moderation (langgenius#5222)
  feat: new editor user permission profile (langgenius#4435)
  Fix: http_request delete method not working (langgenius#4975)
  Update README, deploy dify with YAML file on Kubernetes (langgenius#5131)
  feat: support tencent vector db (langgenius#3568)
  fix: add repo check for build-push.yml (langgenius#5141)
  feat: Add Optional API Key, Proxy Server, and Bypass Cache Parameters to Jina Tools (langgenius#5197)
  ...

# Conflicts:
#	api/core/helper/code_executor/code_executor.py
#	api/requirements.txt
HuberyHuV1 pushed a commit to HuberyHuV1/dify that referenced this pull request Jul 22, 2024
Co-authored-by: Nicolas <[email protected]>
Co-authored-by: chenhe <[email protected]>
Co-authored-by: takatost <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
📚 feat:datasource Data sources like web, Notion, Logseq, Lark, Docs lgtm This PR has been approved by a maintainer size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants