jyapayne/langflow
1
0
Fork 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

feat: add Spider Web Scraper & Crawler (#2439)

Browse source 
* spider files

* rebuild required

* add spider-client here

* Feat: Spider Web Crawler & Scraper

* Feat: spider integration

* new input not working

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

* fix: add outputs and configure build method

* style: run ruff

* Refactor SpiderTool to use 'crawl' instead of 'build' for generating Markdown content

* chore: add type ignore

* chore: new lock

* chore: Update mem0ai dependency to version 0.0.5

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@langflow.org>
This commit is contained in:
William Espegren 2024-08-08 14:53:05 +02:00 • committed by GitHub
commit 7a36cc9ebf
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
12 changed files with 2580 additions and 68 deletions
Download patch file Download diff file Expand all files Collapse all files

4
src/backend/base/pyproject.toml
View file

@ -77,6 +77,10 @@ setuptools = ">=70"
nanoid = "^2.0.0"
filelock = "^3.15.4"
grandalf = "^0.8.0"
crewai = "^0.36.0"
spider-client = "^0.0.27"
[tool.poetry.extras]
deploy = ["celery", "redis", "flower"]
local = ["llama-cpp-python", "sentence-transformers", "ctransformers"]