jyapayne/langflow
1
0
Fork 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

feat: enhance URL component with recursive crawling (#7006)

Browse source 
* feat: Enhance URL component with recursive crawling and advanced options.

- Implement advanced URL loading with configurable depth, domain prevention, and async options.

* chore: move component to a new file and restore the old one

* test: add test to new component

* [autofix.ci] apply automated fixes

* feat: enhance URLComponent for recursive URL loading

- Updated URLComponent to support recursive loading and parsing of child links from a root URL.
- Introduced new inputs: max_depth for controlling link traversal depth and prevent_outside to restrict crawling to the same domain.
- Replaced AsyncHtmlLoader with RecursiveUrlLoader for improved document loading.
- Enhanced error handling and logging for better debugging.
- Updated output methods to return structured data and text content more effectively.

* chore: delete component

* chore: update component tests

* ✨ (freeze.spec.ts): refactor click actions to improve readability and maintainability
♻️ (loop-component.spec.ts): refactor click actions to improve readability and maintainability
🔧 (chatInputOutputUser-shard-1.spec.ts): add click action to fit view element for better user experience

* 🐛 (url.py): fix issue where data was being returned before error handling, causing potential errors to be missed
🐛 (freeze.spec.ts): fix incorrect test assertions for text comparison, ensuring correct comparison logic is applied

* [autofix.ci] apply automated fixes

* fix: ruff error

* ✨ (stop-building.spec.ts): refactor test to use click method instead of hover and mouse events for better readability and reliability
✨ (loop-component.spec.ts): update test to use consistent naming convention for handle-urlcomponent-shownode-data-right
✨ (chatInputOutputUser-shard-1.spec.ts): update test to use consistent naming convention for handle-urlcomponent-shownode-message-right and handle-urlcomponent-shownode-dataframe-right

* ✨ (chatInputOutputUser-shard-1.spec.ts): add additional wait time before running and verifying data output to ensure proper loading and display of data

* fix: playwrite test

* fix: ruff error

* ✅ (auto-login-off.spec.ts): add delay before continuing test to ensure proper execution
✅ (freeze-path.spec.ts): increase timeout for certain actions to prevent test failures due to timing issues
✅ (freeze-path.spec.ts): add delay before continuing test to ensure proper execution
✅ (dropdownComponent.spec.ts): refactor test to use a more reliable assertion for dropdown value length

* ✨ (freeze.spec.ts): add additional wait time before clicking on a button to improve test stability
✨ (freeze.spec.ts): increase timeout for waiting for "built successfully" text to appear to improve test reliability
✨ (globalVariables.spec.ts): add initial setup function to skip certain steps and improve test efficiency

---------

Co-authored-by: italojohnny <italojohnnydosanjos@gmail.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Edwin Jose <edwin.jose@datastax.com>
Co-authored-by: cristhianzl <cristhian.lousa@gmail.com>
This commit is contained in:
Rodrigo Nader 2025-03-21 19:39:03 -03:00 • committed by GitHub
commit 35d62a033f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 307 additions and 334 deletions
Download patch file Download diff file Expand all files Collapse all files

104
src/frontend/tests/extended/features/loop-component.spec.ts
View file

@ -77,20 +77,15 @@ test(
targetPosition: { x: 700, y: 400 },
});
const secondParseDataOutput = await page
await page
.getByTestId("handle-parsedata-shownode-data list-right")
.nth(1);
.nth(1)
.click();
const loopItemInput = await page
.getByTestId("handle-loopcomponent-shownode-item-left")
.first();
// Connecting the second parse data to the loop item to test the wrong loop message
await secondParseDataOutput.hover();
await page.mouse.down();
await loopItemInput.hover();
await page.mouse.up();
.first()
.click();
// Add Chat Output component
await page.getByTestId("sidebar-search-input").click();
@ -111,64 +106,56 @@ test(
// Loop Item -> Update Data
const loopItemHandle = await page
await page
.getByTestId("handle-loopcomponent-shownode-item-right")
.first();
const updateDataInput = await page
.first()
.click();
await page
.getByTestId("handle-updatedata-shownode-data-left")
.first();
await loopItemHandle.hover();
await page.mouse.down();
await updateDataInput.hover();
await page.mouse.up();
.first()
.click();
// URL -> Loop Data
const urlOutput = await page
.getByTestId("handle-url-shownode-data-right")
.first();
const loopInput = await page
await page
.getByTestId("handle-urlcomponent-shownode-data-right")
.first()
.click();
await page
.getByTestId("handle-loopcomponent-shownode-data-left")
.first();
await urlOutput.hover();
await page.mouse.down();
await loopInput.hover();
await page.mouse.up();
.first()
.click();
// Loop Done -> Parse Data
const loopDoneHandle = await page
await page
.getByTestId("handle-loopcomponent-shownode-done-right")
.first();
const parseDataInput = await page
.first()
.click();
await page
.getByTestId("handle-parsedata-shownode-data-left")
.first();
.first()
.click();
await loopDoneHandle.hover();
await page.mouse.down();
await parseDataInput.hover();
await page.mouse.up();
// Parse Data -> Chat Output
await page
.getByTestId("handle-parsedata-shownode-message-right")
.first()
.click();
await page
.getByTestId("handle-chatoutput-noshownode-text-target")
.first()
.click();
await page.getByTestId("div-generic-node").nth(5).click();
await page.waitForTimeout(1000);
await page.getByTestId("more-options-modal").click();
await page.waitForTimeout(500);
await page.getByTestId("expand-button-modal").click();
// Parse Data -> Chat Output
const parseDataOutput = await page
.getByTestId("handle-parsedata-shownode-message-right")
.first();
const chatOutputInput = await page
.getByTestId("handle-chatoutput-shownode-text-left")
.first();
await parseDataOutput.hover();
await page.mouse.down();
await chatOutputInput.hover();
await page.mouse.up();
await page.getByTestId("input-list-plus-btn_urls-0").click();
// Configure components
@ -204,14 +191,15 @@ test(
await page.getByText("Delete").first().click();
// Update Data -> Loop Item (left side)
const updateDataOutput = await page
.getByTestId("handle-updatedata-shownode-data-right")
.first();
await updateDataOutput.hover();
await page.mouse.down();
await loopItemInput.hover();
await page.mouse.up();
await page
.getByTestId("handle-updatedata-shownode-data-right")
.first()
.click();
await page
.getByTestId("handle-loopcomponent-shownode-item-left")
.first()
.click();
// Build and run
await page.getByTestId("button_run_chat output").click();

11
src/frontend/tests/extended/integrations/chatInputOutputUser-shard-1.spec.ts
View file

@ -97,12 +97,17 @@ test(
targetPosition: { x: 700, y: 400 },
});
await page.getByTestId("fit_view").click();
// Fill URL input
await page
.getByTestId("inputlist_str_urls_0")
.fill("https://www.example.com");
await page.getByTestId("handle-url-shownode-text-right").nth(0).click();
await page
.getByTestId("handle-urlcomponent-shownode-message-right")
.nth(0)
.click();
await page.waitForTimeout(600);
await page
@ -127,7 +132,7 @@ test(
// Connect dataframe output to second chat output
await page
.getByTestId("handle-url-shownode-dataframe-right")
.getByTestId("handle-urlcomponent-shownode-dataframe-right")
.nth(0)
.click();
await page.waitForTimeout(600);
@ -182,6 +187,8 @@ test(
await page.keyboard.press("Backspace");
await page.waitForTimeout(600);
await page.waitForTimeout(5000);
// Run and verify data output is shown
await page.getByTestId("button_run_url").first().click();
await page.waitForSelector("text=built successfully", {