test: Enhance test stability (#4566)
* 🐛 (genericIconComponent/index.tsx): fix optional chaining for split and map functions to prevent errors 🐛 (actionsMainPage-shard-1.spec.ts): remove redundant zoom_out click 🐛 (auto-login-off.spec.ts): reduce wait time from 2000ms to 1000ms 🐛 (freeze-path.spec.ts): change click target from "openai" to "OpenAI" for consistency 🐛 (freeze.spec.ts): change click target from "Split Text" to "Split Text" for consistency 🐛 (Dynamic Agent.spec.ts): skip test execution 🐛 (Hierarchical Agent.spec.ts): skip test execution 🐛 (Sequential Task Agent.spec.ts): skip test execution 🐛 (Simple Agent.spec.ts): skip test execution 🐛 (decisionFlow.spec.ts): change search input from "conditional router" to "if else" 🐛 (similarity.spec.ts): change search input from "openai" to "openai embedding" 🐛 (fileUploadComponent.spec.ts): enable test for file upload functionality 📝 (deleteComponents.spec.ts): Remove commented out code and enable test for deleting a component 📝 (deleteFlows.spec.ts): Enable test for deleting a flow 📝 (starter-projects.spec.ts): Enable test for interacting with starter projects 📝 (store-shard-3.spec.ts): Enable tests for ordering visualization and filtering by type 📝 (generalBugs-shard-13.spec.ts): Enable test for sharing a component on the store * ⬆️ (typescript_test.yml): update matrix configuration to run tests on multiple shards for better parallelization 🔧 (typescript_test.yml): enable playwright test execution with shard index and total for parallel test execution * 🐛 (genericIconComponent/index.tsx): fix optional chaining for split and map functions to prevent errors 🐛 (actionsMainPage-shard-1.spec.ts): remove redundant zoom_out click 🐛 (auto-login-off.spec.ts): reduce wait time from 2000ms to 1000ms 🐛 (freeze-path.spec.ts): change click target from "openai" to "OpenAI" for consistency 🐛 (freeze.spec.ts): change click target from "Split Text" to "Split Text" for consistency 🐛 (Dynamic Agent.spec.ts): skip test execution 🐛 (Hierarchical Agent.spec.ts): skip test execution 🐛 (Sequential Task Agent.spec.ts): skip test execution 🐛 (Simple Agent.spec.ts): skip test execution 🐛 (decisionFlow.spec.ts): change search input from "conditional router" to "if else" 🐛 (similarity.spec.ts): change search input from "openai" to "openai embedding" 🐛 (fileUploadComponent.spec.ts): enable test for file upload functionality 📝 (deleteComponents.spec.ts): Remove commented out code and enable test for deleting a component 📝 (deleteFlows.spec.ts): Enable test for deleting a flow 📝 (starter-projects.spec.ts): Enable test for interacting with starter projects 📝 (store-shard-3.spec.ts): Enable tests for ordering visualization and filtering by type 📝 (generalBugs-shard-13.spec.ts): Enable test for sharing a component on the store * ⬆️ (typescript_test.yml): update matrix configuration to run tests on multiple shards for better parallelization 🔧 (typescript_test.yml): enable playwright test execution with shard index and total for parallel test execution * 🐛 (Travel Planning Agent.spec.ts): Fix issue with setting API keys for OpenAI models and Search API 🐛 (Travel Planning Agent.spec.ts): Fix issue with clicking the wrong button for running chat output 🐛 (Travel Planning Agent.spec.ts): Fix issue with checking rate limit and skipping test logic 🐛 (Travel Planning Agent.spec.ts): Fix issue with checking output text for lowercase city names 🐛 (Vector Store.spec.ts): Fix issue with clicking the wrong button for creating a new flow 🐛 (Vector Store.spec.ts): Fix issue with filling the wrong input field for collection name in Vector Store RAG test * 🔧 (generalBugs-shard-4.spec.ts): skip test case temporarily due to functionality not working with uplift designs * Update test to use 'message history' instead of 'chat memory' in sidebar search input * Update test selectors and expected values in dropdownComponent.spec.ts * 📝 (Simple Agent.spec.ts): Remove unnecessary test.skip and commented out code 📝 (dropdownComponent.spec.ts): Update dropdown selection to match new text 📝 (starter-projects.spec.ts): Remove unused template IDs 📝 (store-shard-1.spec.ts): Update test.skip to test.skip for consistency 📝 (generalBugs-shard-10.spec.ts): Refactor test cases for promptText and newPromptText 📝 (generalBugs-shard-6.spec.ts): Add delay before interacting with elements to ensure stability * Refactor tests to use `ControlOrMeta` for cross-platform keyboard shortcuts * ✅ (Simple Agent.spec.ts): update timeout value for waitForSelector to improve test reliability and prevent false negatives * Handle undefined stars in GithubStarComponent to prevent errors * Add waitForSelector to ensure sidebar search input is visible before interaction in chatInputOutput test * ✅ (Simple Agent.spec.ts): update timeout value to 5 minutes for "built successfully" message to allow for longer build times ✨ (Simple Agent.spec.ts): add additional test cases to check for the presence of "apple" in chat messages and ensure the total length of chat messages is greater than 100 characters * 🐛 (Simple Agent.spec.ts): reduce timeout for "built successfully" text to improve test efficiency * ✅ (auto-login-off.spec.ts): add delay before checking visibility of elements to ensure proper rendering and improve test reliability * Fix indentation in TypeScript test workflow file * Update test IDs and expected values in dropdownComponent.spec.ts * 📝 (playground.spec.ts): remove redundant code related to zooming out and dragging elements in the playground test 📝 (deleteComponents.spec.ts): update selectors for clicking buttons and elements in the deleteComponents test 📝 (deleteFlows.spec.ts): update selectors for clicking buttons and elements in the deleteFlows test 📝 (starter-projects.spec.ts): update selectors for starter projects in the extended features test to match changes in the application's UI. * ✅ (playground.spec.ts): remove skip from the test case to enable it to run during testing. * Refactor timeout in playground test for better stability --------- Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@langflow.org> Co-authored-by: anovazzi1 <otavio2204@gmail.com>
This commit is contained in:
Cristhian Zanforlin Lousa
GitHub
parent
563b90ef28
commit
4e2ee25da8
40 changed files with 2142 additions and 1053 deletions
|
|
@ -2,191 +2,194 @@ import { expect, test } from "@playwright/test";
|
|||
import * as dotenv from "dotenv";
|
||||
import path from "path";
|
||||
|
||||
test("TextInputOutputComponent", async ({ page }) => {
|
||||
test.skip("TextInputOutputComponent", async ({ page }) => {
|
||||
// commented out because new playground does not support text io yet
|
||||
// test.skip(
|
||||
// !process?.env?.OPENAI_API_KEY,
|
||||
// "OPENAI_API_KEY required to run this test",
|
||||
// );
|
||||
// if (!process.env.CI) {
|
||||
// dotenv.config({ path: path.resolve(__dirname, "../../.env") });
|
||||
// }
|
||||
// await page.goto("/");
|
||||
// await page.waitForSelector('[data-testid="mainpage_title"]', {
|
||||
// timeout: 30000,
|
||||
// });
|
||||
// await page.waitForSelector('[id="new-project-btn"]', {
|
||||
// timeout: 30000,
|
||||
// });
|
||||
// let modalCount = 0;
|
||||
// try {
|
||||
// const modalTitleElement = await page?.getByTestId("modal-title");
|
||||
// if (modalTitleElement) {
|
||||
// modalCount = await modalTitleElement.count();
|
||||
// }
|
||||
// } catch (error) {
|
||||
// modalCount = 0;
|
||||
// }
|
||||
// while (modalCount === 0) {
|
||||
// await page.getByText("New Flow", { exact: true }).click();
|
||||
// await page.waitForTimeout(3000);
|
||||
// modalCount = await page.getByTestId("modal-title")?.count();
|
||||
// }
|
||||
// await page.waitForSelector('[data-testid="blank-flow"]', {
|
||||
// timeout: 30000,
|
||||
// });
|
||||
// await page.getByTestId("blank-flow").click();
|
||||
// await page.getByTestId("sidebar-search-input").click();
|
||||
// await page.getByTestId("sidebar-search-input").fill("text input");
|
||||
// await page.waitForTimeout(1000);
|
||||
// await page
|
||||
// .getByTestId("inputsText Input")
|
||||
// .dragTo(page.locator('//*[@id="react-flow-id"]'));
|
||||
// await page.mouse.up();
|
||||
// await page.mouse.down();
|
||||
// await page.getByTestId("sidebar-search-input").click();
|
||||
// await page.getByTestId("sidebar-search-input").fill("openai");
|
||||
// await page.waitForTimeout(1000);
|
||||
// await page
|
||||
// .getByTestId("modelsOpenAI")
|
||||
// .dragTo(page.locator('//*[@id="react-flow-id"]'));
|
||||
// await page.mouse.up();
|
||||
// await page.mouse.down();
|
||||
// await page.waitForSelector('[data-testid="fit_view"]', {
|
||||
// timeout: 100000,
|
||||
// });
|
||||
// await page.getByTestId("fit_view").click();
|
||||
// await page.getByTestId("zoom_out").click();
|
||||
// await page.getByTestId("zoom_out").click();
|
||||
// await page.getByTestId("zoom_out").click();
|
||||
// let visibleElementHandle;
|
||||
// const elementsTextInputOutput = await page
|
||||
// .getByTestId("handle-textinput-shownode-text-right")
|
||||
// .all();
|
||||
// for (const element of elementsTextInputOutput) {
|
||||
// if (await element.isVisible()) {
|
||||
// visibleElementHandle = element;
|
||||
// break;
|
||||
// }
|
||||
// }
|
||||
// await visibleElementHandle.waitFor({
|
||||
// state: "visible",
|
||||
// timeout: 30000,
|
||||
// });
|
||||
// await visibleElementHandle.hover();
|
||||
// await page.mouse.down();
|
||||
// for (const element of elementsTextInputOutput) {
|
||||
// if (await element.isVisible()) {
|
||||
// visibleElementHandle = element;
|
||||
// break;
|
||||
// }
|
||||
// }
|
||||
// await visibleElementHandle.waitFor({
|
||||
// state: "visible",
|
||||
// timeout: 30000,
|
||||
// });
|
||||
// // Move to the second element
|
||||
// await visibleElementHandle.hover();
|
||||
// // Release the mouse
|
||||
// await page.mouse.up();
|
||||
// await page.getByTestId("sidebar-search-input").click();
|
||||
// await page.getByTestId("sidebar-search-input").fill("text output");
|
||||
// await page
|
||||
// .getByTestId("outputsText Output")
|
||||
// .dragTo(page.locator('//*[@id="react-flow-id"]'));
|
||||
// await page.mouse.up();
|
||||
// await page.mouse.down();
|
||||
// await page.waitForSelector('[data-testid="fit_view"]', {
|
||||
// timeout: 100000,
|
||||
// });
|
||||
// await page.getByTestId("zoom_out").click();
|
||||
// await page.getByTestId("zoom_out").click();
|
||||
// await page.getByTestId("zoom_out").click();
|
||||
// await page.getByTestId("zoom_out").click();
|
||||
// await page.getByTestId("zoom_out").click();
|
||||
// await page.getByTestId("zoom_out").click();
|
||||
// const elementsOpenAiOutput = await page
|
||||
// .getByTestId("handle-openaimodel-shownode-text-right")
|
||||
// .all();
|
||||
// for (const element of elementsOpenAiOutput) {
|
||||
// if (await element.isVisible()) {
|
||||
// visibleElementHandle = element;
|
||||
// break;
|
||||
// }
|
||||
// }
|
||||
// await visibleElementHandle.waitFor({
|
||||
// state: "visible",
|
||||
// timeout: 30000,
|
||||
// });
|
||||
// // Click and hold on the first element
|
||||
// await visibleElementHandle.hover();
|
||||
// await page.mouse.down();
|
||||
// const elementTextOutputInput = await page
|
||||
// .getByTestId("handle-textoutput-shownode-text-left")
|
||||
// .all();
|
||||
// for (const element of elementTextOutputInput) {
|
||||
// if (await element.isVisible()) {
|
||||
// visibleElementHandle = element;
|
||||
// break;
|
||||
// }
|
||||
// }
|
||||
// await visibleElementHandle.waitFor({
|
||||
// state: "visible",
|
||||
// timeout: 30000,
|
||||
// });
|
||||
// // Move to the second element
|
||||
// await visibleElementHandle.hover();
|
||||
// // Release the mouse
|
||||
// await page.mouse.up();
|
||||
// await page
|
||||
// .getByTestId(/^rf__node-TextInput-[a-zA-Z0-9]+$/)
|
||||
// .getByTestId("textarea_str_input_value")
|
||||
// .fill("This is a test!");
|
||||
// let outdatedComponents = await page.getByTestId("icon-AlertTriangle").count();
|
||||
// while (outdatedComponents > 0) {
|
||||
// await page.getByTestId("icon-AlertTriangle").first().click();
|
||||
// await page.waitForTimeout(1000);
|
||||
// outdatedComponents = await page.getByTestId("icon-AlertTriangle").count();
|
||||
// }
|
||||
// let filledApiKey = await page.getByTestId("remove-icon-badge").count();
|
||||
// while (filledApiKey > 0) {
|
||||
// await page.getByTestId("remove-icon-badge").first().click();
|
||||
// await page.waitForTimeout(1000);
|
||||
// filledApiKey = await page.getByTestId("remove-icon-badge").count();
|
||||
// }
|
||||
// const apiKeyInput = page.getByTestId("popover-anchor-input-api_key");
|
||||
// const isApiKeyInputVisible = await apiKeyInput.isVisible();
|
||||
// if (isApiKeyInputVisible) {
|
||||
// await apiKeyInput.fill(process.env.OPENAI_API_KEY ?? "");
|
||||
// }
|
||||
// await page.getByTestId("dropdown_str_model_name").click();
|
||||
// await page.getByTestId("gpt-4o-1-option").click();
|
||||
// await page.waitForTimeout(1000);
|
||||
// await page.getByText("Playground", { exact: true }).last().click();
|
||||
// await page.getByTestId("button_run_text_output").click();
|
||||
// await page.getByTestId(/^rf__node-TextOutput-[a-zA-Z0-9]+$/).getByTestId("output-inspection-text").click();
|
||||
// await page.getByText("Run Flow", { exact: true }).click();
|
||||
// await page.waitForTimeout(5000);
|
||||
// let textInputContent = await page
|
||||
// .getByPlaceholder("Enter text...")
|
||||
// .textContent();
|
||||
// expect(textInputContent).toBe("This is a test!");
|
||||
// await page.getByText("Outputs", { exact: true }).nth(1).click();
|
||||
// await page.getByText("Text Output", { exact: true }).nth(2).click();
|
||||
// let contentOutput = await page.getByPlaceholder("Enter text...").inputValue();
|
||||
// expect(contentOutput).not.toBe(null);
|
||||
// await page.keyboard.press("Escape");
|
||||
// await page
|
||||
// .getByTestId(/^rf__node-TextInput-[a-zA-Z0-9]+$/)
|
||||
// .getByTestId("textarea_str_input_value")
|
||||
// .fill("This is a test, again just to be sure!");
|
||||
// await page.getByText("Playground", { exact: true }).last().click();
|
||||
// await page.getByText("Run Flow", { exact: true }).click();
|
||||
// await page.waitForTimeout(5000);
|
||||
// textInputContent = await page.getByPlaceholder("Enter text...").textContent();
|
||||
// expect(textInputContent).toBe("This is a test, again just to be sure!");
|
||||
// await page.getByText("Outputs", { exact: true }).nth(1).click();
|
||||
// await page.getByText("Text Output", { exact: true }).nth(2).click();
|
||||
// contentOutput = await page.getByPlaceholder("Enter text...").inputValue();
|
||||
// expect(contentOutput).not.toBe(null);
|
||||
test.skip(
|
||||
!process?.env?.OPENAI_API_KEY,
|
||||
"OPENAI_API_KEY required to run this test",
|
||||
);
|
||||
if (!process.env.CI) {
|
||||
dotenv.config({ path: path.resolve(__dirname, "../../.env") });
|
||||
}
|
||||
await page.goto("/");
|
||||
await page.waitForSelector('[data-testid="mainpage_title"]', {
|
||||
timeout: 30000,
|
||||
});
|
||||
await page.waitForSelector('[id="new-project-btn"]', {
|
||||
timeout: 30000,
|
||||
});
|
||||
let modalCount = 0;
|
||||
try {
|
||||
const modalTitleElement = await page?.getByTestId("modal-title");
|
||||
if (modalTitleElement) {
|
||||
modalCount = await modalTitleElement.count();
|
||||
}
|
||||
} catch (error) {
|
||||
modalCount = 0;
|
||||
}
|
||||
while (modalCount === 0) {
|
||||
await page.getByText("New Flow", { exact: true }).click();
|
||||
await page.waitForTimeout(3000);
|
||||
modalCount = await page.getByTestId("modal-title")?.count();
|
||||
}
|
||||
await page.waitForSelector('[data-testid="blank-flow"]', {
|
||||
timeout: 30000,
|
||||
});
|
||||
await page.getByTestId("blank-flow").click();
|
||||
await page.getByTestId("sidebar-search-input").click();
|
||||
await page.getByTestId("sidebar-search-input").fill("text input");
|
||||
await page.waitForTimeout(1000);
|
||||
await page
|
||||
.getByTestId("inputsText Input")
|
||||
.dragTo(page.locator('//*[@id="react-flow-id"]'));
|
||||
await page.mouse.up();
|
||||
await page.mouse.down();
|
||||
await page.getByTestId("sidebar-search-input").click();
|
||||
await page.getByTestId("sidebar-search-input").fill("openai");
|
||||
await page.waitForTimeout(1000);
|
||||
await page
|
||||
.getByTestId("modelsOpenAI")
|
||||
.dragTo(page.locator('//*[@id="react-flow-id"]'));
|
||||
await page.mouse.up();
|
||||
await page.mouse.down();
|
||||
await page.waitForSelector('[data-testid="fit_view"]', {
|
||||
timeout: 100000,
|
||||
});
|
||||
await page.getByTestId("fit_view").click();
|
||||
await page.getByTestId("zoom_out").click();
|
||||
await page.getByTestId("zoom_out").click();
|
||||
await page.getByTestId("zoom_out").click();
|
||||
let visibleElementHandle;
|
||||
const elementsTextInputOutput = await page
|
||||
.getByTestId("handle-textinput-shownode-text-right")
|
||||
.all();
|
||||
for (const element of elementsTextInputOutput) {
|
||||
if (await element.isVisible()) {
|
||||
visibleElementHandle = element;
|
||||
break;
|
||||
}
|
||||
}
|
||||
await visibleElementHandle.waitFor({
|
||||
state: "visible",
|
||||
timeout: 30000,
|
||||
});
|
||||
await visibleElementHandle.hover();
|
||||
await page.mouse.down();
|
||||
for (const element of elementsTextInputOutput) {
|
||||
if (await element.isVisible()) {
|
||||
visibleElementHandle = element;
|
||||
break;
|
||||
}
|
||||
}
|
||||
await visibleElementHandle.waitFor({
|
||||
state: "visible",
|
||||
timeout: 30000,
|
||||
});
|
||||
// Move to the second element
|
||||
await visibleElementHandle.hover();
|
||||
// Release the mouse
|
||||
await page.mouse.up();
|
||||
await page.getByTestId("sidebar-search-input").click();
|
||||
await page.getByTestId("sidebar-search-input").fill("text output");
|
||||
await page
|
||||
.getByTestId("outputsText Output")
|
||||
.dragTo(page.locator('//*[@id="react-flow-id"]'));
|
||||
await page.mouse.up();
|
||||
await page.mouse.down();
|
||||
await page.waitForSelector('[data-testid="fit_view"]', {
|
||||
timeout: 100000,
|
||||
});
|
||||
await page.getByTestId("zoom_out").click();
|
||||
await page.getByTestId("zoom_out").click();
|
||||
await page.getByTestId("zoom_out").click();
|
||||
await page.getByTestId("zoom_out").click();
|
||||
await page.getByTestId("zoom_out").click();
|
||||
await page.getByTestId("zoom_out").click();
|
||||
const elementsOpenAiOutput = await page
|
||||
.getByTestId("handle-openaimodel-shownode-text-right")
|
||||
.all();
|
||||
for (const element of elementsOpenAiOutput) {
|
||||
if (await element.isVisible()) {
|
||||
visibleElementHandle = element;
|
||||
break;
|
||||
}
|
||||
}
|
||||
await visibleElementHandle.waitFor({
|
||||
state: "visible",
|
||||
timeout: 30000,
|
||||
});
|
||||
// Click and hold on the first element
|
||||
await visibleElementHandle.hover();
|
||||
await page.mouse.down();
|
||||
const elementTextOutputInput = await page
|
||||
.getByTestId("handle-textoutput-shownode-text-left")
|
||||
.all();
|
||||
for (const element of elementTextOutputInput) {
|
||||
if (await element.isVisible()) {
|
||||
visibleElementHandle = element;
|
||||
break;
|
||||
}
|
||||
}
|
||||
await visibleElementHandle.waitFor({
|
||||
state: "visible",
|
||||
timeout: 30000,
|
||||
});
|
||||
// Move to the second element
|
||||
await visibleElementHandle.hover();
|
||||
// Release the mouse
|
||||
await page.mouse.up();
|
||||
await page
|
||||
.getByTestId(/^rf__node-TextInput-[a-zA-Z0-9]+$/)
|
||||
.getByTestId("textarea_str_input_value")
|
||||
.fill("This is a test!");
|
||||
let outdatedComponents = await page.getByTestId("icon-AlertTriangle").count();
|
||||
while (outdatedComponents > 0) {
|
||||
await page.getByTestId("icon-AlertTriangle").first().click();
|
||||
await page.waitForTimeout(1000);
|
||||
outdatedComponents = await page.getByTestId("icon-AlertTriangle").count();
|
||||
}
|
||||
let filledApiKey = await page.getByTestId("remove-icon-badge").count();
|
||||
while (filledApiKey > 0) {
|
||||
await page.getByTestId("remove-icon-badge").first().click();
|
||||
await page.waitForTimeout(1000);
|
||||
filledApiKey = await page.getByTestId("remove-icon-badge").count();
|
||||
}
|
||||
const apiKeyInput = page.getByTestId("popover-anchor-input-api_key");
|
||||
const isApiKeyInputVisible = await apiKeyInput.isVisible();
|
||||
if (isApiKeyInputVisible) {
|
||||
await apiKeyInput.fill(process.env.OPENAI_API_KEY ?? "");
|
||||
}
|
||||
await page.getByTestId("dropdown_str_model_name").click();
|
||||
await page.getByTestId("gpt-4o-1-option").click();
|
||||
await page.waitForTimeout(1000);
|
||||
await page.getByText("Playground", { exact: true }).last().click();
|
||||
await page.getByTestId("button_run_text_output").click();
|
||||
await page
|
||||
.getByTestId(/^rf__node-TextOutput-[a-zA-Z0-9]+$/)
|
||||
.getByTestId("output-inspection-text")
|
||||
.click();
|
||||
await page.getByText("Run Flow", { exact: true }).click();
|
||||
await page.waitForTimeout(5000);
|
||||
let textInputContent = await page
|
||||
.getByPlaceholder("Enter text...")
|
||||
.textContent();
|
||||
expect(textInputContent).toBe("This is a test!");
|
||||
await page.getByText("Outputs", { exact: true }).nth(1).click();
|
||||
await page.getByText("Text Output", { exact: true }).nth(2).click();
|
||||
let contentOutput = await page.getByPlaceholder("Enter text...").inputValue();
|
||||
expect(contentOutput).not.toBe(null);
|
||||
await page.keyboard.press("Escape");
|
||||
await page
|
||||
.getByTestId(/^rf__node-TextInput-[a-zA-Z0-9]+$/)
|
||||
.getByTestId("textarea_str_input_value")
|
||||
.fill("This is a test, again just to be sure!");
|
||||
await page.getByText("Playground", { exact: true }).last().click();
|
||||
await page.getByText("Run Flow", { exact: true }).click();
|
||||
await page.waitForTimeout(5000);
|
||||
textInputContent = await page.getByPlaceholder("Enter text...").textContent();
|
||||
expect(textInputContent).toBe("This is a test, again just to be sure!");
|
||||
await page.getByText("Outputs", { exact: true }).nth(1).click();
|
||||
await page.getByText("Text Output", { exact: true }).nth(2).click();
|
||||
contentOutput = await page.getByPlaceholder("Enter text...").inputValue();
|
||||
expect(contentOutput).not.toBe(null);
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue