feat: truncate parsed uploads to prevent database and frontend blocking caused by excessively large files (#3914)
* 📝 (constants.ts): increase maxSizeFilesInBytes constant value from 10MB to 100MB to allow larger file uploads * 🐛 (inputFileComponent): fix bug in setting the maximum file size alert message to display the correct file size limit of 100 bytes instead of 10 bytes * 📝 (schemas.py): Add a new field_serializer method to serialize data in VertexBuildResponse class 📝 (schemas.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries 📝 (model.py): Add a new field_serializer method to serialize outputs in TransactionBase class 📝 (model.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries 📝 (model.py): Add a new field_serializer method to serialize data and artifacts in VertexBuildBase class 📝 (model.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries * 🐛 (schemas.py): fix truncation length of text fields to 10 characters instead of 99999 🐛 (model.py): fix truncation length of text fields to 10 characters instead of 99999 🐛 (model.py): fix truncation length of text fields to 10 characters instead of 99999 🐛 (index.tsx): truncate resultMessage to 99999 characters and add message if text is too long * 🔧 (switchOutputView/index.tsx): Use useMemo to memoize resultMessage transformations for performance optimization * 🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (index.tsx): Fix logic to correctly handle resultMessageMemoized when it is an object * 📝 (model.py): refactor truncate_text function to truncate_long_strings for better clarity and consistency 📝 (model.py): update serialize_outputs and serialize_artifacts functions to use truncate_long_strings for string truncation 📝 (model.py): introduce MAX_TEXT_LENGTH constant for defining the maximum length of text to truncate in the models * 📝 (schemas.py): refactor serialize_data method in VertexBuildResponse class to use a new helper function truncate_long_strings for better code readability and maintainability * 🔧 (schemas.py): Move the `truncate_long_strings` function to a separate module to improve code organization and reusability 🔧 (model.py): Import the `truncate_long_strings` function from the correct module to fix the reference error 🔧 (model.py): Import the `truncate_long_strings` function from the correct module to fix the reference error * 📝 (util.py): add function truncate_long_strings to recursively truncate long strings in dictionaries and lists to prevent exceeding the maximum text length. * 📝 (constants.py): add constant MAX_TEXT_LENGTH with value 99999 for defining maximum text length allowed in the application * 📝 (model.py): update import path for truncate_long_strings function to match new location in util module * ✨ (test_truncate_long_strings_on_objects.py): Add unit tests for the function truncate_long_strings to ensure correct behavior when truncating long strings in various data structures 🐛 (switchOutputView/index.tsx): Fix truncation logic to correctly truncate long strings by adding ellipsis at the end instead of displaying additional text about truncation. * [autofix.ci] apply automated fixes * ✨ (test_truncate_long_strings_on_objects.py): Update import path for truncate_long_strings function 📝 (test_truncate_long_strings_on_objects.py): Add additional tests for handling negative, zero, and small max_length values in truncate_long_strings function * ♻️ (schemas.py): refactor import statement to use the updated module name util_strings instead of util for better clarity and consistency. * 📝 (model.py): Update import path for util_strings module to fix module import error 📝 (util.py): Remove redundant code for truncating long strings and move it to a separate util_strings module for better organization and separation of concerns. * 📝 (schemas.py): refactor serialize_data method to handle both BaseModel and non-BaseModel data inputs in VertexBuildResponse class * 📝 (util_strings.py): Update util_strings.py to improve string truncation function for dictionaries and lists 🔧 (test_truncate_long_strings_on_objects.py): Update test cases for string truncation function to cover additional scenarios and edge cases * Update src/backend/base/langflow/utils/util_strings.py Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@langflow.org> * 📝 (vite.config.mts): update environment variable MAX_FILE_SIZE to be defined in vite config for frontend to use in the application. * 📝 (constants.ts): update maxSizeFilesInBytes constant to use process.env.MAX_FILE_SIZE environment variable for configurable file size limit 📝 (constants.ts): add MAX_TEXT_LENGTH constant with a value of 99999 for maximum text length limit * 📝 (switchOutputView/index.tsx): import MAX_TEXT_LENGTH constant from shared constants file to improve code organization and reusability * ✨ (langflow/__main__.py): add support for defining maximum file size for upload in MB to improve user experience and prevent large file uploads * 🐛 (files.py): add validation to check if uploaded file size exceeds the maximum allowed size before processing it * ✨ (schemas.py): add max_file_size_upload field to ConfigResponse schema to handle maximum file size allowed for upload * 🔧 (vite.config.mts): remove MAX_FILE_SIZE environment variable configuration as it is no longer needed * ✨ (base.py): introduce max_file_size_upload setting to limit the file size for uploads in MB * 🐛 (util.py): add support for setting max_file_size_upload in update_settings function to allow configuring maximum file size for uploads * 📝 (inputFileComponent/index.tsx): add support for retrieving max file size upload from utility store to improve code modularity and reusability 🐛 (inputFileComponent/index.tsx): fix error handling logic to display error message when uploading a file fails * 📝 (constants.ts): remove maxSizeFilesInBytes constant as it is no longer used and update MAX_TEXT_LENGTH constant to a higher value * ✨ (use-get-config.ts): add functionality to set max file size upload value from config response * ✨ (utilityStore.ts): introduce maxFileSizeUpload property and setMaxFileSizeUpload function to handle maximum file size upload in bytes * ✨ (frontend): introduce maxFileSizeUpload property and setMaxFileSizeUpload method to handle maximum file size upload functionality in the UtilityStoreType * ♻️ (util_strings.py): refactor truncate_long_strings function to improve code readability and consistency by removing unnecessary whitespace and aligning assignment operators. * 🐛 (files.py): fix formatting issue in the raise statement to improve code readability and maintain consistency --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@langflow.org>
This commit is contained in:
Cristhian Zanforlin Lousa
GitHub
parent
b34a7c7f02
commit
948b150946
21 changed files with 414 additions and 21 deletions
|
|
@ -0,0 +1,128 @@
|
|||
import { expect, test } from "@playwright/test";
|
||||
import * as dotenv from "dotenv";
|
||||
import { readFileSync } from "fs";
|
||||
import path from "path";
|
||||
|
||||
test("user should not be able to upload a file larger than the limit", async ({
|
||||
page,
|
||||
}) => {
|
||||
const maxFileSizeUpload = 0.001;
|
||||
await page.route("**/api/v1/config", (route) => {
|
||||
route.fulfill({
|
||||
status: 200,
|
||||
contentType: "application/json",
|
||||
body: JSON.stringify({
|
||||
max_file_size_upload: maxFileSizeUpload,
|
||||
}),
|
||||
headers: {
|
||||
"content-type": "application/json",
|
||||
...route.request().headers(),
|
||||
},
|
||||
});
|
||||
});
|
||||
test.skip(
|
||||
!process?.env?.OPENAI_API_KEY,
|
||||
"OPENAI_API_KEY required to run this test",
|
||||
);
|
||||
|
||||
if (!process.env.CI) {
|
||||
dotenv.config({ path: path.resolve(__dirname, "../../.env") });
|
||||
}
|
||||
|
||||
await page.goto("/");
|
||||
|
||||
await page.waitForTimeout(1000);
|
||||
|
||||
let modalCount = 0;
|
||||
try {
|
||||
const modalTitleElement = await page?.getByTestId("modal-title");
|
||||
if (modalTitleElement) {
|
||||
modalCount = await modalTitleElement.count();
|
||||
}
|
||||
} catch (error) {
|
||||
modalCount = 0;
|
||||
}
|
||||
|
||||
while (modalCount === 0) {
|
||||
await page.getByText("New Project", { exact: true }).click();
|
||||
await page.waitForTimeout(3000);
|
||||
modalCount = await page.getByTestId("modal-title")?.count();
|
||||
}
|
||||
|
||||
await page.getByRole("heading", { name: "Basic Prompting" }).click();
|
||||
await page.waitForSelector('[title="fit view"]', {
|
||||
timeout: 100000,
|
||||
});
|
||||
|
||||
await page.getByTitle("fit view").click();
|
||||
await page.getByTitle("zoom out").click();
|
||||
await page.getByTitle("zoom out").click();
|
||||
await page.getByTitle("zoom out").click();
|
||||
|
||||
let outdatedComponents = await page.getByTestId("icon-AlertTriangle").count();
|
||||
|
||||
while (outdatedComponents > 0) {
|
||||
await page.getByTestId("icon-AlertTriangle").first().click();
|
||||
await page.waitForTimeout(1000);
|
||||
outdatedComponents = await page.getByTestId("icon-AlertTriangle").count();
|
||||
}
|
||||
|
||||
await page
|
||||
.getByTestId("popover-anchor-input-api_key")
|
||||
.fill(process.env.OPENAI_API_KEY ?? "");
|
||||
|
||||
await page.getByTestId("dropdown_str_model_name").click();
|
||||
await page.getByTestId("gpt-4o-1-option").click();
|
||||
|
||||
await page.waitForSelector("text=Chat Input", { timeout: 30000 });
|
||||
|
||||
await page.getByText("Chat Input", { exact: true }).click();
|
||||
await page.getByTestId("more-options-modal").click();
|
||||
await page.getByTestId("edit-button-modal").click();
|
||||
await page.getByText("Close").last().click();
|
||||
|
||||
await page.getByText("Playground", { exact: true }).click();
|
||||
|
||||
// Read the image file as a binary string
|
||||
const filePath = "tests/assets/chain.png";
|
||||
const fileContent = readFileSync(filePath, "base64");
|
||||
|
||||
// Create the DataTransfer and File objects within the browser context
|
||||
const dataTransfer = await page.evaluateHandle(
|
||||
({ fileContent }) => {
|
||||
const dt = new DataTransfer();
|
||||
const byteCharacters = atob(fileContent);
|
||||
const byteNumbers = new Array(byteCharacters.length);
|
||||
for (let i = 0; i < byteCharacters.length; i++) {
|
||||
byteNumbers[i] = byteCharacters.charCodeAt(i);
|
||||
}
|
||||
const byteArray = new Uint8Array(byteNumbers);
|
||||
const file = new File([byteArray], "chain.png", { type: "image/png" });
|
||||
dt.items.add(file);
|
||||
return dt;
|
||||
},
|
||||
{ fileContent },
|
||||
);
|
||||
|
||||
await page.waitForSelector('[data-testid="input-chat-playground"]', {
|
||||
timeout: 100000,
|
||||
});
|
||||
|
||||
// Locate the target element
|
||||
const element = await page.getByTestId("input-chat-playground");
|
||||
|
||||
// Dispatch the drop event on the target element
|
||||
await element.dispatchEvent("drop", { dataTransfer });
|
||||
|
||||
await page.waitForTimeout(1000);
|
||||
|
||||
await page.waitForSelector("text=The file size is too large", {
|
||||
timeout: 10000,
|
||||
});
|
||||
|
||||
await expect(
|
||||
page.getByText(
|
||||
`The file size is too large. Please select a file smaller than ${maxFileSizeUpload}MB`,
|
||||
),
|
||||
).toBeVisible();
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue