feat: truncate parsed uploads to prevent database and frontend blocking caused by excessively large files (#3914)

* 📝 (constants.ts): increase maxSizeFilesInBytes constant value from 10MB to 100MB to allow larger file uploads

* 🐛 (inputFileComponent): fix bug in setting the maximum file size alert message to display the correct file size limit of 100 bytes instead of 10 bytes

* 📝 (schemas.py): Add a new field_serializer method to serialize data in VertexBuildResponse class
📝 (schemas.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries
📝 (model.py): Add a new field_serializer method to serialize outputs in TransactionBase class
📝 (model.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries
📝 (model.py): Add a new field_serializer method to serialize data and artifacts in VertexBuildBase class
📝 (model.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries

* 🐛 (schemas.py): fix truncation length of text fields to 10 characters instead of 99999
🐛 (model.py): fix truncation length of text fields to 10 characters instead of 99999
🐛 (model.py): fix truncation length of text fields to 10 characters instead of 99999
🐛 (index.tsx): truncate resultMessage to 99999 characters and add message if text is too long

* 🔧 (switchOutputView/index.tsx): Use useMemo to memoize resultMessage transformations for performance optimization

* 🐛 (model.py): Fix typo in the path for 'base_retriever' data field
🐛 (model.py): Fix typo in the path for 'base_retriever' data field
🐛 (model.py): Fix typo in the path for 'base_retriever' data field
🐛 (model.py): Fix typo in the path for 'base_retriever' data field
🐛 (index.tsx): Fix logic to correctly handle resultMessageMemoized when it is an object

* 📝 (model.py): refactor truncate_text function to truncate_long_strings for better clarity and consistency
📝 (model.py): update serialize_outputs and serialize_artifacts functions to use truncate_long_strings for string truncation
📝 (model.py): introduce MAX_TEXT_LENGTH constant for defining the maximum length of text to truncate in the models

* 📝 (schemas.py): refactor serialize_data method in VertexBuildResponse class to use a new helper function truncate_long_strings for better code readability and maintainability

* 🔧 (schemas.py): Move the `truncate_long_strings` function to a separate module to improve code organization and reusability
🔧 (model.py): Import the `truncate_long_strings` function from the correct module to fix the reference error
🔧 (model.py): Import the `truncate_long_strings` function from the correct module to fix the reference error

* 📝 (util.py): add function truncate_long_strings to recursively truncate long strings in dictionaries and lists to prevent exceeding the maximum text length.

* 📝 (constants.py): add constant MAX_TEXT_LENGTH with value 99999 for defining maximum text length allowed in the application

* 📝 (model.py): update import path for truncate_long_strings function to match new location in util module

* ✨ (test_truncate_long_strings_on_objects.py): Add unit tests for the function truncate_long_strings to ensure correct behavior when truncating long strings in various data structures
🐛 (switchOutputView/index.tsx): Fix truncation logic to correctly truncate long strings by adding ellipsis at the end instead of displaying additional text about truncation.

* [autofix.ci] apply automated fixes

* ✨ (test_truncate_long_strings_on_objects.py): Update import path for truncate_long_strings function
📝 (test_truncate_long_strings_on_objects.py): Add additional tests for handling negative, zero, and small max_length values in truncate_long_strings function

* ♻️ (schemas.py): refactor import statement to use the updated module name util_strings instead of util for better clarity and consistency.

* 📝 (model.py): Update import path for util_strings module to fix module import error
📝 (util.py): Remove redundant code for truncating long strings and move it to a separate util_strings module for better organization and separation of concerns.

* 📝 (schemas.py): refactor serialize_data method to handle both BaseModel and non-BaseModel data inputs in VertexBuildResponse class

* 📝 (util_strings.py): Update util_strings.py to improve string truncation function for dictionaries and lists
🔧 (test_truncate_long_strings_on_objects.py): Update test cases for string truncation function to cover additional scenarios and edge cases

* Update src/backend/base/langflow/utils/util_strings.py

Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@langflow.org>

* 📝 (vite.config.mts): update environment variable MAX_FILE_SIZE to be defined in vite config for frontend to use in the application.

* 📝 (constants.ts): update maxSizeFilesInBytes constant to use process.env.MAX_FILE_SIZE environment variable for configurable file size limit
📝 (constants.ts): add MAX_TEXT_LENGTH constant with a value of 99999 for maximum text length limit

* 📝 (switchOutputView/index.tsx): import MAX_TEXT_LENGTH constant from shared constants file to improve code organization and reusability

* ✨ (langflow/__main__.py): add support for defining maximum file size for upload in MB to improve user experience and prevent large file uploads

* 🐛 (files.py): add validation to check if uploaded file size exceeds the maximum allowed size before processing it

* ✨ (schemas.py): add max_file_size_upload field to ConfigResponse schema to handle maximum file size allowed for upload

* 🔧 (vite.config.mts): remove MAX_FILE_SIZE environment variable configuration as it is no longer needed

* ✨ (base.py): introduce max_file_size_upload setting to limit the file size for uploads in MB

* 🐛 (util.py): add support for setting max_file_size_upload in update_settings function to allow configuring maximum file size for uploads

* 📝 (inputFileComponent/index.tsx): add support for retrieving max file size upload from utility store to improve code modularity and reusability
🐛 (inputFileComponent/index.tsx): fix error handling logic to display error message when uploading a file fails

* 📝 (constants.ts): remove maxSizeFilesInBytes constant as it is no longer used and update MAX_TEXT_LENGTH constant to a higher value

* ✨ (use-get-config.ts): add functionality to set max file size upload value from config response

* ✨ (utilityStore.ts): introduce maxFileSizeUpload property and setMaxFileSizeUpload function to handle maximum file size upload in bytes

* ✨ (frontend): introduce maxFileSizeUpload property and setMaxFileSizeUpload method to handle maximum file size upload functionality in the UtilityStoreType

* ♻️ (util_strings.py): refactor truncate_long_strings function to improve code readability and consistency by removing unnecessary whitespace and aligning assignment operators.

* 🐛 (files.py): fix formatting issue in the raise statement to improve code readability and maintain consistency

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@langflow.org>
This commit is contained in:
Cristhian Zanforlin Lousa 2024-09-27 12:44:05 -03:00 • committed by GitHub
commit 948b150946
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
21 changed files with 414 additions and 21 deletions

View file

@ -1,4 +1,6 @@
import { MAX_TEXT_LENGTH } from "@/constants/constants";
import { LogsLogType, OutputLogType } from "@/types/api";
import { useMemo } from "react";
import DataOutputComponent from "../../../../../../components/dataOutputComponent";
import ForwardedIconComponent from "../../../../../../components/genericIconComponent";
import {
@ -16,6 +18,7 @@ interface SwitchOutputViewProps {
outputName: string;
type: "Outputs" | "Logs";
}
const SwitchOutputView: React.FC<SwitchOutputViewProps> = ({
nodeId,
outputName,
@ -35,6 +38,35 @@ const SwitchOutputView: React.FC<SwitchOutputViewProps> = ({
if (resultMessage?.raw) {
resultMessage = resultMessage.raw;
}
const resultMessageMemoized = useMemo(() => {
if (
typeof resultMessage === "string" &&
resultMessage.length > MAX_TEXT_LENGTH
) {
resultMessage = `${resultMessage.substring(0, MAX_TEXT_LENGTH)}...`;
}
if (Array.isArray(resultMessage)) {
resultMessage = resultMessage.map((item) => {
if (item && typeof item.data === "object") {
const truncatedData = Object.fromEntries(
Object.entries(item.data).map(([key, value]) => {
if (typeof value === "string" && value.length > MAX_TEXT_LENGTH) {
return [key, `${value.substring(0, MAX_TEXT_LENGTH)}...`];
}
return [key, value];
}),
);
return { ...item, data: truncatedData };
}
return item;
});
}
return resultMessage;
}, [resultMessage]);
return type === "Outputs" ? (
<>
<Case condition={!resultType || resultType === "unknown"}>
@ -42,23 +74,23 @@ const SwitchOutputView: React.FC<SwitchOutputViewProps> = ({
</Case>
<Case condition={resultType === "error" || resultType === "ValueError"}>
<ErrorOutput
value={`${resultMessage.errorMessage}\n\n${resultMessage.stackTrace}`}
value={`${resultMessageMemoized.errorMessage}\n\n${resultMessageMemoized.stackTrace}`}
></ErrorOutput>
</Case>
<Case condition={resultType === "text"}>
<TextOutputView left={false} value={resultMessage} />
<TextOutputView left={false} value={resultMessageMemoized} />
</Case>
<Case condition={RECORD_TYPES.includes(resultType)}>
<DataOutputComponent
rows={
Array.isArray(resultMessage)
? (resultMessage as Array<any>).every((item) => item.data)
? (resultMessage as Array<any>).map((item) => item.data)
: resultMessage
: Object.keys(resultMessage).length > 0
? [resultMessage]
Array.isArray(resultMessageMemoized)
? (resultMessageMemoized as Array<any>).every((item) => item.data)
? (resultMessageMemoized as Array<any>).map((item) => item.data)
: resultMessageMemoized
: Object.keys(resultMessageMemoized).length > 0
? [resultMessageMemoized]
: []
}
pagination={true}

View file

@ -1,6 +1,6 @@
import { maxSizeFilesInBytes } from "@/constants/constants";
import { usePostUploadFile } from "@/controllers/API/queries/files/use-post-upload-file";
import { createFileUpload } from "@/helpers/create-file-upload";
import { useUtilityStore } from "@/stores/utilityStore";
import { useEffect } from "react";
import {
CONSOLE_ERROR_MSG,
@ -23,7 +23,7 @@ export default function InputFileComponent({
}: FileComponentType): JSX.Element {
const currentFlowId = useFlowsManagerStore((state) => state.currentFlowId);
const setErrorData = useAlertStore((state) => state.setErrorData);
const maxFileSizeUpload = useUtilityStore((state) => state.maxFileSizeUpload);
// Clear component state
useEffect(() => {
if (disabled && value !== "") {
@ -47,9 +47,9 @@ export default function InputFileComponent({
createFileUpload({ multiple: false, accept: fileTypes?.join(",") }).then(
(files) => {
const file = files[0];
if (file.size > maxSizeFilesInBytes) {
if (file.size > maxFileSizeUpload) {
setErrorData({
title: INVALID_FILE_SIZE_ALERT(10),
title: INVALID_FILE_SIZE_ALERT(maxFileSizeUpload / 1024 / 1024),
});
return;
}
@ -68,8 +68,12 @@ export default function InputFileComponent({
// sets the value to the user
handleOnNewValue({ value: file.name, file_path });
},
onError: () => {
onError: (error) => {
console.error(CONSOLE_ERROR_MSG);
setErrorData({
title: "Error uploading file",
list: [error.response?.data?.detail],
});
},
},
);

View file

@ -916,4 +916,4 @@ export const COLOR_OPTIONS = {
red: "var(--note-red)",
};
export const maxSizeFilesInBytes = 10 * 1024 * 1024; // 10MB in bytes
export const MAX_TEXT_LENGTH = 99999;

View file

@ -1,4 +1,5 @@
import useFlowsManagerStore from "@/stores/flowsManagerStore";
import { useUtilityStore } from "@/stores/utilityStore";
import axios from "axios";
import { useQueryFunctionType } from "../../../../types/api";
import { api } from "../../api";
@ -10,6 +11,7 @@ export interface ConfigResponse {
auto_saving: boolean;
auto_saving_interval: number;
health_check_max_retries: number;
max_file_size_upload: number;
}
export const useGetConfig: useQueryFunctionType<undefined, ConfigResponse> = (
@ -22,6 +24,9 @@ export const useGetConfig: useQueryFunctionType<undefined, ConfigResponse> = (
const setHealthCheckMaxRetries = useFlowsManagerStore(
(state) => state.setHealthCheckMaxRetries,
);
const setMaxFileSizeUpload = useUtilityStore(
(state) => state.setMaxFileSizeUpload,
);
const { query } = UseRequestProcessor();
@ -37,6 +42,7 @@ export const useGetConfig: useQueryFunctionType<undefined, ConfigResponse> = (
setAutoSaving(data.auto_saving);
setAutoSavingInterval(data.auto_saving_interval);
setHealthCheckMaxRetries(data.health_check_max_retries);
setMaxFileSizeUpload(data.max_file_size_upload);
}
return data;
};

View file

@ -1,7 +1,10 @@
import { Button } from "../../../../../../components/ui/button";
import { INVALID_FILE_SIZE_ALERT } from "@/constants/alerts_constants";
import { usePostUploadFile } from "@/controllers/API/queries/files/use-post-upload-file";
import { createFileUpload } from "@/helpers/create-file-upload";
import useAlertStore from "@/stores/alertStore";
import { useUtilityStore } from "@/stores/utilityStore";
import { useEffect, useState } from "react";
import IconComponent from "../../../../../../components/genericIconComponent";
import {
@ -18,6 +21,8 @@ export default function IOFileInput({ field, updateValue }: IOFileInputProps) {
const [isDragging, setIsDragging] = useState(false);
const [filePath, setFilePath] = useState("");
const [image, setImage] = useState<string | null>(null);
const setErrorData = useAlertStore((state) => state.setErrorData);
const maxFileSizeUpload = useUtilityStore((state) => state.maxFileSizeUpload);
useEffect(() => {
if (filePath) {
@ -74,6 +79,13 @@ export default function IOFileInput({ field, updateValue }: IOFileInputProps) {
const upload = async (file) => {
if (file) {
if (file.size > maxFileSizeUpload) {
setErrorData({
title: INVALID_FILE_SIZE_ALERT(maxFileSizeUpload / 1024 / 1024),
});
return;
}
// Check if a file was selected
const fileReader = new FileReader();
fileReader.onload = (event) => {
@ -93,7 +105,11 @@ export default function IOFileInput({ field, updateValue }: IOFileInputProps) {
const { file_path } = data;
setFilePath(file_path);
},
onError: () => {
onError: (error) => {
setErrorData({
title: "Error uploading file",
list: [error.response?.data?.detail],
});
console.error("Error occurred while uploading file");
},
},

View file

@ -1,5 +1,7 @@
import { INVALID_FILE_SIZE_ALERT } from "@/constants/alerts_constants";
import { usePostUploadFile } from "@/controllers/API/queries/files/use-post-upload-file";
import useAlertStore from "@/stores/alertStore";
import { useUtilityStore } from "@/stores/utilityStore";
import { useEffect, useRef, useState } from "react";
import ShortUniqueId from "short-unique-id";
import {
@ -36,6 +38,7 @@ export default function ChatInput({
const [inputFocus, setInputFocus] = useState<boolean>(false);
const fileInputRef = useRef<HTMLInputElement>(null);
const setErrorData = useAlertStore((state) => state.setErrorData);
const maxFileSizeUpload = useUtilityStore((state) => state.maxFileSizeUpload);
useFocusOnUnlock(lockChat, inputRef);
useAutoResizeTextArea(chatValue, inputRef);
@ -62,10 +65,16 @@ export default function ChatInput({
const fileInput = event.target as HTMLInputElement;
file = fileInput.files?.[0] ?? null;
}
if (file) {
const fileExtension = file.name.split(".").pop()?.toLowerCase();
if (file.size > maxFileSizeUpload) {
setErrorData({
title: INVALID_FILE_SIZE_ALERT(maxFileSizeUpload / 1024 / 1024),
});
return;
}
if (
!fileExtension ||
!ALLOWED_IMAGE_INPUT_EXTENSIONS.includes(fileExtension)
@ -99,7 +108,7 @@ export default function ChatInput({
return newFiles;
});
},
onError: () => {
onError: (error) => {
setFiles((prev) => {
const newFiles = [...prev];
const updatedIndex = newFiles.findIndex((file) => file.id === id);
@ -107,6 +116,10 @@ export default function ChatInput({
newFiles[updatedIndex].error = true;
return newFiles;
});
setErrorData({
title: "Error uploading file",
list: [error.response?.data?.detail],
});
},
},
);

View file

@ -1,6 +1,8 @@
import { INVALID_FILE_SIZE_ALERT } from "@/constants/alerts_constants";
import { useDeleteBuilds } from "@/controllers/API/queries/_builds";
import { usePostUploadFile } from "@/controllers/API/queries/files/use-post-upload-file";
import { track } from "@/customization/utils/analytics";
import { useUtilityStore } from "@/stores/utilityStore";
import { useEffect, useRef, useState } from "react";
import ShortUniqueId from "short-unique-id";
import IconComponent from "../../../../components/genericIconComponent";
@ -42,6 +44,7 @@ export default function ChatView({
const updateFlowPool = useFlowStore((state) => state.updateFlowPool);
const [id, setId] = useState<string>("");
const { mutate: mutateDeleteFlowPool } = useDeleteBuilds();
const maxFileSizeUpload = useUtilityStore((state) => state.maxFileSizeUpload);
//build chat history
useEffect(() => {
@ -173,6 +176,13 @@ export default function ChatView({
if (files) {
const file = files?.[0];
const fileExtension = file.name.split(".").pop()?.toLowerCase();
if (file.size > maxFileSizeUpload) {
setErrorData({
title: INVALID_FILE_SIZE_ALERT(maxFileSizeUpload / 1024 / 1024),
});
return;
}
if (
!fileExtension ||
!ALLOWED_IMAGE_INPUT_EXTENSIONS.includes(fileExtension)
@ -208,7 +218,7 @@ export default function ChatView({
return newFiles;
});
},
onError: () => {
onError: (error) => {
setFiles((prev) => {
const newFiles = [...prev];
const updatedIndex = newFiles.findIndex((file) => file.id === id);
@ -216,6 +226,10 @@ export default function ChatView({
newFiles[updatedIndex].error = true;
return newFiles;
});
setErrorData({
title: "Error uploading file",
list: [error.response?.data?.detail],
});
},
},
);

View file

@ -18,4 +18,7 @@ export const useUtilityStore = create<UtilityStoreType>((set, get) => ({
playgroundScrollBehaves: "instant",
setPlaygroundScrollBehaves: (behaves: ScrollBehavior) =>
set({ playgroundScrollBehaves: behaves }),
maxFileSizeUpload: 100 * 1024 * 1024, // 100MB in bytes
setMaxFileSizeUpload: (maxFileSizeUpload: number) =>
set({ maxFileSizeUpload: maxFileSizeUpload * 1024 * 1024 }),
}));

View file

@ -5,4 +5,6 @@ export type UtilityStoreType = {
setHealthCheckTimeout: (timeout: string | null) => void;
playgroundScrollBehaves: ScrollBehavior;
setPlaygroundScrollBehaves: (behaves: ScrollBehavior) => void;
maxFileSizeUpload: number;
setMaxFileSizeUpload: (maxFileSizeUpload: number) => void;
};

View file

@ -0,0 +1,128 @@
import { expect, test } from "@playwright/test";
import * as dotenv from "dotenv";
import { readFileSync } from "fs";
import path from "path";
test("user should not be able to upload a file larger than the limit", async ({
page,
}) => {
const maxFileSizeUpload = 0.001;
await page.route("**/api/v1/config", (route) => {
route.fulfill({
status: 200,
contentType: "application/json",
body: JSON.stringify({
max_file_size_upload: maxFileSizeUpload,
}),
headers: {
"content-type": "application/json",
...route.request().headers(),
},
});
});
test.skip(
!process?.env?.OPENAI_API_KEY,
"OPENAI_API_KEY required to run this test",
);
if (!process.env.CI) {
dotenv.config({ path: path.resolve(__dirname, "../../.env") });
}
await page.goto("/");
await page.waitForTimeout(1000);
let modalCount = 0;
try {
const modalTitleElement = await page?.getByTestId("modal-title");
if (modalTitleElement) {
modalCount = await modalTitleElement.count();
}
} catch (error) {
modalCount = 0;
}
while (modalCount === 0) {
await page.getByText("New Project", { exact: true }).click();
await page.waitForTimeout(3000);
modalCount = await page.getByTestId("modal-title")?.count();
}
await page.getByRole("heading", { name: "Basic Prompting" }).click();
await page.waitForSelector('[title="fit view"]', {
timeout: 100000,
});
await page.getByTitle("fit view").click();
await page.getByTitle("zoom out").click();
await page.getByTitle("zoom out").click();
await page.getByTitle("zoom out").click();
let outdatedComponents = await page.getByTestId("icon-AlertTriangle").count();
while (outdatedComponents > 0) {
await page.getByTestId("icon-AlertTriangle").first().click();
await page.waitForTimeout(1000);
outdatedComponents = await page.getByTestId("icon-AlertTriangle").count();
}
await page
.getByTestId("popover-anchor-input-api_key")
.fill(process.env.OPENAI_API_KEY ?? "");
await page.getByTestId("dropdown_str_model_name").click();
await page.getByTestId("gpt-4o-1-option").click();
await page.waitForSelector("text=Chat Input", { timeout: 30000 });
await page.getByText("Chat Input", { exact: true }).click();
await page.getByTestId("more-options-modal").click();
await page.getByTestId("edit-button-modal").click();
await page.getByText("Close").last().click();
await page.getByText("Playground", { exact: true }).click();
// Read the image file as a binary string
const filePath = "tests/assets/chain.png";
const fileContent = readFileSync(filePath, "base64");
// Create the DataTransfer and File objects within the browser context
const dataTransfer = await page.evaluateHandle(
({ fileContent }) => {
const dt = new DataTransfer();
const byteCharacters = atob(fileContent);
const byteNumbers = new Array(byteCharacters.length);
for (let i = 0; i < byteCharacters.length; i++) {
byteNumbers[i] = byteCharacters.charCodeAt(i);
}
const byteArray = new Uint8Array(byteNumbers);
const file = new File([byteArray], "chain.png", { type: "image/png" });
dt.items.add(file);
return dt;
},
{ fileContent },
);
await page.waitForSelector('[data-testid="input-chat-playground"]', {
timeout: 100000,
});
// Locate the target element
const element = await page.getByTestId("input-chat-playground");
// Dispatch the drop event on the target element
await element.dispatchEvent("drop", { dataTransfer });
await page.waitForTimeout(1000);
await page.waitForSelector("text=The file size is too large", {
timeout: 10000,
});
await expect(
page.getByText(
`The file size is too large. Please select a file smaller than ${maxFileSizeUpload}MB`,
),
).toBeVisible();
});

View file

@ -36,7 +36,9 @@ export default defineConfig(({ mode }) => {
},
define: {
"process.env.BACKEND_URL": JSON.stringify(env.BACKEND_URL),
"process.env.ACCESS_TOKEN_EXPIRE_SECONDS": JSON.stringify(env.ACCESS_TOKEN_EXPIRE_SECONDS),
"process.env.ACCESS_TOKEN_EXPIRE_SECONDS": JSON.stringify(
env.ACCESS_TOKEN_EXPIRE_SECONDS,
),
"process.env.CI": JSON.stringify(env.CI),
},
plugins: [react(), svgr(), tsconfigPaths()],
@ -47,4 +49,4 @@ export default defineConfig(({ mode }) => {
},
},
};
});
});