feat: truncate parsed uploads to prevent database and frontend blocking caused by excessively large files (#3914)
* 📝 (constants.ts): increase maxSizeFilesInBytes constant value from 10MB to 100MB to allow larger file uploads * 🐛 (inputFileComponent): fix bug in setting the maximum file size alert message to display the correct file size limit of 100 bytes instead of 10 bytes * 📝 (schemas.py): Add a new field_serializer method to serialize data in VertexBuildResponse class 📝 (schemas.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries 📝 (model.py): Add a new field_serializer method to serialize outputs in TransactionBase class 📝 (model.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries 📝 (model.py): Add a new field_serializer method to serialize data and artifacts in VertexBuildBase class 📝 (model.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries * 🐛 (schemas.py): fix truncation length of text fields to 10 characters instead of 99999 🐛 (model.py): fix truncation length of text fields to 10 characters instead of 99999 🐛 (model.py): fix truncation length of text fields to 10 characters instead of 99999 🐛 (index.tsx): truncate resultMessage to 99999 characters and add message if text is too long * 🔧 (switchOutputView/index.tsx): Use useMemo to memoize resultMessage transformations for performance optimization * 🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (index.tsx): Fix logic to correctly handle resultMessageMemoized when it is an object * 📝 (model.py): refactor truncate_text function to truncate_long_strings for better clarity and consistency 📝 (model.py): update serialize_outputs and serialize_artifacts functions to use truncate_long_strings for string truncation 📝 (model.py): introduce MAX_TEXT_LENGTH constant for defining the maximum length of text to truncate in the models * 📝 (schemas.py): refactor serialize_data method in VertexBuildResponse class to use a new helper function truncate_long_strings for better code readability and maintainability * 🔧 (schemas.py): Move the `truncate_long_strings` function to a separate module to improve code organization and reusability 🔧 (model.py): Import the `truncate_long_strings` function from the correct module to fix the reference error 🔧 (model.py): Import the `truncate_long_strings` function from the correct module to fix the reference error * 📝 (util.py): add function truncate_long_strings to recursively truncate long strings in dictionaries and lists to prevent exceeding the maximum text length. * 📝 (constants.py): add constant MAX_TEXT_LENGTH with value 99999 for defining maximum text length allowed in the application * 📝 (model.py): update import path for truncate_long_strings function to match new location in util module * ✨ (test_truncate_long_strings_on_objects.py): Add unit tests for the function truncate_long_strings to ensure correct behavior when truncating long strings in various data structures 🐛 (switchOutputView/index.tsx): Fix truncation logic to correctly truncate long strings by adding ellipsis at the end instead of displaying additional text about truncation. * [autofix.ci] apply automated fixes * ✨ (test_truncate_long_strings_on_objects.py): Update import path for truncate_long_strings function 📝 (test_truncate_long_strings_on_objects.py): Add additional tests for handling negative, zero, and small max_length values in truncate_long_strings function * ♻️ (schemas.py): refactor import statement to use the updated module name util_strings instead of util for better clarity and consistency. * 📝 (model.py): Update import path for util_strings module to fix module import error 📝 (util.py): Remove redundant code for truncating long strings and move it to a separate util_strings module for better organization and separation of concerns. * 📝 (schemas.py): refactor serialize_data method to handle both BaseModel and non-BaseModel data inputs in VertexBuildResponse class * 📝 (util_strings.py): Update util_strings.py to improve string truncation function for dictionaries and lists 🔧 (test_truncate_long_strings_on_objects.py): Update test cases for string truncation function to cover additional scenarios and edge cases * Update src/backend/base/langflow/utils/util_strings.py Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@langflow.org> * 📝 (vite.config.mts): update environment variable MAX_FILE_SIZE to be defined in vite config for frontend to use in the application. * 📝 (constants.ts): update maxSizeFilesInBytes constant to use process.env.MAX_FILE_SIZE environment variable for configurable file size limit 📝 (constants.ts): add MAX_TEXT_LENGTH constant with a value of 99999 for maximum text length limit * 📝 (switchOutputView/index.tsx): import MAX_TEXT_LENGTH constant from shared constants file to improve code organization and reusability * ✨ (langflow/__main__.py): add support for defining maximum file size for upload in MB to improve user experience and prevent large file uploads * 🐛 (files.py): add validation to check if uploaded file size exceeds the maximum allowed size before processing it * ✨ (schemas.py): add max_file_size_upload field to ConfigResponse schema to handle maximum file size allowed for upload * 🔧 (vite.config.mts): remove MAX_FILE_SIZE environment variable configuration as it is no longer needed * ✨ (base.py): introduce max_file_size_upload setting to limit the file size for uploads in MB * 🐛 (util.py): add support for setting max_file_size_upload in update_settings function to allow configuring maximum file size for uploads * 📝 (inputFileComponent/index.tsx): add support for retrieving max file size upload from utility store to improve code modularity and reusability 🐛 (inputFileComponent/index.tsx): fix error handling logic to display error message when uploading a file fails * 📝 (constants.ts): remove maxSizeFilesInBytes constant as it is no longer used and update MAX_TEXT_LENGTH constant to a higher value * ✨ (use-get-config.ts): add functionality to set max file size upload value from config response * ✨ (utilityStore.ts): introduce maxFileSizeUpload property and setMaxFileSizeUpload function to handle maximum file size upload in bytes * ✨ (frontend): introduce maxFileSizeUpload property and setMaxFileSizeUpload method to handle maximum file size upload functionality in the UtilityStoreType * ♻️ (util_strings.py): refactor truncate_long_strings function to improve code readability and consistency by removing unnecessary whitespace and aligning assignment operators. * 🐛 (files.py): fix formatting issue in the raise statement to improve code readability and maintain consistency --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@langflow.org>
This commit is contained in:
Cristhian Zanforlin Lousa
GitHub
parent
b34a7c7f02
commit
948b150946
21 changed files with 414 additions and 21 deletions
|
|
@ -138,6 +138,11 @@ def run(
|
|||
help="Defines the number of retries for the health check.",
|
||||
envvar="LANGFLOW_HEALTH_CHECK_MAX_RETRIES",
|
||||
),
|
||||
max_file_size_upload: int = typer.Option(
|
||||
100,
|
||||
help="Defines the maximum file size for the upload in MB.",
|
||||
envvar="LANGFLOW_MAX_FILE_SIZE_UPLOAD",
|
||||
),
|
||||
):
|
||||
"""
|
||||
Run Langflow.
|
||||
|
|
@ -158,6 +163,7 @@ def run(
|
|||
auto_saving=auto_saving,
|
||||
auto_saving_interval=auto_saving_interval,
|
||||
health_check_max_retries=health_check_max_retries,
|
||||
max_file_size_upload=max_file_size_upload,
|
||||
)
|
||||
# create path object if path is provided
|
||||
static_files_dir: Path | None = Path(path) if path else None
|
||||
|
|
|
|||
|
|
@ -43,6 +43,12 @@ async def upload_file(
|
|||
storage_service: StorageService = Depends(get_storage_service),
|
||||
):
|
||||
try:
|
||||
max_file_size_upload = get_storage_service().settings_service.settings.max_file_size_upload
|
||||
if file.size > max_file_size_upload * 1024 * 1024:
|
||||
raise HTTPException(
|
||||
status_code=413, detail=f"File size is larger than the maximum file size {max_file_size_upload}MB."
|
||||
)
|
||||
|
||||
flow_id_str = str(flow_id)
|
||||
file_content = await file.read()
|
||||
timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@ from langflow.services.database.models.base import orjson_dumps
|
|||
from langflow.services.database.models.flow import FlowCreate, FlowRead
|
||||
from langflow.services.database.models.user import UserRead
|
||||
from langflow.services.tracing.schema import Log
|
||||
from langflow.utils.util_strings import truncate_long_strings
|
||||
|
||||
|
||||
class BuildStatus(Enum):
|
||||
|
|
@ -281,6 +282,12 @@ class VertexBuildResponse(BaseModel):
|
|||
timestamp: datetime | None = Field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
"""Timestamp of the build."""
|
||||
|
||||
@field_serializer("data")
|
||||
def serialize_data(self, data: ResultDataResponse) -> dict:
|
||||
data_dict = data.model_dump() if isinstance(data, BaseModel) else data
|
||||
truncated_data = truncate_long_strings(data_dict)
|
||||
return truncated_data
|
||||
|
||||
|
||||
class VerticesBuiltResponse(BaseModel):
|
||||
vertices: list[VertexBuildResponse]
|
||||
|
|
@ -341,3 +348,4 @@ class ConfigResponse(BaseModel):
|
|||
auto_saving: bool
|
||||
auto_saving_interval: int
|
||||
health_check_max_retries: int
|
||||
max_file_size_upload: int
|
||||
|
|
|
|||
|
|
@ -2,12 +2,14 @@ from datetime import datetime, timezone
|
|||
from typing import TYPE_CHECKING
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from pydantic import field_validator
|
||||
from pydantic import field_serializer, field_validator
|
||||
from sqlmodel import JSON, Column, Field, Relationship, SQLModel
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from langflow.services.database.models.flow.model import Flow
|
||||
|
||||
from langflow.utils.util_strings import truncate_long_strings
|
||||
|
||||
|
||||
class TransactionBase(SQLModel):
|
||||
timestamp: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
|
|
@ -32,6 +34,11 @@ class TransactionBase(SQLModel):
|
|||
value = UUID(value)
|
||||
return value
|
||||
|
||||
@field_serializer("outputs")
|
||||
def serialize_outputs(self, data) -> dict:
|
||||
truncated_data = truncate_long_strings(data)
|
||||
return truncated_data
|
||||
|
||||
|
||||
class TransactionTable(TransactionBase, table=True): # type: ignore
|
||||
__tablename__ = "transaction"
|
||||
|
|
|
|||
|
|
@ -8,6 +8,8 @@ from sqlmodel import JSON, Column, Field, Relationship, SQLModel
|
|||
if TYPE_CHECKING:
|
||||
from langflow.services.database.models.flow.model import Flow
|
||||
|
||||
from langflow.utils.util_strings import truncate_long_strings
|
||||
|
||||
|
||||
class VertexBuildBase(SQLModel):
|
||||
timestamp: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
|
|
@ -38,6 +40,16 @@ class VertexBuildBase(SQLModel):
|
|||
value = value.replace(tzinfo=timezone.utc)
|
||||
return value
|
||||
|
||||
@field_serializer("data")
|
||||
def serialize_data(self, data: dict) -> dict:
|
||||
truncated_data = truncate_long_strings(data)
|
||||
return truncated_data
|
||||
|
||||
@field_serializer("artifacts")
|
||||
def serialize_artifacts(self, data) -> dict:
|
||||
truncated_data = truncate_long_strings(data)
|
||||
return truncated_data
|
||||
|
||||
|
||||
class VertexBuildTable(VertexBuildBase, table=True): # type: ignore
|
||||
__tablename__ = "vertex_build"
|
||||
|
|
|
|||
|
|
@ -153,6 +153,8 @@ class Settings(BaseSettings):
|
|||
"""The interval in ms at which Langflow will auto save flows."""
|
||||
health_check_max_retries: int = 5
|
||||
"""The maximum number of retries for the health check."""
|
||||
max_file_size_upload: int = 100
|
||||
"""The maximum file size for the upload in MB."""
|
||||
|
||||
@field_validator("dev")
|
||||
@classmethod
|
||||
|
|
|
|||
|
|
@ -183,3 +183,5 @@ MESSAGE_SENDER_AI = "Machine"
|
|||
MESSAGE_SENDER_USER = "User"
|
||||
MESSAGE_SENDER_NAME_AI = "AI"
|
||||
MESSAGE_SENDER_NAME_USER = "User"
|
||||
|
||||
MAX_TEXT_LENGTH = 99999
|
||||
|
|
|
|||
|
|
@ -431,6 +431,7 @@ def update_settings(
|
|||
auto_saving: bool = True,
|
||||
auto_saving_interval: int = 1000,
|
||||
health_check_max_retries: int = 5,
|
||||
max_file_size_upload: int = 100,
|
||||
):
|
||||
"""Update the settings from a config file."""
|
||||
from langflow.services.utils import initialize_settings_service
|
||||
|
|
@ -463,6 +464,9 @@ def update_settings(
|
|||
if health_check_max_retries is not None:
|
||||
logger.debug(f"Setting health_check_max_retries to {health_check_max_retries}")
|
||||
settings_service.settings.update_settings(health_check_max_retries=health_check_max_retries)
|
||||
if max_file_size_upload is not None:
|
||||
logger.debug(f"Setting max_file_size_upload to {max_file_size_upload}")
|
||||
settings_service.settings.update_settings(max_file_size_upload=max_file_size_upload)
|
||||
|
||||
|
||||
def is_class_method(func, cls):
|
||||
|
|
|
|||
28
src/backend/base/langflow/utils/util_strings.py
Normal file
28
src/backend/base/langflow/utils/util_strings.py
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
from langflow.utils import constants
|
||||
|
||||
|
||||
def truncate_long_strings(data, max_length=None):
|
||||
"""
|
||||
Recursively traverse the dictionary or list and truncate strings longer than max_length.
|
||||
"""
|
||||
|
||||
if max_length is None:
|
||||
max_length = constants.MAX_TEXT_LENGTH
|
||||
|
||||
if max_length < 0 or not isinstance(data, dict | list):
|
||||
return data
|
||||
|
||||
if isinstance(data, dict):
|
||||
for key, value in data.items():
|
||||
if isinstance(value, str) and len(value) > max_length:
|
||||
data[key] = value[:max_length] + "..."
|
||||
elif isinstance(value, (dict | list)):
|
||||
truncate_long_strings(value, max_length)
|
||||
elif isinstance(data, list):
|
||||
for index, item in enumerate(data):
|
||||
if isinstance(item, str) and len(item) > max_length:
|
||||
data[index] = item[:max_length] + "..."
|
||||
elif isinstance(item, (dict | list)):
|
||||
truncate_long_strings(item, max_length)
|
||||
|
||||
return data
|
||||
|
|
@ -0,0 +1,98 @@
|
|||
from langflow.utils.util_strings import truncate_long_strings
|
||||
from langflow.utils.constants import MAX_TEXT_LENGTH
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"input_data, max_length, expected",
|
||||
[
|
||||
# Test case 1: Simple string truncation
|
||||
({"key": "a" * 100}, 10, {"key": "a" * 10 + "..."}),
|
||||
# Test case 2: Nested dictionary
|
||||
({"outer": {"inner": "b" * 100}}, 5, {"outer": {"inner": "b" * 5 + "..."}}),
|
||||
# Test case 3: List of strings
|
||||
(["short", "a" * 100, "also short"], 7, ["short", "a" * 7 + "...", "also sh" + "..."]),
|
||||
# Test case 4: Mixed nested structure
|
||||
(
|
||||
{"key1": ["a" * 100, {"nested": "b" * 100}], "key2": "c" * 100},
|
||||
8,
|
||||
{"key1": ["a" * 8 + "...", {"nested": "b" * 8 + "..."}], "key2": "c" * 8 + "..."},
|
||||
),
|
||||
# Test case 5: Empty structures
|
||||
({}, 10, {}),
|
||||
([], 10, []),
|
||||
# Test case 6: Strings at exact max_length
|
||||
({"exact": "a" * 10}, 10, {"exact": "a" * 10}),
|
||||
# Test case 7: Non-string values
|
||||
({"num": 12345, "bool": True, "none": None}, 5, {"num": 12345, "bool": True, "none": None}),
|
||||
# Test case 8: Unicode characters
|
||||
({"unicode": "こんにちは世界"}, 3, {"unicode": "こんに..."}),
|
||||
# Test case 9: Very large structure
|
||||
(
|
||||
{"key" + str(i): "value" * i for i in range(1000)},
|
||||
10,
|
||||
{"key" + str(i): ("value" * i)[:10] + "..." if len("value" * i) > 10 else "value" * i for i in range(1000)},
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_truncate_long_strings(input_data, max_length, expected):
|
||||
result = truncate_long_strings(input_data, max_length)
|
||||
assert result == expected
|
||||
|
||||
|
||||
def test_truncate_long_strings_default_max_length():
|
||||
long_string = "a" * (MAX_TEXT_LENGTH + 1)
|
||||
input_data = {"key": long_string}
|
||||
result = truncate_long_strings(input_data)
|
||||
assert len(result["key"]) == MAX_TEXT_LENGTH + 3 # +3 for the "..."
|
||||
|
||||
|
||||
def test_truncate_long_strings_no_modification():
|
||||
input_data = {"short": "short string", "nested": {"also_short": "another short string"}}
|
||||
result = truncate_long_strings(input_data, 100)
|
||||
assert result == input_data
|
||||
|
||||
|
||||
# Test for type preservation
|
||||
def test_truncate_long_strings_type_preservation():
|
||||
input_data = {"str": "a" * 100, "list": ["b" * 100], "dict": {"nested": "c" * 100}}
|
||||
result = truncate_long_strings(input_data, 10)
|
||||
assert isinstance(result, dict)
|
||||
assert isinstance(result["str"], str)
|
||||
assert isinstance(result["list"], list)
|
||||
assert isinstance(result["dict"], dict)
|
||||
|
||||
|
||||
# Test for in-place modification
|
||||
def test_truncate_long_strings_in_place_modification():
|
||||
input_data = {"key": "a" * 100}
|
||||
result = truncate_long_strings(input_data, 10)
|
||||
assert result is input_data # Check if the same object is returned
|
||||
|
||||
|
||||
# Test for invalid input
|
||||
def test_truncate_long_strings_invalid_input():
|
||||
input_string = "not a dict or list"
|
||||
result = truncate_long_strings(input_string, 10)
|
||||
assert result == input_string # The function should return the input unchanged
|
||||
|
||||
|
||||
# Updated test for negative max_length
|
||||
def test_truncate_long_strings_negative_max_length():
|
||||
input_data = {"key": "value"}
|
||||
result = truncate_long_strings(input_data, -1)
|
||||
assert result == input_data # Assuming the function ignores negative max_length
|
||||
|
||||
|
||||
# Additional test for zero max_length
|
||||
def test_truncate_long_strings_zero_max_length():
|
||||
input_data = {"key": "value"}
|
||||
result = truncate_long_strings(input_data, 0)
|
||||
assert result == {"key": "..."} # Assuming the function truncates to just "..."
|
||||
|
||||
|
||||
# Test for very small positive max_length
|
||||
def test_truncate_long_strings_small_max_length():
|
||||
input_data = {"key": "value"}
|
||||
result = truncate_long_strings(input_data, 1)
|
||||
assert result == {"key": "v..."} # Assuming the function keeps at least one character
|
||||
|
|
@ -1,4 +1,6 @@
|
|||
import { MAX_TEXT_LENGTH } from "@/constants/constants";
|
||||
import { LogsLogType, OutputLogType } from "@/types/api";
|
||||
import { useMemo } from "react";
|
||||
import DataOutputComponent from "../../../../../../components/dataOutputComponent";
|
||||
import ForwardedIconComponent from "../../../../../../components/genericIconComponent";
|
||||
import {
|
||||
|
|
@ -16,6 +18,7 @@ interface SwitchOutputViewProps {
|
|||
outputName: string;
|
||||
type: "Outputs" | "Logs";
|
||||
}
|
||||
|
||||
const SwitchOutputView: React.FC<SwitchOutputViewProps> = ({
|
||||
nodeId,
|
||||
outputName,
|
||||
|
|
@ -35,6 +38,35 @@ const SwitchOutputView: React.FC<SwitchOutputViewProps> = ({
|
|||
if (resultMessage?.raw) {
|
||||
resultMessage = resultMessage.raw;
|
||||
}
|
||||
|
||||
const resultMessageMemoized = useMemo(() => {
|
||||
if (
|
||||
typeof resultMessage === "string" &&
|
||||
resultMessage.length > MAX_TEXT_LENGTH
|
||||
) {
|
||||
resultMessage = `${resultMessage.substring(0, MAX_TEXT_LENGTH)}...`;
|
||||
}
|
||||
|
||||
if (Array.isArray(resultMessage)) {
|
||||
resultMessage = resultMessage.map((item) => {
|
||||
if (item && typeof item.data === "object") {
|
||||
const truncatedData = Object.fromEntries(
|
||||
Object.entries(item.data).map(([key, value]) => {
|
||||
if (typeof value === "string" && value.length > MAX_TEXT_LENGTH) {
|
||||
return [key, `${value.substring(0, MAX_TEXT_LENGTH)}...`];
|
||||
}
|
||||
return [key, value];
|
||||
}),
|
||||
);
|
||||
return { ...item, data: truncatedData };
|
||||
}
|
||||
return item;
|
||||
});
|
||||
}
|
||||
|
||||
return resultMessage;
|
||||
}, [resultMessage]);
|
||||
|
||||
return type === "Outputs" ? (
|
||||
<>
|
||||
<Case condition={!resultType || resultType === "unknown"}>
|
||||
|
|
@ -42,23 +74,23 @@ const SwitchOutputView: React.FC<SwitchOutputViewProps> = ({
|
|||
</Case>
|
||||
<Case condition={resultType === "error" || resultType === "ValueError"}>
|
||||
<ErrorOutput
|
||||
value={`${resultMessage.errorMessage}\n\n${resultMessage.stackTrace}`}
|
||||
value={`${resultMessageMemoized.errorMessage}\n\n${resultMessageMemoized.stackTrace}`}
|
||||
></ErrorOutput>
|
||||
</Case>
|
||||
|
||||
<Case condition={resultType === "text"}>
|
||||
<TextOutputView left={false} value={resultMessage} />
|
||||
<TextOutputView left={false} value={resultMessageMemoized} />
|
||||
</Case>
|
||||
|
||||
<Case condition={RECORD_TYPES.includes(resultType)}>
|
||||
<DataOutputComponent
|
||||
rows={
|
||||
Array.isArray(resultMessage)
|
||||
? (resultMessage as Array<any>).every((item) => item.data)
|
||||
? (resultMessage as Array<any>).map((item) => item.data)
|
||||
: resultMessage
|
||||
: Object.keys(resultMessage).length > 0
|
||||
? [resultMessage]
|
||||
Array.isArray(resultMessageMemoized)
|
||||
? (resultMessageMemoized as Array<any>).every((item) => item.data)
|
||||
? (resultMessageMemoized as Array<any>).map((item) => item.data)
|
||||
: resultMessageMemoized
|
||||
: Object.keys(resultMessageMemoized).length > 0
|
||||
? [resultMessageMemoized]
|
||||
: []
|
||||
}
|
||||
pagination={true}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
import { maxSizeFilesInBytes } from "@/constants/constants";
|
||||
import { usePostUploadFile } from "@/controllers/API/queries/files/use-post-upload-file";
|
||||
import { createFileUpload } from "@/helpers/create-file-upload";
|
||||
import { useUtilityStore } from "@/stores/utilityStore";
|
||||
import { useEffect } from "react";
|
||||
import {
|
||||
CONSOLE_ERROR_MSG,
|
||||
|
|
@ -23,7 +23,7 @@ export default function InputFileComponent({
|
|||
}: FileComponentType): JSX.Element {
|
||||
const currentFlowId = useFlowsManagerStore((state) => state.currentFlowId);
|
||||
const setErrorData = useAlertStore((state) => state.setErrorData);
|
||||
|
||||
const maxFileSizeUpload = useUtilityStore((state) => state.maxFileSizeUpload);
|
||||
// Clear component state
|
||||
useEffect(() => {
|
||||
if (disabled && value !== "") {
|
||||
|
|
@ -47,9 +47,9 @@ export default function InputFileComponent({
|
|||
createFileUpload({ multiple: false, accept: fileTypes?.join(",") }).then(
|
||||
(files) => {
|
||||
const file = files[0];
|
||||
if (file.size > maxSizeFilesInBytes) {
|
||||
if (file.size > maxFileSizeUpload) {
|
||||
setErrorData({
|
||||
title: INVALID_FILE_SIZE_ALERT(10),
|
||||
title: INVALID_FILE_SIZE_ALERT(maxFileSizeUpload / 1024 / 1024),
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
|
@ -68,8 +68,12 @@ export default function InputFileComponent({
|
|||
// sets the value to the user
|
||||
handleOnNewValue({ value: file.name, file_path });
|
||||
},
|
||||
onError: () => {
|
||||
onError: (error) => {
|
||||
console.error(CONSOLE_ERROR_MSG);
|
||||
setErrorData({
|
||||
title: "Error uploading file",
|
||||
list: [error.response?.data?.detail],
|
||||
});
|
||||
},
|
||||
},
|
||||
);
|
||||
|
|
|
|||
|
|
@ -916,4 +916,4 @@ export const COLOR_OPTIONS = {
|
|||
red: "var(--note-red)",
|
||||
};
|
||||
|
||||
export const maxSizeFilesInBytes = 10 * 1024 * 1024; // 10MB in bytes
|
||||
export const MAX_TEXT_LENGTH = 99999;
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import useFlowsManagerStore from "@/stores/flowsManagerStore";
|
||||
import { useUtilityStore } from "@/stores/utilityStore";
|
||||
import axios from "axios";
|
||||
import { useQueryFunctionType } from "../../../../types/api";
|
||||
import { api } from "../../api";
|
||||
|
|
@ -10,6 +11,7 @@ export interface ConfigResponse {
|
|||
auto_saving: boolean;
|
||||
auto_saving_interval: number;
|
||||
health_check_max_retries: number;
|
||||
max_file_size_upload: number;
|
||||
}
|
||||
|
||||
export const useGetConfig: useQueryFunctionType<undefined, ConfigResponse> = (
|
||||
|
|
@ -22,6 +24,9 @@ export const useGetConfig: useQueryFunctionType<undefined, ConfigResponse> = (
|
|||
const setHealthCheckMaxRetries = useFlowsManagerStore(
|
||||
(state) => state.setHealthCheckMaxRetries,
|
||||
);
|
||||
const setMaxFileSizeUpload = useUtilityStore(
|
||||
(state) => state.setMaxFileSizeUpload,
|
||||
);
|
||||
|
||||
const { query } = UseRequestProcessor();
|
||||
|
||||
|
|
@ -37,6 +42,7 @@ export const useGetConfig: useQueryFunctionType<undefined, ConfigResponse> = (
|
|||
setAutoSaving(data.auto_saving);
|
||||
setAutoSavingInterval(data.auto_saving_interval);
|
||||
setHealthCheckMaxRetries(data.health_check_max_retries);
|
||||
setMaxFileSizeUpload(data.max_file_size_upload);
|
||||
}
|
||||
return data;
|
||||
};
|
||||
|
|
|
|||
|
|
@ -1,7 +1,10 @@
|
|||
import { Button } from "../../../../../../components/ui/button";
|
||||
|
||||
import { INVALID_FILE_SIZE_ALERT } from "@/constants/alerts_constants";
|
||||
import { usePostUploadFile } from "@/controllers/API/queries/files/use-post-upload-file";
|
||||
import { createFileUpload } from "@/helpers/create-file-upload";
|
||||
import useAlertStore from "@/stores/alertStore";
|
||||
import { useUtilityStore } from "@/stores/utilityStore";
|
||||
import { useEffect, useState } from "react";
|
||||
import IconComponent from "../../../../../../components/genericIconComponent";
|
||||
import {
|
||||
|
|
@ -18,6 +21,8 @@ export default function IOFileInput({ field, updateValue }: IOFileInputProps) {
|
|||
const [isDragging, setIsDragging] = useState(false);
|
||||
const [filePath, setFilePath] = useState("");
|
||||
const [image, setImage] = useState<string | null>(null);
|
||||
const setErrorData = useAlertStore((state) => state.setErrorData);
|
||||
const maxFileSizeUpload = useUtilityStore((state) => state.maxFileSizeUpload);
|
||||
|
||||
useEffect(() => {
|
||||
if (filePath) {
|
||||
|
|
@ -74,6 +79,13 @@ export default function IOFileInput({ field, updateValue }: IOFileInputProps) {
|
|||
|
||||
const upload = async (file) => {
|
||||
if (file) {
|
||||
if (file.size > maxFileSizeUpload) {
|
||||
setErrorData({
|
||||
title: INVALID_FILE_SIZE_ALERT(maxFileSizeUpload / 1024 / 1024),
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
// Check if a file was selected
|
||||
const fileReader = new FileReader();
|
||||
fileReader.onload = (event) => {
|
||||
|
|
@ -93,7 +105,11 @@ export default function IOFileInput({ field, updateValue }: IOFileInputProps) {
|
|||
const { file_path } = data;
|
||||
setFilePath(file_path);
|
||||
},
|
||||
onError: () => {
|
||||
onError: (error) => {
|
||||
setErrorData({
|
||||
title: "Error uploading file",
|
||||
list: [error.response?.data?.detail],
|
||||
});
|
||||
console.error("Error occurred while uploading file");
|
||||
},
|
||||
},
|
||||
|
|
|
|||
|
|
@ -1,5 +1,7 @@
|
|||
import { INVALID_FILE_SIZE_ALERT } from "@/constants/alerts_constants";
|
||||
import { usePostUploadFile } from "@/controllers/API/queries/files/use-post-upload-file";
|
||||
import useAlertStore from "@/stores/alertStore";
|
||||
import { useUtilityStore } from "@/stores/utilityStore";
|
||||
import { useEffect, useRef, useState } from "react";
|
||||
import ShortUniqueId from "short-unique-id";
|
||||
import {
|
||||
|
|
@ -36,6 +38,7 @@ export default function ChatInput({
|
|||
const [inputFocus, setInputFocus] = useState<boolean>(false);
|
||||
const fileInputRef = useRef<HTMLInputElement>(null);
|
||||
const setErrorData = useAlertStore((state) => state.setErrorData);
|
||||
const maxFileSizeUpload = useUtilityStore((state) => state.maxFileSizeUpload);
|
||||
|
||||
useFocusOnUnlock(lockChat, inputRef);
|
||||
useAutoResizeTextArea(chatValue, inputRef);
|
||||
|
|
@ -62,10 +65,16 @@ export default function ChatInput({
|
|||
const fileInput = event.target as HTMLInputElement;
|
||||
file = fileInput.files?.[0] ?? null;
|
||||
}
|
||||
|
||||
if (file) {
|
||||
const fileExtension = file.name.split(".").pop()?.toLowerCase();
|
||||
|
||||
if (file.size > maxFileSizeUpload) {
|
||||
setErrorData({
|
||||
title: INVALID_FILE_SIZE_ALERT(maxFileSizeUpload / 1024 / 1024),
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (
|
||||
!fileExtension ||
|
||||
!ALLOWED_IMAGE_INPUT_EXTENSIONS.includes(fileExtension)
|
||||
|
|
@ -99,7 +108,7 @@ export default function ChatInput({
|
|||
return newFiles;
|
||||
});
|
||||
},
|
||||
onError: () => {
|
||||
onError: (error) => {
|
||||
setFiles((prev) => {
|
||||
const newFiles = [...prev];
|
||||
const updatedIndex = newFiles.findIndex((file) => file.id === id);
|
||||
|
|
@ -107,6 +116,10 @@ export default function ChatInput({
|
|||
newFiles[updatedIndex].error = true;
|
||||
return newFiles;
|
||||
});
|
||||
setErrorData({
|
||||
title: "Error uploading file",
|
||||
list: [error.response?.data?.detail],
|
||||
});
|
||||
},
|
||||
},
|
||||
);
|
||||
|
|
|
|||
|
|
@ -1,6 +1,8 @@
|
|||
import { INVALID_FILE_SIZE_ALERT } from "@/constants/alerts_constants";
|
||||
import { useDeleteBuilds } from "@/controllers/API/queries/_builds";
|
||||
import { usePostUploadFile } from "@/controllers/API/queries/files/use-post-upload-file";
|
||||
import { track } from "@/customization/utils/analytics";
|
||||
import { useUtilityStore } from "@/stores/utilityStore";
|
||||
import { useEffect, useRef, useState } from "react";
|
||||
import ShortUniqueId from "short-unique-id";
|
||||
import IconComponent from "../../../../components/genericIconComponent";
|
||||
|
|
@ -42,6 +44,7 @@ export default function ChatView({
|
|||
const updateFlowPool = useFlowStore((state) => state.updateFlowPool);
|
||||
const [id, setId] = useState<string>("");
|
||||
const { mutate: mutateDeleteFlowPool } = useDeleteBuilds();
|
||||
const maxFileSizeUpload = useUtilityStore((state) => state.maxFileSizeUpload);
|
||||
|
||||
//build chat history
|
||||
useEffect(() => {
|
||||
|
|
@ -173,6 +176,13 @@ export default function ChatView({
|
|||
if (files) {
|
||||
const file = files?.[0];
|
||||
const fileExtension = file.name.split(".").pop()?.toLowerCase();
|
||||
if (file.size > maxFileSizeUpload) {
|
||||
setErrorData({
|
||||
title: INVALID_FILE_SIZE_ALERT(maxFileSizeUpload / 1024 / 1024),
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (
|
||||
!fileExtension ||
|
||||
!ALLOWED_IMAGE_INPUT_EXTENSIONS.includes(fileExtension)
|
||||
|
|
@ -208,7 +218,7 @@ export default function ChatView({
|
|||
return newFiles;
|
||||
});
|
||||
},
|
||||
onError: () => {
|
||||
onError: (error) => {
|
||||
setFiles((prev) => {
|
||||
const newFiles = [...prev];
|
||||
const updatedIndex = newFiles.findIndex((file) => file.id === id);
|
||||
|
|
@ -216,6 +226,10 @@ export default function ChatView({
|
|||
newFiles[updatedIndex].error = true;
|
||||
return newFiles;
|
||||
});
|
||||
setErrorData({
|
||||
title: "Error uploading file",
|
||||
list: [error.response?.data?.detail],
|
||||
});
|
||||
},
|
||||
},
|
||||
);
|
||||
|
|
|
|||
|
|
@ -18,4 +18,7 @@ export const useUtilityStore = create<UtilityStoreType>((set, get) => ({
|
|||
playgroundScrollBehaves: "instant",
|
||||
setPlaygroundScrollBehaves: (behaves: ScrollBehavior) =>
|
||||
set({ playgroundScrollBehaves: behaves }),
|
||||
maxFileSizeUpload: 100 * 1024 * 1024, // 100MB in bytes
|
||||
setMaxFileSizeUpload: (maxFileSizeUpload: number) =>
|
||||
set({ maxFileSizeUpload: maxFileSizeUpload * 1024 * 1024 }),
|
||||
}));
|
||||
|
|
|
|||
|
|
@ -5,4 +5,6 @@ export type UtilityStoreType = {
|
|||
setHealthCheckTimeout: (timeout: string | null) => void;
|
||||
playgroundScrollBehaves: ScrollBehavior;
|
||||
setPlaygroundScrollBehaves: (behaves: ScrollBehavior) => void;
|
||||
maxFileSizeUpload: number;
|
||||
setMaxFileSizeUpload: (maxFileSizeUpload: number) => void;
|
||||
};
|
||||
|
|
|
|||
|
|
@ -0,0 +1,128 @@
|
|||
import { expect, test } from "@playwright/test";
|
||||
import * as dotenv from "dotenv";
|
||||
import { readFileSync } from "fs";
|
||||
import path from "path";
|
||||
|
||||
test("user should not be able to upload a file larger than the limit", async ({
|
||||
page,
|
||||
}) => {
|
||||
const maxFileSizeUpload = 0.001;
|
||||
await page.route("**/api/v1/config", (route) => {
|
||||
route.fulfill({
|
||||
status: 200,
|
||||
contentType: "application/json",
|
||||
body: JSON.stringify({
|
||||
max_file_size_upload: maxFileSizeUpload,
|
||||
}),
|
||||
headers: {
|
||||
"content-type": "application/json",
|
||||
...route.request().headers(),
|
||||
},
|
||||
});
|
||||
});
|
||||
test.skip(
|
||||
!process?.env?.OPENAI_API_KEY,
|
||||
"OPENAI_API_KEY required to run this test",
|
||||
);
|
||||
|
||||
if (!process.env.CI) {
|
||||
dotenv.config({ path: path.resolve(__dirname, "../../.env") });
|
||||
}
|
||||
|
||||
await page.goto("/");
|
||||
|
||||
await page.waitForTimeout(1000);
|
||||
|
||||
let modalCount = 0;
|
||||
try {
|
||||
const modalTitleElement = await page?.getByTestId("modal-title");
|
||||
if (modalTitleElement) {
|
||||
modalCount = await modalTitleElement.count();
|
||||
}
|
||||
} catch (error) {
|
||||
modalCount = 0;
|
||||
}
|
||||
|
||||
while (modalCount === 0) {
|
||||
await page.getByText("New Project", { exact: true }).click();
|
||||
await page.waitForTimeout(3000);
|
||||
modalCount = await page.getByTestId("modal-title")?.count();
|
||||
}
|
||||
|
||||
await page.getByRole("heading", { name: "Basic Prompting" }).click();
|
||||
await page.waitForSelector('[title="fit view"]', {
|
||||
timeout: 100000,
|
||||
});
|
||||
|
||||
await page.getByTitle("fit view").click();
|
||||
await page.getByTitle("zoom out").click();
|
||||
await page.getByTitle("zoom out").click();
|
||||
await page.getByTitle("zoom out").click();
|
||||
|
||||
let outdatedComponents = await page.getByTestId("icon-AlertTriangle").count();
|
||||
|
||||
while (outdatedComponents > 0) {
|
||||
await page.getByTestId("icon-AlertTriangle").first().click();
|
||||
await page.waitForTimeout(1000);
|
||||
outdatedComponents = await page.getByTestId("icon-AlertTriangle").count();
|
||||
}
|
||||
|
||||
await page
|
||||
.getByTestId("popover-anchor-input-api_key")
|
||||
.fill(process.env.OPENAI_API_KEY ?? "");
|
||||
|
||||
await page.getByTestId("dropdown_str_model_name").click();
|
||||
await page.getByTestId("gpt-4o-1-option").click();
|
||||
|
||||
await page.waitForSelector("text=Chat Input", { timeout: 30000 });
|
||||
|
||||
await page.getByText("Chat Input", { exact: true }).click();
|
||||
await page.getByTestId("more-options-modal").click();
|
||||
await page.getByTestId("edit-button-modal").click();
|
||||
await page.getByText("Close").last().click();
|
||||
|
||||
await page.getByText("Playground", { exact: true }).click();
|
||||
|
||||
// Read the image file as a binary string
|
||||
const filePath = "tests/assets/chain.png";
|
||||
const fileContent = readFileSync(filePath, "base64");
|
||||
|
||||
// Create the DataTransfer and File objects within the browser context
|
||||
const dataTransfer = await page.evaluateHandle(
|
||||
({ fileContent }) => {
|
||||
const dt = new DataTransfer();
|
||||
const byteCharacters = atob(fileContent);
|
||||
const byteNumbers = new Array(byteCharacters.length);
|
||||
for (let i = 0; i < byteCharacters.length; i++) {
|
||||
byteNumbers[i] = byteCharacters.charCodeAt(i);
|
||||
}
|
||||
const byteArray = new Uint8Array(byteNumbers);
|
||||
const file = new File([byteArray], "chain.png", { type: "image/png" });
|
||||
dt.items.add(file);
|
||||
return dt;
|
||||
},
|
||||
{ fileContent },
|
||||
);
|
||||
|
||||
await page.waitForSelector('[data-testid="input-chat-playground"]', {
|
||||
timeout: 100000,
|
||||
});
|
||||
|
||||
// Locate the target element
|
||||
const element = await page.getByTestId("input-chat-playground");
|
||||
|
||||
// Dispatch the drop event on the target element
|
||||
await element.dispatchEvent("drop", { dataTransfer });
|
||||
|
||||
await page.waitForTimeout(1000);
|
||||
|
||||
await page.waitForSelector("text=The file size is too large", {
|
||||
timeout: 10000,
|
||||
});
|
||||
|
||||
await expect(
|
||||
page.getByText(
|
||||
`The file size is too large. Please select a file smaller than ${maxFileSizeUpload}MB`,
|
||||
),
|
||||
).toBeVisible();
|
||||
});
|
||||
|
|
@ -36,7 +36,9 @@ export default defineConfig(({ mode }) => {
|
|||
},
|
||||
define: {
|
||||
"process.env.BACKEND_URL": JSON.stringify(env.BACKEND_URL),
|
||||
"process.env.ACCESS_TOKEN_EXPIRE_SECONDS": JSON.stringify(env.ACCESS_TOKEN_EXPIRE_SECONDS),
|
||||
"process.env.ACCESS_TOKEN_EXPIRE_SECONDS": JSON.stringify(
|
||||
env.ACCESS_TOKEN_EXPIRE_SECONDS,
|
||||
),
|
||||
"process.env.CI": JSON.stringify(env.CI),
|
||||
},
|
||||
plugins: [react(), svgr(), tsconfigPaths()],
|
||||
|
|
@ -47,4 +49,4 @@ export default defineConfig(({ mode }) => {
|
|||
},
|
||||
},
|
||||
};
|
||||
});
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue