Commit graph

12 commits

Author SHA1 Message Date
Gabriel Luiz Freitas Almeida
c73070cd52
refactor: Implement unified serialization function (#6044)
* feat: Implement serialization functions for various data types and add a unified serialize method

* feat: Enhance serialization by adding support for primitive types, enums, and generic types

* fix: Update Pinecone integration to use VectorStore and handle import errors gracefully

* test: Add hypothesis-based tests for serialization functions across various data types

* refactor: Replace custom serialization logic with unified serialize function for consistency and maintainability

* refactor: Replace recursive serialization function with unified serialize method for improved clarity and maintainability

* refactor: Replace custom serialization logic with unified serialize function for improved consistency and clarity

* refactor: Enhance serialization logic by adding instance handling and streamlining type checks

* refactor: Remove custom dictionary serialization from ResultDataResponse for streamlined handling

* refactor: Enhance serialization in ResultDataResponse by adding max_items_length for improved handling of outputs, logs, messages, and artifacts

* refactor: Move MAX_ITEMS_LENGTH and MAX_TEXT_LENGTH constants to serialization module for better organization

* refactor: Simplify message serialization in Log model by utilizing unified serialize function

* refactor: Remove unnecessary pytest marker from TestSerializationHypothesis class

* optimize _serialize_bytes

Co-authored-by: codeflash-ai[bot] <148906541+codeflash-ai[bot]@users.noreply.github.com>

* feat: Add support for numpy integer type serialization

* feat: Enhance serialization with support for pandas and numpy types

* test: Add comprehensive serialization tests for numpy and pandas types

* fix: Update _serialize_dispatcher to return string representation for unsupported types

* fix: Update _serialize_dispatcher to return the object directly instead of its string representation

* optmize conditional

Co-authored-by: codeflash-ai[bot] <148906541+codeflash-ai[bot]@users.noreply.github.com>

* optimize length check

Co-authored-by: codeflash-ai[bot] <148906541+codeflash-ai[bot]@users.noreply.github.com>

* fix: Update string and list truncation to include ellipsis for clarity

* fix: Update _serialize_primitive to exclude string type from primitive handling

* feat: Enhance serialization to handle numpy types and introduce unserializable sentinel

* fix: Update test cases for serialization of numpy boolean values for consistency

---------

Co-authored-by: codeflash-ai[bot] <148906541+codeflash-ai[bot]@users.noreply.github.com>
2025-02-03 15:12:03 +00:00
Christophe Bornet
e91bcc2520
ref: Some ruff fixes from preview (#5420)
* Some ruff fixes from preview

* [autofix.ci] apply automated fixes

* [autofix.ci] apply automated fixes (attempt 2/3)

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-12-28 21:25:35 +00:00
Ítalo Johnny
4be6b04d8c
fix: validate and test database connection URLs (#5178)
* test: add unit test for database url validation

* feat: add function to validate database urls

* refactor: use new database url validation function

* fix: ruff errors

* refactor: validate database urls using sqlalchemy

* test: add more cases for database url validation
2024-12-17 17:29:53 +00:00
Gabriel Luiz Freitas Almeida
98274633db
fix: remove use of ImagePromptTemplate in image handling and adds image utils (#4467)
* remove poetry.lock

* fix: update langchain-core dependency version to 0.3.15

* feat: add functions to convert images to base64 and create data URLs

* refactor: simplify image URL handling by replacing ImagePromptTemplate with create_data_url function

* Fix image URL structure in data schema to use nested dictionary format

* Add unit tests for Data schema message conversion with text and images

* test: add unit tests for image utility functions to validate base64 conversion and data URL creation

* Refactor image URL generation to use `create_data_url` utility function instead of `ImagePromptTemplate`

* Add unit tests for message handling and image processing in schema module

- Introduce fixtures for temporary cache directory and sample image creation.
- Add tests for message creation from human and AI text.
- Implement tests for messages with single and multiple images.
- Include tests for invalid image paths and messages without sender.
- Add message serialization and conversion tests.
- Ensure cleanup of cache directory after tests.

* Use platformdirs to determine cache directory paths in test_schema_message.py
2024-11-08 08:24:19 -08:00
Christophe Bornet
e4e1e1bd86
ref: Add missing __init__.py files in tests (#4180)
Add missing __init__.py files in tests
2024-10-17 18:21:56 +00:00
Christophe Bornet
45c8f98692
ref: Auto-fix ruff rules in tests (#4154) 2024-10-16 15:42:36 +00:00
Christophe Bornet
047077cee4
ref: Remove autouse from pytest client fixture (#4158)
Remove autouse from pytest client fixture
2024-10-15 22:55:10 +00:00
Cristhian Zanforlin Lousa
844efc1f73
fix: escape directory to prevent \n on Windows directory name to fail on Pathlib + Tests (#4101)
* 📝 (utils.py): add format_directory_path function to properly escape and format directory paths for consistency and validity

*  (test_format_directory_path.py): add unit tests for the format_directory_path function to ensure correct formatting of directory paths
📝 (test_format_directory_path.py): add documentation and examples for different types of directory paths in the unit tests to improve code readability and maintainability

* 🐛 (utils.py): fix the incorrect replacement of backslashes with newline characters in the format_directory_path function
📝 (test_rewrite_file_path.py): update test cases and function names to reflect the changes made in the format_directory_path function in utils.py

* 🐛 (test_format_directory_path.py): update parameter name from 'path' to 'input_path' for clarity and consistency
📝 (test_format_directory_path.py): improve test case descriptions and handle newline characters in paths correctly
2024-10-14 16:54:05 +00:00
Cristhian Zanforlin Lousa
d0fdc56890
fix: truncate "params" column on vertex_build table to prevent memory heap on database (#4118)
*  (model.py): add new method `serialize_params` to serialize parameters data in VertexBuildBase class

* 🐛 (util_strings.py): fix truncation logic to correctly handle long strings by truncating them with ellipsis if they exceed the maximum length

*  (test_truncate_long_strings.py): add unit tests for the truncate_long_strings function to ensure it works correctly with various input scenarios

*  (test_truncate_long_strings_on_objects.py): update test assertion message to reflect the expected behavior of the function
2024-10-11 18:10:36 +00:00
Cristhian Zanforlin Lousa
948b150946
feat: truncate parsed uploads to prevent database and frontend blocking caused by excessively large files (#3914)
* 📝 (constants.ts): increase maxSizeFilesInBytes constant value from 10MB to 100MB to allow larger file uploads

* 🐛 (inputFileComponent): fix bug in setting the maximum file size alert message to display the correct file size limit of 100 bytes instead of 10 bytes

* 📝 (schemas.py): Add a new field_serializer method to serialize data in VertexBuildResponse class
📝 (schemas.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries
📝 (model.py): Add a new field_serializer method to serialize outputs in TransactionBase class
📝 (model.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries
📝 (model.py): Add a new field_serializer method to serialize data and artifacts in VertexBuildBase class
📝 (model.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries

* 🐛 (schemas.py): fix truncation length of text fields to 10 characters instead of 99999
🐛 (model.py): fix truncation length of text fields to 10 characters instead of 99999
🐛 (model.py): fix truncation length of text fields to 10 characters instead of 99999
🐛 (index.tsx): truncate resultMessage to 99999 characters and add message if text is too long

* 🔧 (switchOutputView/index.tsx): Use useMemo to memoize resultMessage transformations for performance optimization

* 🐛 (model.py): Fix typo in the path for 'base_retriever' data field
🐛 (model.py): Fix typo in the path for 'base_retriever' data field
🐛 (model.py): Fix typo in the path for 'base_retriever' data field
🐛 (model.py): Fix typo in the path for 'base_retriever' data field
🐛 (index.tsx): Fix logic to correctly handle resultMessageMemoized when it is an object

* 📝 (model.py): refactor truncate_text function to truncate_long_strings for better clarity and consistency
📝 (model.py): update serialize_outputs and serialize_artifacts functions to use truncate_long_strings for string truncation
📝 (model.py): introduce MAX_TEXT_LENGTH constant for defining the maximum length of text to truncate in the models

* 📝 (schemas.py): refactor serialize_data method in VertexBuildResponse class to use a new helper function truncate_long_strings for better code readability and maintainability

* 🔧 (schemas.py): Move the `truncate_long_strings` function to a separate module to improve code organization and reusability
🔧 (model.py): Import the `truncate_long_strings` function from the correct module to fix the reference error
🔧 (model.py): Import the `truncate_long_strings` function from the correct module to fix the reference error

* 📝 (util.py): add function truncate_long_strings to recursively truncate long strings in dictionaries and lists to prevent exceeding the maximum text length.

* 📝 (constants.py): add constant MAX_TEXT_LENGTH with value 99999 for defining maximum text length allowed in the application

* 📝 (model.py): update import path for truncate_long_strings function to match new location in util module

*  (test_truncate_long_strings_on_objects.py): Add unit tests for the function truncate_long_strings to ensure correct behavior when truncating long strings in various data structures
🐛 (switchOutputView/index.tsx): Fix truncation logic to correctly truncate long strings by adding ellipsis at the end instead of displaying additional text about truncation.

* [autofix.ci] apply automated fixes

*  (test_truncate_long_strings_on_objects.py): Update import path for truncate_long_strings function
📝 (test_truncate_long_strings_on_objects.py): Add additional tests for handling negative, zero, and small max_length values in truncate_long_strings function

* ♻️ (schemas.py): refactor import statement to use the updated module name util_strings instead of util for better clarity and consistency.

* 📝 (model.py): Update import path for util_strings module to fix module import error
📝 (util.py): Remove redundant code for truncating long strings and move it to a separate util_strings module for better organization and separation of concerns.

* 📝 (schemas.py): refactor serialize_data method to handle both BaseModel and non-BaseModel data inputs in VertexBuildResponse class

* 📝 (util_strings.py): Update util_strings.py to improve string truncation function for dictionaries and lists
🔧 (test_truncate_long_strings_on_objects.py): Update test cases for string truncation function to cover additional scenarios and edge cases

* Update src/backend/base/langflow/utils/util_strings.py

Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@langflow.org>

* 📝 (vite.config.mts): update environment variable MAX_FILE_SIZE to be defined in vite config for frontend to use in the application.

* 📝 (constants.ts): update maxSizeFilesInBytes constant to use process.env.MAX_FILE_SIZE environment variable for configurable file size limit
📝 (constants.ts): add MAX_TEXT_LENGTH constant with a value of 99999 for maximum text length limit

* 📝 (switchOutputView/index.tsx): import MAX_TEXT_LENGTH constant from shared constants file to improve code organization and reusability

*  (langflow/__main__.py): add support for defining maximum file size for upload in MB to improve user experience and prevent large file uploads

* 🐛 (files.py): add validation to check if uploaded file size exceeds the maximum allowed size before processing it

*  (schemas.py): add max_file_size_upload field to ConfigResponse schema to handle maximum file size allowed for upload

* 🔧 (vite.config.mts): remove MAX_FILE_SIZE environment variable configuration as it is no longer needed

*  (base.py): introduce max_file_size_upload setting to limit the file size for uploads in MB

* 🐛 (util.py): add support for setting max_file_size_upload in update_settings function to allow configuring maximum file size for uploads

* 📝 (inputFileComponent/index.tsx): add support for retrieving max file size upload from utility store to improve code modularity and reusability
🐛 (inputFileComponent/index.tsx): fix error handling logic to display error message when uploading a file fails

* 📝 (constants.ts): remove maxSizeFilesInBytes constant as it is no longer used and update MAX_TEXT_LENGTH constant to a higher value

*  (use-get-config.ts): add functionality to set max file size upload value from config response

*  (utilityStore.ts): introduce maxFileSizeUpload property and setMaxFileSizeUpload function to handle maximum file size upload in bytes

*  (frontend): introduce maxFileSizeUpload property and setMaxFileSizeUpload method to handle maximum file size upload functionality in the UtilityStoreType

* ♻️ (util_strings.py): refactor truncate_long_strings function to improve code readability and consistency by removing unnecessary whitespace and aligning assignment operators.

* 🐛 (files.py): fix formatting issue in the raise statement to improve code readability and maintain consistency

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@langflow.org>
2024-09-27 15:44:05 +00:00
Cristhian Zanforlin Lousa
ad97ee9830
fix: image not been sent on ChatInputComponent during runtime of building a flow (#3862)
* refactor: Refactor file path rewriting logic

This commit refactors the logic for rewriting file paths in the `rewrite_file_path` function. The function now splits the file path by "/" and checks if it has at least two parts. If it does, it creates a consistent file path by concatenating the last two parts. If not, it returns the original file path. This change improves the consistency of file paths in the codebase.

Refactor the file path rewriting logic in the `rewrite_file_path` function.

* refactor: Refactor file path rewriting logic and treat file paths in InterfaceVertex

* Refactor file path rewriting logic and treat file paths in InterfaceVertex

* Refactor file path rewriting logic and treat file paths in InterfaceVertex

* Refactor file path rewriting logic and treat file paths in InterfaceVertex

*  (general-bugs-shard-3836.spec.ts): update test description to be more specific about the tool used for sending images on chat
📝 (general-bugs-shard-3836.spec.ts): remove unused import of readFileSync from fs module

*  (test_rewrite_file_path.py): add unit tests for the rewrite_file_path function to ensure correct behavior with various file path scenarios

* ♻️ (utils.py): refactor file_path function to handle both forward and backward slashes and extract file path after drive letter if present

* style: fix single quotes, commas, and spaces

---------

Co-authored-by: italojohnny <italojohnnydosanjos@gmail.com>
2024-09-23 12:19:09 -03:00
Cristhian Zanforlin Lousa
5dc3ff2b1f
bugfix: parse password on db connection string when it has @ on It (#3173)
*  (test_connection_string.py): add unit test for the transform_connection_string function to ensure correct transformation of connection strings

* 📝 (pgvector.py): add support for parsing and transforming connection string in PGVectorStoreComponent to improve security and maintainability
📝 (connection_string_parser.py): create utility function to transform connection string by encoding password to improve security
📝 (test_connection_string_parser.py): add unit tests for transform_connection_string function to ensure correct transformation of connection string

* test: fix import

---------

Co-authored-by: italojohnny <italojohnnydosanjos@gmail.com>
2024-08-05 09:15:33 -03:00