fix: don't send duplicate messages to Agent (#8909)

* fix: update message input handling in LCAgentComponent and improve memory data retrieval

* Refactor MessageTextInput to MessageInput for consistency.
* Enhance input dictionary construction to handle different input types in LCAgentComponent.
* Update get_memory_data method to filter out current input value from retrieved messages.

* fix: update AgentComponent to include documentation link and improve input handling

* Added documentation link for AgentComponent.
* Removed memory inputs from the agent component for cleaner input management.
* Enhanced error handling in message_response method to ensure better validation and logging of exceptions.

* fix: enhance input handling in LCAgentComponent by updating message conversion

* Updated input dictionary construction in LCAgentComponent to use to_lc_message() for Message instances, improving input handling consistency.

* test: add regression test for message duplication in agent component

* Introduced a new test to verify that mathematical expressions do not experience message duplication when processed by the agent component.
* The test checks both input and output JSON to ensure correct handling of expressions like "2+2" without duplication errors.

* test: add workspace tag to regression test for message duplication in agent component

* Updated the regression test for mathematical expressions to include the "@workspace" tag, enhancing test categorization and organization.
* This change ensures better tracking and management of tests related to the agent component.

* fix: add temporary comment in get_memory_data to address message duplication

* Added a TODO comment in the get_memory_data method of AgentComponent to indicate a temporary fix for message duplication issues. This serves as a reminder to develop a more robust solution in the future.

* feat: add message extraction utility for BaseMessage

* Introduced a new helper function, _get_message_from_base_message, to extract and concatenate text content from BaseMessage instances, improving message handling.
* Updated input handling in handle_on_chain_start to utilize the new extraction function, ensuring consistent processing of input messages.

* refactor: standardize code snippets across starter project JSON files

* Updated the "value" field in multiple starter project JSON files to ensure consistent formatting and structure of code snippets.
* This change enhances readability and maintainability of the code examples provided in the starter projects.

* feat: add caching and content dictionary creation for images

* Introduced a new function, create_image_content_dict, to generate a content dictionary for multimodal inputs from image files, enhancing image handling capabilities.
* Implemented LRU caching to optimize performance for repeated image processing.
* Added comprehensive error handling and documentation for better usability and maintainability.

* refactor: update message handling to utilize create_image_content_dict

* Replaced direct image URL creation with create_image_content_dict for improved image content handling in the Data and Message classes.
* Adjusted the order of content in human messages to ensure text appears first, enhancing message structure and clarity.
* Removed deprecated to_lc_message method to streamline the codebase and improve maintainability.

* docs: enhance _get_message_from_base_message docstring for clarity

* Expanded the docstring for the _get_message_from_base_message function to provide detailed information on input types, expected behavior, and examples of usage.
* Improved documentation aims to enhance usability and maintainability of the code by clarifying how to extract text content from BaseMessage instances.

* refactor: enhance image path handling and update message content structure

* Modified the get_file_paths function to support both Image objects and string paths for improved flexibility in file handling.
* Updated test cases to reflect changes in image content structure, ensuring consistency in type and source type attributes.
* Introduced new tests for create_image_content_dict to validate successful creation and error handling for image content dictionaries.

* refactor: streamline message extraction in handle_on_chain_start

* Removed the _get_message_from_base_message function to simplify the codebase.
* Updated handle_on_chain_start to directly use the text method of BaseMessage for extracting message content, enhancing clarity and maintainability.

* feat: enhance input handling for multimodal messages

* Added functionality to process image content within input messages, ensuring images are included in chat history as HumanMessage instances.
* Updated input handling logic to separate image types from text, improving the structure and clarity of message content.
* This enhancement supports better management of multimodal inputs in the agent's chat history.

* feat: add to_lc_message method for converting Data to BaseMessage

* Introduced the to_lc_message method in the Message class to facilitate conversion of Data instances to BaseMessage.
* Implemented logic to handle both HumanMessage and AIMessage based on the presence of required keys and sender type.
* Added logging for missing required keys to improve debugging and maintainability.

* refactor: simplify sender check in Message class

* Updated the sender validation logic in the Message class to remove unnecessary checks for missing sender values.
* This change enhances code clarity and maintains the intended functionality for handling user messages with associated files.

* test: update test_message_from_human_text to reflect content type change

* Modified the test for message conversion to assert that lc_message.content is a string instead of a list.
* Updated assertions to ensure the content matches the expected text, enhancing test accuracy and reliability.

* fix: update sender validation in Message class and adjust test case

* Modified the sender validation logic to handle cases where the sender is not specified, defaulting to HumanMessage.
* Updated the corresponding test case to reflect this change, ensuring accurate type assertion for lc_message when no sender is provided.

* refactor: update import statements for consistency and clarity

* Replaced the import of BaseModel from langchain.pydantic_v1 with the direct import from pydantic to streamline dependencies.
* This change enhances code clarity and aligns with best practices for managing imports in the codebase.

---------

Co-authored-by: Edwin Jose <edwin.jose@datastax.com>
Co-authored-by: Carlos Coelho <80289056+carlosrcoelho@users.noreply.github.com>

src/backend/tests/unit/schema/test_schema_data.py

@@ -39,9 +39,10 @@ class TestDataSchema:
         assert message.content[0] == {"type": "text", "text": "Check out this image"}
 
         # Check image content
-        assert message.content[1]["type"] == "image_url"
-        assert "url" in message.content[1]["image_url"]
-        assert message.content[1]["image_url"]["url"].startswith("data:image/png;base64,")
+        assert message.content[1]["type"] == "image"
+        assert message.content[1]["source_type"] == "url"
+        assert "url" in message.content[1]
+        assert message.content[1]["url"].startswith("data:image/png;base64,")
 
     def test_data_to_message_with_multiple_images(self, sample_image, tmp_path):
         """Test conversion of Data to Message with multiple images."""
@@ -66,9 +67,15 @@ class TestDataSchema:
         assert message.content[0]["type"] == "text"
 
         # Check both images
-        assert message.content[1]["type"] == "image_url"
-        assert message.content[2]["type"] == "image_url"
-        assert all(content["image_url"]["url"].startswith("data:image/png;base64,") for content in message.content[1:])
+        assert message.content[1]["type"] == "image"
+        assert message.content[1]["source_type"] == "url"
+        assert "url" in message.content[1]
+        assert message.content[1]["url"].startswith("data:image/png;base64,")
+
+        assert message.content[2]["type"] == "image"
+        assert message.content[2]["source_type"] == "url"
+        assert "url" in message.content[2]
+        assert message.content[2]["url"].startswith("data:image/png;base64,")
 
     def test_data_to_message_ai_response(self):
         """Test conversion of Data to AI Message."""

src/backend/tests/unit/schema/test_schema_message.py

@@ -65,6 +65,7 @@ def test_message_from_human_text():
     lc_message = message.to_lc_message()
 
     assert isinstance(lc_message, HumanMessage)
+    assert isinstance(lc_message.content, str)
     assert lc_message.content == text
 
 
@@ -94,9 +95,10 @@ def test_message_with_single_image(sample_image):
     assert lc_message.content[0] == {"type": "text", "text": text}
 
     # Check image content
-    assert lc_message.content[1]["type"] == "image_url"
-    assert "url" in lc_message.content[1]["image_url"]
-    assert lc_message.content[1]["image_url"]["url"].startswith("data:image/png;base64,")
+    assert lc_message.content[1]["type"] == "image"
+    assert lc_message.content[1]["source_type"] == "url"
+    assert "url" in lc_message.content[1]
+    assert lc_message.content[1]["url"].startswith("data:image/png;base64,")
 
 
 def test_message_with_multiple_images(sample_image, langflow_cache_dir):
@@ -129,7 +131,10 @@ def test_message_with_multiple_images(sample_image, langflow_cache_dir):
 
     # Check both images
     assert all(
-        content["type"] == "image_url" and content["image_url"]["url"].startswith("data:image/png;base64,")
+        content["type"] == "image"
+        and content["source_type"] == "url"
+        and "url" in content
+        and content["url"].startswith("data:image/png;base64,")
         for content in lc_message.content[1:]
     )
 
@@ -169,10 +174,9 @@ def test_message_serialization():
 def test_message_to_lc_without_sender():
     """Test converting a message without sender to langchain message."""
     message = Message(text="Test message")
-    # When no sender is specified, it defaults to HumanMessage
+    # When no sender is specified, it defaults to AIMessage
     lc_message = message.to_lc_message()
     assert isinstance(lc_message, HumanMessage)
-    assert lc_message.content == "Test message"
 
 
 def test_timestamp_serialization():

src/backend/tests/unit/utils/test_image_utils.py

@@ -1,7 +1,7 @@
 import base64
 
 import pytest
-from langflow.utils.image import convert_image_to_base64, create_data_url
+from langflow.utils.image import convert_image_to_base64, create_data_url, create_image_content_dict
 
 
 @pytest.fixture
@@ -70,3 +70,39 @@ def test_create_data_url_unrecognized_extension(tmp_path):
     invalid_file.touch()
     with pytest.raises(ValueError, match="Could not determine MIME type"):
         create_data_url(invalid_file)
+
+
+def test_create_image_content_dict_success(sample_image):
+    """Test successful creation of image content dict."""
+    content_dict = create_image_content_dict(sample_image)
+    assert content_dict["type"] == "image"
+    assert content_dict["source_type"] == "url"
+    assert "url" in content_dict
+    assert content_dict["url"].startswith("data:image/png;base64,")
+    # Verify the base64 part is valid
+    base64_part = content_dict["url"].split(",")[1]
+    assert base64.b64decode(base64_part)
+
+
+def test_create_image_content_dict_with_custom_mime(sample_image):
+    """Test creation of image content dict with custom MIME type."""
+    custom_mime = "image/custom"
+    content_dict = create_image_content_dict(sample_image, mime_type=custom_mime)
+    assert content_dict["type"] == "image"
+    assert content_dict["source_type"] == "url"
+    assert "url" in content_dict
+    assert content_dict["url"].startswith(f"data:{custom_mime};base64,")
+
+
+def test_create_image_content_dict_invalid_file():
+    """Test creation of image content dict with invalid file."""
+    with pytest.raises(FileNotFoundError):
+        create_image_content_dict("nonexistent.jpg")
+
+
+def test_create_image_content_dict_unrecognized_extension(tmp_path):
+    """Test creation of image content dict with unrecognized file extension."""
+    invalid_file = tmp_path / "test.unknown"
+    invalid_file.touch()
+    with pytest.raises(ValueError, match="Could not determine MIME type"):
+        create_image_content_dict(invalid_file)