refactor: add pagination on folders (#4020)

* ⬆️ (uv.lock): Update authlib version from 1.3.2 to 1.3.1
⬆️ (uv.lock): Update httpx version from 0.27.2 to 0.27.0
⬆️ (uv.lock): Add new package grpcio-health-checking version 1.62.3

⬆️ (uv.lock): upgrade weaviate-client package from version 3.26.7 to 4.8.1 and add new dependencies grpcio, grpcio-health-checking, grpcio-tools, httpx, and pydantic. Update package source and distribution URLs accordingly.

* ✨ (flows.py): Add pagination support for retrieving a list of flows to improve performance and user experience
📝 (flows.py): Update documentation to reflect changes made for pagination and additional query parameters
📝 (folders.py): Add a new endpoint to retrieve a folder with paginated flows for better organization and readability

* ✨ (model.py): add PaginatedFlowResponse class to handle paginated flow responses in API calls

* ✨ (test_database.py): add support for fetching all flows by adding "get_all" parameter to the request
🐛 (test_database.py): fix endpoint for fetching read-only starter projects to use correct URL path

* packages changes

* packages changes

* 📝 (paginatorComponent/index.tsx): refactor PaginatorComponent to use constants for default values and improve code readability
🐛 (paginatorComponent/index.tsx): fix issue with setting maxPageIndex when pages prop is provided to PaginatorComponent

* 🐛 (storeCardComponent): fix height of store card component to prevent overflow and improve layout aesthetics

* ✨ (constants.ts): introduce new constants for search tabs, pagination settings, and store pagination settings to enhance user experience and improve functionality

* ✨ (use-post-login-user.ts): add queryClient to UseRequestProcessor to enable refetching queries on mutation settled state
📝 (use-post-login-user.ts): update useLoginUser function to include onSettled callback in options to refetch queries after mutation settles

* ✨ (use-get-basic-examples.ts): introduce a new query function to fetch basic examples of flows from the API and update the state with the fetched data.

* ✨ (use-get-refresh-flows.ts): introduce new functionality to fetch and process flows with query parameters for components_only, get_all, folder_id, remove_example_flows, page, and size. This allows for more flexible and specific retrieval of flow data.

* ✨ (use-get-folder.ts): Add support for pagination and filtering options in the useGetFolderQuery function to enhance flexibility and customization for fetching folder data.

* 🔧 (use-get-folders.ts): Remove unused code related to refreshFlows and setStarterProjectId to improve code readability and maintainability
✨ (use-get-folders.ts): Update useGetFoldersQuery to setFolders with the fetched data instead of filtering out starter projects to simplify the logic and improve performance

* ✨ (use-upload-flow.ts): update refreshFlows function call to include an object with get_all property set to true to fetch all flows when refreshing

* 🔧 (codeAreaModal/index.tsx): remove unused import postCustomComponent to clean up code and improve readability

* 📝 (AdminPage/index.tsx): Update initial page size and index values to use constants for better maintainability
📝 (AdminPage/index.tsx): Update resetFilter function to set page size and index using constants for consistency
📝 (AdminPage/index.tsx): Update logic to handle loading state and empty user list based on isIdle state for better user experience
📝 (AdminPage/index.tsx): Update PaginatorComponent props to use constant for rows count and simplify paginate function assignment

* ✨ (AppInitPage/index.tsx): introduce new queries to fetch basic examples and folders data for improved functionality and user experience

* ✨ (FlowPage/index.tsx): update refreshFlows function call to include a parameter to get all flows at once for better performance.

* ✨ (frontend): introduce flowsPagination and setFlowsPagination functions to manage pagination for flows in the UtilityStoreType

* ✨ (frontend): introduce new 'folders' and 'setFolders' properties to FoldersStoreType to manage folder data efficiently

* ✨ (types.ts): introduce Pagination type to define structure for pagination data in frontend application

* ✨ (frontend): introduce PaginatedFlowsType to represent paginated flow data structure in the application.

* ✨ (frontend): add optional 'pages' property to PaginatorComponentType to allow customization of the number of pages displayed in the paginator

* ✨ (utilityStore.ts): introduce flowsPagination object with default values for page and size to manage pagination in the utility store

* ✨ (foldersStore.tsx): introduce new state 'folders' and setter function 'setFolders' to manage folder data in the store

* ✨ (ViewPage/index.tsx): update refreshFlows function call to include a parameter to get all flows at once, improving efficiency and reducing unnecessary calls to the backend.

* 📝 (StorePage/index.tsx): update constants import to include new pagination constants for better organization
♻️ (StorePage/index.tsx): refactor pagination logic to use new pagination constants for clarity and consistency throughout the file

* ✨ (componentsComponent/index.tsx): Add support for pagination feature in ComponentsComponent to improve user experience and performance.

* ✨ (myCollectionComponent/index.tsx): introduce pagination and search functionality to improve user experience and data handling in the MyCollectionComponent component

* ✨ (Playground/index.tsx): Update refreshFlows function call to include a parameter to get all flows at once for better performance.

* ✨ (entities/index.tsx): introduce PaginatedFolderType to represent a folder with pagination information for better organization and handling of paginated data.

* ✨ (headerTabsSearchComponent/index.tsx): add support for changing tabs and searching functionality in headerTabsSearchComponent
📝 (headerTabsSearchComponent/index.tsx): update tabsOptions to use constant SEARCH_TABS for consistency and reusability

* 📝 (folders.py): Remove redundant import and unused code related to FolderWithPaginatedFlows class
🔧 (folders.py): Update query conditions to use explicit boolean values instead of implicit truthiness for better clarity and readability

* 📝 (folders.py): reorganize imports to improve readability and maintain consistency

* 📝 (flows.py): import FlowSummary model to support new functionality in the API
📝 (flows.py): add header_flows parameter to read_flows function to return a simplified list of flows if set to true

* 📝 (folders.py): remove unnecessary empty line to improve code readability

* ✨ (model.py): introduce new FlowSummary class to represent a summary of flow data for better organization and readability

* ✨ (pagination_model.py): introduce a new model 'FolderWithPaginatedFlows' to represent a folder with paginated flows for better organization and readability.

* 📝 (cardComponent/index.tsx): add missing semicolon to improve code consistency
♻️ (cardComponent/index.tsx): refactor logic to use data parameter directly instead of fetching flow by id to simplify code and improve readability

* ✨ (use-get-refresh-flows.ts): introduce new query parameter 'header_flows' to support fetching header flows in API requests.

* ✨ (use-get-folders.ts): add functionality to refresh flows when fetching folders to ensure data consistency and up-to-date information.

* ✨ (use-upload-flow.ts): add support for fetching header flows along with all flows when refreshing flows in useUploadFlow hook

* ✨ (use-redirect-flow-card-click.tsx): introduce a new custom hook 'useFlowCardClick' to handle flow card click events in the newFlowModal component. This hook utilizes react-router-dom for navigation, custom analytics tracking, and various utility functions to manage flow data and update the UI.

* ✨ (NewFlowCardComponent/index.tsx): refactor onClick event handler into a separate function handleClick for better readability and maintainability

* 📝 (undrawCards/index.tsx): Remove unused imports and variables to clean up the code
♻️ (undrawCards/index.tsx): Refactor onClick event handler to use a separate function for handling flow card click events

* ✨ (flowsManager/index.ts): introduce new state and setter for flowToCanvas to manage the flow displayed on canvas

* ✨ (flowsManagerStore.ts): introduce new feature to set and update the flow to be displayed on the canvas asynchronously

* ✨ (ViewPage/index.tsx): add support for fetching header flows along with all flows when refreshing data to improve data retrieval efficiency

* ✨ (collectionCard/index.tsx): introduce useFlowsManagerStore to manage flows in the application
📝 (collectionCard/index.tsx): update handleClick function to set flow to canvas before navigating to editFlowLink

* ✨ (Playground/index.tsx): add support for fetching header flows along with all flows when refreshing data in the Playground page.

* 🐛 (FlowPage/index.tsx): Fix setCurrentFlow logic to correctly set the current flow based on flowToCanvas value. Add flowToCanvas dependency to useEffect to ensure proper rendering.

* ✨ (use-get-flow.ts): introduce a new file to handle API queries for fetching flow data in the frontend application. This file defines a custom hook 'useGetFlow' that makes a GET request to the API endpoint to retrieve flow data based on the provided flow ID.

* 📝 (flows.py): update import statement to use FlowHeader instead of FlowSummary for better clarity
📝 (flows.py): update response_model in read_flows endpoint to use FlowHeader for consistency and clarity

* ✨ (model.py): rename FlowSummary class to FlowHeader for better clarity and consistency in naming conventions
📝 (model.py): add is_component field to FlowHeader class to indicate if the flow is a component or not

* ✨ (use-get-folder.ts): introduce processFlows function to process flows data
📝 (use-get-folder.ts): add cloneDeep function to safely clone data before processing

* 🔧 (use-get-refresh-flows.ts): refactor useGetRefreshFlows function to simplify processing of dbDataFlows and update state accordingly

* ✨ (FlowPage/index.tsx): Add useGetFlow hook to fetch flow data and update current flow on canvas. Add getFlowToAddToCanvas function to handle fetching and setting flow data on canvas.

* ✨ (nodeToolbarComponent/index.tsx): improve user experience by automatically closing the override modal after successful flow override

* ✨ (use-post-login-user.ts): add queryClient.refetchQueries for "useGetTags" after successful login to update tags data in the frontend.

* ✨ (use-get-flow.ts): add processFlows function to process flow data before returning it to improve code readability and maintainability

* 🐛 (use-get-refresh-flows.ts): fix asynchronous flow to correctly handle data retrieval and processing before setting state and returning flows

* ✨ (use-get-tags.ts): add functionality to set tags in utility store after fetching them from the API

* 📝 (shareModal/index.tsx): Update import statement for useUtilityStore to improve code organization and readability
🔧 (shareModal/index.tsx): Replace useGetTagsQuery with useUtilityStore to manage tags state and remove unnecessary API call for tags data
🔧 (shareModal/index.tsx): Replace references to data with tags variable to ensure consistency and avoid potential bugs
🔧 (shareModal/index.tsx): Update TagsSelector component to use tags variable instead of data for better data management

* ✨ (AppInitPage/index.tsx): introduce useGetTagsQuery to fetch tags data from the store API for AppInitPage.

* ✨ (StorePage/index.tsx): refactor to use useUtilityStore for tags state management instead of useGetTagsQuery for better separation of concerns and code organization. Update TagsSelector component to use tags from useUtilityStore hook.

* ✨ (frontend): introduce Tag type to UtilityStoreType and add tags and setTags functions to manage tags in the store.

* ✨ (types.ts): introduce new Tag type to represent a tag with id and name properties

* ✨ (utilityStore.ts): introduce new 'tags' array and 'setTags' function to manage tags in the utility store

* 📝 (App.css): add a blank line for better readability and consistency in the CSS file

* ✨ (test_database.py): add pagination support for reading flows and folders to improve data retrieval efficiency and user experience

* ✨ (tabsComponent/index.tsx): refactor changeLocation function to use useCallback hook for better performance and stability
✨ (tabsComponent/index.tsx): update onClick event handler to directly call changeLocation function for cleaner code and improved readability

* ✨ (headerTabsSearchComponent/index.tsx): refactor handleChangeTab, handleSearch, handleInputChange, and handleKeyDown functions to use useCallback for better performance and memoization
📝 (headerTabsSearchComponent/index.tsx): update tabActive prop to use the activeTab prop passed from parent component for consistency and clarity

* ✨ (myCollectionComponent/index.tsx): refactor filter state initialization to dynamically set based on current location pathname
♻️ (myCollectionComponent/index.tsx): refactor onSearch and onChangeTab functions to use useCallback for better performance and memoization

* ✨ (create-query-param-string.ts): introduce a new utility function to build query parameter strings for URLs in the frontend controllers.

* ✨ (use-get-folder.ts): introduce buildQueryStringUrl function to create query parameter strings for API requests
📝 (use-get-folder.ts): add comments to explain the purpose of the code and improve code readability
🔧 (use-get-folder.ts): update useGetFolderQuery function to include additional configuration options for the query, such as refetchOnWindowFocus: false

* ✨ (use-delete-folders.ts): add functionality to update local store after deleting a folder to keep it in sync with the server data

* 📝 (use-post-add-flow.ts): import useFolderStore to access myCollectionId state for refetching queries with the correct folder_id
🐛 (use-post-add-flow.ts): fix queryClient.refetchQueries to include the correct queryKey with folder_id or myCollectionId if response.folder_id is null

* ✨ (use-get-refresh-flows.ts): refactor addQueryParams function to use buildQueryStringUrl utility function for better code readability and maintainability

* ♻️ (flows.py): remove unnecessary commented out code and add pagination functionality to the router for better code organization and readability

* 🔧 (NodeDescription/index.tsx): remove console.log statement for better code cleanliness and readability

* ✨ (index.tsx): Add DialogClose component from @radix-ui/react-dialog to handle cancel action in ConfirmationModal. Refactor handleCancel function to improve code readability and maintainability.

* ♻️ (use-redirect-flow-card-click.tsx): remove unused setFlowToCanvas function to clean up code and improve maintainability

* 📝 (use-patch-update-flow.ts): update onSettled callback to refetch useGetFolders query with the updated folder_id after patching a flow

* 🔧 (use-delete-folders.ts): update onSettled function to correctly refetch queries with the specific folder id when deleting folders

* ✨ (use-get-folder.ts): update useGetFolderQuery to include additional query parameters for pagination and filtering options

* 🔧 (use-post-upload-to-folder.ts): update onSettled callback to refetch useGetFolders query with the correct queryKey and folder_id parameter

* ✨ (use-add-flow.ts): add functionality to refresh flows after adding a new flow to ensure the UI is up to date with the latest data.

* ✨ (AppInitPage/index.tsx): enable fetching basic examples and tags only when isFetched is true to improve performance and reduce unnecessary API calls

* ✨ (myCollectionComponent/index.tsx): refactor onPaginate function to handlePageChange callback for better code organization and readability. Update key prop in ComponentsComponent to include filter and search variables for proper re-rendering.

* ✨ (use-get-folder.ts): add placeholderData option to useGetFolderQuery to provide initial data while fetching folder information

* [autofix.ci] apply automated fixes

* 🐛 (use-post-upload-to-folder.ts): fix queryKey values to correctly refetch queries after uploading a file to a folder

* ⬆️ (folders.spec.ts): increase timeout for waiting in test to improve reliability and prevent flakiness

* ✨ (folders.spec.ts): update selectors to target specific elements by using the first() method to improve test reliability

* ✅ (auto-save-off.spec.ts): update test to match changes in UI for auto-save feature and improve test coverage for hover functionality.

* 📝 (model.py): update model fields to set default values for folder_id, is_component, endpoint_name, and description to None for consistency and clarity

* ♻️ (index.tsx): refactor isUpdatingFolder logic to include isFetchingFolder variable for better accuracy and readability

* 🐛 (index.tsx): fix variable name typo in isLoadingFolder assignment to correctly reference isFetchingFolder

* 🐛 (use-patch-update-flow.ts): fix issue with missing closing parenthesis in queryClient.refetchQueries() method
📝 (use-patch-update-flow.ts): update queryKey in queryClient.refetchQueries() to ["useGetFolder"] to correctly refetch the folder data after updating a flow

* ✨ (use-save-flow.ts): add support for fetching flow data before saving if not already present to ensure data consistency and accuracy

* ✅ (folders.spec.ts): update selectors to target specific elements correctly for testing purposes

* ✨ (use-on-file-drop.tsx): add support for checking flow names in a specific collection to prevent duplicates
♻️ (use-add-flow.ts): refactor to use the same logic for checking flow names in a specific collection to prevent duplicates

* ✨ (index.tsx): Add useIsMutating hook to check if a folder is being deleted to update UI accordingly
♻️ (index.tsx): Refactor logic to determine if a folder is being updated to include check for folder deletion
🔧 (index.tsx): Refactor Select component to use a separate function for handling value change
🔧 (index.tsx): Refactor Button components to disable based on separate variables for first and last page
🔧 (index.ts): Remove unused import and refactor code to use useRef hook for storing the latest folder id in useGetFolderQuery

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

src/backend/tests/unit/test_database.py

@@ -67,8 +67,30 @@ async def test_read_flows(client: TestClient, json_flow: str, active_user, logge
     assert len(response.json()) > 0
 
 
+async def test_read_flows_pagination(client: TestClient, json_flow: str, active_user, logged_in_headers):
+    response = await client.get("api/v1/flows/", headers=logged_in_headers)
+    assert response.status_code == 200
+    assert response.json()["page"] == 1
+    assert response.json()["size"] == 50
+    assert response.json()["pages"] == 0
+    assert response.json()["total"] == 0
+    assert len(response.json()["items"]) == 0
+
+
+async def test_read_flows_pagination_with_params(client: TestClient, json_flow: str, active_user, logged_in_headers):
+    response = await client.get("api/v1/flows/", headers=logged_in_headers, params={"page": 3, "size": 10})
+    assert response.status_code == 200
+    assert response.json()["page"] == 3
+    assert response.json()["size"] == 10
+    assert response.json()["pages"] == 0
+    assert response.json()["total"] == 0
+    assert len(response.json()["items"]) == 0
+
+
 async def test_read_flows_components_only(client: TestClient, flow_component: dict, logged_in_headers):
-    response = await client.get("api/v1/flows/", headers=logged_in_headers, params={"components_only": True})
+    response = await client.get(
+        "api/v1/flows/", headers=logged_in_headers, params={"components_only": True, "get_all": True}
+    )
     assert response.status_code == 200
     names = [flow["name"] for flow in response.json()]
     assert any("Chat Input Component" in name for name in names)
@@ -267,6 +289,52 @@ async def test_delete_folder_with_flows_with_transaction_and_build(
         assert response.json() == {"vertex_builds": {}}
 
 
+async def test_get_flows_from_folder_pagination(client: TestClient, logged_in_headers):
+    # Create a new folder
+    folder_name = f"Test Folder {uuid4()}"
+    folder = FolderCreate(name=folder_name, description="Test folder description", components_list=[], flows_list=[])
+
+    response = await client.post("api/v1/folders/", json=folder.model_dump(), headers=logged_in_headers)
+    assert response.status_code == 201, f"Expected status code 201, but got {response.status_code}"
+
+    created_folder = response.json()
+    folder_id = created_folder["id"]
+
+    response = await client.get(f"api/v1/folders/{folder_id}", headers=logged_in_headers)
+    assert response.status_code == 200
+    assert response.json()["folder"]["name"] == folder_name
+    assert response.json()["folder"]["description"] == "Test folder description"
+    assert response.json()["flows"]["page"] == 1
+    assert response.json()["flows"]["size"] == 50
+    assert response.json()["flows"]["pages"] == 0
+    assert response.json()["flows"]["total"] == 0
+    assert len(response.json()["flows"]["items"]) == 0
+
+
+async def test_get_flows_from_folder_pagination_with_params(client: TestClient, logged_in_headers):
+    # Create a new folder
+    folder_name = f"Test Folder {uuid4()}"
+    folder = FolderCreate(name=folder_name, description="Test folder description", components_list=[], flows_list=[])
+
+    response = await client.post("api/v1/folders/", json=folder.model_dump(), headers=logged_in_headers)
+    assert response.status_code == 201, f"Expected status code 201, but got {response.status_code}"
+
+    created_folder = response.json()
+    folder_id = created_folder["id"]
+
+    response = await client.get(
+        f"api/v1/folders/{folder_id}", headers=logged_in_headers, params={"page": 3, "size": 10}
+    )
+    assert response.status_code == 200
+    assert response.json()["folder"]["name"] == folder_name
+    assert response.json()["folder"]["description"] == "Test folder description"
+    assert response.json()["flows"]["page"] == 3
+    assert response.json()["flows"]["size"] == 10
+    assert response.json()["flows"]["pages"] == 0
+    assert response.json()["flows"]["total"] == 0
+    assert len(response.json()["flows"]["items"]) == 0
+
+
 async def test_create_flows(client: TestClient, session: Session, json_flow: str, logged_in_headers):
     flow = orjson.loads(json_flow)
     data = flow["data"]
@@ -412,7 +480,7 @@ async def test_delete_nonexistent_flow(client: TestClient, active_user, logged_i
 
 
 async def test_read_only_starter_projects(client: TestClient, active_user, logged_in_headers):
-    response = await client.get("api/v1/flows/", headers=logged_in_headers)
+    response = await client.get("api/v1/flows/basic_examples/", headers=logged_in_headers)
     starter_projects = load_starter_projects()
     assert response.status_code == 200
     assert len(response.json()) == len(starter_projects)