jyapayne/langflow
1
0
Fork 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

Docs: update components to v1.0.18 (#3903)

Browse source 
* embeddings

* helpers

* vector-stores

* hashes
This commit is contained in:
Mendon Kissling 2024-09-24 14:29:31 -04:00 • committed by GitHub
commit 7b99f01340
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 324 additions and 82 deletions
Download patch file Download diff file Expand all files Collapse all files

126
docs/docs/Components/components-embedding-models.md
View file

@ -48,22 +48,24 @@ This component is used to load embedding models from [Amazon Bedrock](https://aw
|------|------|-------------| |------|------|-------------|
| embeddings | Embeddings | An instance for generating embeddings using Amazon Bedrock | | embeddings | Embeddings | An instance for generating embeddings using Amazon Bedrock |
## Astra DB vectorize
## Astra vectorize Connect this component to the **Embeddings** port of the [Astra DB vector store component](components-vector-stores#astra-db-serverless) to generate embeddings.
This component is used to generate server-side embeddings using [DataStax Astra](https://docs.datastax.com/en/astra-db-serverless/databases/embedding-generation.html). This component requires that your Astra DB database has a collection that uses a vectorize embedding provider integration.
For more information and instructions, see [Embedding Generation](https://docs.datastax.com/en/astra-db-serverless/databases/embedding-generation.html).
### Parameters ### Parameters
#### Inputs #### Inputs
| Name | Type | Description | | Name | Display Name | Info |
|------|------|-------------| |------|--------------|------|
| provider | String | The embedding provider to use | | provider | Embedding Provider | The embedding provider to use |
| model_name | String | The embedding model to use | | model_name | Model Name | The embedding model to use |
| authentication | Dict | Authentication parameters. Use the Astra Portal to add the embedding provider integration to your Astra organization | | authentication | Authentication | The name of the API key in Astra that stores your [vectorize embedding provider credentials](https://docs.datastax.com/en/astra-db-serverless/databases/embedding-generation.html#embedding-provider-authentication). (Not required if using an [Astra-hosted embedding provider](https://docs.datastax.com/en/astra-db-serverless/databases/embedding-generation.html#supported-embedding-providers).) |
| provider_api_key | String | An alternative to the Astra Authentication that lets you use directly the API key of the provider | | provider_api_key | Provider API Key | As an alternative to `authentication`, directly provide your embedding provider credentials. |
| model_parameters | Dict | Additional model parameters | | model_parameters | Model Parameters | Additional model parameters |
#### Outputs #### Outputs
@ -112,29 +114,86 @@ This component is used to load embedding models from [Cohere](https://cohere.com
|------|------|-------------| |------|------|-------------|
| embeddings | Embeddings | An instance for generating embeddings using Cohere | | embeddings | Embeddings | An instance for generating embeddings using Cohere |
## Hugging Face Inference API Embeddings ## Embedding similarity
This component generates embeddings using Hugging Face Inference API models. This component computes selected forms of similarity between two embedding vectors.
### Parameters ### Parameters
#### Inputs #### Inputs
| Name | Type | Description | | Name | Display Name | Info |
|------|------|-------------| |------|--------------|------|
| API Key | String | API key for accessing the Hugging Face Inference API | | embedding_vectors | Embedding Vectors | A list containing exactly two data objects with embedding vectors to compare. |
| API URL | String | URL of the Hugging Face Inference API (default: `http://localhost:8080`) | | similarity_metric | Similarity Metric | Select the similarity metric to use. Options: "Cosine Similarity", "Euclidean Distance", "Manhattan Distance". |
| Model Name | String | Name of the model to use for embeddings (default: `BAAI/bge-large-en-v1.5`) |
| Cache Folder | String | Folder path to cache Hugging Face models |
| Encode Kwargs | Dict | Additional arguments for the encoding process |
| Model Kwargs | Dict | Additional arguments for the model |
| Multi Process | Boolean | Whether to use multiple processes (default: `False`) |
#### Outputs #### Outputs
| Name | Type | Description | | Name | Display Name | Info |
|------|------|-------------| |------|--------------|------|
| embeddings | Embeddings | An instance for generating embeddings using Hugging Face Inference API | | similarity_data | Similarity Data | Data object containing the computed similarity score and additional information. |
## Google generative AI embeddings
This component connects to Google's generative AI embedding service using the GoogleGenerativeAIEmbeddings class from the `langchain-google-genai` package.
### Parameters
#### Inputs
| Name | Display Name | Info |
|------|--------------|------|
| api_key | API Key | Secret API key for accessing Google's generative AI service (required) |
| model_name | Model Name | Name of the embedding model to use (default: "models/text-embedding-004") |
#### Outputs
| Name | Display Name | Info |
|------|--------------|------|
| embeddings | Embeddings | Built GoogleGenerativeAIEmbeddings object |
## Hugging Face Embeddings
:::note
This component is deprecated as of Langflow version 1.0.18.
Instead, use the [Hugging Face API Embeddings component](#hugging-face-embeddings-inference-api).
:::
This component loads embedding models from HuggingFace.
Use this component to generate embeddings using locally downloaded Hugging Face models. Ensure you have sufficient computational resources to run the models.
### Parameters
#### Inputs
| Name | Display Name | Info |
|------|--------------|------|
| Cache Folder | Cache Folder | Folder path to cache HuggingFace models |
| Encode Kwargs | Encoding Arguments | Additional arguments for the encoding process |
| Model Kwargs | Model Arguments | Additional arguments for the model |
| Model Name | Model Name | Name of the HuggingFace model to use |
| Multi Process | Multi-Process | Whether to use multiple processes |
## Hugging Face embeddings Inference API
This component generates embeddings using Hugging Face Inference API models.
Use this component to create embeddings with Hugging Face's hosted models. Ensure you have a valid Hugging Face API key.
### Parameters
#### Inputs
| Name | Display Name | Info |
|------|--------------|------|
| API Key | API Key | API key for accessing the Hugging Face Inference API |
| API URL | API URL | URL of the Hugging Face Inference API |
| Model Name | Model Name | Name of the model to use for embeddings |
| Cache Folder | Cache Folder | Folder path to cache Hugging Face models |
| Encode Kwargs | Encoding Arguments | Additional arguments for the encoding process |
| Model Kwargs | Model Arguments | Additional arguments for the model |
| Multi Process | Multi-Process | Whether to use multiple processes |
## MistralAI ## MistralAI
@ -239,6 +298,25 @@ This component is used to load embedding models from [OpenAI](https://openai.com
|------|------|-------------| |------|------|-------------|
| embeddings | Embeddings | An instance for generating embeddings using OpenAI | | embeddings | Embeddings | An instance for generating embeddings using OpenAI |
## Text embedder
This component generates embeddings for a given message using a specified embedding model.
### Parameters
#### Inputs
| Name | Display Name | Info |
|------|--------------|------|
| embedding_model | Embedding Model | The embedding model to use for generating embeddings. |
| message | Message | The message for which to generate embeddings. |
#### Outputs
| Name | Display Name | Info |
|------|--------------|------|
| embeddings | Embedding Data | Data object containing the original text and its embedding vector. |
## VertexAI Embeddings ## VertexAI Embeddings
This component is a wrapper around [Google Vertex AI](https://cloud.google.com/vertex-ai) [Embeddings API](https://cloud.google.com/vertex-ai/docs/generative-ai/embeddings/get-text-embeddings). This component is a wrapper around [Google Vertex AI](https://cloud.google.com/vertex-ai) [Embeddings API](https://cloud.google.com/vertex-ai/docs/generative-ai/embeddings/get-text-embeddings).
@ -267,5 +345,3 @@ This component is a wrapper around [Google Vertex AI](https://cloud.google.com/v
|------|------|-------------| |------|------|-------------|
| embeddings | Embeddings | An instance for generating embeddings using VertexAI | | embeddings | Embeddings | An instance for generating embeddings using VertexAI |
[Previous Vector Stores](/components-vector-stores)

57
docs/docs/Components/components-helpers.md
View file

@ -65,9 +65,26 @@ This component dynamically creates a record with a specified number of fields.
Use this component as a template to create your custom component. Use this component as a template to create your custom component.
For more, see [Custom Components](components-custom-components).
## Filter Data ## Filter Data
This component converts LangChain documents into Data. This component filters a Data object based on a list of keys.
### Parameters
#### Inputs
| Name | Display Name | Info |
|------|--------------|------|
| data | Data | Data object to filter. |
| filter_criteria | Filter Criteria | List of keys to filter by. |
#### Outputs
| Name | Display Name | Info |
|------|--------------|------|
| filtered_data | Filtered Data | A new Data object containing only the key-value pairs that match the filter criteria. |
## Hierarchical Task ## Hierarchical Task
@ -103,8 +120,46 @@ This component generates a unique ID.
|------|--------------|------| |------|--------------|------|
| value | Value | Unique ID generated. | | value | Value | Unique ID generated. |
## Parse JSON
This component converts and extracts JSON fields using JQ queries.
### Parameters
#### Inputs
| Name | Display Name | Info |
|------|--------------|------|
| input_value | Input | Data object to filter. Can be a Message or Data object. |
| query | JQ Query | JQ Query to filter the data. The input is always a JSON list. |
#### Outputs
| Name | Display Name | Info |
|------|--------------|------|
| filtered_data | Filtered Data | Filtered data as a list of Data objects. |
## Merge Data ## Merge Data
This component combines multiple data sources into a single unified Data object.
The component iterates through the input list of Data objects, merging them into a single Data object. If the input list is empty, it returns an empty Data object. If there's only one input Data object, it returns that object unchanged. The merging process uses the addition operator to combine Data objects.
### Parameters
#### Inputs
| Name | Display Name | Info |
|------|--------------|------|
| data | Data | A list of Data objects to be merged |
#### Outputs
| Name | Display Name | Info |
|------|--------------|------|
| merged_data | Merged Data | A single Data object containing the combined information from all input Data objects |
## Parse Data ## Parse Data
The ParseData component converts Data objects into plain text using a specified template. The ParseData component converts Data objects into plain text using a specified template.

179
docs/docs/Components/components-vector-stores.md
View file

@ -7,39 +7,42 @@ slug: /components-vector-stores
Vector databases are used to store and search for vectors. They can be used to store embeddings, search for similar vectors, and perform other vector operations. Vector databases are used to store and search for vectors. They can be used to store embeddings, search for similar vectors, and perform other vector operations.
## Astra DB Serverless ## Astra DB Vector Store
This component implements a Vector Store using Astra DB with search capabilities.
This component creates an Astra DB Vector Store with search capabilities.
For more information, see the [DataStax documentation](https://docs.datastax.com/en/astra-db-serverless/databases/create-database.html). For more information, see the [DataStax documentation](https://docs.datastax.com/en/astra-db-serverless/databases/create-database.html).
### Parameters ### Parameters
#### Intputs #### Inputs
| Name | Type | Description | | Name | Display Name | Info |
|----------------------|------------------|-------------------------------------------------------| |------|--------------|------|
| collection_name | String | Name of the collection in Astra DB | | collection_name | Collection Name | The name of the collection within Astra DB where the vectors will be stored (required) |
| token | SecretString | Astra DB Application Token | | token | Astra DB Application Token | Authentication token for accessing Astra DB (required) |
| api_endpoint | SecretString | API endpoint URL for Astra DB | | api_endpoint | API Endpoint | API endpoint URL for the Astra DB service (required) |
| search_input | String | Query for similarity search | | search_input | Search Input | Query string for similarity search |
| ingest_data | Data | Data to be ingested into the vector store | | ingest_data | Ingest Data | Data to be ingested into the vector store |
| namespace | String | Optional namespace within Astra DB | | namespace | Namespace | Optional namespace within Astra DB to use for the collection |
| metric | String | Distance metric for vector comparisons | | embedding_service | Embedding Model or Astra Vectorize | Determines whether to use an Embedding Model or Astra Vectorize for the collection |
| batch_size | Integer | Number of data to process in a single batch | | embedding | Embedding Model | Allows an embedding model configuration (when using Embedding Model) |
| setup_mode | String | Configuration mode for setting up the vector store | | provider | Vectorize Provider | Provider for Astra Vectorize (when using Astra Vectorize) |
| pre_delete_collection | Boolean | Whether to delete the collection before creating a new one | | metric | Metric | Optional distance metric for vector comparisons |
| embedding | Embeddings/Dict | Embedding model or Astra Vectorize configuration | | batch_size | Batch Size | Optional number of data to process in a single batch |
| number_of_results | Integer | Number of results to return in search | | setup_mode | Setup Mode | Configuration mode for setting up the vector store (options: "Sync", "Async", "Off", default: "Sync") |
| search_type | String | Type of search to perform | | pre_delete_collection | Pre Delete Collection | Boolean flag to determine whether to delete the collection before creating a new one |
| search_score_threshold| Float | Minimum similarity score for search results | | number_of_results | Number of Results | Number of results to return in similarity search (default: 4) |
| search_filter | Dict | Metadata filters for search query | | search_type | Search Type | Search type to use (options: "Similarity", "Similarity with score threshold", "MMR (Max Marginal Relevance)") |
| search_score_threshold | Search Score Threshold | Minimum similarity score threshold for search results |
| search_filter | Search Metadata Filter | Optional dictionary of filters to apply to the search query |
#### Outputs #### Outputs
| Name | Type | Description | | Name | Display Name | Info |
|----------------|-----------|-----------------------------------| |------|--------------|------|
| vector_store | AstraDB | Astra DB vector store instance | | vector_store | Vector Store | Built Astra DB vector store |
| search_results | List[Data]| Results of similarity search | | search_results | Search Results | Results of the similarity search as a list of Data objects |
## Cassandra ## Cassandra
@ -48,10 +51,10 @@ For more information, see the [Cassandra documentation](https://cassandra.apache
### Parameters ### Parameters
#### Intputs #### Inputs
| Name | Type | Description | | Name | Type | Description |
|---------------------|----------------|----------------------------------------------------| |------|------|-------------|
| database_ref | String | Contact points for the database or AstraDB database ID | | database_ref | String | Contact points for the database or AstraDB database ID |
| username | String | Username for the database (leave empty for AstraDB) | | username | String | Username for the database (leave empty for AstraDB) |
| token | SecretString | User password for the database or AstraDB token | | token | SecretString | User password for the database or AstraDB token |
@ -74,10 +77,43 @@ For more information, see the [Cassandra documentation](https://cassandra.apache
#### Outputs #### Outputs
| Name | Type | Description | | Name | Type | Description |
|----------------|------------|----------------------------------| |------|------|-------------|
| vector_store | Cassandra | Cassandra vector store instance | | vector_store | Cassandra | Cassandra vector store instance |
| search_results | List[Data] | Results of similarity search | | search_results | List[Data] | Results of similarity search |
## Cassandra Graph Vector Store
This component implements a Cassandra Graph Vector Store with search capabilities.
### Parameters
#### Inputs
| Name | Display Name | Info |
|------|--------------|------|
| database_ref | Contact Points / Astra Database ID | Contact points for the database or AstraDB database ID (required) |
| username | Username | Username for the database (leave empty for AstraDB) |
| token | Password / AstraDB Token | User password for the database or AstraDB token (required) |
| keyspace | Keyspace | Table Keyspace or AstraDB namespace (required) |
| table_name | Table Name | The name of the table or AstraDB collection where vectors will be stored (required) |
| setup_mode | Setup Mode | Configuration mode for setting up the Cassandra table (options: "Sync", "Off", default: "Sync") |
| cluster_kwargs | Cluster arguments | Optional dictionary of additional keyword arguments for the Cassandra cluster |
| search_query | Search Query | Query string for similarity search |
| ingest_data | Ingest Data | Data to be ingested into the vector store (list of Data objects) |
| embedding | Embedding | Embedding model to use |
| number_of_results | Number of Results | Number of results to return in similarity search (default: 4) |
| search_type | Search Type | Search type to use (options: "Traversal", "MMR traversal", "Similarity", "Similarity with score threshold", "MMR (Max Marginal Relevance)", default: "Traversal") |
| depth | Depth of traversal | The maximum depth of edges to traverse (for "Traversal" or "MMR traversal" search types, default: 1) |
| search_score_threshold | Search Score Threshold | Minimum similarity score threshold for search results (for "Similarity with score threshold" search type) |
| search_filter | Search Metadata Filter | Optional dictionary of filters to apply to the search query |
#### Outputs
| Name | Display Name | Info |
|------|--------------|------|
| vector_store | Vector Store | Built Cassandra Graph vector store |
| search_results | Search Results | Results of the similarity search as a list of Data objects |
## Chroma DB ## Chroma DB
This component creates a Chroma Vector Store with search capabilities. This component creates a Chroma Vector Store with search capabilities.
@ -85,7 +121,7 @@ For more information, see the [Chroma documentation](https://docs.trychroma.com/
### Parameters ### Parameters
#### Intputs #### Inputs
| Name | Type | Description | | Name | Type | Description |
|------------------------------|---------------|--------------------------------------------------| |------------------------------|---------------|--------------------------------------------------|
@ -111,6 +147,41 @@ For more information, see the [Chroma documentation](https://docs.trychroma.com/
| vector_store | Chroma | Chroma vector store instance | | vector_store | Chroma | Chroma vector store instance |
| search_results | List[Data] | Results of similarity search | | search_results | List[Data] | Results of similarity search |
## Clickhouse
This component implements a Clickhouse Vector Store with search capabilities.
For more information, see the [CLickhouse Documentation](https://clickhouse.com/docs/en/intro).
### Parameters
#### Inputs
| Name | Display Name | Info |
|------|--------------|------|
| host | hostname | Clickhouse server hostname (required, default: "localhost") |
| port | port | Clickhouse server port (required, default: 8123) |
| database | database | Clickhouse database name (required) |
| table | Table name | Clickhouse table name (required) |
| username | The ClickHouse user name. | Username for authentication (required) |
| password | The password for username. | Password for authentication (required) |
| index_type | index_type | Type of the index (options: "annoy", "vector_similarity", default: "annoy") |
| metric | metric | Metric to compute distance (options: "angular", "euclidean", "manhattan", "hamming", "dot", default: "angular") |
| secure | Use https/TLS | Overrides inferred values from the interface or port arguments (default: false) |
| index_param | Param of the index | Index parameters (default: "'L2Distance',100") |
| index_query_params | index query params | Additional index query parameters |
| search_query | Search Query | Query string for similarity search |
| ingest_data | Ingest Data | Data to be ingested into the vector store |
| embedding | Embedding | Embedding model to use |
| number_of_results | Number of Results | Number of results to return in similarity search (default: 4) |
| score_threshold | Score threshold | Threshold for similarity scores |
#### Outputs
| Name | Display Name | Info |
|------|--------------|------|
| vector_store | Vector Store | Built Clickhouse vector store |
| search_results | Search Results | Results of the similarity search as a list of Data objects |
## Couchbase ## Couchbase
This component creates a Couchbase Vector Store with search capabilities. This component creates a Couchbase Vector Store with search capabilities.
@ -118,7 +189,7 @@ For more information, see the [Couchbase documentation](https://docs.couchbase.c
### Parameters ### Parameters
#### Intputs #### Inputs
| Name | Type | Description | | Name | Type | Description |
|-------------------------|---------------|--------------------------------------------------| |-------------------------|---------------|--------------------------------------------------|
@ -147,7 +218,7 @@ For more information, see the [FAISS documentation](https://faiss.ai/index.html)
### Parameters ### Parameters
#### Intputs #### Inputs
| Name | Type | Description | | Name | Type | Description |
|---------------------------|---------------|--------------------------------------------------| |---------------------------|---------------|--------------------------------------------------|
@ -165,6 +236,46 @@ For more information, see the [FAISS documentation](https://faiss.ai/index.html)
|----------------|------------------------|--------------------------------| |----------------|------------------------|--------------------------------|
| vector_store | FAISS | A FAISS vector store instance configured with the specified parameters. | | vector_store | FAISS | A FAISS vector store instance configured with the specified parameters. |
## Hyper-Converged Database (HCD) Vector Store
This component implements a Vector Store using HCD.
### Parameters
#### Inputs
| Name | Display Name | Info |
|------|--------------|------|
| collection_name | Collection Name | The name of the collection within HCD where the vectors will be stored (required) |
| username | HCD Username | Authentication username for accessing HCD (default: "hcd-superuser", required) |
| password | HCD Password | Authentication password for accessing HCD (required) |
| api_endpoint | HCD API Endpoint | API endpoint URL for the HCD service (required) |
| search_input | Search Input | Query string for similarity search |
| ingest_data | Ingest Data | Data to be ingested into the vector store |
| namespace | Namespace | Optional namespace within HCD to use for the collection (default: "default_namespace") |
| ca_certificate | CA Certificate | Optional CA certificate for TLS connections to HCD |
| metric | Metric | Optional distance metric for vector comparisons (options: "cosine", "dot_product", "euclidean") |
| batch_size | Batch Size | Optional number of data to process in a single batch |
| bulk_insert_batch_concurrency | Bulk Insert Batch Concurrency | Optional concurrency level for bulk insert operations |
| bulk_insert_overwrite_concurrency | Bulk Insert Overwrite Concurrency | Optional concurrency level for bulk insert operations that overwrite existing data |
| bulk_delete_concurrency | Bulk Delete Concurrency | Optional concurrency level for bulk delete operations |
| setup_mode | Setup Mode | Configuration mode for setting up the vector store (options: "Sync", "Async", "Off", default: "Sync") |
| pre_delete_collection | Pre Delete Collection | Boolean flag to determine whether to delete the collection before creating a new one |
| metadata_indexing_include | Metadata Indexing Include | Optional list of metadata fields to include in the indexing |
| embedding | Embedding or Astra Vectorize | Allows either an embedding model or an Astra Vectorize configuration |
| metadata_indexing_exclude | Metadata Indexing Exclude | Optional list of metadata fields to exclude from the indexing |
| collection_indexing_policy | Collection Indexing Policy | Optional dictionary defining the indexing policy for the collection |
| number_of_results | Number of Results | Number of results to return in similarity search (default: 4) |
| search_type | Search Type | Search type to use (options: "Similarity", "Similarity with score threshold", "MMR (Max Marginal Relevance)", default: "Similarity") |
| search_score_threshold | Search Score Threshold | Minimum similarity score threshold for search results (default: 0) |
| search_filter | Search Metadata Filter | Optional dictionary of filters to apply to the search query |
#### Outputs
| Name | Display Name | Info |
|------|--------------|------|
| vector_store | Vector Store | Built HCD vector store instance |
| search_results | Search Results | Results of similarity search as a list of Data objects |
## Milvus ## Milvus
@ -173,7 +284,7 @@ For more information, see the [Milvus documentation](https://milvus.io/docs).
### Parameters ### Parameters
#### Intputs #### Inputs
| Name | Type | Description | | Name | Type | Description |
|-------------------------|---------------|--------------------------------------------------| |-------------------------|---------------|--------------------------------------------------|
@ -207,7 +318,7 @@ For more information, see the [MongoDB Atlas documentation](https://www.mongodb.
### Parameters ### Parameters
###### Intputs #### Inputs
| Name | Type | Description | | Name | Type | Description |
| ------------------------ | ------------ | ----------------------------------------- | | ------------------------ | ------------ | ----------------------------------------- |
@ -235,7 +346,7 @@ For more information, see the [PGVector documentation](https://github.com/pgvect
### Parameters ### Parameters
###### Intputs #### Inputs
| Name | Type | Description | | Name | Type | Description |
| --------------- | ------------ | ----------------------------------------- | | --------------- | ------------ | ----------------------------------------- |
@ -349,7 +460,7 @@ For more information, see the [Redis documentation](https://redis.io/docs/latest
## Supabase ## Supabase
This component creates a Supabase Vector Store with search capabilities. This component creates a connection to a Supabase Vector Store with search capabilities.
For more information, see the [Supabase documentation](https://supabase.com/docs/guides/ai). For more information, see the [Supabase documentation](https://supabase.com/docs/guides/ai).
### Parameters ### Parameters