Model Runtime (#1858)

Co-authored-by: StyleZhang <jasonapring2015@outlook.com>
Co-authored-by: Garfield Dai <dai.hai@foxmail.com>
Co-authored-by: chenhe <guchenhe@gmail.com>
Co-authored-by: jyong <jyong@dify.ai>
Co-authored-by: Joel <iamjoel007@gmail.com>
Co-authored-by: Yeuoly <admin@srmxy.cn>

api/core/model_runtime/docs/en_US/interfaces.md

@@ -0,0 +1,668 @@
+# Interface Methods
+
+This section describes the interface methods and parameter explanations that need to be implemented by providers and various model types.
+
+## Provider
+
+Inherit the `__base.model_provider.ModelProvider` base class and implement the following interfaces:
+
+```python
+def validate_provider_credentials(self, credentials: dict) -> None:
+    """
+    Validate provider credentials
+    You can choose any validate_credentials method of model type or implement validate method by yourself,
+    such as: get model list api
+
+    if validate failed, raise exception
+
+    :param credentials: provider credentials, credentials form defined in `provider_credential_schema`.
+    """
+```
+
+- `credentials` (object) Credential information
+
+  The parameters of credential information are defined by the `provider_credential_schema` in the provider's YAML configuration file. Inputs such as `api_key` are included.
+
+If verification fails, throw the `errors.validate.CredentialsValidateFailedError` error.
+
+## Model
+
+Models are divided into 5 different types, each inheriting from different base classes and requiring the implementation of different methods.
+
+All models need to uniformly implement the following 2 methods:
+
+- Model Credential Verification
+
+  Similar to provider credential verification, this step involves verification for an individual model.
+
+
+  ```python
+  def validate_credentials(self, model: str, credentials: dict) -> None:
+      """
+      Validate model credentials
+  
+      :param model: model name
+      :param credentials: model credentials
+      :return:
+      """
+  ```
+
+  Parameters:
+
+  - `model` (string) Model name
+
+  - `credentials` (object) Credential information
+
+    The parameters of credential information are defined by either the `provider_credential_schema` or `model_credential_schema` in the provider's YAML configuration file. Inputs such as `api_key` are included.
+
+  If verification fails, throw the `errors.validate.CredentialsValidateFailedError` error.
+
+- Invocation Error Mapping Table
+
+  When there is an exception in model invocation, it needs to be mapped to the `InvokeError` type specified by Runtime. This facilitates Dify's ability to handle different errors with appropriate follow-up actions.
+
+  Runtime Errors:
+
+  - `InvokeConnectionError` Invocation connection error
+  - `InvokeServerUnavailableError` Invocation service provider unavailable
+  - `InvokeRateLimitError` Invocation reached rate limit
+  - `InvokeAuthorizationError` Invocation authorization failure
+  - `InvokeBadRequestError` Invocation parameter error
+
+  ```python
+  @property
+  def _invoke_error_mapping(self) -> dict[type[InvokeError], list[type[Exception]]]:
+      """
+      Map model invoke error to unified error
+      The key is the error type thrown to the caller
+      The value is the error type thrown by the model,
+      which needs to be converted into a unified error type for the caller.
+  
+      :return: Invoke error mapping
+      """
+  ```
+
+​	You can refer to OpenAI's `_invoke_error_mapping` for an example.
+
+### LLM
+
+Inherit the `__base.large_language_model.LargeLanguageModel` base class and implement the following interfaces:
+
+- LLM Invocation
+
+  Implement the core method for LLM invocation, which can support both streaming and synchronous returns.
+
+
+  ```python
+  def _invoke(self, model: str, credentials: dict,
+              prompt_messages: list[PromptMessage], model_parameters: dict,
+              tools: Optional[list[PromptMessageTool]] = None, stop: Optional[List[str]] = None,
+              stream: bool = True, user: Optional[str] = None) \
+          -> Union[LLMResult, Generator]:
+      """
+      Invoke large language model
+  
+      :param model: model name
+      :param credentials: model credentials
+      :param prompt_messages: prompt messages
+      :param model_parameters: model parameters
+      :param tools: tools for tool calling
+      :param stop: stop words
+      :param stream: is stream response
+      :param user: unique user id
+      :return: full response or stream response chunk generator result
+      """
+  ```
+
+  - Parameters:
+
+    - `model` (string) Model name
+
+    - `credentials` (object) Credential information
+
+      The parameters of credential information are defined by either the `provider_credential_schema` or `model_credential_schema` in the provider's YAML configuration file. Inputs such as `api_key` are included.
+
+    - `prompt_messages` (array[[PromptMessage](#PromptMessage)]) List of prompts
+
+      If the model is of the `Completion` type, the list only needs to include one [UserPromptMessage](#UserPromptMessage) element;
+
+      If the model is of the `Chat` type, it requires a list of elements such as [SystemPromptMessage](#SystemPromptMessage), [UserPromptMessage](#UserPromptMessage), [AssistantPromptMessage](#AssistantPromptMessage), [ToolPromptMessage](#ToolPromptMessage) depending on the message.
+
+    - `model_parameters` (object) Model parameters
+
+      The model parameters are defined by the `parameter_rules` in the model's YAML configuration.
+
+    - `tools` (array[[PromptMessageTool](#PromptMessageTool)]) [optional] List of tools, equivalent to the `function` in `function calling`.
+
+      That is, the tool list for tool calling.
+
+    - `stop` (array[string]) [optional] Stop sequences
+
+      The model output will stop before the string defined by the stop sequence.
+
+    - `stream` (bool) Whether to output in a streaming manner, default is True
+
+      Streaming output returns Generator[[LLMResultChunk](#LLMResultChunk)], non-streaming output returns [LLMResult](#LLMResult).
+
+    - `user` (string) [optional] Unique identifier of the user
+
+      This can help the provider monitor and detect abusive behavior.
+
+  - Returns
+
+    Streaming output returns Generator[[LLMResultChunk](#LLMResultChunk)], non-streaming output returns [LLMResult](#LLMResult).
+
+- Pre-calculating Input Tokens
+
+  If the model does not provide a pre-calculated tokens interface, you can directly return 0.
+
+  ```python
+  def get_num_tokens(self, model: str, credentials: dict, prompt_messages: list[PromptMessage],
+                     tools: Optional[list[PromptMessageTool]] = None) -> int:
+      """
+      Get number of tokens for given prompt messages
+
+      :param model: model name
+      :param credentials: model credentials
+      :param prompt_messages: prompt messages
+      :param tools: tools for tool calling
+      :return:
+      """
+  ```
+
+  For parameter explanations, refer to the above section on `LLM Invocation`.
+
+- Fetch Custom Model Schema [Optional]
+
+  ```python
+  def get_customizable_model_schema(self, model: str, credentials: dict) -> Optional[AIModelEntity]:
+      """
+      Get customizable model schema
+
+      :param model: model name
+      :param credentials: model credentials
+      :return: model schema
+      """
+  ```
+
+  When the provider supports adding custom LLMs, this method can be implemented to allow custom models to fetch model schema. The default return null.
+
+
+### TextEmbedding
+
+Inherit the `__base.text_embedding_model.TextEmbeddingModel` base class and implement the following interfaces:
+
+- Embedding Invocation
+
+  ```python
+  def _invoke(self, model: str, credentials: dict,
+              texts: list[str], user: Optional[str] = None) \
+          -> TextEmbeddingResult:
+      """
+      Invoke large language model
+  
+      :param model: model name
+      :param credentials: model credentials
+      :param texts: texts to embed
+      :param user: unique user id
+      :return: embeddings result
+      """
+  ```
+
+  - Parameters:
+
+    - `model` (string) Model name
+
+    - `credentials` (object) Credential information
+
+      The parameters of credential information are defined by either the `provider_credential_schema` or `model_credential_schema` in the provider's YAML configuration file. Inputs such as `api_key` are included.
+
+    - `texts` (array[string]) List of texts, capable of batch processing
+
+    - `user` (string) [optional] Unique identifier of the user
+
+      This can help the provider monitor and detect abusive behavior.
+
+  - Returns:
+
+    [TextEmbeddingResult](#TextEmbeddingResult) entity.
+
+- Pre-calculating Tokens
+
+  ```python
+  def get_num_tokens(self, model: str, credentials: dict, texts: list[str]) -> int:
+      """
+      Get number of tokens for given prompt messages
+
+      :param model: model name
+      :param credentials: model credentials
+      :param texts: texts to embed
+      :return:
+      """
+  ```
+
+  For parameter explanations, refer to the above section on `Embedding Invocation`.
+
+### Rerank
+
+Inherit the `__base.rerank_model.RerankModel` base class and implement the following interfaces:
+
+- Rerank Invocation
+
+  ```python
+  def _invoke(self, model: str, credentials: dict,
+              query: str, docs: list[str], score_threshold: Optional[float] = None, top_n: Optional[int] = None,
+              user: Optional[str] = None) \
+          -> RerankResult:
+      """
+      Invoke rerank model
+  
+      :param model: model name
+      :param credentials: model credentials
+      :param query: search query
+      :param docs: docs for reranking
+      :param score_threshold: score threshold
+      :param top_n: top n
+      :param user: unique user id
+      :return: rerank result
+      """
+  ```
+
+  - Parameters:
+
+    - `model` (string) Model name
+
+    - `credentials` (object) Credential information
+
+      The parameters of credential information are defined by either the `provider_credential_schema` or `model_credential_schema` in the provider's YAML configuration file. Inputs such as `api_key` are included.
+
+    - `query` (string) Query request content
+
+    - `docs` (array[string]) List of segments to be reranked
+
+    - `score_threshold` (float) [optional] Score threshold
+
+    - `top_n` (int) [optional] Select the top n segments
+
+    - `user` (string) [optional] Unique identifier of the user
+
+      This can help the provider monitor and detect abusive behavior.
+
+  - Returns:
+
+    [RerankResult](#RerankResult) entity.
+
+### Speech2text
+
+Inherit the `__base.speech2text_model.Speech2TextModel` base class and implement the following interfaces:
+
+- Invoke Invocation
+
+  ```python
+  def _invoke(self, model: str, credentials: dict,
+              file: IO[bytes], user: Optional[str] = None) \
+          -> str:
+      """
+      Invoke large language model
+  
+      :param model: model name
+      :param credentials: model credentials
+      :param file: audio file
+      :param user: unique user id
+      :return: text for given audio file
+      """	
+  ```
+
+  - Parameters:
+
+    - `model` (string) Model name
+
+    - `credentials` (object) Credential information
+
+      The parameters of credential information are defined by either the `provider_credential_schema` or `model_credential_schema` in the provider's YAML configuration file. Inputs such as `api_key` are included.
+
+    - `file` (File) File stream
+
+    - `user` (string) [optional] Unique identifier of the user
+
+      This can help the provider monitor and detect abusive behavior.
+
+  - Returns:
+
+    The string after speech-to-text conversion.
+
+### Moderation
+
+Inherit the `__base.moderation_model.ModerationModel` base class and implement the following interfaces:
+
+- Invoke Invocation
+
+  ```python
+  def _invoke(self, model: str, credentials: dict,
+              text: str, user: Optional[str] = None) \
+          -> bool:
+      """
+      Invoke large language model
+  
+      :param model: model name
+      :param credentials: model credentials
+      :param text: text to moderate
+      :param user: unique user id
+      :return: false if text is safe, true otherwise
+      """
+  ```
+
+  - Parameters:
+
+    - `model` (string) Model name
+
+    - `credentials` (object) Credential information
+
+      The parameters of credential information are defined by either the `provider_credential_schema` or `model_credential_schema` in the provider's YAML configuration file. Inputs such as `api_key` are included.
+
+    - `text` (string) Text content
+
+    - `user` (string) [optional] Unique identifier of the user
+
+      This can help the provider monitor and detect abusive behavior.
+
+  - Returns:
+
+    False indicates that the input text is safe, True indicates otherwise.
+
+
+
+## Entities
+
+### PromptMessageRole 
+
+Message role
+
+```python
+class PromptMessageRole(Enum):
+    """
+    Enum class for prompt message.
+    """
+    SYSTEM = "system"
+    USER = "user"
+    ASSISTANT = "assistant"
+    TOOL = "tool"
+```
+
+### PromptMessageContentType
+
+Message content types, divided into text and image.
+
+```python
+class PromptMessageContentType(Enum):
+    """
+    Enum class for prompt message content type.
+    """
+    TEXT = 'text'
+    IMAGE = 'image'
+```
+
+### PromptMessageContent
+
+Message content base class, used only for parameter declaration and cannot be initialized.
+
+```python
+class PromptMessageContent(BaseModel):
+    """
+    Model class for prompt message content.
+    """
+    type: PromptMessageContentType
+    data: str
+```
+
+Currently, two types are supported: text and image. It's possible to simultaneously input text and multiple images.
+
+You need to initialize `TextPromptMessageContent` and `ImagePromptMessageContent` separately for input.
+
+### TextPromptMessageContent
+
+```python
+class TextPromptMessageContent(PromptMessageContent):
+    """
+    Model class for text prompt message content.
+    """
+    type: PromptMessageContentType = PromptMessageContentType.TEXT
+```
+
+If inputting a combination of text and images, the text needs to be constructed into this entity as part of the `content` list.
+
+### ImagePromptMessageContent
+
+```python
+class ImagePromptMessageContent(PromptMessageContent):
+    """
+    Model class for image prompt message content.
+    """
+    class DETAIL(Enum):
+        LOW = 'low'
+        HIGH = 'high'
+
+    type: PromptMessageContentType = PromptMessageContentType.IMAGE
+    detail: DETAIL = DETAIL.LOW  # Resolution
+```
+
+If inputting a combination of text and images, the images need to be constructed into this entity as part of the `content` list.
+
+`data` can be either a `url` or a `base64` encoded string of the image.
+
+### PromptMessage
+
+The base class for all Role message bodies, used only for parameter declaration and cannot be initialized.
+
+```python
+class PromptMessage(ABC, BaseModel):
+    """
+    Model class for prompt message.
+    """
+    role: PromptMessageRole
+    content: Optional[str | list[PromptMessageContent]] = None  # Supports two types: string and content list. The content list is designed to meet the needs of multimodal inputs. For more details, see the PromptMessageContent explanation.
+    name: Optional[str] = None
+```
+
+### UserPromptMessage
+
+UserMessage message body, representing a user's message.
+
+```python
+class UserPromptMessage(PromptMessage):
+    """
+    Model class for user prompt message.
+    """
+    role: PromptMessageRole = PromptMessageRole.USER
+```
+
+### AssistantPromptMessage
+
+Represents a message returned by the model, typically used for `few-shots` or inputting chat history.
+
+```python
+class AssistantPromptMessage(PromptMessage):
+    """
+    Model class for assistant prompt message.
+    """
+    class ToolCall(BaseModel):
+        """
+        Model class for assistant prompt message tool call.
+        """
+        class ToolCallFunction(BaseModel):
+            """
+            Model class for assistant prompt message tool call function.
+            """
+            name: str  # tool name
+            arguments: str  # tool arguments
+
+        id: str  # Tool ID, effective only in OpenAI tool calls. It's the unique ID for tool invocation and the same tool can be called multiple times.
+        type: str  # default: function
+        function: ToolCallFunction  # tool call information
+
+    role: PromptMessageRole = PromptMessageRole.ASSISTANT
+    tool_calls: list[ToolCall] = []  # The result of tool invocation in response from the model (returned only when tools are input and the model deems it necessary to invoke a tool).
+```
+
+Where `tool_calls` are the list of `tool calls` returned by the model after invoking the model with the `tools` input.
+
+### SystemPromptMessage
+
+Represents system messages, usually used for setting system commands given to the model.
+
+```python
+class SystemPromptMessage(PromptMessage):
+    """
+    Model class for system prompt message.
+    """
+    role: PromptMessageRole = PromptMessageRole.SYSTEM
+```
+
+### ToolPromptMessage
+
+Represents tool messages, used for conveying the results of a tool execution to the model for the next step of processing.
+
+```python
+class ToolPromptMessage(PromptMessage):
+    """
+    Model class for tool prompt message.
+    """
+    role: PromptMessageRole = PromptMessageRole.TOOL
+    tool_call_id: str  # Tool invocation ID. If OpenAI tool call is not supported, the name of the tool can also be inputted.
+```
+
+The base class's `content` takes in the results of tool execution.
+
+### PromptMessageTool
+
+```python
+class PromptMessageTool(BaseModel):
+    """
+    Model class for prompt message tool.
+    """
+    name: str
+    description: str
+    parameters: dict
+```
+
+---
+
+### LLMResult
+
+```python
+class LLMResult(BaseModel):
+    """
+    Model class for llm result.
+    """
+    model: str  # Actual used modele
+    prompt_messages: list[PromptMessage]  # prompt messages
+    message: AssistantPromptMessage  # response message
+    usage: LLMUsage  # usage info
+    system_fingerprint: Optional[str] = None  # request fingerprint, refer to OpenAI definition
+```
+
+### LLMResultChunkDelta
+
+In streaming returns, each iteration contains the `delta` entity.
+
+```python
+class LLMResultChunkDelta(BaseModel):
+    """
+    Model class for llm result chunk delta.
+    """
+    index: int
+    message: AssistantPromptMessage  # response message
+    usage: Optional[LLMUsage] = None  # usage info
+    finish_reason: Optional[str] = None  # finish reason, only the last one returns
+```
+
+### LLMResultChunk
+
+Each iteration entity in streaming returns.
+
+```python
+class LLMResultChunk(BaseModel):
+    """
+    Model class for llm result chunk.
+    """
+    model: str  # Actual used modele
+    prompt_messages: list[PromptMessage]  # prompt messages
+    system_fingerprint: Optional[str] = None  # request fingerprint, refer to OpenAI definition
+    delta: LLMResultChunkDelta
+```
+
+### LLMUsage
+
+```python
+class LLMUsage(ModelUsage):
+    """
+    Model class for LLM usage.
+    """
+    prompt_tokens: int  # Tokens used for prompt
+    prompt_unit_price: Decimal  # Unit price for prompt
+    prompt_price_unit: Decimal  # Price unit for prompt, i.e., the unit price based on how many tokens
+    prompt_price: Decimal  # Cost for prompt
+    completion_tokens: int  # Tokens used for response
+    completion_unit_price: Decimal  # Unit price for response
+    completion_price_unit: Decimal  # Price unit for response, i.e., the unit price based on how many tokens
+    completion_price: Decimal  # Cost for response
+    total_tokens: int  # Total number of tokens used
+    total_price: Decimal  # Total cost
+    currency: str  # Currency unit
+    latency: float  # Request latency (s)
+```
+
+---
+
+### TextEmbeddingResult
+
+```python
+class TextEmbeddingResult(BaseModel):
+    """
+    Model class for text embedding result.
+    """
+    model: str  # Actual model used
+    embeddings: list[list[float]]  # List of embedding vectors, corresponding to the input texts list
+    usage: EmbeddingUsage  # Usage information
+```
+
+### EmbeddingUsage
+
+```python
+class EmbeddingUsage(ModelUsage):
+    """
+    Model class for embedding usage.
+    """
+    tokens: int  # Number of tokens used
+    total_tokens: int  # Total number of tokens used
+    unit_price: Decimal  # Unit price
+    price_unit: Decimal  # Price unit, i.e., the unit price based on how many tokens
+    total_price: Decimal  # Total cost
+    currency: str  # Currency unit
+    latency: float  # Request latency (s)
+```
+
+---
+
+### RerankResult
+
+```python
+class RerankResult(BaseModel):
+    """
+    Model class for rerank result.
+    """
+    model: str  # Actual model used
+    docs: list[RerankDocument]  # Reranked document list	
+```
+
+### RerankDocument
+
+```python
+class RerankDocument(BaseModel):
+    """
+    Model class for rerank document.
+    """
+    index: int  # original index
+    text: str
+    score: float
+```

api/core/model_runtime/docs/en_US/provider_scale_out.md

@@ -0,0 +1,264 @@
+## Adding a New Provider
+
+Providers support three types of model configuration methods:
+
+- `predefined-model` Predefined model
+
+  This indicates that users only need to configure the unified provider credentials to use the predefined models under the provider.
+
+- `customizable-model` Customizable model
+
+  Users need to add credential configurations for each model.
+
+- `fetch-from-remote` Fetch from remote
+
+  This is consistent with the `predefined-model` configuration method. Only unified provider credentials need to be configured, and models are obtained from the provider through credential information.
+
+These three configuration methods **can coexist**, meaning a provider can support `predefined-model` + `customizable-model` or `predefined-model` + `fetch-from-remote`, etc. In other words, configuring the unified provider credentials allows the use of predefined and remotely fetched models, and if new models are added, they can be used in addition to the custom models.
+
+## Getting Started
+
+Adding a new provider starts with determining the English identifier of the provider, such as `anthropic`, and using this identifier to create a `module` in `model_providers`.
+
+Under this `module`, we first need to prepare the provider's YAML configuration.
+
+### Preparing Provider YAML
+
+Here, using `Anthropic` as an example, we preset the provider's basic information, supported model types, configuration methods, and credential rules.
+
+```YAML
+provider: anthropic  # Provider identifier
+label:  # Provider display name, can be set in en_US English and zh_Hans Chinese, zh_Hans will default to en_US if not set.
+  en_US: Anthropic
+icon_small:  # Small provider icon, stored in the _assets directory under the corresponding provider implementation directory, same language strategy as label
+  en_US: icon_s_en.png
+icon_large:  # Large provider icon, stored in the _assets directory under the corresponding provider implementation directory, same language strategy as label
+  en_US: icon_l_en.png
+supported_model_types:  # Supported model types, Anthropic only supports LLM
+- llm
+configurate_methods:  # Supported configuration methods, Anthropic only supports predefined models
+- predefined-model
+provider_credential_schema:  # Provider credential rules, as Anthropic only supports predefined models, unified provider credential rules need to be defined
+  credential_form_schemas:  # List of credential form items
+  - variable: anthropic_api_key  # Credential parameter variable name
+    label:  # Display name
+      en_US: API Key
+    type: secret-input  # Form type, here secret-input represents an encrypted information input box, showing masked information when editing.
+    required: true  # Whether required
+    placeholder:  # Placeholder information
+      zh_Hans: Enter your API Key here
+      en_US: Enter your API Key
+  - variable: anthropic_api_url
+    label:
+      en_US: API URL
+    type: text-input  # Form type, here text-input represents a text input box
+    required: false
+    placeholder:
+      zh_Hans: Enter your API URL here
+      en_US: Enter your API URL
+```
+
+You can also refer to the YAML configuration information under other provider directories in `model_providers`. The complete YAML rules are available at: [Schema](schema.md#Provider).
+
+### Implementing Provider Code
+
+Providers need to inherit the `__base.model_provider.ModelProvider` base class and implement the `validate_provider_credentials` method for unified provider credential verification. For reference, see [AnthropicProvider](https://github.com/langgenius/dify-runtime/blob/main/lib/model_providers/anthropic/anthropic.py).
+> If the provider is the type of `customizable-model`, there is no need to implement the `validate_provider_credentials` method.
+
+```python
+def validate_provider_credentials(self, credentials: dict) -> None:
+    """
+    Validate provider credentials
+    You can choose any validate_credentials method of model type or implement validate method by yourself,
+    such as: get model list api
+
+    if validate failed, raise exception
+
+    :param credentials: provider credentials, credentials form defined in `provider_credential_schema`.
+    """
+```
+
+Of course, you can also preliminarily reserve the implementation of `validate_provider_credentials` and directly reuse it after the model credential verification method is implemented.
+
+---
+
+### Adding Models
+
+After the provider integration is complete, the next step is to integrate models under the provider.
+
+First, we need to determine the type of the model to be integrated and create a `module` for the corresponding model type in the provider's directory.
+
+The currently supported model types are as follows:
+
+- `llm` Text generation model
+- `text_embedding` Text Embedding model
+- `rerank` Rerank model
+- `speech2text` Speech to text
+- `moderation` Moderation
+
+Continuing with `Anthropic` as an example, since `Anthropic` only supports LLM, we create a `module` named `llm` in `model_providers.anthropic`.
+
+For predefined models, we first need to create a YAML file named after the model, such as `claude-2.1.yaml`, under the `llm` `module`.
+
+#### Preparing Model YAML
+
+```yaml
+model: claude-2.1  # Model identifier
+# Model display name, can be set in en_US English and zh_Hans Chinese, zh_Hans will default to en_US if not set.
+# Alternatively, if the label is not set, use the model identifier content.
+label:
+  en_US: claude-2.1
+model_type: llm  # Model type, claude-2.1 is an LLM
+features:  # Supported features, agent-thought for Agent reasoning, vision for image understanding
+- agent-thought
+model_properties:  # Model properties
+  mode: chat  # LLM mode, complete for text completion model, chat for dialogue model
+  context_size: 200000  # Maximum supported context size
+parameter_rules:  # Model invocation parameter rules, only required for LLM
+- name: temperature  # Invocation parameter variable name
+  # Default preset with 5 variable content configuration templates: temperature/top_p/max_tokens/presence_penalty/frequency_penalty
+  # Directly set the template variable name in use_template, which will use the default configuration in entities.defaults.PARAMETER_RULE_TEMPLATE
+  # If additional configuration parameters are set, they will override the default configuration
+  use_template: temperature
+- name: top_p
+  use_template: top_p
+- name: top_k
+  label:  # Invocation parameter display name
+    zh_Hans: Sampling quantity
+    en_US: Top k
+  type: int  # Parameter type, supports float/int/string/boolean
+  help:  # Help information, describing the role of the parameter
+    zh_Hans: Only sample from the top K options for each subsequent token.
+    en_US: Only sample from the top K options for each subsequent token.
+  required: false  # Whether required, can be left unset
+- name: max_tokens_to_sample
+  use_template: max_tokens
+  default: 4096  # Default parameter value
+  min: 1  # Minimum parameter value, only applicable for float/int
+  max: 4096  # Maximum parameter value, only applicable for float/int
+pricing:  # Pricing information
+  input: '8.00'  # Input price, i.e., Prompt price
+  output: '24.00'  # Output price, i.e., returned content price
+  unit: '0.000001'  # Pricing unit, i.e., the above prices are per 100K
+  currency: USD  # Currency
+```
+
+It is recommended to prepare all model configurations before starting the implementation of the model code.
+
+Similarly, you can also refer to the YAML configuration information for corresponding model types of other providers in the `model_providers` directory. The complete YAML rules can be found at: [Schema](schema.md#AIModel).
+
+#### Implementing Model Invocation Code
+
+Next, you need to create a python file named `llm.py` under the `llm` `module` to write the implementation code.
+
+In `llm.py`, create an Anthropic LLM class, which we name `AnthropicLargeLanguageModel` (arbitrarily), inheriting the `__base.large_language_model.LargeLanguageModel` base class, and implement the following methods:
+
+- LLM Invocation
+
+  Implement the core method for LLM invocation, which can support both streaming and synchronous returns.
+
+  ```python
+  def _invoke(self, model: str, credentials: dict,
+              prompt_messages: list[PromptMessage], model_parameters: dict,
+              tools: Optional[list[PromptMessageTool]] = None, stop: Optional[List[str]] = None,
+              stream: bool = True, user: Optional[str] = None) \
+          -> Union[LLMResult, Generator]:
+      """
+      Invoke large language model
+  
+      :param model: model name
+      :param credentials: model credentials
+      :param prompt_messages: prompt messages
+      :param model_parameters: model parameters
+      :param tools: tools for tool calling
+      :param stop: stop words
+      :param stream: is stream response
+      :param user: unique user id
+      :return: full response or stream response chunk generator result
+      """
+  ```
+
+- Pre-calculating Input Tokens
+
+  If the model does not provide a pre-calculated tokens interface, you can directly return 0.
+
+  ```python
+  def get_num_tokens(self, model: str, credentials: dict, prompt_messages: list[PromptMessage],
+                   tools: Optional[list[PromptMessageTool]] = None) -> int:
+    """
+    Get number of tokens for given prompt messages
+
+    :param model: model name
+    :param credentials: model credentials
+    :param prompt_messages: prompt messages
+    :param tools: tools for tool calling
+    :return:
+    """
+  ```
+
+- Model Credential Verification
+
+  Similar to provider credential verification, this step involves verification for an individual model.
+
+  ```python
+  def validate_credentials(self, model: str, credentials: dict) -> None:
+      """
+      Validate model credentials
+  
+      :param model: model name
+      :param credentials: model credentials
+      :return:
+      """
+  ```
+
+- Invocation Error Mapping Table
+
+  When there is an exception in model invocation, it needs to be mapped to the `InvokeError` type specified by Runtime. This facilitates Dify's ability to handle different errors with appropriate follow-up actions.
+
+  Runtime Errors:
+
+  - `InvokeConnectionError` Invocation connection error
+  - `InvokeServerUnavailableError` Invocation service provider unavailable
+  - `InvokeRateLimitError` Invocation reached rate limit
+  - `InvokeAuthorizationError` Invocation authorization failure
+  - `InvokeBadRequestError` Invocation parameter error
+
+  ```python
+  @property
+  def _invoke_error_mapping(self) -> dict[type[InvokeError], list[type[Exception]]]:
+      """
+      Map model invoke error to unified error
+      The key is the error type thrown to the caller
+      The value is the error type thrown by the model,
+      which needs to be converted into a unified error type for the caller.
+  
+      :return: Invoke error mapping
+      """
+  ```
+
+For details on the interface methods, see: [Interfaces](interfaces.md). For specific implementations, refer to: [llm.py](https://github.com/langgenius/dify-runtime/blob/main/lib/model_providers/anthropic/llm/llm.py).
+
+### Testing
+
+To ensure the availability of integrated providers/models, each method written needs corresponding integration test code in the `tests` directory.
+
+Continuing with `Anthropic` as an example:
+
+Before writing test code, you need to first add the necessary credential environment variables for the test provider in `.env.example`, such as: `ANTHROPIC_API_KEY`.
+
+Before execution, copy `.env.example` to `.env` and then execute.
+
+#### Writing Test Code
+
+Create a `module` with the same name as the provider in the `tests` directory: `anthropic`, and continue to create `test_provider.py` and test py files for the corresponding model types within this module, as shown below:
+
+```shell
+.
+├── __init__.py
+├── anthropic
+│   ├── __init__.py
+│   ├── test_llm.py       # LLM Testing
+│   └── test_provider.py  # Provider Testing
+```
+
+Write test code for all the various cases implemented above and submit the code after passing the tests.

api/core/model_runtime/docs/en_US/schema.md

@@ -0,0 +1,194 @@
+# Configuration Rules
+
+- Provider rules are based on the [Provider](#Provider) entity.
+- Model rules are based on the [AIModelEntity](#AIModelEntity) entity.
+
+> All entities mentioned below are based on `Pydantic BaseModel` and can be found in the `entities` module.
+
+### Provider
+
+- `provider` (string) Provider identifier, e.g., `openai`
+- `label` (object) Provider display name, i18n, with `en_US` English and `zh_Hans` Chinese language settings
+  - `zh_Hans` (string) [optional] Chinese label name, if `zh_Hans` is not set, `en_US` will be used by default.
+  - `en_US` (string) English label name
+- `description` (object) Provider description, i18n
+  - `zh_Hans` (string) [optional] Chinese description
+  - `en_US` (string) English description
+- `icon_small` (string) [optional] Small provider ICON, stored in the `_assets` directory under the corresponding provider implementation directory, with the same language strategy as `label`
+  - `zh_Hans` (string) Chinese ICON
+  - `en_US` (string) English ICON
+- `icon_large` (string) [optional] Large provider ICON, stored in the `_assets` directory under the corresponding provider implementation directory, with the same language strategy as `label`
+  - `zh_Hans` (string) Chinese ICON
+  - `en_US` (string) English ICON
+- `background` (string) [optional] Background color value, e.g., #FFFFFF, if empty, the default frontend color value will be displayed.
+- `help` (object) [optional] help information
+  - `title` (object) help title, i18n
+    - `zh_Hans` (string) [optional] Chinese title
+    - `en_US` (string) English title
+  - `url` (object) help link, i18n
+    - `zh_Hans` (string) [optional] Chinese link
+    - `en_US` (string) English link
+- `supported_model_types` (array[[ModelType](#ModelType)]) Supported model types
+- `configurate_methods` (array[[ConfigurateMethod](#ConfigurateMethod)]) Configuration methods
+- `provider_credential_schema` ([ProviderCredentialSchema](#ProviderCredentialSchema)) Provider credential specification
+- `model_credential_schema` ([ModelCredentialSchema](#ModelCredentialSchema)) Model credential specification
+
+### AIModelEntity
+
+- `model` (string) Model identifier, e.g., `gpt-3.5-turbo`
+- `label` (object) [optional] Model display name, i18n, with `en_US` English and `zh_Hans` Chinese language settings
+  - `zh_Hans` (string) [optional] Chinese label name
+  - `en_US` (string) English label name
+- `model_type` ([ModelType](#ModelType)) Model type
+- `features` (array[[ModelFeature](#ModelFeature)]) [optional] Supported feature list
+- `model_properties` (object) Model properties
+  - `mode` ([LLMMode](#LLMMode)) Mode (available for model type `llm`)
+  - `context_size` (int) Context size (available for model types `llm`, `text-embedding`)
+  - `max_chunks` (int) Maximum number of chunks (available for model types `text-embedding`, `moderation`)
+  - `file_upload_limit` (int) Maximum file upload limit, in MB (available for model type `speech2text`)
+  - `supported_file_extensions` (string) Supported file extension formats, e.g., mp3, mp4 (available for model type `speech2text`)
+  - `max_characters_per_chunk` (int) Maximum characters per chunk (available for model type `moderation`)
+- `parameter_rules` (array[[ParameterRule](#ParameterRule)]) [optional] Model invocation parameter rules
+- `pricing` ([PriceConfig](#PriceConfig)) [optional] Pricing information
+- `deprecated` (bool) Whether deprecated. If deprecated, the model will no longer be displayed in the list, but those already configured can continue to be used. Default False.
+
+### ModelType
+
+- `llm` Text generation model
+- `text-embedding` Text Embedding model
+- `rerank` Rerank model
+- `speech2text` Speech to text
+- `moderation` Moderation
+
+### ConfigurateMethod
+
+- `predefined-model` Predefined model
+
+  Indicates that users can use the predefined models under the provider by configuring the unified provider credentials.
+- `customizable-model` Customizable model
+
+  Users need to add credential configuration for each model.
+
+- `fetch-from-remote` Fetch from remote
+
+  Consistent with the `predefined-model` configuration method, only unified provider credentials need to be configured, and models are obtained from the provider through credential information.
+
+### ModelFeature
+
+- `agent-thought` Agent reasoning, generally over 70B with thought chain capability.
+- `vision` Vision, i.e., image understanding.
+
+### FetchFrom
+
+- `predefined-model` Predefined model
+- `fetch-from-remote` Remote model
+
+### LLMMode
+
+- `complete` Text completion
+- `chat` Dialogue
+
+### ParameterRule
+
+- `name` (string) Actual model invocation parameter name
+- `use_template` (string) [optional] Using template
+
+  By default, 5 variable content configuration templates are preset:
+
+  - `temperature`
+  - `top_p`
+  - `frequency_penalty`
+  - `presence_penalty`
+  - `max_tokens`
+  
+  In use_template, you can directly set the template variable name, which will use the default configuration in entities.defaults.PARAMETER_RULE_TEMPLATE
+  No need to set any parameters other than `name` and `use_template`. If additional configuration parameters are set, they will override the default configuration.
+  Refer to `openai/llm/gpt-3.5-turbo.yaml`.
+
+- `label` (object) [optional] Label, i18n
+
+  - `zh_Hans`(string) [optional] Chinese label name
+  - `en_US` (string) English label name
+
+- `type`(string) [optional] Parameter type
+
+  - `int` Integer
+  - `float` Float
+  - `string` String
+  - `boolean` Boolean
+
+- `help` (string) [optional] Help information
+
+  - `zh_Hans` (string) [optional] Chinese help information
+  - `en_US` (string) English help information
+
+- `required` (bool) Required, default False.
+
+- `default`(int/float/string/bool) [optional] Default value
+
+- `min`(int/float) [optional] Minimum value, applicable only to numeric types
+
+- `max`(int/float) [optional] Maximum value, applicable only to numeric types
+
+- `precision`(int) [optional] Precision, number of decimal places to keep, applicable only to numeric types
+
+- `options` (array[string]) [optional] Dropdown option values, applicable only when `type` is `string`, if not set or null, option values are not restricted
+
+### PriceConfig
+
+- `input` (float) Input price, i.e., Prompt price
+- `output` (float) Output price, i.e., returned content price
+- `unit` (float) Pricing unit, e.g., per 100K price is `0.000001`
+- `currency` (string) Currency unit
+
+### ProviderCredentialSchema
+
+- `credential_form_schemas` (array[[CredentialFormSchema](#CredentialFormSchema)]) Credential form standard
+
+### ModelCredentialSchema
+
+- `model` (object) Model identifier, variable name defaults to `model`
+  - `label` (object) Model form item display name
+    - `en_US` (string) English
+    - `zh_Hans`(string) [optional] Chinese
+  - `placeholder` (object) Model prompt content
+    - `en_US`(string) English
+    - `zh_Hans`(string) [optional] Chinese
+- `credential_form_schemas` (array[[CredentialFormSchema](#CredentialFormSchema)]) Credential form standard
+
+### CredentialFormSchema
+
+- `variable` (string) Form item variable name
+- `label` (object) Form item label name
+  - `en_US`(string) English
+  - `zh_Hans` (string) [optional] Chinese
+- `type` ([FormType](#FormType)) Form item type
+- `required` (bool) Whether required
+- `default`(string) Default value
+- `options` (array[[FormOption](#FormOption)]) Specific property of form items of type `select` or `radio`, defining dropdown content
+- `placeholder`(object) Specific property of form items of type `text-input`, placeholder content
+  - `en_US`(string) English
+  - `zh_Hans` (string) [optional] Chinese
+- `max_length` (int) Specific property of form items of type `text-input`, defining maximum input length, 0 for no limit.
+- `show_on` (array[[FormShowOnObject](#FormShowOnObject)]) Displayed when other form item values meet certain conditions, displayed always if empty.
+
+### FormType
+
+- `text-input` Text input component
+- `secret-input` Password input component
+- `select` Single-choice dropdown
+- `radio` Radio component
+- `switch` Switch component, only supports `true` and `false` values
+
+### FormOption
+
+- `label` (object) Label
+  - `en_US`(string) English
+  - `zh_Hans`(string) [optional] Chinese
+- `value` (string) Dropdown option value
+- `show_on` (array[[FormShowOnObject](#FormShowOnObject)]) Displayed when other form item values meet certain conditions, displayed always if empty.
+
+### FormShowOnObject
+
+- `variable` (string) Variable name of other form items
+- `value` (string) Variable value of other form items

api/core/model_runtime/docs/zh_Hans/customizable_model_scale_out.md

@@ -0,0 +1,296 @@
+## 自定义预定义模型接入
+
+### 介绍
+
+供应商集成完成后，接下来为供应商下模型的接入，为了帮助理解整个接入过程，我们以`Xinference`为例，逐步完成一个完整的供应商接入。
+
+需要注意的是，对于自定义模型，每一个模型的接入都需要填写一个完整的供应商凭据。
+
+而不同于预定义模型，自定义供应商接入时永远会拥有如下两个参数，不需要在供应商yaml中定义。
+
+![Alt text](images/index/image-3.png)
+
+
+在前文中，我们已经知道了供应商无需实现`validate_provider_credential`，Runtime会自行根据用户在此选择的模型类型和模型名称调用对应的模型层的`validate_credentials`来进行验证。
+
+### 编写供应商yaml
+
+我们首先要确定，接入的这个供应商支持哪些类型的模型。
+
+当前支持模型类型如下：
+
+- `llm` 文本生成模型
+- `text_embedding` 文本 Embedding 模型
+- `rerank` Rerank 模型
+- `speech2text` 语音转文字
+- `moderation` 审查
+
+`Xinference`支持`LLM`和`Text Embedding`和Rerank，那么我们开始编写`xinference.yaml`。
+
+```yaml
+provider: xinference #确定供应商标识
+label: # 供应商展示名称，可设置 en_US 英文、zh_Hans 中文两种语言，zh_Hans 不设置将默认使用 en_US。
+  en_US: Xorbots Inference
+icon_small: # 小图标，可以参考其他供应商的图标，存储在对应供应商实现目录下的 _assets 目录，中英文策略同 label
+  en_US: icon_s_en.svg
+icon_large: # 大图标
+  en_US: icon_l_en.svg
+help: # 帮助
+  title:
+    en_US: How to deploy Xinference
+    zh_Hans: 如何部署 Xinference
+  url:
+    en_US: https://github.com/xorbitsai/inference
+supported_model_types: # 支持的模型类型，Xinference同时支持LLM/Text Embedding/Rerank
+- llm
+- text-embedding
+- rerank
+configurate_methods: # 因为Xinference为本地部署的供应商，并且没有预定义模型，需要用什么模型需要根据Xinference的文档自己部署，所以这里只支持自定义模型
+- customizable-model
+provider_credential_schema:
+  credential_form_schemas:
+```
+
+随后，我们需要思考在Xinference中定义一个模型需要哪些凭据
+
+- 它支持三种不同的模型，因此，我们需要有`model_type`来指定这个模型的类型，它有三种类型，所以我们这么编写
+```yaml
+provider_credential_schema:
+  credential_form_schemas:
+  - variable: model_type
+    type: select
+    label:
+      en_US: Model type
+      zh_Hans: 模型类型
+    required: true
+    options:
+    - value: text-generation
+      label:
+        en_US: Language Model
+        zh_Hans: 语言模型
+    - value: embeddings
+      label:
+        en_US: Text Embedding
+    - value: reranking
+      label:
+        en_US: Rerank
+```
+- 每一个模型都有自己的名称`model_name`，因此需要在这里定义
+```yaml
+  - variable: model_name
+    type: text-input
+    label:
+      en_US: Model name
+      zh_Hans: 模型名称
+    required: true
+    placeholder:
+      zh_Hans: 填写模型名称
+      en_US: Input model name
+```
+- 填写Xinference本地部署的地址
+```yaml
+  - variable: server_url
+    label:
+      zh_Hans: 服务器URL
+      en_US: Server url
+    type: text-input
+    required: true
+    placeholder:
+      zh_Hans: 在此输入Xinference的服务器地址，如 https://example.com/xxx
+      en_US: Enter the url of your Xinference, for example https://example.com/xxx
+```
+- 每个模型都有唯一的model_uid，因此需要在这里定义
+```yaml
+  - variable: model_uid
+    label:
+      zh_Hans: 模型UID
+      en_US: Model uid
+    type: text-input
+    required: true
+    placeholder:
+      zh_Hans: 在此输入您的Model UID
+      en_US: Enter the model uid
+```
+现在，我们就完成了供应商的基础定义。
+
+### 编写模型代码
+
+然后我们以`llm`类型为例，编写`xinference.llm.llm.py`
+
+在 `llm.py` 中创建一个 Xinference LLM 类，我们取名为 `XinferenceAILargeLanguageModel`（随意），继承 `__base.large_language_model.LargeLanguageModel` 基类，实现以下几个方法：
+
+- LLM 调用
+
+  实现 LLM 调用的核心方法，可同时支持流式和同步返回。
+
+  ```python
+  def _invoke(self, model: str, credentials: dict,
+              prompt_messages: list[PromptMessage], model_parameters: dict,
+              tools: Optional[list[PromptMessageTool]] = None, stop: Optional[List[str]] = None,
+              stream: bool = True, user: Optional[str] = None) \
+          -> Union[LLMResult, Generator]:
+      """
+      Invoke large language model
+  
+      :param model: model name
+      :param credentials: model credentials
+      :param prompt_messages: prompt messages
+      :param model_parameters: model parameters
+      :param tools: tools for tool calling
+      :param stop: stop words
+      :param stream: is stream response
+      :param user: unique user id
+      :return: full response or stream response chunk generator result
+      """
+  ```
+
+  在实现时，需要注意使用两个函数来返回数据，分别用于处理同步返回和流式返回，因为Python会将函数中包含 `yield` 关键字的函数识别为生成器函数，返回的数据类型固定为 `Generator`，因此同步和流式返回需要分别实现，就像下面这样（注意下面例子使用了简化参数，实际实现时需要按照上面的参数列表进行实现）：
+
+  ```python
+  def _invoke(self, stream: bool, **kwargs) \
+          -> Union[LLMResult, Generator]:
+      if stream:
+            return self._handle_stream_response(**kwargs)
+      return self._handle_sync_response(**kwargs)
+
+  def _handle_stream_response(self, **kwargs) -> Generator:
+      for chunk in response:
+            yield chunk
+  def _handle_sync_response(self, **kwargs) -> LLMResult:
+      return LLMResult(**response)
+  ```
+
+- 预计算输入 tokens
+
+  若模型未提供预计算 tokens 接口，可直接返回 0。
+
+  ```python
+  def get_num_tokens(self, model: str, credentials: dict, prompt_messages: list[PromptMessage],
+                   tools: Optional[list[PromptMessageTool]] = None) -> int:
+    """
+    Get number of tokens for given prompt messages
+
+    :param model: model name
+    :param credentials: model credentials
+    :param prompt_messages: prompt messages
+    :param tools: tools for tool calling
+    :return:
+    """
+  ```
+
+  有时候，也许你不需要直接返回0，所以你可以使用`self._get_num_tokens_by_gpt2(text: str)`来获取预计算的tokens，这个方法位于`AIModel`基类中，它会使用GPT2的Tokenizer进行计算，但是只能作为替代方法，并不完全准确。
+
+- 模型凭据校验
+
+  与供应商凭据校验类似，这里针对单个模型进行校验。
+
+  ```python
+  def validate_credentials(self, model: str, credentials: dict) -> None:
+      """
+      Validate model credentials
+  
+      :param model: model name
+      :param credentials: model credentials
+      :return:
+      """
+  ```
+
+- 模型参数Schema
+  
+  与自定义类型不同，由于没有在yaml文件中定义一个模型支持哪些参数，因此，我们需要动态时间模型参数的Schema。
+  
+  如Xinference支持`max_tokens` `temperature` `top_p` 这三个模型参数。
+  
+  但是有的供应商根据不同的模型支持不同的参数，如供应商`OpenLLM`支持`top_k`，但是并不是这个供应商提供的所有模型都支持`top_k`，我们这里举例A模型支持`top_k`，B模型不支持`top_k`，那么我们需要在这里动态生成模型参数的Schema，如下所示：
+  
+    ```python
+    def get_customizable_model_schema(self, model: str, credentials: dict) -> AIModelEntity | None:
+        """
+            used to define customizable model schema
+        """
+        rules = [
+            ParameterRule(
+                name='temperature', type=ParameterType.FLOAT,
+                use_template='temperature',
+                label=I18nObject(
+                    zh_Hans='温度', en_US='Temperature'
+                )
+            ),
+            ParameterRule(
+                name='top_p', type=ParameterType.FLOAT,
+                use_template='top_p',
+                label=I18nObject(
+                    zh_Hans='Top P', en_US='Top P'
+                )
+            ),
+            ParameterRule(
+                name='max_tokens', type=ParameterType.INT,
+                use_template='max_tokens',
+                min=1,
+                default=512,
+                label=I18nObject(
+                    zh_Hans='最大生成长度', en_US='Max Tokens'
+                )
+            )
+        ]
+
+        # if model is A, add top_k to rules
+        if model == 'A':
+            rules.append(
+                ParameterRule(
+                    name='top_k', type=ParameterType.INT,
+                    use_template='top_k',
+                    min=1,
+                    default=50,
+                    label=I18nObject(
+                        zh_Hans='Top K', en_US='Top K'
+                    )
+                )
+            )
+
+        """
+            some NOT IMPORTANT code here
+        """
+
+        entity = AIModelEntity(
+            model=model,
+            label=I18nObject(
+                en_US=model
+            ),
+            fetch_from=FetchFrom.CUSTOMIZABLE_MODEL,
+            model_type=model_type,
+            model_properties={ 
+                'mode':  ModelType.LLM,
+            },
+            parameter_rules=rules
+        )
+
+        return entity
+    ```
+    
+- 调用异常错误映射表
+
+  当模型调用异常时需要映射到 Runtime 指定的 `InvokeError` 类型，方便 Dify 针对不同错误做不同后续处理。
+
+  Runtime Errors:
+
+  - `InvokeConnectionError` 调用连接错误
+  - `InvokeServerUnavailableError ` 调用服务方不可用
+  - `InvokeRateLimitError ` 调用达到限额
+  - `InvokeAuthorizationError`  调用鉴权失败
+  - `InvokeBadRequestError ` 调用传参有误
+
+  ```python
+  @property
+  def _invoke_error_mapping(self) -> dict[type[InvokeError], list[type[Exception]]]:
+      """
+      Map model invoke error to unified error
+      The key is the error type thrown to the caller
+      The value is the error type thrown by the model,
+      which needs to be converted into a unified error type for the caller.
+  
+      :return: Invoke error mapping
+      """
+  ```
+
+接口方法说明见：[Interfaces](./interfaces.md)，具体实现可参考：[llm.py](https://github.com/langgenius/dify-runtime/blob/main/lib/model_providers/anthropic/llm/llm.py)。