🚀 feat(config.yaml, base.py): enable RecursiveCharacterTextSplitter and update TextSplitterCreator to handle it
The RecursiveCharacterTextSplitter is now enabled in the config.yaml file. The TextSplitterCreator class in base.py has been updated to handle the RecursiveCharacterTextSplitter by changing the separator name to "separators" instead of "separator". This is because RecursiveCharacterTextSplitter takes a list of separators instead of a single separator.
This commit is contained in:
parent
38d0c9b170
commit
5798750b32
2 changed files with 8 additions and 4 deletions
|
|
@ -65,7 +65,7 @@ prompts:
|
|||
- ZeroShotPrompt
|
||||
textsplitters:
|
||||
- CharacterTextSplitter
|
||||
# - RecursiveCharacterTextSplitter
|
||||
- RecursiveCharacterTextSplitter
|
||||
# - LatexTextSplitter
|
||||
# - PythonCodeTextSplitter
|
||||
toolkits:
|
||||
|
|
|
|||
|
|
@ -25,14 +25,18 @@ class TextSplitterCreator(LangChainTypeCreator):
|
|||
"show": True,
|
||||
"name": "documents",
|
||||
}
|
||||
if name == "RecursiveCharacterTextSplitter":
|
||||
separator_name = "separators"
|
||||
else:
|
||||
separator_name = "separator"
|
||||
|
||||
signature["template"]["separator"] = {
|
||||
signature["template"][separator_name] = {
|
||||
"type": "str",
|
||||
"required": True,
|
||||
"show": True,
|
||||
"value": ".",
|
||||
"name": "separator",
|
||||
"display_name": "Separator",
|
||||
"name": separator_name,
|
||||
"display_name": separator_name.title(),
|
||||
}
|
||||
|
||||
signature["template"]["chunk_size"] = {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue