🚀 feat(config.yaml, base.py): enable RecursiveCharacterTextSplitter and update TextSplitterCreator to handle it

The RecursiveCharacterTextSplitter is now enabled in the config.yaml file. The TextSplitterCreator class in base.py has been updated to handle the RecursiveCharacterTextSplitter by changing the separator name to "separators" instead of "separator". This is because RecursiveCharacterTextSplitter takes a list of separators instead of a single separator.
This commit is contained in:
Gabriel Almeida 2023-05-30 23:38:57 -03:00
commit 5798750b32
2 changed files with 8 additions and 4 deletions

View file

@ -65,7 +65,7 @@ prompts:
- ZeroShotPrompt
textsplitters:
- CharacterTextSplitter
# - RecursiveCharacterTextSplitter
- RecursiveCharacterTextSplitter
# - LatexTextSplitter
# - PythonCodeTextSplitter
toolkits:

View file

@ -25,14 +25,18 @@ class TextSplitterCreator(LangChainTypeCreator):
"show": True,
"name": "documents",
}
if name == "RecursiveCharacterTextSplitter":
separator_name = "separators"
else:
separator_name = "separator"
signature["template"]["separator"] = {
signature["template"][separator_name] = {
"type": "str",
"required": True,
"show": True,
"value": ".",
"name": "separator",
"display_name": "Separator",
"name": separator_name,
"display_name": separator_name.title(),
}
signature["template"]["chunk_size"] = {