Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added .codex/design/assets/create-dataset-modal-ui.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .codex/design/assets/file-chunk-modal-ui.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .codex/design/assets/file-chunk-qa-modal-ui.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .codex/design/assets/image-chunk-modal-ui.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .codex/design/assets/search-test-base-ui.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .codex/design/assets/search-test-upload-ui.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
971 changes: 971 additions & 0 deletions .codex/design/图搜图-当前需求-功能开发文档.md

Large diffs are not rendered by default.

515 changes: 515 additions & 0 deletions .codex/design/图搜图-当前需求-需求设计文档.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,12 @@ For detailed parameters and internal logic, see: [FastGPT Knowledge Base Search]

Select one or more Knowledge Bases using the **same embedding model** for vector search.

### Input - Search Content

Search content is an `Array<string>`. New nodes reference the user's question by default. You can also add file links to the same input for mixed text-image retrieval.

At runtime, FastGPT separates text from file links automatically: plain text goes into text search, image links go into image search, and non-image file links such as PDF, docx, xlsx, txt, and pptx are filtered out. Filtered file links are not searched as text and do not make the node fail.

### Input - Search Parameters

[View parameter details](../../../dataset/dataset_engine.en.mdx#搜索参数)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,12 @@ description: FastGPT AI 知识库搜索模块介绍

可以选择一个或多个**相同向量模型**的知识库,用于向量搜索。

### 输入 - 检索内容

检索内容为`Array<string>`,新建节点默认引用用户问题。你也可以在同一个输入里追加文件链接,用于图文混合检索。

运行时会自动拆分文本和文件链接:普通文本进入文本检索;图片链接进入图片检索;PDF、docx、xlsx、txt、pptx 等非图片文件链接会被过滤,不会作为文本参与检索,也不会让节点报错。

### 输入 - 搜索参数

[点击查看参数介绍](../../../dataset/dataset_engine.mdx#搜索参数)
Expand Down
12 changes: 12 additions & 0 deletions document/content/guide/dataset/dataset_engine.en.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,18 @@ This means you can continuously improve data chunk accuracy through annotation.

![](/imgs/dataset_search_process.png)

### Image and Mixed Text-Image Search

When `Support Image Recognition` is enabled for an embedding model in model settings, FastGPT treats it as a multimodal embedding model. Image data can generate an `imageEmbedding` vector index for image-to-image search. If the Knowledge Base also has an image understanding model, images can also generate text-description indexes so text queries can match image content.

Search automatically branches based on the Knowledge Base configuration:

- Multimodal embedding model: text queries use text vector search, image queries use image vector search, and mixed text-image queries are merged with RRF.
- Multimodal embedding model + image understanding model: in addition to image vector search, query images are captioned and used for text search.
- Text-only embedding model + image understanding model: query images are captioned first, then searched as text.
- Text-only embedding model without an image understanding model: image inputs are ignored; mixed text-image queries use only the text portion.

The search test page supports local image uploads for image-only and mixed text-image tests, up to 10 images. Images uploaded for search testing are temporary, expire after 3 hours, and are not saved as Knowledge Base data.

## Search Parameters
| | | |
Expand Down
12 changes: 12 additions & 0 deletions document/content/guide/dataset/dataset_engine.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,18 @@ FastGPT 采用了`PostgresSQL`的`PG Vector`插件作为向量检索器,索引

![](/imgs/dataset_search_process.png)

### 图片与图文混合检索

当索引模型在模型配置中开启`支持图片识别`后,FastGPT 会把它视为多模态索引模型:图片数据可以生成`imageEmbedding`图片向量索引,用于以图搜图。若知识库同时配置了图片理解模型,图片还会生成文本描述索引,便于通过文字命中图片内容。

搜索时会按知识库能力自动分支:

- 多模态索引模型:文本查询走文本向量检索,图片查询走图片向量检索,图文混合时通过 RRF 合并结果。
- 多模态索引模型 + 图片理解模型:除图片向量检索外,查询图片还会先转成文本描述,再参与文本检索。
- 普通索引模型 + 图片理解模型:图片会先转成文本描述,再走普通文本检索。
- 普通索引模型且无图片理解模型:图片输入不会参与检索;图文混合时仅使用文字部分。

搜索测试页支持本地上传图片进行纯图或图文混合测试,最多 10 张。测试上传的图片只用于临时检索,3 小时后过期,不会作为知识库正式数据保存。

## 搜索参数
| | | |
Expand Down
32 changes: 31 additions & 1 deletion document/content/openapi/dataset.en.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1248,6 +1248,9 @@ curl --location --request POST 'http://localhost:3000/api/core/dataset/searchTes
--data-raw '{
"datasetId": "Dataset ID",
"text": "Who is the director",
"queryImageUrls": [
"temp/teamId/search-image.png"
],
"limit": 5000,
"similarity": 0,
"searchMode": "embedding",
Expand All @@ -1265,7 +1268,8 @@ curl --location --request POST 'http://localhost:3000/api/core/dataset/searchTes

<div>
- datasetId - Dataset ID
- text - Text to test
- text - Text to test. It can be empty if queryImageUrls contains at least one image.
- queryImageUrls - Temporary image keys for search testing, up to 10 images. Supports image-only search and mixed text-image search. Public image URLs, dataset keys, and chat keys are not supported here. For API calls, first upload the local image through `/api/core/dataset/file/uploadSearchTestImage`, then pass the returned `key` (`temp/${teamId}/...`) in this field. The temporary key expires after 3 hours.
- limit - Maximum tokens
- similarity - Minimum similarity (0~1, optional)
- searchMode - Search mode: embedding | fullTextRecall | mixedRecall
Expand Down Expand Up @@ -1303,3 +1307,29 @@ Returns top k results. limit is the maximum tokens, up to 20000 tokens.

</Tab>
</Tabs>

### Upload Search Test Image

Use this endpoint to upload a local image from the search test page or an OpenAPI client. Only image files are supported; PDF, docx, xlsx, txt, pptx, and other document files are not accepted. Image format and size limits follow the system upload configuration. Uploaded objects are only used for temporary retrieval and expire after 3 hours.

To use image retrieval with `/api/core/dataset/searchTest`, call this endpoint first and pass `data.key` from the response to `queryImageUrls`. Public image URLs are not supported directly in `queryImageUrls`.

```bash
curl --location --request POST 'http://localhost:3000/api/core/dataset/file/uploadSearchTestImage' \
--header 'Authorization: Bearer fastgpt-xxxxx' \
--form 'datasetId="Dataset ID"' \
--form 'file=@"./query.png"'
```

Response example:

```json
{
"code": 200,
"statusText": "",
"data": {
"key": "temp/teamId/query.png",
"previewUrl": "https://..."
}
}
```
32 changes: 31 additions & 1 deletion document/content/openapi/dataset.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1230,6 +1230,9 @@ curl --location --request POST 'http://localhost:3000/api/core/dataset/searchTes
--data-raw '{
"datasetId": "知识库的ID",
"text": "导演是谁",
"queryImageUrls": [
"temp/teamId/search-image.png"
],
"limit": 5000,
"similarity": 0,
"searchMode": "embedding",
Expand All @@ -1247,7 +1250,8 @@ curl --location --request POST 'http://localhost:3000/api/core/dataset/searchTes

<div>
- datasetId - 知识库ID
- text - 需要测试的文本
- text - 需要测试的文本,可为空;当 text 为空时,queryImageUrls 至少需要传 1 张图片
- queryImageUrls - 搜索测试图片临时 key,最多 10 张;支持纯图片搜索和图文混合搜索。当前不支持直接传公网 URL、dataset key 或 chat key。API 调用时需要先通过 `/api/core/dataset/file/uploadSearchTestImage` 上传本地图片,使用响应中的 `key`(格式为 `temp/${teamId}/...`)填入该字段。该临时 key 3 小时后过期
- limit - 最大 tokens 数量
- similarity - 最低相关度(0~1,可选)
- searchMode - 搜索模式:embedding | fullTextRecall | mixedRecall
Expand Down Expand Up @@ -1285,3 +1289,29 @@ curl --location --request POST 'http://localhost:3000/api/core/dataset/searchTes

</Tab>
</Tabs>

### 上传搜索测试图片

该接口用于搜索测试页或 OpenAPI 调用方上传本地图片。仅支持图片文件,不支持 PDF、docx、xlsx、txt、pptx 等普通文件;图片格式和大小限制跟随系统上传配置。上传对象只用于临时检索,过期时间固定为 3 小时。

如果要在 `/api/core/dataset/searchTest` 中使用图片检索,先调用本接口上传图片,然后将响应中的 `data.key` 填入 `queryImageUrls`。当前不支持把公网图片 URL 直接填入 `queryImageUrls`。

```bash
curl --location --request POST 'http://localhost:3000/api/core/dataset/file/uploadSearchTestImage' \
--header 'Authorization: Bearer fastgpt-xxxxx' \
--form 'datasetId="知识库的ID"' \
--form 'file=@"./query.png"'
```

响应示例:

```json
{
"code": 200,
"statusText": "",
"data": {
"key": "temp/teamId/query.png",
"previewUrl": "https://..."
}
}
```
Loading
Loading