OpenAIMultiModalTool
public class OpenAIMultiModalTool源码Source
io/agentscope/extensions/model/openai/tool/OpenAIMultiModalTool.java:57–594OpenAI Multi Modal 工具，封装可被智能体调用的能力。位于 io.agentscope.extensions.model.openai.tool 包。核心成员包括 openaiTextToImage、openaiImageToText、openaiTextToAudio、openaiAudioToText、OpenAIMultiModalTool。
OpenAI multimodal tool.
Supports:
-
Text to image(s)
-
Image to text (via vision models)
-
Text to audio (speech)
-
Audio to text (transcription)
This implementation uses the custom OpenAIClient HTTP client instead of the OpenAI
Java SDK, keeping the core module lightweight and dependency-free.
Please refer to the OpenAI documentation for more details.
构造器摘要Constructor Summary
| 构造器Constructor | 说明Description |
|---|---|
public OpenAIMultiModalTool(String apiKey) | 创建新的OpenAIMultiModalTool 使用默认基类 URL。 Create a new OpenAIMultiModalTool with default base URL. |
public OpenAIMultiModalTool(String apiKey, String baseUrl) | 创建新的OpenAIMultiModalTool 使用自定义基类 URL。 Create a new OpenAIMultiModalTool with custom base URL. |
public OpenAIMultiModalTool(String apiKey, String baseUrl, String defaultModelName) | 创建新的OpenAIMultiModalTool 使用自定义基类 URL 和默认视觉模型。 Create a new OpenAIMultiModalTool with custom base URL and default vision model. |
protected OpenAIMultiModalTool(OpenAIClient client) | 初始化 `OpenAIMultiModalTool` 实例。 Create a new OpenAIMultiModalTool with custom client (for testing). |
protected OpenAIMultiModalTool(OpenAIClient client, String defaultModelName) | 初始化 `OpenAIMultiModalTool` 实例。 Create a new OpenAIMultiModalTool with custom client and default model (for testing). |
方法摘要Method Summary
| 方法Method | 说明Description |
|---|---|
public Mono<ToolResultBlock> openaiTextToImage(String prompt, String model, Integer n, String size, String quality, String responseFormat) | 执行OpenAI 文本到图像相关操作。 Generate image(s) based on the given prompt. |
public Mono<ToolResultBlock> openaiImageToText(String imageUrls, String prompt, String model, Integer maxTokens) | 转换图像(s)到文本使用视觉模型。 Convert image(s) to text using vision models. |
public Mono<ToolResultBlock> openaiTextToAudio(String text, String model, String voice, String responseFormat, Double speed) | 转换文本到音频(语音)使用 OpenAI TTS 模型。 Convert text to audio (speech) using OpenAI TTS models. |
public Mono<ToolResultBlock> openaiAudioToText(String audioUrl, String model, String language, String prompt, String responseFormat, Double temperature) | 执行OpenAI 音频到文本相关操作。 Convert audio to text (transcription) using OpenAI Whisper models. |
构造器详情Constructor Detail
OpenAIMultiModalTool
public OpenAIMultiModalTool(String apiKey)创建新的OpenAIMultiModalTool 使用默认基类 URL。
Create a new OpenAIMultiModalTool with default base URL.
参数Parameters
| 名称Name | 类型Type | 说明Description |
|---|---|---|
apiKey | String | OpenAI API 密钥。the OpenAI API key |
源码Source
io/agentscope/extensions/model/openai/tool/OpenAIMultiModalTool.java:78–80OpenAIMultiModalTool
public OpenAIMultiModalTool(String apiKey, String baseUrl)创建新的OpenAIMultiModalTool 使用自定义基类 URL。
Create a new OpenAIMultiModalTool with custom base URL.
参数Parameters
| 名称Name | 类型Type | 说明Description |
|---|---|---|
apiKey | String | OpenAI API 密钥。the OpenAI API key |
baseUrl | String | 基类 URL（类型 `String`）。the base URL (null for default https://api.openai.com) |
源码Source
io/agentscope/extensions/model/openai/tool/OpenAIMultiModalTool.java:88–90OpenAIMultiModalTool
public OpenAIMultiModalTool(String apiKey, String baseUrl, String defaultModelName)创建新的OpenAIMultiModalTool 使用自定义基类 URL 和默认视觉模型。
Create a new OpenAIMultiModalTool with custom base URL and default vision model.
参数Parameters
| 名称Name | 类型Type | 说明Description |
|---|---|---|
apiKey | String | OpenAI API 密钥。the OpenAI API key |
baseUrl | String | 基类 URL（类型 `String`）。the base URL (null for default https://api.openai.com) |
defaultModelName | String | 默认模型名称（类型 `String`）。the default vision model name used when the caller omits the model parameter (e.g., "gpt-4o" for OpenAI, or your backend's vision model name) |
源码Source
io/agentscope/extensions/model/openai/tool/OpenAIMultiModalTool.java:100–111OpenAIMultiModalTool
protected OpenAIMultiModalTool(OpenAIClient client)初始化 OpenAIMultiModalTool 实例。
Create a new OpenAIMultiModalTool with custom client (for testing).
参数Parameters
| 名称Name | 类型Type | 说明Description |
|---|---|---|
client | OpenAIClient | OpenAI 客户端。the OpenAI client |
源码Source
io/agentscope/extensions/model/openai/tool/OpenAIMultiModalTool.java:118–120OpenAIMultiModalTool
protected OpenAIMultiModalTool(OpenAIClient client, String defaultModelName)初始化 OpenAIMultiModalTool 实例。
Create a new OpenAIMultiModalTool with custom client and default model (for testing).
参数Parameters
| 名称Name | 类型Type | 说明Description |
|---|---|---|
client | OpenAIClient | OpenAI 客户端。the OpenAI client |
defaultModelName | String | 默认视觉模型名称。the default vision model name |
源码Source
io/agentscope/extensions/model/openai/tool/OpenAIMultiModalTool.java:128–136方法详情Method Detail
openaiTextToImage
public Mono<ToolResultBlock> openaiTextToImage(String prompt, String model, Integer n, String size, String quality, String responseFormat)执行OpenAI 文本到图像相关操作。
Generate image(s) based on the given prompt.
参数Parameters
| 名称Name | 类型Type | 说明Description |
|---|---|---|
prompt | String | 提示词（类型 `String`）。the text prompt to generate image |
model | String | 模型实例（类型 `String`）。the model to use (e.g., "dall-e-3", "dall-e-2") |
n | Integer | n（类型 `Integer`）。the number of images to generate (1 for dall-e-3, 1-10 for dall-e-2) |
size | String | 大小（类型 `String`）。the size of the image (e.g., "1024x1024", "1792x1024", "1024x1792") |
quality | String | quality（类型 `String`）。the quality of the image ("standard" or "hd" for dall-e-3) |
responseFormat | String | 响应("URL"或"b64_JSON")的格式。the format of the response ("url" or "b64_json") |
Mono<ToolResultBlock>返回Mono。a ToolResultBlock containing the generated image(s)
源码Source
io/agentscope/extensions/model/openai/tool/OpenAIMultiModalTool.java:149–278openaiImageToText
public Mono<ToolResultBlock> openaiImageToText(String imageUrls, String prompt, String model, Integer maxTokens)转换图像(s)到文本使用视觉模型。
Convert image(s) to text using vision models.
参数Parameters
| 名称Name | 类型Type | 说明Description |
|---|---|---|
imageUrls | String | 图像 Urls（类型 `String`）。the URLs of the images to analyze |
prompt | String | 提示词（类型 `String`）。the text prompt describing what to extract from the images |
model | String | 模型实例（类型 `String`）。the vision model to use (leave empty to use the configured default) |
maxTokens | Integer | 最大令牌数（类型 `Integer`）。the maximum number of tokens in the response |
Mono<ToolResultBlock>返回Mono。a ToolResultBlock containing the text description of the images
源码Source
io/agentscope/extensions/model/openai/tool/OpenAIMultiModalTool.java:289–407openaiTextToAudio
public Mono<ToolResultBlock> openaiTextToAudio(String text, String model, String voice, String responseFormat, Double speed)转换文本到音频(语音)使用 OpenAI TTS 模型。
Convert text to audio (speech) using OpenAI TTS models.
参数Parameters
| 名称Name | 类型Type | 说明Description |
|---|---|---|
text | String | 文本到转换到语音。the text to convert to speech |
model | String | 模型实例（类型 `String`）。the TTS model to use (e.g., "tts-1", "tts-1-hd") |
voice | String | 语音（类型 `String`）。the voice to use ("alloy", "echo", "fable", "onyx", "nova", "shimmer") |
responseFormat | String | 响应格式（类型 `String`）。the audio format ("mp3", "opus", "aac", "flac") |
speed | Double | speed（类型 `Double`）。the speed of the speech (0.25 to 4.0) |
Mono<ToolResultBlock>返回Mono。a ToolResultBlock containing the audio as base64 data
源码Source
io/agentscope/extensions/model/openai/tool/OpenAIMultiModalTool.java:419–491openaiAudioToText
public Mono<ToolResultBlock> openaiAudioToText(String audioUrl, String model, String language, String prompt, String responseFormat, Double temperature)执行OpenAI 音频到文本相关操作。
Convert audio to text (transcription) using OpenAI Whisper models.
参数Parameters
| 名称Name | 类型Type | 说明Description |
|---|---|---|
audioUrl | String | 音频 URL（类型 `String`）。the URL of the audio file to transcribe |
model | String | 模型实例（类型 `String`）。the transcription model to use (e.g., "whisper-1") |
language | String | language（类型 `String`）。the language of the audio (ISO-639-1 code, optional) |
prompt | String | 提示词（类型 `String`）。optional text to guide the model's style |
responseFormat | String | 响应格式（类型 `String`）。the format of the response ("json", "text", "verbose_json", etc.) |
temperature | Double | temperature（类型 `Double`）。the temperature for sampling (0.0 to 1.0) |
Mono<ToolResultBlock>返回Mono。a ToolResultBlock containing the transcribed text
源码Source
io/agentscope/extensions/model/openai/tool/OpenAIMultiModalTool.java:507–593