Class ChatModelIntegrationTests<CallOptions, OutputMessageType, ConstructorArgs>Abstract

constructor

• new ChatModelIntegrationTests<CallOptions, OutputMessageType, ConstructorArgs>(fields): ChatModelIntegrationTests<CallOptions, OutputMessageType, ConstructorArgs>

• Type Parameters

  • CallOptions extends BaseChatModelCallOptions = BaseChatModelCallOptions
  • OutputMessageType extends BaseMessageChunk = BaseMessageChunk
  • ConstructorArgs extends RecordStringAny = RecordStringAny

  Parameters

  • fields: ChatModelIntegrationTestsFields<CallOptions, OutputMessageType, ConstructorArgs>

  Returns ChatModelIntegrationTests<CallOptions, OutputMessageType, ConstructorArgs>

functionId

invokeResponseType

supportsParallelToolCalls

runTests

• runTests(): Promise<boolean>

• Run all unit tests for the chat model. Each test is wrapped in a try/catch block to prevent the entire test suite from failing. If a test fails, the error is logged to the console, and the test suite continues.

  Returns Promise<boolean>

testBatch

• testBatch(callOptions?): Promise<void>

• Tests the batch processing capability of the chat model. This test ensures that the model can handle multiple inputs simultaneously and return appropriate responses for each.

  It verifies that:

  1. The batch results are defined and in array format.
  2. The number of results matches the number of inputs.
  3. Each result is of the correct type and has non-empty content.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testBindToolsWithOpenAIFormattedTools

• testBindToolsWithOpenAIFormattedTools(callOptions?): Promise<void>

• Tests the chat model's ability to bind and use OpenAI-formatted tools. This test ensures that the model can correctly process and use tools formatted in the OpenAI function calling style.

  It verifies that:

  1. The model supports tool calling functionality.
  2. The model can successfully bind an OpenAI-formatted tool.
  3. The model invokes the bound tool correctly when prompted.
  4. The result contains a tool call with the expected name.

  This test is crucial for ensuring compatibility with OpenAI's function calling format, which is a common standard in AI tool integration.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testBindToolsWithRunnableToolLike

• testBindToolsWithRunnableToolLike(callOptions?): Promise<void>

• Tests the chat model's ability to bind and use Runnable-like tools. This test ensures that the model can correctly process and use tools that are created from Runnable objects using the asTool method.

  It verifies that:

  1. The model supports tool calling functionality.
  2. The model can successfully bind a Runnable-like tool.
  3. The model invokes the bound tool correctly when prompted.
  4. The result contains a tool call with the expected name.

  This test is crucial for ensuring compatibility with tools created from Runnable objects, which provides a flexible way to integrate custom logic into the model's tool-calling capabilities.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testCacheComplexMessageTypes

• testCacheComplexMessageTypes(callOptions?): Promise<void>

• Tests the chat model's ability to cache and retrieve complex message types. This test ensures that the model can correctly cache and retrieve messages with complex content structures, such as arrays of content objects.

  It verifies that:

  1. The model can be instantiated with caching enabled.
  2. A complex HumanMessage can be created and invoked.
  3. The result is correctly cached after the first invocation.
  4. A subsequent invocation with the same input retrieves the cached result.
  5. The cached result matches the original result in both content and structure.
  6. No additional cache entries are created for repeated invocations.

  This test is crucial for ensuring that the caching mechanism works correctly with various message structures, maintaining consistency and efficiency.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testConversation

• testConversation(callOptions?): Promise<void>

• Tests the chat model's ability to handle a conversation with multiple messages. This test ensures that the model can process a sequence of messages from different roles (Human and AI) and generate an appropriate response.

  It verifies that:

  1. The result is defined and is an instance of the correct response type.
  2. The content of the response is a non-empty string.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testInvoke

• testInvoke(callOptions?): Promise<void>

• Tests the basic invoke method of the chat model. This test ensures that the model can process a simple input and return a valid response.

  It verifies that:

  1. The result is defined and is an instance of the correct type.
  2. The content of the response is a non-empty string.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testInvokeMoreComplexTools

• testInvokeMoreComplexTools(callOptions?): Promise<void>

• Tests the chat model's ability to handle a more complex tool schema. This test verifies that the model can correctly process and use a tool with a schema that includes a z.record(z.unknown()) field, which represents an object with unknown/any fields.

  The test performs the following steps:

  1. Defines a complex schema with nested objects and unknown fields.
  2. Creates a chat prompt template that instructs the model to use the tool.
  3. Invokes the model with structured output using the complex schema.
  4. Verifies that the result contains all expected fields and types.

  This test is particularly important for ensuring compatibility with APIs that may not accept JSON schemas with unknown object fields (e.g., Google's API).

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testModelCanAcceptStructuredToolParamsSchema

• testModelCanAcceptStructuredToolParamsSchema(callOptions?): Promise<void>

• Tests the chat model's ability to accept and use a StructuredToolParams schema. This schema contains the same fields as StructuredToolInterface, but does not require a function to be passed when the tool is created.

  This test verifies that the model can:

  1. Correctly bind a tool defined using StructuredToolParams
  2. Process a prompt that should trigger the use of the bound tool
  3. Generate a response that includes appropriate tool calls

  The test uses a simple weather tool to simulate a scenario where the model needs to make a tool call to retrieve weather information.

  It ensures that the model can correctly interpret the tool's schema, make the appropriate tool call, and include the required arguments.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testModelCanUseToolUseAIMessage

• testModelCanUseToolUseAIMessage(callOptions?): Promise<void>

• Tests the chat model's ability to use tool calls in a multi-turn conversation. This test verifies that the model can:

  1. Invoke a tool in response to a user query.
  2. Use the AIMessage containing the tool call in a followup request.
  3. Process the tool's response and generate a final answer.

  This capability is crucial for building agents or other pipelines that involve tool usage.

  The test follows these steps:

  1. Bind a weather tool to the model.
  2. Send an initial query about the weather.
  3. Verify the model makes a tool call.
  4. Simulate the tool's response.
  5. Send a followup request including the tool call and response.
  6. Verify the model generates a non-empty final response.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testModelCanUseToolUseAIMessageWithStreaming

• testModelCanUseToolUseAIMessageWithStreaming(callOptions?): Promise<void>

• Tests the chat model's ability to use tool calls in a multi-turn conversation with streaming. This test verifies that the model can:

  1. Stream a response that includes a tool call.
  2. Use the AIMessage containing the tool call in a followup request.
  3. Stream a final response that processes the tool's output.

  This test is crucial for ensuring that the model can handle tool usage in a streaming context, which is important for building responsive agents or other AI systems that require real-time interaction.

  The test follows these steps:

  1. Bind a weather tool to the model.
  2. Stream an initial query about the weather.
  3. Verify the streamed result contains a tool call.
  4. Simulate the tool's response.
  5. Stream a followup request including the tool call and response.
  6. Verify the model generates a non-empty final streamed response.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testParallelToolCalling

• testParallelToolCalling(callOptions?, onlyVerifyHistory?): Promise<void>

• Tests the chat model's ability to handle parallel tool calls in various scenarios. This comprehensive test covers three aspects of parallel tool calling:

  1. Invoking multiple tools simultaneously
  2. Streaming responses with parallel tool calls
  3. Processing message histories containing parallel tool calls

  The test uses a weather tool and a current time tool to simulate complex, multi-tool scenarios. It ensures that the model can correctly process and respond to prompts requiring multiple tool calls, both in streaming and non-streaming contexts, and can handle message histories with parallel tool calls.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model.
  • onlyVerifyHistory: boolean = false

    If true, only verifies the message history test.

  Returns Promise<void>

testStream

• testStream(callOptions?): Promise<void>

• Tests the streaming capability of the chat model. This test ensures that the model can properly stream responses and that each streamed token is a valid AIMessageChunk.

  It verifies that:

  1. Each streamed token is defined and is an instance of AIMessageChunk.
  2. The content of each token is a string.
  3. The total number of characters streamed is greater than zero.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testStreamEvents

• testStreamEvents(callOptions?): Promise<void>

• Tests the model can properly use the .streamEvents method. This test ensures the .streamEvents method yields at least three event types: on_chat_model_start, on_chat_model_stream, and on_chat_model_end.

  It also verifies the first chunk is an on_chat_model_start event, and the last chunk is an on_chat_model_end event. The middle chunk should be an on_chat_model_stream event.

  Finally, it verifies the final chunk's event.data.output field matches the concatenated content of all on_chat_model_stream events.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testStreamTokensWithToolCalls

• testStreamTokensWithToolCalls(callOptions?): Promise<void>

• Tests the chat model's ability to stream tokens while using tool calls. This test ensures that the model can correctly stream responses that include tool calls, and that the streamed response contains the expected information.

  It verifies that:

  1. The model can be bound with a tool and streamed successfully.
  2. The streamed result contains at least one tool call.
  3. The usage metadata is present in the streamed result.
  4. Both input and output tokens are present and greater than zero in the usage metadata.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testStructuredFewShotExamples

• testStructuredFewShotExamples(callOptions?): Promise<void>

• Tests the chat model's ability to process few-shot examples with tool calls. This test ensures that the model can correctly handle and respond to a conversation that includes tool calls within the context of few-shot examples.

  The test performs the following steps:

  1. Creates a chat model and binds an AdderTool to it.
  2. Constructs a message history that simulates a few-shot example scenario:
     • A human message asking about addition
     • An AI message with a tool call to the AdderTool
     • A ToolMessage with the result of the tool call
     • An AI message with the result
     • A new human message asking about a different addition
  3. Invokes the model with this message history.
  4. Verifies that the result is of the expected type (AIMessage or AIMessageChunk).

  This test is crucial for ensuring that the model can learn from and apply the patterns demonstrated in few-shot examples, particularly when those examples involve tool usage.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testToolMessageHistoriesListContent

• testToolMessageHistoriesListContent(callOptions?): Promise<void>

• Tests the chat model's ability to handle message histories with list tool contents. This test is specifically designed for models that support tool calling with list-based content, such as Anthropic's Claude.

  The test performs the following steps:

  1. Creates a chat model and binds an AdderTool to it.
  2. Constructs a message history that includes a HumanMessage, an AIMessage with list content (simulating a tool call), and a ToolMessage with the tool's response.
  3. Invokes the model with this message history.
  4. Verifies that the result is of the expected type (AIMessage or AIMessageChunk).

  This test ensures that the model can correctly process and respond to complex message histories that include tool calls with list-based content structures.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model.

  Returns Promise<void>

testToolMessageHistoriesStringContent

• testToolMessageHistoriesStringContent(callOptions?): Promise<void>

• Tests the chat model's ability to handle message histories with string tool contents. This test is specifically designed for models that support tool calling with string-based content, such as OpenAI's GPT models.

  The test performs the following steps:

  1. Creates a chat model and binds an AdderTool to it.
  2. Constructs a message history that includes a HumanMessage, an AIMessage with string content (simulating a tool call), and a ToolMessage with the tool's response.
  3. Invokes the model with this message history.
  4. Verifies that the result is of the expected type (AIMessage or AIMessageChunk).

  This test ensures that the model can correctly process and respond to complex message histories that include tool calls with string-based content structures.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testUsageMetadata

• testUsageMetadata(callOptions?): Promise<void>

• Tests the usage metadata functionality of the chat model. This test ensures that the model returns proper usage metadata after invoking it with a simple message.

  It verifies that:

  1. The result is defined and is an instance of the correct response type.
  2. The result contains the usage_metadata field.
  3. The usage_metadata field contains input_tokens, output_tokens, and total_tokens, all of which are numbers.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testUsageMetadataStreaming

• testUsageMetadataStreaming(callOptions?): Promise<void>

• Tests the usage metadata functionality for streaming responses from the chat model. This test ensures that the model returns proper usage metadata after streaming a response for a simple message.

  It verifies that:

  1. Each streamed chunk is defined and is an instance of AIMessageChunk.
  2. The final concatenated result contains the usage_metadata field.
  3. The usage_metadata field contains input_tokens, output_tokens, and total_tokens, all of which are numbers.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testWithStructuredOutput

• testWithStructuredOutput(callOptions?): Promise<void>

• Tests the chat model's ability to generate structured output using the withStructuredOutput method. This test ensures that the model can correctly process a prompt and return a response that adheres to a predefined schema (adderSchema).

  It verifies that:

  1. The model supports structured output functionality.
  2. The result contains the expected fields ('a' and 'b') from the adderSchema.
  3. The values of these fields are of the correct type (number).

  This test is crucial for ensuring that the model can generate responses in a specific format, which is useful for tasks requiring structured data output.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>

testWithStructuredOutputIncludeRaw

• testWithStructuredOutputIncludeRaw(callOptions?): Promise<void>

• Tests the chat model's ability to generate structured output with raw response included. This test ensures that the model can correctly process a prompt and return a response that adheres to a predefined schema (adderSchema) while also including the raw model output.

  It verifies that:

  1. The model supports structured output functionality with raw response inclusion.
  2. The result contains both 'raw' and 'parsed' properties.
  3. The 'raw' property is an instance of the expected response type.
  4. The 'parsed' property contains the expected fields ('a' and 'b') from the adderSchema.
  5. The values of these fields in the 'parsed' property are of the correct type (number).

  This test is crucial for ensuring that the model can generate responses in a specific format while also providing access to the original, unprocessed model output.

  Parameters

  • OptionalcallOptions: any

    Optional call options to pass to the model. These options will be applied to the model at runtime.

  Returns Promise<void>