Struct LlmBroker 

pub struct LlmBroker { /* private fields */ }

Main interface for LLM interactions

Implementations

impl LlmBroker

pub fn new( model: impl Into<String>, gateway: Arc<dyn LlmGateway>, tracer: Option<Arc<TracerSystem>>, ) -> Self

Create a new LLM broker

Arguments

• model - The name of the LLM model to use
• gateway - The gateway to use for LLM communication
• tracer - Optional tracer system for observability

pub async fn generate( &self, messages: &[LlmMessage], tools: Option<&[Box<dyn LlmTool>]>, config: Option<CompletionConfig>, correlation_id: Option<String>, ) -> Result<String>

Generate text response from LLM

Arguments

• messages - The messages to send to the LLM
• tools - Optional tools available to the LLM
• config - Optional completion configuration
• correlation_id - Optional correlation ID for tracing (generates UUID if None)

pub async fn generate_object<T>( &self, messages: &[LlmMessage], config: Option<CompletionConfig>, correlation_id: Option<String>, ) -> Result<T>

where T: for<'de> Deserialize<'de> + Serialize + JsonSchema + Send,

Generate structured object response from LLM

Arguments

• messages - The messages to send to the LLM
• config - Optional completion configuration
• correlation_id - Optional correlation ID for tracing (generates UUID if None)

pub fn generate_stream<'a>( &'a self, messages: &'a [LlmMessage], tools: Option<&'a [Box<dyn LlmTool>]>, config: Option<CompletionConfig>, correlation_id: Option<String>, ) -> Pin<Box<dyn Stream<Item = Result<String>> + 'a>>

Generate streaming text response from LLM

Returns a stream that yields content chunks as they arrive. When tool calls are detected, the broker executes them and recursively streams the LLM’s follow-up response.

Arguments

• messages - The messages to send to the LLM
• tools - Optional tools available to the LLM
• config - Optional completion configuration
• correlation_id - Optional correlation ID for tracing (generates UUID if None)

Example

use futures::stream::StreamExt;

let broker = LlmBroker::new("qwen3:32b", gateway, None);
let messages = vec![LlmMessage::user("Tell me a story")];

let mut stream = broker.generate_stream(&messages, None, None, None);
while let Some(result) = stream.next().await {
    match result {
        Ok(chunk) => print!("{}", chunk),
        Err(e) => eprintln!("Error: {}", e),
    }
}

Trait Implementations

impl Clone for LlmBroker

fn clone(&self) -> LlmBroker

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.