Combines the P0 event-id fix with the three transport fixes that rewrite the same stream loop; they are not separable in these two files. fix(astrbot): scope stream event ids per request (P0-1 / F-01) `AstrBotStreamMapper` restarts its sequence at zero for every stream, but `seen_event_ids` lives on the client and outlives a request. Event ids were built from `server_instance_id + sequence` alone, so the second message on a connection replayed the first message's ids and every one of its events was discarded as a duplicate: the assistant's reply never reached the UI or the database. Ids now carry a per-request `stream_id`. fix(protocol): serialize model options in camelCase (P1-7b / F-07) `AstrBotModelOption` had no `rename_all`, so the frontend's `isDefault` was always undefined and no model was preselected. fix(net): stop total timeouts from killing streams (P1-1 / F-10) `Client::timeout` runs from the start of the request until the body is finished and keeps counting while `bytes_stream()` is consumed. The 30s desktop-v1 budget therefore tore the long-lived SSE connection down every 30 seconds no matter how healthy it was — heartbeats could not keep it alive — and the 120s AstrBot budget cut off any generation longer than two minutes. Streaming now uses a separate client with no total deadline, detecting a stalled peer with `read_timeout` (which resets on every successful read) plus a TCP keepalive. REST endpoints keep their total deadline. fix(astrbot): decode sse bytes on line boundaries (P1-3 / F-11) Decoding each network chunk with `String::from_utf8_lossy` corrupted any character straddling a chunk boundary. For CJK text that is one replacement character per unlucky chunk, persisted into the message. The stream is now split into lines at the byte level and only complete lines are decoded. fix(astrbot): finalize streams on cancel and error (P2-2 / F-13) Cancellation and every error path returned without a `MessageCompleted` event, leaving the row `streaming = 1` so it rendered as "generating" again after a restart. All exit paths now finalize with `Cancelled` or `Error` and keep the text received so far. Cancelling also no longer flips the connection to Disconnected and back, which was a visible flicker (P3-4 / F-56). Hardening found while reviewing the above: a 64 MiB per-response ceiling (removing the total timeout removed the only bound on a response, and `read_timeout` cannot replace it), and a cursor-based line buffer instead of draining from the front, which was quadratic in the chunk size. Tests drive a real `AstrBotClient` against a hand-rolled HTTP/1.1 chunked-SSE server on a raw `TcpListener`: byte-exact control over chunk boundaries and dropping a socket mid-body without the terminal chunk are not expressible through an HTTP framework, and both are needed here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
334 行
11 KiB
Rust
334 行
11 KiB
Rust
use desktop_protocol::{ActiveSlotKind, ConnectionState, RawProtocolEnvelope};
|
|
use secrecy::{ExposeSecret, SecretString};
|
|
use std::{
|
|
collections::{HashSet, VecDeque},
|
|
sync::Arc,
|
|
time::Duration,
|
|
};
|
|
|
|
use eventsource_stream::Eventsource;
|
|
use futures_util::StreamExt;
|
|
use reqwest::header::{ACCEPT, HeaderValue};
|
|
use serde::Serialize;
|
|
use serde_json::Value;
|
|
use thiserror::Error;
|
|
use tokio::sync::{Mutex, mpsc, watch};
|
|
use tokio_util::sync::CancellationToken;
|
|
|
|
use super::SlotKind;
|
|
|
|
#[derive(Debug, Clone)]
|
|
pub struct GatewayClientConfig {
|
|
pub profile_id: String,
|
|
pub slot: SlotKind,
|
|
pub base_url: String,
|
|
pub expected_server_instance_id: String,
|
|
pub bearer_token: Option<SecretString>,
|
|
}
|
|
|
|
#[derive(Debug, Clone, Serialize)]
|
|
#[serde(rename_all = "camelCase")]
|
|
pub struct BoundGatewayEvent {
|
|
pub profile_id: String,
|
|
pub slot: SlotKind,
|
|
pub event_id: String,
|
|
pub event_type: String,
|
|
pub server_instance_id: String,
|
|
pub session_id: Option<String>,
|
|
pub message_id: Option<String>,
|
|
pub payload: Value,
|
|
}
|
|
|
|
#[derive(Debug, Error)]
|
|
pub enum GatewayError {
|
|
#[error("gateway request failed")]
|
|
Request(#[from] reqwest::Error),
|
|
#[error("gateway returned HTTP {0}")]
|
|
HttpStatus(reqwest::StatusCode),
|
|
#[error("invalid SSE event")]
|
|
Event(#[from] eventsource_stream::EventStreamError<reqwest::Error>),
|
|
#[error("invalid gateway event payload")]
|
|
Json(#[from] serde_json::Error),
|
|
#[error("gateway protocol validation failed")]
|
|
Protocol(#[from] desktop_protocol::ProtocolError),
|
|
#[error("server identity changed")]
|
|
ServerIdentityChanged,
|
|
#[error("gateway event channel closed")]
|
|
ChannelClosed,
|
|
#[error("gateway adapter error: {0}")]
|
|
Adapter(String),
|
|
}
|
|
|
|
const MAX_DEDUPLICATED_EVENTS: usize = 4_096;
|
|
|
|
#[derive(Default)]
|
|
struct EventDeduplicator {
|
|
ids: HashSet<String>,
|
|
order: VecDeque<String>,
|
|
}
|
|
|
|
impl EventDeduplicator {
|
|
fn insert(&mut self, event_id: String) -> bool {
|
|
if !self.ids.insert(event_id.clone()) {
|
|
return false;
|
|
}
|
|
|
|
self.order.push_back(event_id);
|
|
if self.order.len() > MAX_DEDUPLICATED_EVENTS
|
|
&& let Some(expired) = self.order.pop_front()
|
|
{
|
|
self.ids.remove(&expired);
|
|
}
|
|
true
|
|
}
|
|
}
|
|
|
|
/// Total budget for `/desktop/v1/health` and any other short exchange.
|
|
const REST_TOTAL_TIMEOUT: Duration = Duration::from_secs(30);
|
|
|
|
/// How long the event stream may stay silent before it is torn down and resumed.
|
|
///
|
|
/// A total `Client::timeout` keeps counting while `bytes_stream()` is consumed,
|
|
/// so a 30s total deadline meant the long-lived desktop-v1 connection was
|
|
/// killed and reconnected every 30 seconds no matter how healthy it was, and
|
|
/// heartbeats could not keep it alive. `read_timeout` resets on every received
|
|
/// byte, so heartbeats now do exactly that. Reconnects still resume from
|
|
/// `Last-Event-ID`, so an idle teardown is lossless.
|
|
const STREAM_IDLE_TIMEOUT: Duration = Duration::from_secs(90);
|
|
|
|
const CONNECT_TIMEOUT: Duration = Duration::from_secs(5);
|
|
|
|
#[derive(Clone)]
|
|
pub struct GatewayClient {
|
|
config: GatewayClientConfig,
|
|
/// Short request/response endpoints; keeps a total deadline.
|
|
http: reqwest::Client,
|
|
/// `/desktop/v1/events/stream` only; no total deadline.
|
|
stream_http: reqwest::Client,
|
|
seen_event_ids: Arc<Mutex<EventDeduplicator>>,
|
|
last_event_id: Arc<Mutex<Option<String>>>,
|
|
state_tx: watch::Sender<ConnectionState>,
|
|
}
|
|
|
|
impl GatewayClient {
|
|
pub fn new(config: GatewayClientConfig) -> Result<Self, GatewayError> {
|
|
let (state_tx, _) = watch::channel(ConnectionState::Disconnected);
|
|
let http = reqwest::Client::builder()
|
|
.redirect(reqwest::redirect::Policy::none())
|
|
.connect_timeout(CONNECT_TIMEOUT)
|
|
.timeout(REST_TOTAL_TIMEOUT)
|
|
.build()?;
|
|
|
|
let stream_http = reqwest::Client::builder()
|
|
.redirect(reqwest::redirect::Policy::none())
|
|
.connect_timeout(CONNECT_TIMEOUT)
|
|
.read_timeout(STREAM_IDLE_TIMEOUT)
|
|
.tcp_keepalive(Duration::from_secs(60))
|
|
.build()?;
|
|
|
|
Ok(Self {
|
|
config,
|
|
http,
|
|
stream_http,
|
|
seen_event_ids: Arc::new(Mutex::new(EventDeduplicator::default())),
|
|
last_event_id: Arc::new(Mutex::new(None)),
|
|
state_tx,
|
|
})
|
|
}
|
|
|
|
pub fn state_receiver(&self) -> watch::Receiver<ConnectionState> {
|
|
self.state_tx.subscribe()
|
|
}
|
|
|
|
fn set_state(&self, state: ConnectionState) {
|
|
self.state_tx.send_replace(state);
|
|
}
|
|
|
|
pub async fn health(&self) -> Result<Value, GatewayError> {
|
|
let response = self
|
|
.http
|
|
.get(format!("{}/desktop/v1/health", self.config.base_url))
|
|
.send()
|
|
.await?;
|
|
|
|
if !response.status().is_success() {
|
|
return Err(GatewayError::HttpStatus(response.status()));
|
|
}
|
|
|
|
Ok(response.json().await?)
|
|
}
|
|
|
|
pub async fn stream_once(
|
|
&self,
|
|
last_event_id: Option<&str>,
|
|
sender: &mpsc::Sender<BoundGatewayEvent>,
|
|
cancellation: &CancellationToken,
|
|
) -> Result<(), GatewayError> {
|
|
self.set_state(ConnectionState::Connecting);
|
|
let mut request = self
|
|
.stream_http
|
|
.get(format!("{}/desktop/v1/events/stream", self.config.base_url))
|
|
.header(ACCEPT, HeaderValue::from_static("text/event-stream"));
|
|
if let Some(token) = &self.config.bearer_token {
|
|
request = request.bearer_auth(token.expose_secret());
|
|
}
|
|
|
|
if let Some(event_id) = last_event_id {
|
|
request = request.header("Last-Event-ID", event_id);
|
|
}
|
|
|
|
let response = request.send().await?;
|
|
if !response.status().is_success() {
|
|
return Err(GatewayError::HttpStatus(response.status()));
|
|
}
|
|
self.set_state(ConnectionState::Connected);
|
|
|
|
let mut stream = response.bytes_stream().eventsource();
|
|
|
|
loop {
|
|
tokio::select! {
|
|
_ = cancellation.cancelled() => {
|
|
self.set_state(ConnectionState::Disconnected);
|
|
return Ok(());
|
|
},
|
|
next = stream.next() => {
|
|
let Some(next) = next else {
|
|
return Ok(());
|
|
};
|
|
let event = next?;
|
|
if event.data.is_empty() {
|
|
continue;
|
|
}
|
|
|
|
let raw: RawProtocolEnvelope = serde_json::from_str(&event.data)?;
|
|
let payload = raw.payload.clone();
|
|
let bound = desktop_protocol::BoundGatewayEvent::bind(
|
|
raw,
|
|
self.config.profile_id.clone(),
|
|
protocol_slot(self.config.slot),
|
|
&self.config.expected_server_instance_id,
|
|
)?;
|
|
let event_id = bound.event.metadata.event_id.clone();
|
|
let event_type = bound.event.event_type().to_owned();
|
|
let server_instance_id = bound.event.metadata.server_instance_id.clone();
|
|
|
|
let mut seen = self.seen_event_ids.lock().await;
|
|
if !seen.insert(event_id.clone()) {
|
|
continue;
|
|
}
|
|
drop(seen);
|
|
|
|
sender
|
|
.send(BoundGatewayEvent {
|
|
profile_id: bound.profile_id,
|
|
slot: self.config.slot,
|
|
event_id: event_id.clone(),
|
|
event_type,
|
|
server_instance_id,
|
|
session_id: bound.event.metadata.session_id.clone(),
|
|
message_id: bound.event.metadata.message_id.clone(),
|
|
payload,
|
|
})
|
|
.await
|
|
.map_err(|_| GatewayError::ChannelClosed)?;
|
|
*self.last_event_id.lock().await = Some(event_id);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
pub async fn run_reconnecting(
|
|
&self,
|
|
sender: &mpsc::Sender<BoundGatewayEvent>,
|
|
cancellation: &CancellationToken,
|
|
) -> Result<(), GatewayError> {
|
|
let mut retry_delay = Duration::from_millis(100);
|
|
|
|
loop {
|
|
if cancellation.is_cancelled() {
|
|
self.set_state(ConnectionState::Disconnected);
|
|
return Ok(());
|
|
}
|
|
|
|
let last_event_id = self.last_event_id.lock().await.clone();
|
|
let result = self
|
|
.stream_once(last_event_id.as_deref(), sender, cancellation)
|
|
.await;
|
|
|
|
if cancellation.is_cancelled() {
|
|
self.set_state(ConnectionState::Disconnected);
|
|
return Ok(());
|
|
}
|
|
|
|
match result {
|
|
Ok(()) => retry_delay = Duration::from_millis(100),
|
|
Err(error) => {
|
|
if let Some(state) = terminal_state(&error) {
|
|
self.set_state(state);
|
|
return Err(error);
|
|
}
|
|
tracing::warn!(
|
|
profile_id = %self.config.profile_id,
|
|
error = %error,
|
|
"gateway SSE connection will retry"
|
|
);
|
|
}
|
|
}
|
|
|
|
self.set_state(ConnectionState::Reconnecting);
|
|
tokio::select! {
|
|
_ = cancellation.cancelled() => {
|
|
self.set_state(ConnectionState::Disconnected);
|
|
return Ok(());
|
|
}
|
|
_ = tokio::time::sleep(retry_delay) => {}
|
|
}
|
|
retry_delay = std::cmp::min(retry_delay.saturating_mul(2), Duration::from_secs(5));
|
|
}
|
|
}
|
|
}
|
|
|
|
fn protocol_slot(slot: SlotKind) -> ActiveSlotKind {
|
|
match slot {
|
|
SlotKind::Companion => ActiveSlotKind::Companion,
|
|
SlotKind::Operations => ActiveSlotKind::Operations,
|
|
}
|
|
}
|
|
|
|
fn terminal_state(error: &GatewayError) -> Option<ConnectionState> {
|
|
match error {
|
|
GatewayError::HttpStatus(status)
|
|
if matches!(
|
|
*status,
|
|
reqwest::StatusCode::UNAUTHORIZED | reqwest::StatusCode::FORBIDDEN
|
|
) =>
|
|
{
|
|
Some(ConnectionState::AuthExpired)
|
|
}
|
|
GatewayError::Protocol(_) | GatewayError::ServerIdentityChanged => {
|
|
Some(ConnectionState::Incompatible)
|
|
}
|
|
GatewayError::ChannelClosed => Some(ConnectionState::FatalError),
|
|
_ => None,
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
#[tokio::test]
|
|
async fn duplicate_cache_is_scoped_to_one_client() {
|
|
let client = GatewayClient::new(GatewayClientConfig {
|
|
bearer_token: None,
|
|
profile_id: "profile-a".into(),
|
|
slot: SlotKind::Companion,
|
|
base_url: "http://127.0.0.1:1".into(),
|
|
expected_server_instance_id: "mock-a".into(),
|
|
})
|
|
.expect("client");
|
|
|
|
assert!(client.seen_event_ids.lock().await.insert("same-id".into()));
|
|
assert!(!client.seen_event_ids.lock().await.insert("same-id".into()));
|
|
}
|
|
}
|