"""Material library service.""" from __future__ import annotations import base64 import binascii from dataclasses import replace from io import BytesIO from pathlib import PurePosixPath from typing import Any, Callable from uuid import uuid4 try: from PIL import Image except ImportError: # pragma: no cover - AstrBot ships Pillow; standalone users may not. Image = None # type: ignore[assignment] from backend.dto.common import ApiError, ErrorCode from backend.dto.material_dto import Material, MaterialCategory, MaterialListResult from backend.services.material_normalization_service import ( MaterialNormalizationError, MaterialNormalizationService, validate_material_structures, ) from core.audit_service import AuditService from core.image_formats import ( IMAGE_EXTENSIONS_BY_MIME, SUPPORTED_IMAGE_EXTENSIONS, detect_image_format, ) from core.identity import RequestContext from core.storage import VfsStorage from database.repositories.material_repository import MaterialRepository from engines.vision_provider import ( VisionProvider, VisionProviderError, VisionRecognitionRequest, VisionRecognitionResult, ) MATERIAL_STATUSES = {"pending_upload", "pending_recognition", "pending_confirm", "confirmed", "archived"} MAX_UPLOAD_BYTES = 20 * 1024 * 1024 VISION_INPUT_MAX_EDGE = 1536 VISION_INPUT_REENCODE_THRESHOLD = 1024 * 1024 VISION_INPUT_JPEG_QUALITY = 88 UPLOAD_SIZE_FIELDS = ("size_bytes", "content_length", "file_size", "size") UPLOAD_ALLOWED_EXTENSIONS = set(SUPPORTED_IMAGE_EXTENSIONS) UPLOAD_ALLOWED_CONTENT_TYPES = { mime_type: set(extensions) for mime_type, extensions in IMAGE_EXTENSIONS_BY_MIME.items() } class MaterialService: def __init__( self, repository: MaterialRepository, audit_service: AuditService, *, vision_provider_factory: Callable[[dict[str, Any]], VisionProvider | None] | None = None, default_recognition_route: str = "material_vlm", recognition_prompt_config: dict[str, Any] | None = None, normalization_service: MaterialNormalizationService | None = None, storage: VfsStorage | None = None, ): self.repository = repository self.audit_service = audit_service self.vision_provider_factory = vision_provider_factory self.default_recognition_route = default_recognition_route self.recognition_prompt_config = recognition_prompt_config or {} self.normalization_service = normalization_service or MaterialNormalizationService() self.storage = storage or VfsStorage() def create_category(self, payload: dict[str, Any], ctx: RequestContext) -> MaterialCategory: name = str(payload.get("name") or "").strip() if not name: raise ApiError(ErrorCode.VALIDATION_FAILED, "category name is required", {"field": "name"}) parent_id = payload.get("parent_id") if parent_id and self.repository.get_category(parent_id) is None: raise ApiError(ErrorCode.RESOURCE_NOT_FOUND, "parent category not found", {"parent_id": parent_id}) category = self.repository.create_category( { "category_id": payload.get("category_id") or new_category_id(), "name": name, "parent_id": parent_id, "category_type": payload.get("category_type", "material"), "sort_order": payload.get("sort_order", 0), "metadata": payload.get("metadata") or {}, } ) self.audit_service.record( ctx=ctx, action="create_material_category", target_type="material_category", target_id=category.category_id, after_snapshot=category.to_dict(), ) return category def list_categories(self) -> list[MaterialCategory]: return self.repository.list_categories() def update_category(self, category_id: str, payload: dict[str, Any], ctx: RequestContext) -> MaterialCategory: before = self.get_category(category_id) if "name" in payload and not str(payload["name"]).strip(): raise ApiError(ErrorCode.VALIDATION_FAILED, "category name is required", {"field": "name"}) parent_id = payload.get("parent_id") if "parent_id" in payload else None if parent_id: if parent_id == category_id: raise ApiError(ErrorCode.VALIDATION_FAILED, "category cannot be its own parent", {"field": "parent_id"}) if self.repository.get_category(parent_id) is None: raise ApiError(ErrorCode.RESOURCE_NOT_FOUND, "parent category not found", {"parent_id": parent_id}) after = self.repository.update_category(category_id, payload) if after is None: raise ApiError(ErrorCode.RESOURCE_NOT_FOUND, "category not found", {"category_id": category_id}) self.audit_service.record( ctx=ctx, action="update_material_category", target_type="material_category", target_id=category_id, before_snapshot=before.to_dict(), after_snapshot=after.to_dict(), ) return after def delete_category(self, category_id: str, ctx: RequestContext) -> MaterialCategory: before = self.repository.delete_category(category_id) if before is None: raise ApiError(ErrorCode.RESOURCE_NOT_FOUND, "category not found", {"category_id": category_id}) self.audit_service.record( ctx=ctx, action="delete_material_category", target_type="material_category", target_id=category_id, before_snapshot=before.to_dict(), ) return before def get_category(self, category_id: str) -> MaterialCategory: category = self.repository.get_category(category_id) if category is None: raise ApiError(ErrorCode.RESOURCE_NOT_FOUND, "category not found", {"category_id": category_id}) return category def create_material(self, payload: dict[str, Any], ctx: RequestContext) -> Material: self.validate_material_payload(payload, creating=True) display_name = str(payload.get("display_name") or "").strip() status = payload.get("recognition_status", "pending_upload") material = self.repository.create_material( { "material_id": payload.get("material_id") or new_material_id(), "category_id": payload.get("category_id"), "display_name": display_name, "image_uri": payload.get("image_uri"), "thumbnail_uri": payload.get("thumbnail_uri"), "prompt": payload.get("prompt"), "negative_prompt": payload.get("negative_prompt"), "tags": payload.get("tags") or [], "recognition_status": status, "metadata": payload.get("metadata") or {}, } ) self.audit_service.record( ctx=ctx, action="create_material", target_type="material", target_id=material.material_id, after_snapshot=material.to_dict(), ) return material def list_materials( self, *, category_id: str | None = None, keyword: str | None = None, page: int = 1, page_size: int = 30, color: str | None = None, material_type: str | None = None, body_slot: str | None = None, wear_layer: str | None = None, min_confidence: float | None = None, confirmation_state: str | None = None, ) -> MaterialListResult: normalized_page = max(page, 1) normalized_page_size = min(max(page_size, 1), 100) advanced_filters = any(value not in (None, "") for value in (color, material_type, body_slot, wear_layer, min_confidence, confirmation_state)) if advanced_filters: all_items: list[Material] = [] repository_page = 1 while True: batch, total = self.repository.list_materials( category_id=category_id, keyword=keyword, page=repository_page, page_size=100, ) all_items.extend(self._with_material_structures(item) for item in batch) if len(all_items) >= total or not batch: break repository_page += 1 filtered = [ item for item in all_items if _matches_material_filters( item, color=color, material_type=material_type, body_slot=body_slot, wear_layer=wear_layer, min_confidence=min_confidence, confirmation_state=confirmation_state, ) ] start = (normalized_page - 1) * normalized_page_size return MaterialListResult(items=filtered[start : start + normalized_page_size], total=len(filtered), page=normalized_page, page_size=normalized_page_size) items, total = self.repository.list_materials( category_id=category_id, keyword=keyword, page=normalized_page, page_size=normalized_page_size, ) return MaterialListResult( items=[self._with_material_structures(item) for item in items], total=total, page=normalized_page, page_size=normalized_page_size, ) def get_material(self, material_id: str) -> Material: material = self.repository.get_material(material_id) if material is None or material.recognition_status == "archived": raise ApiError(ErrorCode.RESOURCE_NOT_FOUND, "material not found", {"material_id": material_id}) return self._with_material_structures(material) def update_material(self, material_id: str, payload: dict[str, Any], ctx: RequestContext) -> Material: before = self.get_material(material_id) self.validate_material_payload(payload, creating=False) update_payload = dict(payload) structured_keys = {"raw_analysis", "normalized_asset_profile", "generation_capsules"} if structured_keys & update_payload.keys(): metadata = {**before.metadata} for key in structured_keys: if key in update_payload: metadata[key] = update_payload.pop(key) validate_material_structures(metadata) update_payload["metadata"] = metadata if "display_name" in update_payload: update_payload["display_name"] = str(update_payload["display_name"]).strip() after = self.repository.update_material(material_id, update_payload) if after is None: raise ApiError(ErrorCode.RESOURCE_NOT_FOUND, "material not found", {"material_id": material_id}) self.audit_service.record( ctx=ctx, action="update_material", target_type="material", target_id=material_id, before_snapshot=before.to_dict(), after_snapshot=after.to_dict(), ) return after def validate_material_payload(self, payload: dict[str, Any], *, creating: bool) -> None: if creating or "display_name" in payload: if not str(payload.get("display_name") or "").strip(): raise ApiError(ErrorCode.VALIDATION_FAILED, "display_name is required", {"field": "display_name"}) if creating or "category_id" in payload: self._validate_category(payload.get("category_id")) if creating or "image_uri" in payload: self._validate_storage_uri(payload.get("image_uri"), "image_uri") if creating or "thumbnail_uri" in payload: self._validate_storage_uri(payload.get("thumbnail_uri"), "thumbnail_uri") if creating or "recognition_status" in payload: self._validate_status(payload.get("recognition_status", "pending_upload")) metadata = payload.get("metadata") if isinstance(metadata, dict) and any(key in metadata for key in ("raw_analysis", "normalized_asset_profile", "generation_capsules")): try: validate_material_structures(metadata) except MaterialNormalizationError as exc: raise ApiError(ErrorCode.VALIDATION_FAILED, str(exc), {"field": "metadata"}) from None def delete_material(self, material_id: str, ctx: RequestContext) -> Material: return self.update_material(material_id, {"recognition_status": "archived"}, ctx) def upload_material(self, payload: dict[str, Any], ctx: RequestContext) -> Material: filename = _normalize_upload_filename(payload.get("filename")) content_type = _normalize_upload_content_type(payload.get("content_type")) declared_size = _normalize_upload_size(payload) upload_metadata = _normalize_upload_metadata(payload.get("metadata")) image_data_url = str(payload.get("image_data_url") or "").strip() image_uri = str(payload.get("image_uri") or "").strip() persisted_path = None if image_data_url: content, detected_content_type, detected_extension = _decode_upload_data_url(image_data_url) if len(content) > MAX_UPLOAD_BYTES: raise ApiError( ErrorCode.UPLOAD_TOO_LARGE, "upload exceeds maximum allowed size", {"size_bytes": len(content), "max_size_bytes": MAX_UPLOAD_BYTES}, ) if declared_size is not None and declared_size != len(content): raise ApiError( ErrorCode.VALIDATION_FAILED, "declared upload size does not match image bytes", {"declared_size_bytes": declared_size, "actual_size_bytes": len(content)}, ) if content_type and content_type != detected_content_type: raise ApiError( ErrorCode.UNSUPPORTED_FILE_TYPE, "upload content type does not match image bytes", {"content_type": content_type, "detected_content_type": detected_content_type}, ) filename_extension = _suffix(filename) if filename_extension and filename_extension not in UPLOAD_ALLOWED_CONTENT_TYPES[detected_content_type]: raise ApiError( ErrorCode.UNSUPPORTED_FILE_TYPE, "upload filename extension does not match image bytes", {"filename": filename, "detected_content_type": detected_content_type}, ) content_type = detected_content_type declared_size = len(content) safe_filename = filename or f"material-{uuid4().hex}{detected_extension}" stored_filename = f"{uuid4().hex}-{PurePosixPath(safe_filename).stem}{detected_extension}" image_uri = f"storage://materials/uploads/{stored_filename}" persisted_path = self.storage.resolve_uri(image_uri) try: persisted_path.parent.mkdir(parents=True, exist_ok=True) temporary_path = persisted_path.with_suffix(f"{persisted_path.suffix}.part") temporary_path.write_bytes(content) temporary_path.replace(persisted_path) except OSError as exc: raise ApiError( ErrorCode.STORAGE_WRITE_FAILED, "material image could not be stored", {"uri": image_uri, "exception_type": exc.__class__.__name__}, ) from None upload_metadata["browser_upload"] = { "source": str(payload.get("upload_source") or "file"), "persisted": True, } filename = safe_filename if not image_uri: raise ApiError( ErrorCode.VALIDATION_FAILED, "image_uri or image_data_url is required", {"fields": ["image_uri", "image_data_url"]}, ) recognition_route = _optional_text(payload.get("recognition_model_route")) if recognition_route: upload_metadata["recognition_model_route"] = recognition_route extension = _validate_upload_file_type(filename=filename, image_uri=image_uri, content_type=content_type) display_name = str(payload.get("display_name") or _display_name_from_filename(filename) or "Uploaded material").strip() thumbnail_uri = str(payload.get("thumbnail_uri") or image_uri).strip() try: return self.create_material( { "category_id": payload.get("category_id"), "display_name": display_name, "image_uri": image_uri, "thumbnail_uri": thumbnail_uri, "recognition_status": "pending_recognition", "metadata": { **upload_metadata, "upload": { "filename": filename, "content_type": content_type, "extension": extension, "size_bytes": declared_size, "max_size_bytes": MAX_UPLOAD_BYTES, "storage_reference_only": not bool(image_data_url), } }, }, ctx, ) except Exception: if persisted_path is not None: try: persisted_path.unlink(missing_ok=True) except OSError: pass raise def recognize_material(self, material_id: str, ctx: RequestContext) -> Material: material = self.get_material(material_id) if material.recognition_status == "confirmed": return material return self._recognize_material(material, ctx) def re_recognize_material(self, material_id: str, ctx: RequestContext) -> Material: material = self.get_material(material_id) return self._recognize_material(material, ctx) def renormalize_material(self, material_id: str, ctx: RequestContext) -> Material: material = self.get_material(material_id) raw = material.metadata.get("raw_analysis") if isinstance(material.metadata.get("raw_analysis"), dict) else {} source_payload = raw.get("source_payload") if isinstance(raw.get("source_payload"), dict) else {"legacy": True} recognition = material.metadata.get("recognition") if isinstance(material.metadata.get("recognition"), dict) else {} structures = self.normalization_service.normalize_recognition( source_payload, prompt=material.prompt or str(raw.get("description") or "material reference"), negative_prompt=material.negative_prompt, tags=material.tags, confidence=recognition.get("confidence"), provider_id=str(recognition.get("provider_id") or "stored_analysis"), model=str(recognition.get("model") or "stored_analysis"), route_name=str(recognition.get("route_name") or "stored_analysis"), category_hint=self._material_category_hint(material), previous_metadata=material.metadata, ) after = self.update_material(material_id, {"metadata": {**material.metadata, **structures}}, ctx) self.audit_service.record( ctx=ctx, action="renormalize_material", target_type="material", target_id=material_id, before_snapshot=material.to_dict(), after_snapshot=after.to_dict(), metadata={"vision_provider_called": False}, ) return after def renormalize_batch(self, payload: dict[str, Any], ctx: RequestContext) -> dict[str, Any]: requested = [str(item).strip() for item in payload.get("material_ids", []) if str(item).strip()] if isinstance(payload.get("material_ids"), list) else [] if not requested: items: list[Material] = [] page = 1 while True: batch, total = self.repository.list_materials(page=page, page_size=100) items.extend(batch) if len(items) >= total or not batch: break page += 1 requested = [item.material_id for item in items if item.recognition_status in {"pending_confirm", "confirmed"}] limit = min(max(int(payload.get("limit") or 500), 1), 5000) succeeded: list[str] = [] failed: list[dict[str, str]] = [] for material_id in requested[:limit]: try: self.renormalize_material(material_id, ctx) succeeded.append(material_id) except Exception as exc: failed.append({"material_id": material_id, "error_type": exc.__class__.__name__}) return {"requested": min(len(requested), limit), "succeeded": succeeded, "failed": failed, "vision_provider_called": False} def _recognize_material(self, material: Material, ctx: RequestContext) -> Material: route_name = _recognition_route(material.metadata, self.default_recognition_route) provider = self._vision_provider(material, route_name) if provider is None: raise ApiError( ErrorCode.MODEL_CAPABILITY_UNCONFIRMED, "material vision route is not configured or its credentials are unavailable", { "material_id": material.material_id, "route_name": route_name, "modality": "vision_recognition", }, ) request = _vision_request(material, self.storage, self._recognition_instruction(material)) try: recognition = provider.recognize_material(request) except VisionProviderError as exc: failure = _vision_provider_error(exc) self._record_recognition_failure(material, route_name, failure, ctx) raise _vision_api_error(material.material_id, route_name, failure) from None except Exception as exc: failure = _unexpected_vision_provider_error(provider, exc) self._record_recognition_failure(material, route_name, failure, ctx) raise _vision_api_error(material.material_id, route_name, failure) from None generated_tags = recognition.tags metadata = { key: value for key, value in material.metadata.items() if key not in {"confirmed", "recognition", "recognition_failure"} } category_hint = self._material_category_hint(material) try: structures = self.normalization_service.normalize_recognition( recognition.structured_data, prompt=recognition.prompt, negative_prompt=recognition.negative_prompt, tags=generated_tags, confidence=recognition.confidence, provider_id=recognition.provider_id, model=recognition.model, route_name=route_name, category_hint=category_hint, previous_metadata=material.metadata, ) except MaterialNormalizationError as exc: failure = { "source": "material_normalization", "provider_id": recognition.provider_id, "status_code": None, "error_type": "invalid_structured_analysis", "reason": str(exc), } self._record_recognition_failure(material, route_name, failure, ctx) raise _vision_api_error(material.material_id, route_name, failure) from None after = self.update_material( material.material_id, { "prompt": material.prompt or recognition.prompt, "negative_prompt": material.negative_prompt or recognition.negative_prompt, "tags": sorted(set(material.tags + generated_tags)), "recognition_status": "pending_confirm", "metadata": { **metadata, "recognition": _recognition_metadata( recognition, route_name=route_name, ), **structures, }, }, ctx, ) self.audit_service.record( ctx=ctx, action="recognize_material", target_type="material", target_id=material.material_id, before_snapshot=material.to_dict(), after_snapshot=after.to_dict(), ) return after def _recognition_instruction(self, material: Material) -> str: base = str(self.recognition_prompt_config.get("default_instruction") or "").strip() requirements = self.recognition_prompt_config.get("category_requirements") category_name = "" if material.category_id: category = self.repository.get_category(material.category_id) category_name = category.name if category else "" additions: list[str] = [] if isinstance(requirements, list): for item in requirements: if not isinstance(item, dict) or item.get("enabled") is False: continue category = str(item.get("category") or "").strip().lower() requirement = str(item.get("requirement") or "").strip() if requirement and (not category or category in {str(material.category_id or "").lower(), category_name.lower()}): additions.append(requirement) body_slot = str(material.metadata.get("body_slot") or "").strip() if body_slot: additions.append(f"The user marked this material for body slot: {body_slot}. Verify and preserve that slot in the structured result.") upload_note = str(material.metadata.get("note") or "").strip() if upload_note: additions.append(f"Additional user recognition requirement: {upload_note}") return "\n\n".join([part for part in [base, *additions] if part]) def _vision_provider(self, material: Material, route_name: str) -> VisionProvider | None: if self.vision_provider_factory is None: return None return self.vision_provider_factory( { "route_name": route_name, "modality": "vision_recognition", "material_id": material.material_id, "image_uri": material.image_uri, "thumbnail_uri": material.thumbnail_uri, } ) def _record_recognition_failure( self, material: Material, route_name: str, failure: dict[str, Any], ctx: RequestContext, ) -> Material: metadata = { key: value for key, value in material.metadata.items() if key not in {"confirmed", "recognition", "recognition_failure"} } after = self.update_material( material.material_id, { "recognition_status": "pending_recognition", "metadata": { **metadata, "recognition_failure": {**failure, "route_name": route_name}, }, }, ctx, ) self.audit_service.record( ctx=ctx, action="recognize_material_failed", target_type="material", target_id=material.material_id, before_snapshot=material.to_dict(), after_snapshot=after.to_dict(), metadata={"route_name": route_name, "error_type": failure.get("error_type")}, ) return after def confirm_material(self, material_id: str, payload: dict[str, Any], ctx: RequestContext) -> Material: material = self.get_material(material_id) if material.recognition_status not in {"pending_confirm", "confirmed"}: raise ApiError( ErrorCode.VALIDATION_FAILED, "material must have a successful recognition result before confirmation", { "material_id": material_id, "recognition_status": material.recognition_status, "required_status": "pending_confirm", }, ) metadata_patch = payload.get("metadata") if isinstance(payload.get("metadata"), dict) else {} metadata = {**material.metadata, **metadata_patch, "confirmed": True} profile = metadata.get("normalized_asset_profile") if isinstance(profile, dict): profile = {**profile, "confirmed": True} for key in ("category", "role", "body_slots", "wear_layer", "must_preserve", "forbidden_changes", "stackable", "user_locked"): if key in payload: profile[key] = payload[key] metadata["normalized_asset_profile"] = profile update_payload = { "recognition_status": "confirmed", "display_name": payload.get("display_name", material.display_name), "prompt": payload.get("prompt", material.prompt), "negative_prompt": payload.get("negative_prompt", material.negative_prompt), "tags": payload.get("tags", material.tags), "metadata": metadata, } after = self.update_material(material_id, update_payload, ctx) self.audit_service.record( ctx=ctx, action="confirm_material", target_type="material", target_id=material_id, before_snapshot=material.to_dict(), after_snapshot=after.to_dict(), ) return after def _material_category_hint(self, material: Material) -> str | None: if not material.category_id: return None category = self.repository.get_category(material.category_id) return category.name if category else None def _with_material_structures(self, material: Material) -> Material: if material.recognition_status not in {"pending_confirm", "confirmed"}: return material metadata = self.normalization_service.ensure_legacy_structures( prompt=material.prompt, negative_prompt=material.negative_prompt, tags=material.tags, metadata=material.metadata, category_hint=self._material_category_hint(material), ) return material if metadata == material.metadata else replace(material, metadata=metadata) def _validate_category(self, category_id: Any) -> None: if category_id and self.repository.get_category(str(category_id)) is None: raise ApiError(ErrorCode.RESOURCE_NOT_FOUND, "category not found", {"category_id": category_id}) def _validate_status(self, status: Any) -> None: if status not in MATERIAL_STATUSES: raise ApiError( ErrorCode.VALIDATION_FAILED, "unsupported material status", {"field": "recognition_status", "allowed": sorted(MATERIAL_STATUSES)}, ) def _validate_storage_uri(self, uri: Any, field: str) -> None: if uri is None: return if not str(uri).startswith("storage://"): raise ApiError(ErrorCode.VALIDATION_FAILED, f"{field} must use storage://", {"field": field}) def new_category_id() -> str: return f"cat_{uuid4().hex}" def new_material_id() -> str: return f"mat_{uuid4().hex}" def _vision_request(material: Material, storage: VfsStorage, recognition_instruction: str = "") -> VisionRecognitionRequest: image_data_url, content_type, size_bytes = _storage_image_data_url( str(material.image_uri or ""), material.metadata, storage, ) return VisionRecognitionRequest( display_name=material.display_name, image_uri=image_data_url, thumbnail_uri=None, prompt=material.prompt, negative_prompt=material.negative_prompt, tags=tuple(material.tags), metadata={ **material.metadata, "recognition_instruction": recognition_instruction, "vision_input": { "source_uri": material.image_uri, "content_type": content_type, "size_bytes": size_bytes, }, }, ) def _storage_image_data_url( image_uri: str, metadata: dict[str, Any], storage: VfsStorage, ) -> tuple[str, str, int]: if not image_uri: raise ApiError( ErrorCode.VALIDATION_FAILED, "material image_uri is required for recognition", {"field": "image_uri"}, ) path = storage.ensure_file(image_uri) try: reported_size = path.stat().st_size except OSError: raise ApiError(ErrorCode.MEDIA_NOT_FOUND, "material image could not be inspected", {"uri": image_uri}) from None if reported_size > MAX_UPLOAD_BYTES: raise ApiError( ErrorCode.UPLOAD_TOO_LARGE, "material image exceeds the recognition size limit", {"uri": image_uri, "size_bytes": reported_size, "max_size_bytes": MAX_UPLOAD_BYTES}, ) try: with path.open("rb") as handle: content = handle.read(MAX_UPLOAD_BYTES + 1) except OSError: raise ApiError(ErrorCode.MEDIA_NOT_FOUND, "material image could not be read", {"uri": image_uri}) from None if len(content) > MAX_UPLOAD_BYTES: raise ApiError( ErrorCode.UPLOAD_TOO_LARGE, "material image exceeds the recognition size limit", {"uri": image_uri, "size_bytes": len(content), "max_size_bytes": MAX_UPLOAD_BYTES}, ) content_type = _image_content_type(content) if content_type is None: raise ApiError( ErrorCode.UNSUPPORTED_FILE_TYPE, "material image bytes do not match a supported image format", {"uri": image_uri, "allowed": sorted(UPLOAD_ALLOWED_CONTENT_TYPES)}, ) extension = _suffix(image_uri) if extension and extension not in UPLOAD_ALLOWED_CONTENT_TYPES[content_type]: raise ApiError( ErrorCode.UNSUPPORTED_FILE_TYPE, "material image extension does not match its file signature", {"uri": image_uri, "extension": extension, "detected_content_type": content_type}, ) upload = metadata.get("upload") if isinstance(metadata.get("upload"), dict) else {} declared_content_type = _optional_text(upload.get("content_type")) if declared_content_type and declared_content_type != content_type: raise ApiError( ErrorCode.UNSUPPORTED_FILE_TYPE, "material image content type does not match its file signature", { "uri": image_uri, "declared_content_type": declared_content_type, "detected_content_type": content_type, }, ) vision_content, vision_content_type = _optimized_vision_input(content, content_type) encoded = base64.b64encode(vision_content).decode("ascii") return f"data:{vision_content_type};base64,{encoded}", vision_content_type, len(vision_content) def _optimized_vision_input(content: bytes, content_type: str) -> tuple[bytes, str]: """Bound multimodal payload size without modifying the stored original. Chat-completion providers receive images inline as base64. Multi-megabyte PNG screenshots become substantially larger on the wire and can time out at reverse proxies even when their visual resolution is modest. AstrBot already depends on Pillow, so large or over-dimensioned inputs are reduced to a high-quality JPEG used only for recognition. """ if Image is None: return content, content_type try: with Image.open(BytesIO(content)) as source: width, height = source.size needs_resize = max(width, height) > VISION_INPUT_MAX_EDGE needs_reencode = len(content) > VISION_INPUT_REENCODE_THRESHOLD if not needs_resize and not needs_reencode: return content, content_type source.seek(0) frame = source.convert("RGBA") if needs_resize: frame.thumbnail( (VISION_INPUT_MAX_EDGE, VISION_INPUT_MAX_EDGE), Image.Resampling.LANCZOS, ) background = Image.new("RGB", frame.size, (255, 255, 255)) background.paste(frame, mask=frame.getchannel("A")) output = BytesIO() background.save( output, format="JPEG", quality=VISION_INPUT_JPEG_QUALITY, optimize=True, progressive=True, ) optimized = output.getvalue() if not optimized or len(optimized) >= len(content): return content, content_type return optimized, "image/jpeg" except Exception: # Signature validation above already proved that the original is a # supported image. Optimization is best-effort and must never make a # previously valid material unusable. return content, content_type def _image_content_type(content: bytes) -> str | None: image_format = detect_image_format(content) return image_format.mime_type if image_format is not None else None def _recognition_route(metadata: dict[str, Any], default_route: str) -> str: upload = metadata.get("upload") if isinstance(metadata.get("upload"), dict) else {} return ( _optional_text(metadata.get("recognition_model_route")) or _optional_text(metadata.get("recognition_route")) or _optional_text(upload.get("recognition_model_route")) or default_route ) def _recognition_metadata( recognition: VisionRecognitionResult, *, route_name: str, ) -> dict[str, Any]: metadata = { "engine": "mock_vlm" if recognition.provider_id == "mock" else "vlm_adapter", "provider_id": recognition.provider_id, "model": recognition.model, "confidence": recognition.confidence, "route_name": route_name, "fallback_used": False, "description": recognition.description, "raw_metadata": recognition.raw_metadata, } return metadata def _vision_provider_error(exc: VisionProviderError) -> dict[str, Any]: failure = { "source": "vision_provider", "provider_id": _optional_text(exc.provider_id) or "unknown", "status_code": exc.status_code, "error_type": _optional_text(exc.error_type) or "provider_error", "response_issue": str(exc)[:300], } if isinstance(exc.details.get("retryable"), bool): failure["retryable"] = exc.details["retryable"] reason = _optional_text(exc.details.get("reason") or exc.details.get("parse_error")) if reason: failure["reason"] = reason[:500] structured_keys = exc.details.get("structured_keys") if isinstance(structured_keys, list): failure["structured_keys"] = [str(item) for item in structured_keys[:50]] return failure def _unexpected_vision_provider_error(provider: VisionProvider, exc: Exception) -> dict[str, Any]: return { "source": "vision_provider", "provider_id": _optional_text(getattr(provider, "provider_id", None)) or "unknown", "status_code": None, "error_type": "unexpected_provider_error", "exception_type": exc.__class__.__name__, } def _vision_api_error(material_id: str, route_name: str, failure: dict[str, Any]) -> ApiError: details = { "material_id": material_id, "route_name": route_name, "provider_id": failure.get("provider_id"), "status_code": failure.get("status_code"), "error_type": failure.get("error_type"), } if "retryable" in failure: details["retryable"] = failure["retryable"] return ApiError( ErrorCode.GENERATION_FAILED, "material vision provider request failed", details, ) def _normalize_upload_filename(value: Any) -> str | None: if value is None: return None filename = str(value).strip() if not filename: return None if len(filename) > 255 or filename in {".", ".."} or any(marker in filename for marker in ("/", "\\", "\x00", ":")): raise ApiError( ErrorCode.VALIDATION_FAILED, "filename must be a safe basename", {"field": "filename"}, ) return filename def _decode_upload_data_url(value: str) -> tuple[bytes, str, str]: if not value.startswith("data:") or ";base64," not in value: raise ApiError( ErrorCode.VALIDATION_FAILED, "image_data_url must be a base64 data URL", {"field": "image_data_url"}, ) header, encoded = value.split(",", 1) declared_content_type = header[5:].split(";", 1)[0].strip().lower() if declared_content_type not in UPLOAD_ALLOWED_CONTENT_TYPES: raise ApiError( ErrorCode.UNSUPPORTED_FILE_TYPE, "unsupported upload content type", {"content_type": declared_content_type, "allowed": sorted(UPLOAD_ALLOWED_CONTENT_TYPES)}, ) if len(encoded) > ((MAX_UPLOAD_BYTES + 2) // 3) * 4 + 8: raise ApiError( ErrorCode.UPLOAD_TOO_LARGE, "upload exceeds maximum allowed size", {"max_size_bytes": MAX_UPLOAD_BYTES}, ) try: content = base64.b64decode(encoded, validate=True) except (binascii.Error, ValueError): raise ApiError( ErrorCode.VALIDATION_FAILED, "image_data_url contains invalid base64 data", {"field": "image_data_url"}, ) from None detected = detect_image_format(content) if detected is None: raise ApiError( ErrorCode.UNSUPPORTED_FILE_TYPE, "uploaded bytes do not match a supported image format", {"allowed": sorted(UPLOAD_ALLOWED_CONTENT_TYPES)}, ) if declared_content_type != detected.mime_type: raise ApiError( ErrorCode.UNSUPPORTED_FILE_TYPE, "data URL content type does not match image bytes", {"content_type": declared_content_type, "detected_content_type": detected.mime_type}, ) return content, detected.mime_type, detected.extension def _normalize_upload_content_type(value: Any) -> str | None: if value is None: return None content_type = str(value).split(";", 1)[0].strip().lower() if not content_type: return None if content_type not in UPLOAD_ALLOWED_CONTENT_TYPES: raise ApiError( ErrorCode.UNSUPPORTED_FILE_TYPE, "unsupported upload content type", { "field": "content_type", "content_type": content_type, "allowed": sorted(UPLOAD_ALLOWED_CONTENT_TYPES), }, ) return content_type def _normalize_upload_size(payload: dict[str, Any]) -> int | None: for field in UPLOAD_SIZE_FIELDS: if field not in payload or payload[field] in (None, ""): continue try: size = int(payload[field]) except (TypeError, ValueError): raise ApiError(ErrorCode.VALIDATION_FAILED, "upload size must be an integer", {"field": field}) from None if size < 0: raise ApiError(ErrorCode.VALIDATION_FAILED, "upload size must not be negative", {"field": field}) if size > MAX_UPLOAD_BYTES: raise ApiError( ErrorCode.UPLOAD_TOO_LARGE, "upload exceeds maximum allowed size", {"field": field, "size_bytes": size, "max_size_bytes": MAX_UPLOAD_BYTES}, ) return size return None def _normalize_upload_metadata(value: Any) -> dict[str, Any]: if value in (None, ""): return {} if not isinstance(value, dict): raise ApiError(ErrorCode.VALIDATION_FAILED, "metadata must be an object", {"field": "metadata"}) return dict(value) def _validate_upload_file_type(*, filename: str | None, image_uri: str, content_type: str | None) -> str: extension = _suffix(filename) or _suffix(image_uri) if not extension: raise ApiError( ErrorCode.UNSUPPORTED_FILE_TYPE, "upload filename or image_uri must include a supported image extension", {"field": "filename", "allowed": sorted(UPLOAD_ALLOWED_EXTENSIONS)}, ) if extension not in UPLOAD_ALLOWED_EXTENSIONS: raise ApiError( ErrorCode.UNSUPPORTED_FILE_TYPE, "unsupported upload file extension", {"field": "filename", "extension": extension, "allowed": sorted(UPLOAD_ALLOWED_EXTENSIONS)}, ) if content_type and extension not in UPLOAD_ALLOWED_CONTENT_TYPES[content_type]: raise ApiError( ErrorCode.UNSUPPORTED_FILE_TYPE, "upload extension does not match content type", {"field": "content_type", "extension": extension, "content_type": content_type}, ) return extension def _suffix(value: str | None) -> str | None: if not value: return None suffix = PurePosixPath(str(value).split("?", 1)[0]).suffix.lower() return suffix or None def _display_name_from_filename(filename: str | None) -> str | None: if not filename: return None return PurePosixPath(filename).stem.replace("_", " ").strip() or None def _optional_text(value: Any) -> str | None: text = str(value or "").strip() return text or None def _matches_material_filters( material: Material, *, color: str | None, material_type: str | None, body_slot: str | None, wear_layer: str | None, min_confidence: float | None, confirmation_state: str | None, ) -> bool: profile = material.metadata.get("normalized_asset_profile") if isinstance(material.metadata.get("normalized_asset_profile"), dict) else {} attributes = profile.get("attributes") if isinstance(profile.get("attributes"), dict) else {} searchable = " ".join( [material.display_name, material.prompt or "", " ".join(material.tags), str(attributes)] ).casefold() if color and str(color).casefold() not in searchable: return False if material_type and str(material_type).casefold() not in searchable: return False if body_slot and str(body_slot) not in (profile.get("body_slots") or []): return False if wear_layer and str(profile.get("wear_layer") or "") != str(wear_layer): return False if min_confidence is not None: try: if float(profile.get("confidence") or 0.0) < float(min_confidence): return False except (TypeError, ValueError): return False state = str(confirmation_state or "").lower() if state == "confirmed" and not profile.get("confirmed"): return False if state == "needs_confirmation" and not profile.get("requires_confirmation"): return False if state == "unconfirmed" and profile.get("confirmed"): return False return True