MulmoChat

Product Introduction

1. Definition: MulmoChat is an open-source, research-prototype multimodal AI chat application and orchestration framework designed to transcend the limitations of traditional text-based interfaces. Built using a modern technical stack—including TypeScript, Vue.js, and Vite—it functions as a "Shared Canvas" environment where natural language conversations trigger real-time visual and interactive outputs. It is categorized as an AI-native interface (LLM OS) prototype that integrates voice-first interaction with dynamic UI rendering.

2. Core Value Proposition: MulmoChat exists to bridge the gap between generative AI outputs and functional workspace utility. By utilizing a "shared canvas" paradigm, it eliminates the friction of switching between tools. Users can engage in voice or text conversations that result in immediate, actionable visual artifacts such as interactive maps, AI-generated artwork, playable games, and data spreadsheets. Its primary value lies in its provider-agnostic architecture, allowing seamless switching between OpenAI, Anthropic, Google Gemini, and local models like Ollama.

Main Features

1. Multimodal Shared Canvas UI: Unlike standard chat interfaces that stream text in a vertical column, MulmoChat utilizes a spatial canvas. When a user requests a location, a Google Maps instance materializes; when they ask for a design, an image generation block appears. This is achieved through a coordinated frontend architecture that maps LLM tool-calling events to specific Vue.js components, allowing the conversation and the visual workspace to coexist and interact.

2. Provider-Agnostic Text Generation API: MulmoChat features a unified backend abstraction layer for Large Language Models (LLMs). Through a standardized API (POST /api/text/generate), developers can swap between GPT-4, Claude 3.5, and Gemini 1.5 Pro without changing client-side logic. The system normalizes various vendor responses into a consistent JSON format, handling credential availability and default model suggestions automatically.

3. Advanced ComfyUI & Local Image Integration: The platform offers deep integration with ComfyUI Desktop for local image generation, specifically optimized for high-performance models like FLUX.1 (Schnell and Dev). The system includes model-specific optimization logic that automatically adjusts parameters such as CFG scale, sampling steps (e.g., 4 steps for Schnell), and noise schedulers based on the detected model. This allows for professional-grade, privacy-focused image synthesis directly within the chat flow.

4. Extensible Plugin Architecture: Designed with developers in mind, MulmoChat employs a modular plugin system defined via TypeScript contracts. Developers can extend the platform's capabilities by building "Tool Plugins" that define both the backend logic (agent tools) and the frontend representation (Vue views). This allows for the rapid creation of custom visual experiences, such as specialized data visualizations or proprietary internal tools.

Problems Solved

1. Pain Point: Fragmented AI Workflows: Traditional AI interactions require users to copy-paste prompts or results between different browser tabs (e.g., a chat window, a map tool, and an image generator). MulmoChat solves this "context-switching tax" by consolidating these functions into a single, voice-responsive visual environment.

2. Target Audience:

• Product Strategists and UX Designers: Those exploring the "AI-native OS" mindset and looking to prototype the next generation of human-computer interaction.
• AI Developers and Researchers: Engineers who need an extensible, open-source stack to test multi-provider LLM orchestration and custom tool-calling.
• Creative Professionals: Users seeking a voice-controlled environment for brainstorming that integrates local image generation (via ComfyUI) and research (via Exa search).

3. Use Cases:

• Interactive Brainstorming: Converting spoken ideas into visual mind maps or spreadsheets in real-time.
• Geospatial Exploration: Asking "Show me the best coffee shops in Tokyo" and having an interactive, navigable map appear instantly without leaving the chat.
• Rapid Prototyping: Generating HTML/CSS components or artwork prototypes through natural language, then viewing them immediately on the shared canvas.

Unique Advantages

1. Differentiation: Most AI chat platforms are "wrappers" around a text-input box. MulmoChat is a "workspace" built around a chat engine. It treats the visual output not as a static attachment, but as a primary interaction layer. Its ability to run entirely locally (using Ollama for text and ComfyUI for images) provides a significant privacy and cost advantage over cloud-only competitors.

2. Key Innovation: The "Voice-to-Visual" pipeline. By combining high-accuracy voice recognition with an intent-based orchestration layer, MulmoChat can interpret complex multi-step commands—like "Generate a futuristic city and then find me real-world architectural references"—and execute them across different visual modules simultaneously.

Frequently Asked Questions (FAQ)

1. Does MulmoChat support local LLMs for privacy? Yes. MulmoChat integrates with Ollama via a provider-agnostic API. By configuring the OLLAMA_BASE_URL, users can run open-source models like Llama 3 or Mistral locally, ensuring that conversation data does not leave their hardware.

2. How does the ComfyUI integration work for image generation? MulmoChat connects to the ComfyUI Desktop API (typically on port 8000). It includes automated optimizations for FLUX and Stable Diffusion models, handling technical parameters like euler samplers and karras schedulers automatically based on the model selected in the environment configuration.

3. Can I build my own visual tools for MulmoChat? Absolutely. The platform is built on an extensible plugin architecture. Developers can follow the TOOLPLUGIN.md documentation to create custom TypeScript contracts and Vue.js components, allowing the AI to interact with any custom API or data visualization tool.

4. What AI providers are currently supported? MulmoChat supports major cloud providers including OpenAI (GPT series), Anthropic (Claude series), and Google Gemini. It also supports specialized search via the Exa API and local generation via Ollama and ComfyUI.