tsm-hub: Bringing LLMs, Tools, MCPs, and Skills into One Unified Gateway

Introduction: Why tsm-hub?

In large model application development, I encountered a common pain point: each Provider has inconsistent API formats and different rate limiting policies, requiring callers to write adaptation code for each model. Even worse, when a model fails or becomes unavailable, callers need to manually switch configurations, resulting in high operational costs.

Take my own projects as an example: multiple AI applications (development environments, content generation, automation services, etc.) all need to call LLMs, and each project maintains its own Provider configuration. Multiple paid model providers are integrated but cannot be centrally scheduled. When one Provider experiences failures or rate limiting, another may still be serving normally, but the caller has no way to know and can only wait. Even more troublesome, different tasks are suited to different models, but it's difficult for callers to automatically select the most appropriate model based on task type.

Beyond model routing, there's a more fundamental issue: how AI capabilities are organized. Every project repeatedly implements tool call loops, skill pack management, and MCP service integration, but these capabilities are inherently shareable. Callers can carry their own capability declarations, the gateway statistics based on actual usage, and administrators can choose whether to incorporate them into the shared capability pool — this is what makes tsm-hub most distinctive.

To solve these problems, I built tsm-hub (GitHub Repository) — a unified LLM access and capability platform integrating LLM gateway, capability pool platform, fast path engine, runtime environment, and observability. Its core idea is: externally expose only self-issued sk-tr-… Keys, callers use it just like OpenAI, and the platform automatically completes smart routing, capability injection, fast path acceleration, runtime isolation, and full-chain observability on the backend.

Three Core Selling Points:

1. Model Gateway: Multi-Provider / multi-model / unified protocol, smart routing and automatic failover
2. Capability Pool: Discovery, statistics, and incorporation of Tools / MCPs / Skills, configure once and share network-wide
3. Fastpath: Deterministic requests bypass LLM, directly compute answers locally

Other capabilities (Runtime, Sandbox, Memory, Observability, Streaming) serve as capability supplements, detailed in subsequent sections.

Core Capability Matrix

tsm-hub has formed a complete functional matrix covering the full chain from access to runtime:

Capability Domain	Core Features
Unified Entry	OpenAI-compatible API, callers only need to change Key, base_url, and model alias
Smart Routing	Configurable multi-dimensional smart scheduling strategy (smart), supporting four modes: cost_first / stability_first / task_aware / balanced, with individually adjustable dimension weights; also supports explicit routing tables with custom failover / weighted strategies; configurable via config.json, environment variables, or .env files
Multi-Protocol Adaptation	Supports 13 upstream protocols (OpenAI-compatible, Anthropic, Azure OpenAI, Google Gemini, AWS Bedrock, AWS SageMaker, Cohere, Mistral, Hugging Face, Replicate, Together AI, Fireworks AI, Groq), with unified Provider adapter interface
Capability Pool	Three-layer capability system: built-in Tools (general tools), MCPs (Model Context Protocol services), Skills (skill packs), supporting discovery, statistics, and selective adoption
Fast Path Engine	Fastpath optional acceleration (direct answers for deterministic queries like arithmetic, statistics, unit conversion), Codegen code generation fastpath
Runtime Environment	Sandbox (Docker-isolated code execution environment), Memory (SQLite-persisted conversation memory and user preference storage)
Observability	Usage statistics, Provider health monitoring, daily trends, cost attribution
Privacy & Compliance	TLS transport encryption, CORS whitelist, startup security self-check, audit logs

Key Design Decisions and Architectural Benefits

During the implementation of tsm-hub, several key design decisions not only solved immediate problems but more importantly established extensible architectural thinking, bringing long-term benefits to subsequent development.

Decision 1: Self-Issued Token Key for Unified Authentication

Design Thinking: Externally issue only self-issued Keys starting with sk-tr-, and the server stores only hashes (plaintext is returned only once at creation). Each Key can independently configure the allowed model list, quota limits (total Tokens, total cost, daily Tokens, RPM), skill injection mode (no injection / skill list / all skills / specified skills), and Agent switch (whether to go through the gateway's built-in Agent tool loop).

This way, callers don't need to know the real Keys of upstream Providers — the gateway manages them uniformly, which also facilitates subsequent revocation and permission control.

Architectural Benefits:

• Fine-grained Permission Control: Each Key can independently configure model scope, quotas, and skill injection modes to meet different caller needs
• Key Security: The server stores only hashes; even if the database is leaked, plaintext Keys cannot be obtained; plaintext is returned only once at creation, reducing leakage risk
• Unified Revocation: When a caller no longer needs access, simply disable the corresponding Key without modifying upstream Provider configurations
• Traceability: Each request is associated with a specific Key, and usage statistics, cost attribution, and audit logs can all be aggregated by Key dimension

Decision 2: Configurable Multi-Dimensional Smart Scheduling + Explicit Routing Tables

Design Thinking: After integrating multiple model providers, different scenarios have different routing strategy needs: some scenarios are cost-sensitive, some prioritize reliability, and some need to balance quality and speed. Therefore, smart routing should not be a hardcoded single strategy, but rather a configurable multi-dimensional scoring system that allows users to select appropriate strategy modes and weights based on their business needs.

By default, the smart scheduling strategy is adopted. For models without explicit routing tables, the gateway automatically scans all Providers supporting that model, comprehensively evaluates multiple dimensions, and intelligently selects the optimal upstream — no need to manually configure routing for each model.

Four Strategy Modes (configurable via settings.smart.strategy_mode):

Mode	Cost Weight	Stability Weight	Latency Weight	Applicable Scenarios
cost_first	100	40	30	Cost-sensitive scenarios, free/low-cost priority
stability_first	30	100	50	Production services, reliability priority, Providers with high success rate preferred
task_aware	50	80	70	Mixed workloads, balancing quality and speed
balanced	60	70	70	General scenarios, balanced across all dimensions

Each dimension's weight can also be individually overridden via settings.smart.cost_weight / stability_weight / latency_weight (0 means use the strategy default). Configurable via config.json, environment variables (TSM_HUB_SMART_*), or .env files.

The smart strategy's scheduling algorithm comprehensively considers multiple dimensions:

• Cost Dimension: Free model bonus + price tier bonus (configurable free_bonus / price_tiers)
• Stability Dimension: Historical success rate (requests - errors) / requests × 100, consecutive failure penalty; new providers get 70 points to encourage exploration
• Latency Dimension: EWMA latency, 100 points within 100ms, 0 points above 1000ms; 60 points for no data
• Health Filtering: Automatically filters unhealthy Providers with 429 rate limiting, 5xx errors, high latency, etc., faulty nodes are automatically removed
• Capability Matching: Requests with tools prioritize models supporting tool_calls, long-text requests prioritize models supporting long context
• General Penalties: Half-open candidates get point deductions, significant weight reduction (-1000) during 429 rate limiting periods

The scheduling algorithm performs weighted comprehensive scoring across multiple dimensions. The gateway sorts by score from high to low and prioritizes the healthy Provider with the highest score. If the first one fails, it automatically fails over to the next one, achieving seamless switching between multiple model providers.

At the same time, users can also freely configure explicit routing tables, defining a set of upstream targets for specific models or routing aliases (such as chat, fast, code, reason), selecting strategies like failover (strict priority failover), weighted (weighted load balancing), or smart (applying smart scheduling within the target range).

When a request comes in, the gateway makes decisions in the following steps: ① Match explicit routing table (if any) → ② Apply corresponding strategy → ③ Health check filtering → ④ Capability matching filtering → ⑤ Comprehensive scoring and sorting based on configured strategy mode and weights → ⑥ Select final Provider → ⑦ Automatic failover on failure.

Architectural Benefits:

• Flexible and Configurable: Four strategy modes + individually adjustable dimension weights, adapting to different business scenarios; supports three configuration methods: config.json / environment variables / .env
• Optional Stability Priority: When selecting stability_first mode, prioritize Providers with recent high success rates, faulty nodes are automatically removed, service quality is guaranteed
• Task-Model Matching: Configure different model pools for different task types (general conversation, code generation, deep reasoning, etc.) through explicit routing tables, each playing to its strengths
• Multi-Active Disaster Recovery: Integrate multiple model providers, automatically switch to others when one experiences failures or rate limiting, completely transparent to callers
• Zero-Configuration Onboarding: New models don't need manual routing configuration, the smart strategy automatically scans all Providers and intelligently selects
• Quality Assurance: Health status and capability matching are hard filter conditions, unhealthy or mismatched Providers are directly excluded

Decision 3: Three-Layer Capability Pool Architecture + Discovery Statistics Mechanism

Design Thinking: Abstract general capabilities into three layers:

• Tools: Lightweight general tools (get_time, calc, fetch_url, read_file, remember, recall, execute_code, etc.), built-in execution by the gateway, no external dependencies
• MCPs: Model Context Protocol services, connecting to external tool services via stdio or HTTP (such as filesystem fs, code editing serena, memory memory, thinking think, etc.), tool names support three formats: server__tool, server:tool, server/tool
• Skills: Skill packs, loaded via git submodule (such as demo-lab server inspection, weekly-investment investment weekly report, etc.), supporting skill list injection and skill-run calls

All three layers of capabilities support the "discovery — statistics — selective adoption" mechanism: the gateway records external capabilities declared by callers in requests, automatically classifies them by naming conventions (general tool names → Tools, skill: prefix → Skills, MCP style → MCPs), statistics call counts and number of users, and displays candidate lists in the management backend. Administrators can selectively adopt based on actual needs, choosing the implementation method at adoption (record-only marker / HTTP call / MCP tool / built-in implementation).

The core value of this mechanism lies in discovery and statistics — allowing administrators to see what capabilities callers are actually using, avoiding reinventing the wheel; promotion requires manual review and configuration, not full automation. Recognition accuracy is higher for capabilities following naming conventions, and third-party frameworks may require additional adaptation.

Architectural Benefits:

• Configure Once, Share Network-wide: Tools/MCPs/Skills only need to be configured once in the gateway, and all projects passing through the gateway can use them
• Capability Visibility: Automatically discover and count external capabilities declared by callers, allowing administrators to see actual usage and avoid reinventing the wheel
• Progressive Accumulation: Through the "discovery — statistics — selective adoption" mechanism, the capability pool can continuously accumulate best practices from callers
• Flexible Adoption Methods: Multiple adoption methods available: record-only, HTTP call, MCP tool, built-in implementation, adapting to the actual situation of different capabilities
• Decoupling Callers from Capability Implementation: Callers only need to declare tool names, no need to know the specific implementation method of capabilities (built-in/MCP/HTTP), the gateway automatically proxies

Decision 4: Fastpath Fast Path (Optional Acceleration)

Design Thinking: For deterministic queries like explicit arithmetic, statistics, unit conversion, date calculation, base conversion, word count, etc., the gateway can directly compute and answer locally without calling upstream LLM, thereby saving quota and reducing latency. Fastpath is a configurable acceleration option that users can enable or disable based on business scenarios.

To reduce the risk of false matches (for example, chapter numbers in article writing requests being mistaken for arithmetic problems), a three-layer protection mechanism was designed: task-type keyword blacklist, global length limit, query-type keyword whitelist. The design principle of the three-layer protection is "better to miss than to be wrong" — for uncertain requests, conservatively forward them to LLM processing, rather consume a bit more quota than return incorrect calculation results.

Architectural Benefits:

• Low Latency: On hit, no need to call upstream LLM, directly compute locally, response drops from seconds to milliseconds
• Cost Saving: Deterministic queries are executed directly locally, no LLM call quota is consumed
• Safe and Controllable: Three-layer protection + "better to miss than to be wrong" principle, uncertain requests directly fall back to LLM
• Optional Enablement: Users can choose to enable or disable based on business scenarios, cautious use recommended for open-ended conversation or content generation scenarios

Decision 5: Unified Abstraction Layer — Decoupled Design for Multi-Protocol Streaming Responses

Faced with 13 upstream protocols each with different streaming response formats, adopting the unified abstraction layer design thinking: completely decouple "upstream format parsing" and "downstream format output". Each Provider adapter only needs to implement the StreamChunk interface, converting the upstream's raw chunk into a unified internal representation, then a unified SSE encoder is responsible for output. Adding a new protocol only requires implementing one adapter file (usually a few hundred lines), and output format adjustments only require modifying the unified encoder, with 13 protocols automatically taking effect.

Decision 6: Fault-Tolerant Recognition — Elegant Degradation for MCP Tool Name Multi-Format

In the MCP ecosystem, tool naming formats are inconsistent (server__tool, server:tool, server/tool, etc.). Adopting fault-tolerant recognition design: try multiple separators by priority, extract the server name and then filter false matches through validation, successfully recognized tools are aggregated by server, recognition failures gracefully degrade to general Tools. This way, the capability discovery mechanism can automatically adapt to newly emerging MCP framework naming formats, no need to write dedicated recognition logic for each framework.

Decision 7: Immutable Data — Lock-Free Concurrency Safety for Configuration Hot Reload

The management backend runtime configuration modifications need to take effect immediately, while a large number of requests are using the old configuration. Adopting immutable data + atomic replacement design: the configuration object is immutable after loading, when modifying, create a brand new configuration object, and atomically replace the global pointer through atomic.Value. Read operations are completely lock-free, write operations only have atomic overhead, and each request uses the same configuration throughout its lifecycle, no inconsistent state of half-new half-old.

Typical Scenarios and Call Examples

Scenario 1: Multi-Project Unified Access, Centralized Quota Management

Multiple AI applications (development environments, content generation, automation services, etc.) no longer maintain Provider configurations individually, but uniformly call LLMs through tsm-hub. The gateway centrally manages quotas of multiple paid model providers, automatically selects the most appropriate model for different tasks, prioritizes Providers with recent high success rates, and automatically switches to others when one model provider experiences failures or rate limiting, completely transparent to callers. Operations personnel only need to maintain Providers and Keys in the management backend, no need to modify configurations project by project.

Scenario 2: Capability Reuse Platform, Configure Once, Share Network-wide

The three-layer capability pool of Tools, MCPs, and Skills is configured once, and all projects passing through the gateway can share and use them. For example: the filesystem MCP service only needs to be configured once in the gateway, and all projects can call it through tools like fs__read_file; the server inspection skill pack only needs to be loaded once, and all projects can call it through skill-run. The gateway also automatically discovers and counts external capabilities declared by callers, displays candidate lists in the management backend, and administrators can selectively adopt based on actual needs, avoiding reinventing the wheel.

Scenario 3: Automatic Failover, High Service Availability

When an upstream Provider experiences failures (429 rate limiting, 500 errors, network timeout), the gateway automatically fails over to the next healthy Provider in the routing table, and callers don't need to do anything. The health check mechanism periodically probes the availability of each model, automatically marks problematic ones as degraded status, and automatically skips them during routing. For critical businesses, cross-Provider multi-active routing can also be configured to ensure services run normally even if one provider is completely unavailable.

Call Example 1: Built-in Tools Automatic Execution (Client Doesn't Pass tools)

The client sends a normal conversation request without passing the tools field, the gateway automatically enables the built-in Agent, attaches the tool pool, and executes the tool call loop on the server side:

// Client request
{
  "model": "chat",
  "messages": [{"role": "user", "content": "Help me calculate 123 * 456, then get the current time"}]
}

// Gateway internal processing flow
// 1. Automatically attach tool pool (get_time, calc, fetch_url, read_file, remember, etc.)
// 2. Call upstream LLM, model decides to call calc tool
// 3. Gateway executes calc(123 * 456) = 56088 locally
// 4. Return tool result to LLM, model decides to call get_time tool
// 5. Gateway executes get_time() = "current time" locally
// 6. Return tool result to LLM, model generates final answer

Throughout the process, arithmetic calculations and time acquisition are both executed locally by the gateway, no additional LLM calls consumed, latency drops from seconds to tens of milliseconds.

Call Example 2: MCP Tool Call (Proxied Through Gateway)

The client calls an integrated MCP service through the gateway, tool names use the server__tool format:

// Client request
{
  "model": "chat",
  "messages": [{"role": "user", "content": "Read the README.md file in the project root directory"}],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "fs__read_file",
        "description": "Read a file from the filesystem",
        "parameters": {"type": "object", "properties": {"path": {"type": "string"}}}
      }
    }
  ]
}

// Gateway internal processing flow
// 1. Recognize tool name fs__read_file, extract server=fs, tool=read_file
// 2. Find configured fs MCP service (stdio transport, persistent connection already established)
// 3. Call read_file tool of fs service through MCP protocol, parameter path="README.md"
// 4. MCP service executes file reading, returns file content
// 5. Gateway returns tool result to upstream LLM
// 6. LLM generates final answer based on file content

The client doesn't need to know the specific connection method of the MCP service (stdio/HTTP), only needs to declare the tool name in server__tool format, and the gateway automatically proxies the call.

Technical Implementation and Source Code Navigation

The complete source code is open source on GitHub, welcome to Star and submit Issues.

In terms of specific implementation, the Go + Angular tech stack was chosen:

Backend (Go):

• Standard library net/http implements HTTP service, no external Web framework dependencies
• sqlite stores Provider health status, memory, audit logs
• JSON configuration file (data/config.json) stores Provider, routing, Key, and other configurations
• Plugin architecture: Provider adapters, Fastpath matchers, Agent tools can all be independently extended
• Compiled into a single binary, simple deployment

Frontend (Angular):

• Management backend covers all functions: Provider management, routing configuration, Key issuance, model catalog, quota query, usage statistics, observability, Tools/MCPs/Skills management, Sandbox/Memory management, etc.
• Components organized in ts/, html/, css/ subdirectories, templates separated from logic
• Responsive design, supports desktop usage

Source Code Navigation:

File/Directory	Description
cmd/server/main.go	Service entry, loads configuration, initializes modules, starts HTTP service
internal/proxy/proxy.go	Proxy core, request authentication, routing selection, failover, usage recording
internal/proxy/adapter/	Upstream protocol adapter layer, 13 protocols, unified interface + streaming conversion
internal/router/router.go	Routing selection algorithm, supports smart / failover / weighted three strategies
internal/store/	Configuration storage and hot reload, SQLite memory and health status persistence
internal/proxy/fastpath.go	Fastpath fast path matcher (arithmetic, statistics, unit conversion, etc.)
internal/proxy/agent.go	Gateway built-in Agent, tool call loop and multi-turn reasoning
internal/proxy/codegen.go	Codegen code generation fastpath
internal/api/	Admin API (Key management, capability management, model catalog, routing configuration, etc.)
internal/skills/	Skills skill pack loading and injection (git submodule management)
internal/mcp/	MCP service connection and tool proxy (stdio/HTTP transport, tool call forwarding)
internal/sandbox/	Docker sandbox management (container lifecycle, resource limits, security isolation)
web/src/app/	Angular frontend, all management backend pages (organized in ts/html/css subdirectories)

Deployment and Operations

Docker Containerized Deployment

tsm-hub provides a complete Docker containerized deployment solution, using multi-stage builds, the image contains only the Go binary and runtime dependencies, small size and fast startup.

Quick Start:

make docker-up    # Automatically build frontend + image, start container in background

After startup, visit http://localhost:9070 to enter the management console.

Makefile Operations Commands:

Command	Description
make docker-build	Build Docker image (automatically build frontend first)
make docker-up	Build and start container (background, mount ./data)
make docker-down	Stop and remove container (keep ./data)
make docker-stop	Stop container (don't remove, can be restored)
make docker-start	Start a previously stopped container
make docker-restart	Restart container
make docker-logs	Follow container logs
make docker-status	View container status + health check

Data Persistence

The container persists all data through volume mount ./data:/data, including configuration files, usage logs, conversation memory, audit logs, etc. Container deletion and rebuilding won't lose data, just keep the ./data directory intact.

Configuration Methods

Supports four configuration methods, priority from high to low:

1. Command-line flags: such as -addr :9070, -admin-token xxx
2. System environment variables: such as TSM_HUB_ADDR=:9070
3. .env file: .env file in project root (docker compose automatically loads)
4. config.json: mounted ./data/config.json configuration file

Smart routing strategy can be configured via environment variables:

TSM_HUB_SMART_STRATEGY=stability_first  # cost_first / stability_first / task_aware / balanced
TSM_HUB_SMART_COST_WEIGHT=40
TSM_HUB_SMART_STABILITY_WEIGHT=100
TSM_HUB_SMART_LATENCY_WEIGHT=60

Health Check and High Availability

• Health check endpoint: GET /healthz, returns all provider health status, request count, uptime
• docker compose healthcheck: automatically checks every 30s, automatically marks anomalies
• Configuration hot reload: config.json changes are automatically hot-reloaded (fsnotify), no need to restart container
• Automatic failover: when a provider fails or is rate-limited, automatically fails over to other healthy providers

Security Design

• Container runs as non-root user, following the principle of least privilege
• Admin Token recommended to be injected via environment variables, not written to configuration files
• Production environments recommended to enable TLS or put a reverse proxy (Nginx / Caddy) in front
• All management operations are automatically recorded in audit logs, traceable