A high-throughput and memory-efficient inference and serving engine for LLMs
其他人工智能更新于 2026-11-02
其他人工智能MIT
Repository dossier
lightseekorg/tokenspeed
TokenSpeed is a speed-of-light LLM inference engine.
Brief / README 摘要
查看原文TokenSpeed is a speed-of-light LLM inference engine designed for agentic workloads, with TensorRT-LLM-level performance and vLLM-level usability. The goal is to be the most performant inference engine for production agentic workloads. Scheduler: C++ control plane and Python execution plane. Request lifecycle, KV cache ownership, and overlap timing are encoded as a finite-state machine, with safe KV resource reuse enforced by the type system at compile time. Kernels: pluggable, layered kernel system with a portable public API and a centralized registry including one of the fastest MLA (Multi-he…
Topics / 标签
01
浏览全部相关推荐
Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
其他人工智能更新于 2026-11-02
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
其他人工智能更新于 2026-11-02
Never stop coding. Free MIT AI gateway: one endpoint, 375 providers (150+ free), 1,400+ models Kimi, Claude, GPT, Gemini, GLM, DeepSeek, MiniMax. Works with Claude Code, Codex, Cursor, OpenCode, Cline & Copilot. Quota-aware auto-fallback, RTK+Caveman compression saves 15-95% tokens, MCP/A2A, Desktop/PWA. Built by 900+ contributors
其他人工智能更新于 2026-11-02
Build Agentic workflows, RAG pipelines, with rich AI model and tool support on one collaborative workspace. Deploy on cloud, VPC, or self-hosted, so teams move from prototype to production without rebuilding the stack.
其他人工智能更新于 2026-11-02
🤯 LobeHub is your Chief Agent Operator, organizing your agents into 7×24 operations by hiring, scheduling, and reporting on your entire AI team.
其他人工智能更新于 2026-11-02