Latest Research Papers
2025-01-22
arXiv
Accelerate High-Quality Diffusion Models with Inner Loop Feedback
The paper introduces Inner Loop Feedback (ILF), a method to speed up diffusion models' inference by predicting future features using a lightweight module. This approach reduces runtime while maintaining high-quality results, and it is effective for both class-to-image and text-to-image generation. The performance of ILF is validated through various metrics including FID, CLIP score, and qualitative comparisons.
We propose Inner Loop Feedback (ILF), a novel approach to accelerate
diffusion models' inference. ILF trains a lightweight module to predict future
features in the denoising process by leveraging the outputs from a chosen
diffusion backbone block at a given time step. This approach exploits two key
intuitions; (1) the outputs of a given block at adjacent time steps are
similar, and (2) performing partial computations for a step imposes a lower
burden on the model than skipping the step entirely. Our method is highly
flexible, since we find that the feedback module itself can simply be a block
from the diffusion backbone, with all settings copied. Its influence on the
diffusion forward can be tempered with a learnable scaling factor from zero
initialization. We train this module using distillation losses; however, unlike
some prior work where a full diffusion backbone serves as the student, our
model freezes the backbone, training only the feedback module. While many
efforts to optimize diffusion models focus on achieving acceptable image
quality in extremely few steps (1-4 steps), our emphasis is on matching best
case results (typically achieved in 20 steps) while significantly reducing
runtime. ILF achieves this balance effectively, demonstrating strong
performance for both class-to-image generation with diffusion transformer (DiT)
and text-to-image generation with DiT-based PixArt-alpha and PixArt-sigma. The
quality of ILF's 1.7x-1.8x speedups are confirmed by FID, CLIP score, CLIP
Image Quality Assessment, ImageReward, and qualitative comparisons. Project
information is available at https://mgwillia.github.io/ilf.
2024-12-27
arXiv
A Survey on Large Language Model Acceleration based on KV Cache Management
This survey provides a comprehensive overview of Key-Value (KV) cache management strategies for accelerating Large Language Model (LLM) inference, categorizing them into token-level, model-level, and system-level optimizations. It aims to offer insights and support the development of efficient and scalable KV cache management techniques for practical LLM deployment.
Large Language Models (LLMs) have revolutionized a wide range of domains such
as natural language processing, computer vision, and multi-modal tasks due to
their ability to comprehend context and perform logical reasoning. However, the
computational and memory demands of LLMs, particularly during inference, pose
significant challenges when scaling them to real-world, long-context, and
real-time applications. Key-Value (KV) cache management has emerged as a
critical optimization technique for accelerating LLM inference by reducing
redundant computations and improving memory utilization. This survey provides a
comprehensive overview of KV cache management strategies for LLM acceleration,
categorizing them into token-level, model-level, and system-level
optimizations. Token-level strategies include KV cache selection, budget
allocation, merging, quantization, and low-rank decomposition, while
model-level optimizations focus on architectural innovations and attention
mechanisms to enhance KV reuse. System-level approaches address memory
management, scheduling, and hardware-aware designs to improve efficiency across
diverse computing environments. Additionally, the survey provides an overview
of both text and multimodal datasets and benchmarks used to evaluate these
strategies. By presenting detailed taxonomies and comparative analyses, this
work aims to offer useful insights for researchers and practitioners to support
the development of efficient and scalable KV cache management techniques,
contributing to the practical deployment of LLMs in real-world applications.
The curated paper list for KV cache management is in:
\href{https://github.com/TreeAI-Lab/Awesome-KV-Cache-Management}{https://github.com/TreeAI-Lab/Awesome-KV-Cache-Management}.