Model-Compression.md

Awesome Model Compression Paper Collection

• 2024

  • ICLR
  • AAAI

• 2023

  • NeurIPS
  • CVPR
  • ICLR
  • ICCV
  • ICML
  • ACL
  • IJCAI
  • AAAI

• 2022

  • NeurIPS
  • CVPR
  • ICLR
  • ECCV
  • ICML
  • ACL
  • AAAI

• 2021

  • NeurIPS
  • CVPR
  • ICLR
  • ICCV
  • ACL
  • AAAI

• 2020

  • NeurIPS
  • CVPR
  • ICLR
  • ECCV
  • ICML
  • AAAI

• 2019

  • NeurIPS
  • CVPR
  • ICLR
  • ICCV
  • ICML
  • IJCAI
  • AAAI

• 2018

  • NeurIPS
  • CVPR
  • ICLR
  • ECCV
  • ICML
  • ACL

2024

ICLR-2024

• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs [paper] [arxiv] [paper with code] [openreview]

• Compressed Context Memory for Online Language Model Interaction [paper] [arxiv] [paper with code] [code] [openreview]

• In-context Autoencoder for Context Compression in a Large Language Model [paper] [arxiv] [paper with code] [code] [openreview]

• Language Modeling Is Compression [paper] [arxiv] [paper with code] [code] [openreview]

• In defense of parameter sharing for model-compression [paper] [arxiv] [paper with code] [openreview]

AAAI-2024

• EPSD: Early Pruning with Self-Distillation for Efficient Model Compression [paper] [arxiv] [paper with code]

• QCS-SGM+: Improved Quantized Compressed Sensing with Score-Based Generative Models [paper]

2023

NeurIPS-2023

• Towards Efficient Image Compression Without Autoregressive Models [paper] [openreview]

• Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization [paper] [arxiv] [paper with code] [openreview]

• Lossy Image Compression with Conditional Diffusion Models [paper] [arxiv] [paper with code] [code] [openreview]

CVPR-2023

• Efficient Hierarchical Entropy Model for Learned Point Cloud Compression [paper] [paper with code]

ICLR-2023

• Quantized Compressed Sensing with Score-Based Generative Models [paper] [arxiv] [paper with code] [code] [openreview]

• MIMT: Masked Image Modeling Transformer for Video Compression [paper] [openreview]

• LilNetX: Lightweight Networks with EXtreme Model Compression and Structured Sparsification [paper] [arxiv] [paper with code] [code] [openreview]

ICCV-2023

• Lossy and Lossless (L2) Post-training Model Size Compression [paper] [paper with code] [code]

• Scene Matters: Model-based Deep Video Compression [paper] [arxiv] [paper with code]

ICML-2023

• LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation [paper] [arxiv] [paper with code]

• Less is More: Task-aware Layer-wise Distillation for Language Model Compression [paper] [arxiv] [paper with code] [code]

• Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Models [paper] [arxiv] [paper with code]

• COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Models [paper] [arxiv] [paper with code] [code]

ACL-2023

• Deep Model Compression Also Helps Models Capture Ambiguity [paper] [arxiv] [paper with code]

• AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression [paper] [arxiv] [paper with code] [code]

• A Comparative Study on the Impact of Model Compression Techniques on Fairness in Language Models [paper]

• Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models [paper] [arxiv] [paper with code]

IJCAI-2023

• Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data [paper] [arxiv] [paper with code] [code]

AAAI-2023

• DAMix: Exploiting Deep Autoregressive Model Zoo for Improving Lossless Compression Generalization [paper]

• A Survey on Model Compression and Acceleration for Pretrained Language Models [paper] [arxiv] [paper with code]

• Lifelong Compression Mixture Model via Knowledge Relationship Graph [paper]

2022

NeurIPS-2022

• Weighted Mutual Learning with Diversity-Driven Model Compression [paper] [openreview]

• Deep Compression of Pre-trained Transformer Models [paper] [openreview]

• Information-Theoretic GAN Compression with Variational Energy-based Model [paper] [arxiv] [paper with code] [openreview]

• Differentially Private Model Compression [paper] [arxiv] [paper with code] [openreview]

• Model Preserving Compression for Neural Networks [paper] [arxiv] [openreview]

CVPR-2022

• DiSparse: Disentangled Sparsification for Multitask Model Compression [paper] [arxiv] [paper with code] [code]

• CHEX: CHannel EXploration for CNN Model Compression [paper] [arxiv] [paper with code] [code]

• Practical Learned Lossless JPEG Recompression With Multi-Level Cross-Channel Entropy Model in the DCT Domain [paper] [arxiv] [paper with code]

ICLR-2022

• Entroformer: A Transformer-based Entropy Model for Learned Image Compression [paper] [arxiv] [paper with code] [code] [openreview]

• Encoding Weights of Irregular Sparsity for Fixed-to-Fixed Model Compression [paper] [arxiv] [paper with code] [openreview]

• Language model compression with weighted low-rank factorization [paper] [arxiv] [paper with code] [openreview]

• Fast Generic Interaction Detection for Model Interpretability and Compression [paper] [paper with code] [openreview]

• Exploring extreme parameter compression for pre-trained language models [paper] [arxiv] [paper with code] [code] [openreview]

ECCV-2022

• Contextformer: A Transformer with Spatio-Channel Attention for Context Modeling in Learned Image Compression [paper] [arxiv] [paper with code]

• Point Cloud Compression with Range Image-Based Entropy Model for Autonomous Driving [paper]

ICML-2022

• History Compression via Language Models in Reinforcement Learning [paper] [arxiv] [paper with code] [code]

• Uncertainty Modeling in Generative Compressed Sensing [paper]

ACL-2022

• Multi-Granularity Structural Knowledge Distillation for Language Model Compression [paper] [paper with code] [code]

• Compression of Generative Pre-trained Language Models via Quantization [paper] [arxiv] [paper with code]

AAAI-2022

• OctAttention: Octree-Based Large-Scale Contexts Model for Point Cloud Compression [paper] [arxiv] [paper with code] [code]

• From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression [paper] [arxiv] [paper with code] [code]

2021

NeurIPS-2021

• Preserved central model for faster bidirectional compression in distributed settings [paper] [arxiv] [paper with code] [code] [openreview]

• OSOA: One-Shot Online Adaptation of Deep Generative Models for Lossless Compression [paper] [arxiv] [paper with code] [openreview]

• DRONE: Data-aware Low-rank Compression for Large NLP Models [paper] [paper with code] [openreview]

CVPR-2021

• Towards Efficient Tensor Decomposition-Based DNN Model Compression With Optimization Framework [paper] [arxiv] [paper with code]

• Checkerboard Context Model for Efficient Learned Image Compression [paper] [arxiv] [paper with code] [code]

ICLR-2021

• Training with Quantization Noise for Extreme Model Compression [paper] [arxiv] [paper with code] [code] [openreview]

• Lossless Compression of Structured Convolutional Models via Lifting [paper] [arxiv] [paper with code] [code] [openreview]

• Learning Accurate Entropy Model with Global Reference for Image Compression [paper] [arxiv] [paper with code] [code] [openreview]

• UMEC: Unified model and embedding compression for efficient recommendation systems [paper] [paper with code] [openreview]

• Hierarchical Autoregressive Modeling for Neural Video Compression [paper] [arxiv] [paper with code] [code] [openreview]

ICCV-2021

• Exploration and Estimation for Model Compression [paper]

ACL-2021

• Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains [paper] [arxiv] [paper with code] [code]

• Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators [paper] [arxiv] [paper with code] [code]

• Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization [paper] [arxiv] [paper with code] [code]

AAAI-2021

• Robust Model Compression Using Deep Hypotheses [paper] [arxiv] [paper with code] [code]

• Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networks [paper] [arxiv] [paper with code]

2020

NeurIPS-2020

• Robust compressed sensing using generative models [paper] [arxiv]

• MuSCLE: Multi Sweep Compression of LiDAR using Deep Entropy Models [paper] [arxiv] [paper with code]

CVPR-2020

• Discrete Model Compression With Resource Constraint for Deep Neural Networks [paper] [paper with code]

• Multi-Dimensional Pruning: A Unified Framework for Model Compression [paper] [paper with code]

• Towards Efficient Model Compression via Learned Global Ranking [paper] [arxiv] [paper with code] [code]

• OctSqueeze: Octree-Structured Entropy Model for LiDAR Compression [paper] [arxiv] [paper with code] [code]

• The Knowledge Within: Methods for Data-Free Model Compression [paper] [arxiv] [paper with code]

• Structured Multi-Hashing for Model Compression [paper] [arxiv] [paper with code]

ICLR-2020

• Scalable Model Compression by Entropy Penalized Reparameterization [paper] [arxiv] [paper with code] [openreview]

• HiLLoC: lossless image compression with hierarchical latent variable models [paper] [arxiv] [paper with code] [code] [openreview]

ECCV-2020

• Online Ensemble Model Compression using Knowledge Distillation [paper] [arxiv]

• High-quality Single-model Deep Video Compression with Frame-Conv3D and Multi-frame Differential Modulation [paper]

ICML-2020

• Evaluating Lossy Compression Rates of Deep Generative Models [paper] [arxiv] [paper with code] [code]

• On the Power of Compressed Sensing with Generative Models [paper] [paper with code]

AAAI-2020

• Information-Theoretic Understanding of Population Risk Improvement with Model Compression [paper] [arxiv] [paper with code] [code]

• Light Multi-Segment Activation for Model Compression [paper] [arxiv] [paper with code] [code]

• Coarse-to-Fine Hyper-Prior Modeling for Learned Image Compression [paper]

2019

NeurIPS-2019

• Model Compression with Adversarial Robustness: A Unified Optimization Framework [paper] [arxiv] [paper with code] [code]

CVPR-2019

• ComDefend: An Efficient Image Compression Model to Defend Adversarial Examples [paper] [arxiv] [paper with code] [code]

• Cross Domain Model Compression by Structurally Weight Sharing [paper] [paper with code]

• ECC: Platform-Independent Energy-Constrained Deep Neural Network Compression via a Bilinear Regression Model [paper] [arxiv] [paper with code] [code]

ICLR-2019

• Context-adaptive Entropy Model for End-to-end Optimized Image Compression [paper] [arxiv] [paper with code] [code] [openreview]

• Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters [paper] [arxiv] [paper with code] [code] [openreview]

• Integer Networks for Data Compression with Latent-Variable Models [paper] [paper with code] [openreview]

ICCV-2019

• Adversarial Robustness vs. Model Compression, or Both? [paper] [paper with code] [code]

ICML-2019

• Rate Distortion For Model Compression:From Theory To Practice [paper]

• LIT: Learned Intermediate Representation Training for Model Compression [paper] [paper with code] [code]

IJCAI-2019

• Play and Prune: Adaptive Filter Pruning for Deep Model Compression [paper] [arxiv] [paper with code] [code]

• COP: Customized Deep Model Compression via Regularized Correlation-Based Filter-Level Pruning [paper] [arxiv] [paper with code] [code]

AAAI-2019

• Private Model Compression via Knowledge Distillation [paper] [arxiv] [paper with code]

2018

NeurIPS-2018

• Deep Generative Models for Distribution-Preserving Lossy Compression [paper] [arxiv] [paper with code] [code]

CVPR-2018

• Conditional Probability Models for Deep Image Compression [paper] [arxiv] [paper with code] [code]

ICLR-2018

• Model compression via distillation and quantization [paper] [arxiv] [paper with code] [code] [openreview]

ECCV-2018

• AMC: AutoML for Model Compression and Acceleration on Mobile Devices [paper] [arxiv] [paper with code] [code]

ICML-2018

• Modeling Sparse Deviations for Compressed Sensing using Generative Models [paper] [arxiv] [paper with code] [code]

ACL-2018

• A Language Model based Evaluator for Sentence Compression [paper] [paper with code]