Data-Free Knowledge Distillation for Deep Neural Networks
ABSTRACT
Recent advances in model compression have provided
procedures for compressing large neural networks to a
fraction of their original size while retaining most if
not all of their accuracy. However, all of these
approaches rely on access to the original training set,
which might not always be possible if the network to be
compressed was trained on a very large dataset, or on a
dataset whose release poses privacy or safety concerns as
may be the case for biometrics tasks. We present a method
for data-free knowledge distillation, which is able to
compress deep neural networks trained on large-scale
datasets to a fraction of their size leveraging only some
extra metadata to be provided with a pretrained model
release. We also explore different kinds of metadata that
can be used with our method, and discuss tradeoffs
involved in using each of them.
AUTHORS