“Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures” was published at the ACM CCS conference on October 12-16, 2015 by Matt Fredrikson (Carnegie Mellon University), Somesh Jha (University of Wisconsin-Madison), and Thomas Ristenpart (Cornell Tech). It showed that the confidence scores a model returns alongside its predictions can leak information about the data the model was trained on.
A model inversion attack does not try to fool the model; it tries to read sensitive information back out of it. The authors developed attacks that exploit the numeric confidence values exposed by prediction APIs. In one striking demonstration against a facial-recognition model, they reconstructed a recognizable image of a person’s face given only the person’s name and black-box access to the model. They also showed attacks that infer sensitive attributes, such as whether a respondent in a lifestyle survey admitted to a private behavior, from decision-tree models offered as a service. The paper then explored simple countermeasures, like rounding or omitting confidence values, that reduce leakage with little loss of utility.
The work made concrete a privacy risk that is easy to overlook: even when a model is supposed to only output a label or a class, the extra detail it provides about its confidence can be turned into a reconstruction of private inputs.
For a business reader, model inversion is a reason to treat a deployed model as a potential disclosure channel. If a model was trained on personal data, the very richness of its outputs can let an adversary recover aspects of that data, which has clear implications for privacy compliance and how much detail an API should reveal.