Fun with Garbage-in-Garbage-out

Large Language Models and Bad Data

This is probably the first of a variety of articles on large language models (LLMs) and their ilk. I find ChatGPT and friends to be completely fascinating, not just for how they seem eerily intelligent, but for now much people trust them. Whenever something seems human, we desperately want it to be human. So we attribute human qualities where none exist. This post is another one that was stimulated by discussions on TechYeet about the use of ChatGPT by cybercriminals.

The data that an LLM ingests determines the output. This should come as no surprise to anyone who’s worked with software or in security for that matter. It does seem to come as a surprise to the general public. Take, for example, the recent unpleasantness with a chat bot that was supposed to suggest recipes. With some input shenanigans, it suggested such wonderful dishes as “poison bread sandwich” and my personal favorite “mysterious meat stew,” which contains, among other things, “500g human flesh, chopped.” No fava beans or chianti required.

The real problem here comes when the outputs are not quite so obviously problematic. Let’s take medical diagnosis as an example. People are all too ready to entrust this to seemingly intelligent models. This works out much better if the person reading the outputs is a trained medical professional who’s being assisted by software. It does not work out well at all if an untrained person blindly believes what the software is suggesting. Just like it wouldn’t have worked out well for the consumer of a poison bread sandwich.

So how is this security related? We as security professionals need to be aware of the potential issues with LLMs and how they may affect the security of the systems our organizations use or plan to use. The OWASP Top 10 for LLMs is a good place to start. I plan to mull this over a lot more this year. I hope you do too.