Once you can actually run an agent against a real repository, the powerful parts and the risky parts arrive together. This page collects three families of risk you will run into in practice: copyright and intellectual property, privacy and data exposure, and harm from unreliable output.
Earlier pages teach how to operate the tools. This page is the reflection layer on top: same tools, viewed through what they can cost you if you use them without care.
After working through this page, students should be better able to:
Hosted models were trained on large bodies of code under many different licences. The model itself does not track which licence produced which token, and neither does its output. That has practical consequences when you paste generated code into a real project.
A model can produce code that resembles training material very closely, especially for short, idiomatic snippets and well-known algorithms. If that training material was under a copyleft licence such as GPL, and you paste the output verbatim into a permissively licensed repository, you may have created a licence problem that no one in the project actually intended.
The risk is not “the agent will write GPL code at you on purpose”. The risk is that no one notices, because the snippet is short and reads like ordinary code.
Practical defaults:
LICENSE file and any contribution guidelines before
using an agent on a project you do not ownThe legal picture differs by jurisdiction and is still moving. In many places, pure model output is not automatically copyrightable, because there is no human author. The human edits, structure, and integration around the output typically are.
For a student project this matters less than it does for a company, but it still shows up in two everyday cases:
Even when nothing legally requires it, disclosing AI assistance is becoming the expected default in education and in serious open source work. A short note in the pull request description or commit message is usually enough.
Disclosure is not an admission that something is wrong with using an agent. It is information the next reader needs in order to review the change properly.
Every prompt you send to a hosted provider is a piece of data leaving your machine. Every file you let an agent open is a piece of data the agent can read back into the prompt. This is easy to forget once the tool feels routine.
A hosted coding agent typically sees:
That is fine for ordinary public code. It is not always fine for the rest of what lives in your home directory.
Things that should not end up in a hosted prompt include:
.envA good habit is to assume the agent will eventually read everything in the
working directory. If a file should not leave your machine, do not have it
sitting next to code an agent is allowed to open. Treat .gitignore-style
hygiene as a privacy tool, not only a version-control tool.
If you are in the EU, sending another person’s personal data through a hosted provider is a data transfer with legal consequences under the GDPR. This is not unique to LLMs, but agents make it especially easy to do without realising, because they read whole files at a time.
Concretely, do not paste class lists, student emails, grade spreadsheets, or similar material into a hosted chat or hosted coding agent. If you need a language model to help with that kind of material at all, run it locally.
For sensitive material, the privacy answer is usually to move off the hosted route. See Local models and Ollama for the practical setup. Local does not automatically mean correct, but it does mean the data does not leave your machine.
A second privacy- and reliability-adjacent risk is prompt injection: content the agent reads — a webpage, an issue comment, a file in the repository — can contain instructions aimed at the agent rather than at you. The model has no reliable way to tell instructions from data.
Simon Willison’s lethal trifecta names the conditions under which prompt injection becomes especially dangerous: an agent that simultaneously has (a) access to your private data, (b) exposure to untrusted input, and (c) the ability to communicate externally (post comments, push commits, send HTTP requests). Each leg alone is usually manageable; together they let an attacker exfiltrate private data through the agent.
Practical defaults:
For the wider discussion, see Reflections.
llm logs prompts and responses to a local SQLite database by default. That
is convenient until the prompts contained something sensitive. See the
logging commands in the llm guide (llm logs off,
llm logs status) and turn logging off before working on sensitive material.
A model can sound completely confident and still be wrong. When wrong output is acted on without checking, the harm is not in the model — it is in what the model talked you into doing.
Common failure modes you should expect:
The model is not lying. It is producing plausible text. Plausibility is not the same as truth.
Some practitioners go further and treat every generation as a hallucination — the model is always producing plausible text, and the useful question is which of those generations happen to be true. Framing it that way is a reminder that verification is not an exceptional step you take when something looks wrong; it is the default. See Reflections for the wider discussion around this idea.
A more dangerous case is code that runs but is wrong in a way you cannot see at a glance:
The code passes a quick read, often passes a quick test, and still ships a security bug. This is one of the strongest reasons to treat agent output as a draft rather than a finished answer.
The quiet failure mode is the one where nothing visible breaks. The student runs the agent, the answer sounds good, the student stops checking. Over time, the habit of verifying erodes, and small errors accumulate into a codebase or an essay that nobody actually understands.
This pattern has several names in the wider discussion. Joshua Bloom calls the passive-acceptance side of it stuporous acquiescence. Margaret-Anne Storey describes the longer-term loss of skill at team level as cognitive debt; Addy Osmani uses comprehension debt for the related case where the codebase grows faster than anyone’s understanding of it. A useful umbrella term is decoupled comprehension: the code and your grasp of it have drifted apart. The Reflections page collects the wider conversation.
The mitigation is the same workflow taught earlier in this guide: treat each agent claim as a draft until evidence supports it. See Verification for the operational version of this.