A large language model predicts the next word. That is not a simplification hiding something cleverer — it is the whole mechanism, and once you hold it in your head, most of what these tools do well and badly stops being surprising. They produce the most plausible continuation of the text so far, every time, whether or not the underlying fact is real. This lesson is about what that makes them good at, what it makes them dangerous at, and the two numbers a business owner should carry.
Fluency Is Not Knowledge
Because the model always produces a fluent continuation, it produces one even when it has no idea. A confident, well-written wrong answer is not a malfunction — it is the system working exactly as designed, because "sounds right" is precisely what it optimises for. This is usually called hallucination, which makes it sound like a rare glitch. It is not rare and it is not a glitch; it is the flip side of the thing that makes the tool useful.
The practical consequence is that you cannot use the tone to filter. The model's confidence is identical whether the answer is right or invented:
— At 95 percent accuracy, 5 percent of confident-sounding answers are wrong — and look exactly like the right ones — At 85 percent accuracy, 15 percent are wrong — At 70 percent accuracy, 30 percent are wrong
There is no signal in the writing that separates the reliable answer from the fabricated one. Any process that trusts the output because it "sounds sure" is trusting the one property that carries no information about correctness.
Reliability Compounds Against You
Here is the number that decides whether you can automate a whole task or only a step of one. A multi-step job is right end-to-end only if every step is right, so per-step accuracy compounds:
— A 95 percent-per-step model: 3 steps 85.7 percent, 10 steps 59.9 percent, 20 steps 35.8 percent — A 90 percent-per-step model: 10 steps 34.9 percent, 20 steps 12.2 percent — Even a 99 percent-per-step model: 20 steps 81.8 percent
Read the first line again. A 95 percent-per-step model is genuinely good — better than most tasks need in isolation — and it still gets a 20-step job right only about a third of the time. This is why "point it at my entire workflow and walk away" does not work: the errors accumulate, and a long autonomous chain is a product of many numbers below one. The tool is far more reliable as a fast assistant on single steps a human checks than as an unattended operator of a long process.
The Line Between Safe And Dangerous Uses
Put the two facts together and a clean rule falls out. The model is strong when you give it the ground truth and ask it to transform that; it is risky when you ask it to be the ground truth.
— Strong: summarise this document, rewrite this email, translate this text, extract the figures from this table, draft a first version of this. The source is in front of it, so there is little to invent. — Risky: what is the deadline, what does this law say, what were last year's numbers, cite a source. Here it is reaching into memory and will emit a plausible answer rather than a looked-up one.
Almost every reliable business use of these tools is a transformation of text you supplied. Almost every embarrassing failure is a request to recall a fact it was never reliably given. If you can arrange for the answer to be present in the input, you have moved the task from the dangerous column to the safe one — which is what techniques like retrieval and "here is the document, answer only from it" are really doing.
The Catch
The accuracy figures above are illustrative — I am modelling the structure, not quoting a benchmark, and real accuracy depends enormously on the task, the model and how you prompt it. The compounding maths also assumes independent steps; in practice errors can correlate or a later step can catch an earlier one, which softens the decay. Treat 95-percent-to-the-tenth as the shape of the problem, not a measured failure rate for your specific case.
The capabilities are also a moving target. Models improve, tool-use and retrieval close much of the recall gap, and a system built to check its own work behaves better than a raw model. The point is not that these tools are unreliable forever; it is that the naive use — trust the fluent output, chain many steps unattended — fights the mechanism, and understanding the mechanism tells you how to use them well.
And none of this is a reason to avoid them. A tool that drafts, summarises and transforms at speed is genuinely valuable. It is a reason to put a human at the checkpoint where correctness matters, not to remove the human because the output reads confidently.
Do This
— Sort your intended use into "transform text I provide" or "recall a fact from memory". The first is where these tools earn their keep; the second is where they embarrass you. — For any workflow you want to automate, count the steps and multiply an honest per-step accuracy. If the end-to-end number is uncomfortable, keep a human at the steps that matter rather than automating the whole chain. — Never let confidence stand in for verification. The fluent wrong answer looks exactly like the fluent right one, so the check has to be external — a source, a test, a person — not the tone of the reply.
This is education, not financial advice. It's general information, not a recommendation for your situation, and it doesn't account for your circumstances. Rules and figures change — verify before you act, and speak to a qualified professional for decisions that matter.
Go deeper at Agent Smith
FNZ keeps this track short and practical on purpose. For the full treatment, see the Agent Smith academy, for how these systems are actually built and bounded.