What colour keying actually does
Keying compares every pixel against one colour. Anything within the tolerance becomes transparent; everything else is kept. That is the whole algorithm, and its strengths and its limits both follow directly from it.
It is excellent when the background is genuinely one colour and the subject is not: a product on a white sweep, a logo on a flat field, a scanned signature on paper, a portrait against a studio backdrop. In those cases the edge is clean because there really is a hard boundary to find.
It fails where the boundary is soft or the colours overlap. Hair against foliage has thousands of pixels that are genuinely part background and part subject, and no single threshold separates them. The result is a ragged edge — and it is better to say so than to ship a tool that produces one and calls it a cutout.
Flood from the edges, or key everything
Two modes, and the difference is more than cosmetic. Flood from the edges starts at the border and spreads inward through matching pixels, so it only removes background that is actually connected to the outside.
Key every matching pixel removes the colour wherever it appears — including inside the subject. A white mug on a white background loses its white handle-hole under flood, and keeps it; keys everything and the hole goes transparent too.
Which you want depends on the picture. Flood is the safer default and is what most product shots need. Key-everything is right when the subject genuinely has no background-coloured areas, and it deals better with a background interrupted by the subject.
Tolerance, feathering and the edge
Tolerance is how far from the picked colour still counts as background. Too low and you leave a halo of near-background pixels; too high and you start eating the subject. A gently graded backdrop needs more tolerance than a flat one, which is why the setting exists rather than being fixed.
Feathering softens the boundary by a pixel or two. It makes the edge look natural against a new background instead of cut out with scissors, and it is usually worth a small amount even on a clean key.
Work at the edge rather than at the middle. The middle of a plain background will key at almost any setting; the last few pixels beside the subject are where the choice actually shows.
Transparency needs the right format
JPEG cannot store transparency. Save a keyed image as JPEG and the transparent areas come back white — the alpha channel is simply discarded, and the picture looks like the tool did nothing.
PNG and WebP both store alpha. WebP is usually much smaller for the same quality and every current browser reads it; PNG is the safer choice for anything going into older software.
If you do want a JPEG, put a flat colour behind the subject instead of leaving it transparent. The tool will do that for you, and it tells you which formats can carry what.