Coagulation is the process where proteins unfold and bond (or stick) together when exposed to heat, acid, or certain other ingredients. This changes the structure of the proteins from a liquid or soft state into a firmer one. In baking, coagulation of proteins helps set the structure.
You'll often hear me say it "creates an inner scaffolding," which supports the dessert while starches, fats, sugars, and other ingredients fill in the spaces around it. Examples of important proteins that commonly perform this function include egg-based proteins, gluten (from wheat-based flours), milk-based proteins, gelatin (collagen from animal bones) and legume-based proteins (typically used in vegan baking, such as chickpea powders).