When you define a foreign key, the database registers a constraint. On every INSERT or UPDATE to the child table, the engine checks that the referenced value exists in the parent. On every DELETE or UPDATE to the parent, it checks whether any child rows still point to the row you are touching.
To make these checks fast, the parent side relies on the index that already backs its primary key. On the child side you usually want your own index on the foreign key column, because the engine looks up child rows during parent deletes and joins. Postgres, for example, does not auto-create that child index, so forgetting it can make deletes on big parent tables painfully slow.
You also choose what happens when a parent row is removed. ON DELETE RESTRICT, or NO ACTION, which is the actual default and behaves almost the same, blocks the delete if children exist. ON DELETE CASCADE deletes the children too. ON DELETE SET NULL clears the child reference. So deleting a customer can either be refused, wipe their orders, or detach the orders, depending on the rule you picked.