More information on Wikipedia.
Engine braking occurs when the retarding forces within an engine are used to slow a vehicle down, as opposed to using additional external braking mechanisms such as friction brakes or magnetic brakes.
The term is often confused with several other types of braking, most notably compression-release braking or "jake braking" which uses a different mechanism. The term only applies to gasoline engines and other throttled engines (as opposed to diesel engines, electric motors, etc.).
Additionally, traffic regulations in a large number of countries require trucks to always drive with an engaged gear, which in turn provides a certain amount of engine braking (viscous losses to the engine oil and air pumped through the engine and friction losses to the cylinder walls and bearings) when no accelerator pedal is applied.
The term 'engine braking' refers to the braking effect that occurs in gasoline engines when the accelerator pedal is released.
When the accelerator is released, the butterfly valve that controls intake airflow (the throttle valve) is closed and the air flow through the intake is greatly restricted (but not cut off completely). This causes a high manifold vacuum which the cylinders have to work against - this saps energy and is where the majority of the engine braking force comes from. The concept can be illustrated by the amount of effort required to blow/suck through a narrow tube vs. a wider one. It is the work the engine has to do against this restricted air flow that provides the braking effect.
While some of the braking force is due to friction in the drive train, this is negligible compared to the effect from the manifold vacuum caused by the air-flow restriction.
Diesel engines do not have engine braking in the above sense. Unlike gasoline engines, diesel engines vary fuel flow to control power, rather than throttling air intake and maintaining a constant fuel ratio as gasoline engines do. Because they do not maintain a throttle vacuum, they are not subjected to the same engine braking effects. This is partly why non-turbo diesel engined vehicles can coast in-gear for longer than an equivalent gasoline engine.
The higher compression ratio in diesels means they are harder to start, but once they are running the energy expended in compressing air is regained during the expansion stroke when the compressed air is allowed to 'spring' back, so the higher compression ratio doesn't cause any engine braking.
Compression release brake
A compression release brake or "jake brake", is the type of brake most commonly confused with real engine braking; it is used mainly in large diesel trucks and works by opening the exhaust valves at the top of the compression stroke, so the large amount of energy stored in that compressed air is not returned to the crankshaft, but is released into the atmosphere.
Normally during the compression stroke, energy is used as the upward-traveling piston compresses air in the cylinder; the compressed air then acts as a compressed spring and pushes the piston back down. However, with the jake brake in operation, the compressed air is suddenly released just before the piston begins its downward travel. (This sudden release of compressed air creates audible sound waves similar to the expanding gases escaping from the muzzle of a firearm.) Having lost the energy stored within the compressed air, there is no 'spring back' from it so the engine must expend yet more energy pulling the piston back down again
This type of brake is banned or restricted in many locations where people live because it creates a machine gun like sound loud enough to disturb the peace, including waking people at night. It is very effective however, and creates immense amounts of braking force which significantly extends friction brake life - a 565 hp (421 kW) diesel engine can produce up to 600 hp (450 kW) of braking power.[citation needed
An exhaust brake works by causing a restriction in the exhaust, much like the intake throttle causes in a gasoline engine. In simple terms, it works by increasing the back-pressure of the exhaust. Nearly all of these brakes are butterfly valves similar to a throttle valve, mounted downstream of the turbocharger if there is one.