There are two part of formula:
First part: If Lower Layer Value > 127.5, then do the following -
Value Unit = (255-Lower Layer Value)/127.5
Min Value = Lower Layer Value - (255-Lower Layer Value)
Overlay = (Upper Layer Value * Value Unit) + Min Value
Second part: If Lower Layer Value < 127.5, then do the following -
Value Unit=Lower Layer Value/127.5
Overlay = Upper Layer Value * Value Unit
From the formual we can see that the final result is much depend on the upper layer value. If the upper layer value is higher(lighter), then the final result is more likely to be lighter.
This mode is useful for color enhancement or special effects.
To understand Overlay, the best way is to understand its fairly complex mathematic formula. Before we talk about the formula, let's look at the Overylay blending example below:
Instead of immediately showing you how to do a cool layer mode trick that might be rarely use by you, we will explain the true mechanism working behind every layer mode. By fully understanding the fundamental of every layer mode, you will be able to apply layer mode into any problem you might run into when the time come.
Layer mode is about blending the content of two or more layers together to produce intended result. The result of one blending can be both final artistically result or intermediate result that use for further image correction or enchancement.
Layer mode blending the content of layers by calculating the Red, Green and Blue value of every pixels against its lower layer Red, Green and Blue value. Almost all layer mode do calculation on each color channel separately, that is mean upper layer Red value will be only use for calculating against lower layer Red value, and same thing happen to Green and Blue.
While most of the layer mode do calculation on RGB channel, some layer mode do use HSB(Hue, saturation and brightness) for the calculation, e.g. Hue, saturation, color and luminosity mode.