Glossary definition of 'Noise shaping'The process by which quantization and dither noise is processed in order to make it less audible. This is mostly done as part of a bit reduction scheme, for example, creating a 16 bit CD from a 24 bit master. The dither noise and quantization noise from this process has a flat (white) spectrum which results in a noise similar to tape hiss. The ear is more sensitive to some frequencies than others, so by shaping the frequency content of the noise, we can shift it to frequencies where the ear is least sensitive so as to give an improvement in subjective signal to noise ratio.
It is possible to reduce the subjective effect of added dither noise by simply using spectrally weighted ('blue') dither noise, which is quieter in the more sensitive registers. Noise shaping is just like conventional dithering, except that the error signal generated when the unwanted low-order bits are discarded is filtered and subtracted from the input signal. You can't get something for nothing – the error cannot be simply cancelled out, because we already know that the output hasn’t got enough bits to precisely represent the input. But by choosing an appropriate shape for the error filter, we can force the dither noise / error signal to adopt the desired shape in the frequency domain – we usually choose a shape which tracks the low-field perception threshold of the human ear against frequency. This has the effect of actually lowering the noise floor in the more sensitive frequency bands when compared to the flat dither case.
The theory of noise shaping has been around for a long time – certainly since well before DSP in real-time was feasible for audio signals. It has applications in many signal processing and data conversion applications outside audio. It has been well researched, and is not in the least bit mysterious. ‘Proprietary’ wordlength reduction algorithms are generally conventional noise shapers. Assuming that the basic implementation and dither levels are correct, the only significant freedoms available to the designer are to choose the actual shape of the noise floor, and to decide how to adapt this (if at all) to different sample rates.