Precision
Computer words consist of a finite number of bits. This means that the binary encoding of variables is only an approximation of an arbitrarily precise real-world value. Therefore, the limitations of the binary representation automatically introduce limitations on the precision of the value.
Topics
The range of a number gives the limits of the representation, while the precision gives the distance between successive numbers in the representation. The range and precision of a fixed-point number depend on the length of the word and the scaling.
To maximize precision, make the slope as small as possible while keeping the range adequately large.
Rounding involves going from high precision to lower precision and produces quantization errors and computational noise.
- Rounding Mode: Ceiling
- Rounding Mode: Convergent
- Rounding Mode: Floor
- Rounding Mode: Nearest
- Rounding Mode: Round
- Rounding Mode: Simplest
- Rounding Mode: Zero
Rounding Modes for Fixed-Point Simulink Blocks
Fixed-point Simulink® blocks support seven different rounding modes.
Net Slope and Net Bias Precision
Net slope and bias precision, detecting precision loss, underflow, and overflow
Detect Fixed-Point Constant Precision Loss
This example shows how to detect fixed-point constant precision loss.
Use Scaled Doubles to Avoid Precision Loss
How to avoid precision loss by overriding the data types in your model with scaled doubles.
