an algorithm is said to be numerically unstable if real-numbers were used throughout the calculation, but where some approximation can be meaningless if fixed-precision floating-point arithmetic is used

for example, calculating the inverse of a matrix is never numerically stable. there may be certain matrices for which calculating the inverse is straight-forward; for example, calculating the inverse of a diagonal matrix with no non-zero entries on the diagonal, but there is no algorithm that will even guarantee an inversion on even a tri-diagonal matrix (one with non-zero entries both in the super- and sub-diagonals as well as on the diagonal

conditioning

a problem is called ill-conditioned (opposite: well-conditioned) if a relative change in the input can produce a much larger relative change in the solution. the condition number of a problem is

          cond = |relative change in solution| / |relative change in input|  

A problem is ill-conditioned, if cond >> 1.

Evaluating functions

Evaluate function f for approximate input value x̂ = x + h instead of true input value x:

   absolute error : f (x + h) − f (x) ≈ f' (x) · h 
   relative error : (f (x + h) − f (x)) / f (x) ≈ f'(x) * h / f (x) 
   cond = |x * f'(x) / f(x)| 

For f (x) = cos (x) , x → π/2 : cond → ∞

Programming with Floating Point Numbers

When programming, we have to specify which format of floating point number to use:

IEEE	C, C++	FORTRAN
single	float	REAL or REAL*4
double	double	DOUBLE PRECISION or REAL*8
double extended	long double	REAL*16