[rt #69261] fix incorrect image size and color calculations for rotate()

[rt #69261] fix incorrect image size and color calculations for rotate()

and matrix_transform().

For rotate() the calculation used integer abs() instead of fabs() when
calculating the size of the new image.

This made the image a little too small, hiding a bug at the corners
where a very low coverage pixel would reveal another bug where bad
rounding of the output level could result in all samples being maxed
out (8-bit images only.)