Precision

The precision for a value map or a value column determines the 'resolution' of possible values in the value map or value column. All values will be rounded so that they will fit the specified precision.

 

If you specify that a value map or a value column should use:

For the precision of a value map or a value column you can choose any figure you like, also 2, 5, or 250.

During calculations or operations, the system will come up with a suggestion for the precision of the output map or column. If allowed by the domain that you selected for the output object, you can adjust the precision yourself.

You cannot specify the precision for the values in an output map or column when you select a domain that has a fixed precision (e.g. system domains Image, NOAA, or Radar).

The smaller you choose the precision of a map, the larger the size of the map will be (in bytes on disk), as more bytes will be needed to store each value in the map. Values in value maps can be stored as fixed point real numbers (using 1 byte, 2 bytes, 4 bytes per pixel) or as floating point real numbers (using 8 bytes per pixel). The display of maps using 1 byte per pixel is fast and the display of 8 byte maps is slower, also calculation time may increase.

See also: