Because a number isn’t just a representation of a size or amount - that’s called a scalar. A number can also represent a point in a space, the label of a vertex on a graph and probably some other things too.

Wouldnt it be best to think of it more as the representation of the absence of something?