There are non-standard models of arithmetic. They follow the original first-order Peano axioms and any theorem about the naturals is true for them, but they have some wacky extra stuff in them like you mention.