Math problems

# Thread: Math problems

1. Rob Guest

## Math problems

I am having a hell of a time doing some real simple math:<BR><BR>See if you can tell me why it does this.<BR><BR>Dim dd<BR>&#039########## answer = 5454545454545454 <BR>&#039But it print 5454545454545450 *** Notice the last number is 0<BR>dd = 5454545454545454*186<BR>dd = dd/186<BR>dd=formatnumber(dd,0)<BR>Response.write(rep lace(dd,",",""))<BR>

2. Senior Member
Join Date
Dec 1969
Posts
96,118

## You need a course in computer systems...

I would guess you have never taken a CompSci course on "how computers work", yes?<BR><BR>Computers typically do math either with integers or "real numbers".<BR><BR>Computers typically have at least two ranges for integers, -32768 to +32767 (short integers) and -2147483648 to +2147483647 (long integers).<BR><BR>Calculations done using whole numbers (integers) inside those ranges will be "spot on".<BR><BR>Once you get outside those ranges--or once you use any non-integers (e.g., 1.5, 3.25, etc.)--you are into the realm of "real numbers" or "floating point numbers".<BR><BR>The IEEE and ANSI standard for floating point numbers provides for two standard representations, one occupying 4 bytes of memory (on a typical machine and certainly on a Pentium) and one occupying 8 bytes of memory.<BR><BR>Now, how many digits of accuracy do you think can be held in only 8 bytes of memory? It turns out the answer varies a little bit, but it is *roughly* 15 digits.<BR><BR>Once you get beyond 14 digits, actually, it begins to get dicey, depending upon the operations you perform.<BR><BR>So what happened to you? You gave a number with 16 digits. But the computer could only remember 15 of them. So it said to itself "well, I can only remember 15 digits, but I *can* remember that the actual number had another digit on the end, so I&#039ll remember that as 5.45454545454545 x 10^15." That is, the computer remembers as many "significant digits" as it can (about 15) and also remembers the "magnitude" of the number separately, so that at least it can give you an approximately correct answer.<BR><BR>Incidentally: Actually, all of this is done in binary arithmetic internally, so it actually is keeping something like 1.0010101110 x 2^52. But the idea is right.<BR><BR>This is how *ALL* computers work. No exception (well, years ago there were exceptions, but the IEEE/ANSI standard has eliminated them). <BR><BR>There are programming *languages* (Java is one of them, but not JavaScript) that have "extended precision" arithmetic implemented, but it is all done in software, not in the hardware of the computer.<BR><BR>So...<BR><BR>Give it up. You&#039ve got the best answer you can.<BR><BR>

#### Posting Permissions

• You may not post new threads
• You may not post replies
• You may not post attachments
• You may not edit your posts
•