
Math problems
I am having a hell of a time doing some real simple math:<BR><BR>See if you can tell me why it does this.<BR><BR>Dim dd<BR>'########## answer = 5454545454545454 <BR>'But it print 5454545454545450 *** Notice the last number is 0<BR>dd = 5454545454545454*186<BR>dd = dd/186<BR>dd=formatnumber(dd,0)<BR>Response.write(rep lace(dd,",",""))<BR>

You need a course in computer systems...
I would guess you have never taken a CompSci course on "how computers work", yes?<BR><BR>Computers typically do math either with integers or "real numbers".<BR><BR>Computers typically have at least two ranges for integers, 32768 to +32767 (short integers) and 2147483648 to +2147483647 (long integers).<BR><BR>Calculations done using whole numbers (integers) inside those ranges will be "spot on".<BR><BR>Once you get outside those rangesor once you use any nonintegers (e.g., 1.5, 3.25, etc.)you are into the realm of "real numbers" or "floating point numbers".<BR><BR>The IEEE and ANSI standard for floating point numbers provides for two standard representations, one occupying 4 bytes of memory (on a typical machine and certainly on a Pentium) and one occupying 8 bytes of memory.<BR><BR>Now, how many digits of accuracy do you think can be held in only 8 bytes of memory? It turns out the answer varies a little bit, but it is *roughly* 15 digits.<BR><BR>Once you get beyond 14 digits, actually, it begins to get dicey, depending upon the operations you perform.<BR><BR>So what happened to you? You gave a number with 16 digits. But the computer could only remember 15 of them. So it said to itself "well, I can only remember 15 digits, but I *can* remember that the actual number had another digit on the end, so I'll remember that as 5.45454545454545 x 10^15." That is, the computer remembers as many "significant digits" as it can (about 15) and also remembers the "magnitude" of the number separately, so that at least it can give you an approximately correct answer.<BR><BR>Incidentally: Actually, all of this is done in binary arithmetic internally, so it actually is keeping something like 1.0010101110 x 2^52. But the idea is right.<BR><BR>This is how *ALL* computers work. No exception (well, years ago there were exceptions, but the IEEE/ANSI standard has eliminated them). <BR><BR>There are programming *languages* (Java is one of them, but not JavaScript) that have "extended precision" arithmetic implemented, but it is all done in software, not in the hardware of the computer.<BR><BR>So...<BR><BR>Give it up. You've got the best answer you can.<BR><BR>
Posting Permissions
 You may not post new threads
 You may not post replies
 You may not post attachments
 You may not edit your posts

Forum Rules

