Handle Floating Point

Hi guys, I want to ask you about floating point in javascript. I know from the other source for using toFixed() and toPrecision() to handle that. But when I give it a try, it just failed:

FAILED because (of) groovy.lang.MissingMethodException: No signature of method: java.lang.String.toPrecision() is applicable for argument types: (java.lang.Integer) values: [3]

I want to tell u about my case.
- I have an input 10000000 from my data filed and I also have an object with value 156,041.38
- I tried to getAttribute for value from the object, and I got it 156,041.38 in string
- cause I need to calculate that value, I must manipulate and parse it into float, and it successfully parsed with result 156041.38
- next step, I will calculate the value 10000000 - 156041.38
- the problem appear here. that calculate result must be 9,843,958.62, but the katalon just shown me result 9843958.625, which is maybe cause by the floating point

So, I need some suggestion, how can I control that floating point with 2 decimal? I tried round(num*100)/100 but it didn’t work for me. The only solution I think to use toFixed or toPrecision, but the keyword didn’t work at all (or maybe I need import some library again to use it?)

Your suggestion will be great for me, thank you :slight_smile:

correction, that calculate result must be 9843958.62, but the katalon just shown
me result 9843958.625, which is maybe cause by the floating point

Hi Willis,

I use NumberFormat class for such situations. Just set both min and max to 2 fraction digits. Check this:

import java.text.NumberFormatNumberFormat nf = NumberFormat.getInstance()nf.setMinimumFractionDigits(2)nf.setMaximumFractionDigits(2)String result = nf.format(10000000 - 156041.38)

Hi @Marek Melocik
thanks for the suggestion, it worked in String very well. But how if I’m using Float?

NumberFormat nf = NumberFormat.getInstance()
nf.setMinimumFractionDigits(2)
nf.setMaximumFractionDigits(2)
Float a = 10000000
Float b = 156041.38
Float total = 0
total = nf.format(a-b)
println('total = '+total)

it will give me an error “org.codehaus.groovy.runtime.typehandling.GroovyCastException: Cannot cast object ‘9,843,958.62’ with class ‘java.lang.String’ to class ‘java.lang.Float’”

is it possible to using NumberFormat for Float type and still give the correct precision?

I see. This is updated solution:

import java.text.NumberFormatNumberFormat nf = NumberFormat.getInstance()nf.setMinimumFractionDigits(2)nf.setMaximumFractionDigits(2)Double a = 10000000Double b = 156041.38Double total = 0String formattedDouble = nf.format(a-b).replaceAll(",", "")total = Double.parseDouble(formattedDouble)println('total = '+total)

I used Double instead of Float as Double has higher precision and such big numbers behave incorrectly with Float.

You can see there is formattedDouble String, which is your formatted result AND with thousands separators/commas removed (Double.parseDouble cannot parse such a number). Finally, total variable contains Double value of your result.

@Marek Melocik
Ahh, I think it will be worked for me. Thanks for the suggestion, I will give it a try. I will tell you if it worked well or not cause I need to modify my whole script to reach that condition.

Thanks for you help, really appreciate it :slight_smile:

Good to hear! Feel free to add more questions here if something is not clear.

1 Like