The matches(string, regex) function was added in the XPath 2.0 specification. Browsers only supports the XPath 1.0. You can not use any Regular Expression in XPath on browser. It is not a defect of Katalon Studio at all.
However you can write an XPath expression equivalent to an regexp using XPath 1.0 string functions: substring(s, pos, len), substring-before(s1, s2), substring-after(s1, s2), string-length(s), translate(s1, s2, s3) in combination.