I was able to connect to a WebSocket server on my localhost with the following website example. https://www.eclipse.org/jetty/documentation/9.1.5.v20140505/jetty-websocket-client-api.html
I implemented the two classes shown in the example URL as keywords in Katalon, and then I used SimpleEchoClient keyword in a test script to connect and send and receive messages to the server. Adapting the SimpleEchoSocket file should get you something close to your specific needs.