Katalon test cases are executed perfectly in Browser mode and Console mode. But some test cases failed when execute via Bamboo agent

I guess you tried Browser mode and Console mode on your PC, and you executed Bamboo agent on another box (Linux?).

Have you installed Xvfb in the machine where Boomboo agent runs?

See the following post:

this post describes how to setup runtime environment for Jenkins, not Bamboo. But it will be still informative for you how the OS environement should be setup for CI tools in general.