I checked in Katalon Studio 5.6 on Mac with JIS Keyboard and found that I can not capture anything in Spy Web with Alt+`.
However, as http://docs.katalon.com/display/KD/Object+Spy+Preferences describes, you can allocate other key code for Spy Web’s Hot Key. So I allocated Alt+a and it worked OK.
-------------------
As far as I see, Alt+` as Spy Web Hot Key never works for those who are using NON-US keyboard. Why? I have described my thought about this problem in the following post — it depends on Keyboard type you are using:
I regret that so many people who are new to Katalon Studio and who are using NON-US Keyboard get stuck (as I did previously) bacuase Alt+` does not work.