Problem with window frame

I am trying to capture an element (with the ok and cancel buttons) that looks like a pop-up but it actually is window frame. It is either impossible to capture element (you click the button and nothing happens) or you can capture it but when you run a test then it says element not found

Katalon has built-in keyword for keyboard events. ‘Send Keys’
Here’s the sample code
WebUI.sendKeys(findTestObject('Page_Login/btn_Login'), 'Enter')

Here’s the reference API url: https://docs.katalon.com/display/KD/%5BWebUI%5D+Send+Keys

The accept alert did not work; I get ‘no alert found error’.

I noticed that if the dialog appears and I click enter from the keyboard I can progress.

Do you know how to simulate pressing enter from the keyboard?

I assume it’s an alert since you’ve stated that a pop-up with Yes or No. If yes try using Accept Alert Keyword.
If its in frame, you need to swtich to frame first and perform an action and then switch to default content