You can read the source code of WebUI.takeFullPageScreenshot keyword:
If you read the source of the keyword and related classes, you would find the implementation doesn’t take the DPR into account.
Your case seems to be complicated. You case might have some factors that make the keyword to perform wrong. I don’t know what. I don’t have any more clues to resolve your case. You need to find them out on your machine for yourself.