Sikuli library for mobile testing

I’m trying to use Sikuli library to automate some tests in mobile that requires the analysis of the screen of the mobile device. The problem seems to be that when I search for an image in the screen of the device it is searching in the screen of my PC instead… is it possible to use Sikuli with the mobile??? I’m currently using the Sikuli 1.1.3 because the latest version of Sikuli is not compatible with Katalon JNA.
Idon’t want to use Vysor or other app to be able to use Sikuli…

I tried using ADBScreen() instead of Screen() but now I get errors with OpenCV…

I found out how to make it work finally with the help of a dev I’m working with…

First with sikuli 1.1.3 you need also openCV I found out with version 1.1.3 of sikuli and 7.2.1 of Katalon you need openCV 3.4.9. I’m running on a x64 windows 10.

you need to load openCV dll, if you don’t you get this kind of error: java.lang.UnsatisfiedLinkError: org.opencv.core.Mat.n_Mat(III)J

Also in Katalon you need to load the dll with the command (the last argument is the path of the openCV dll in x64):
Runtime.getRuntime().load0(groovy.lang.GroovyClassLoader.class, “D:/jars/opencv_java349.dll”)

here is my code for android

import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import static com.kms.katalon.core.testobject.ObjectRepository.findWindowsObject
import com.kms.katalon.core.checkpoint.Checkpoint as Checkpoint
import com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords as CucumberKW
import as Mobile
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testdata.TestData as TestData
import com.kms.katalon.core.testobject.TestObject as TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import as Windows
import internal.GlobalVariable as GlobalVariable
import com.kms.katalon.core.configuration.RunConfiguration as RunConfiguration
import org.sikuli.script.FindFailed
import org.sikuli.script.Pattern
import org.sikuli.script.Screen
import org.sikuli.script.Key

//load openCV
Runtime.getRuntime().load0(groovy.lang.GroovyClassLoader.class, “D:/jars/opencv_java349.dll”)

//screenshot to analyse
ADBScreen dev = new ADBScreen()

//image of the button I’m looking for (the object is non existent in spy mode)
Pattern ReferenceButton = new Pattern (RunConfiguration.getProjectDir() + ‘\images\ReferencePictures_EN.PNG’)

//do a search in the screen and tap on the image of the button

1 Like


i have the same problem, i use sikuli with katalon to automate android mobile tests and when i search for an image on the device screen, sikuli takes the screen of my pc.
I can have more details please !!!