Offtopic: Appium - OpenGL based app automation testing

AWS Device Farm and Appium

A few days ago i checked "unread" emails in my account. One of them was from Amazon AWS about their AWS Device Farm.
I checked it on one my mobile application, everything was good but only "Fuzzy testing" option was suitable for me.
I saw "Appium python" option and  decided to do some smart tests for my application.
The biggest problem is a testing UI for OpenGL application, because OpenGL application uses one GLView, UI in games rendered by OpenGL and Appium does't see app UI hierarchy and controls - only one root view.
As a first steps i used tap events to simulate pushing of OpenGL drawn buttons. 
This technique was good for devices where width and height is known and buttons always in the same places, but may fail in other cases (like other device orientation or width/height aspect is different). Also there is no feedback. Test does not have ways to determine is button was pressed or not.

Example to simulate button pushing:

window_size = self.driver.get_window_size()
# tap ok button  

positions = []positions.append((window_size['width']/2, window_size['height']*0.7))self.driver.tap(positions)

NOTE: AWS Device Farm and Appuim examples is available here
NOTE2: Good start point for Appium and AWS  is available here 

The feedback

Seems like i need some ways to organize feedback between app and Appium
I had a few ideas: 

1. Hard and long - Instantiate simple http server on app side and organize network pipe between app and python test script. i.e. my python test script recieves data about controls hierarchy (of cause you opengl app should send UI layout and controls to this pipe)
2. Fast but hacky variant - use something simpler - i choosed this one

 Feedback (Hacky)

I used special key codes to send event to my app to activate testing mode (i used key press event with almost impossible keycodes for android app)

def activeTestMode(self):
   # activate test mode   self.driver.press_keycode(152);

 In my app key code 152 is a signal to activate testing mode, when app receive this signal it just adds TextView on top left with "TestActivated" text

        if ((event.getKeyCode()==152) && (event.getAction() == KeyEvent.ACTION_DOWN))
        {
                appiumPipe = new TextView(this);
                appiumPipe.setId(123456);
                appiumPipe.setTag("AppiumPipe");
                appiumPipe.setText("TestActivated");
                appiumPipe.setBackgroundColor(Color.RED);
                appiumPipe.setSingleLine(true);

                FrameLayout.LayoutParams params = new FrameLayout.LayoutParams(
                        FrameLayout.LayoutParams.MATCH_PARENT,
                        FrameLayout.LayoutParams.WRAP_CONTENT, Gravity.TOP);
                params.setMargins(0, 0, 0, 0);

                appiumPipe.setLayoutParams(params);
                frame.addView(appiumPipe);
                return true;
      }

Next i used Apium api to find this control:

els = self.driver.find_elements_by_class_name('android.widget.TextView') # find_elements_by_xpath('//android.widget.TextView[*]')print els
# find textviewfor element in els:
   if element.text=="TestActivated":
      self.AppiumPipe = element
      break

 When control found the control text used as a pipe between app and my Appium python script.
Next - the other key code 153 used to send signal  to my app to update this control with my gui hierarchy in json format.

def updateTestInfo(self):
   self.driver.press_keycode(153);
   time.sleep(1)

   #print "-------------------"   #print self.AppiumPipe.text   #print "-------------------"
   #print AppiumPipe.name   self.data = json.loads(self.AppiumPipe.text)
   #pprint(self.data)

        else if ((event.getKeyCode()==153) && (event.getAction() == KeyEvent.ACTION_DOWN))

        {
                String res = NativeMethods.executeCommand("APPIUM:GetJSON", null);
                appiumPipe.setText(res);
                return true;
        }

String res = NativeMethods.executeCommand("APPIUM:GetJSON", null); is a code in my app returns gson with my internal opengl controls

After parsing json from control text i know elements and places of visible controls in my GUI

def controlCenterByPath(self,json,path):
   pCur = self.controlByPath(json,path);
   self.assertIsNotNone(pCur)
   return (pCur['x']+pCur['w']/2,pCur['y']+pCur['h']/2);

def tapControlByPath(self,json,path):
   pos = self.controlCenterByPath(json,path)
   # set up array of two coordinates   positions = []
   positions.append(pos)
   self.driver.tap(positions)

Simulation tap by Control name

self.updateTestInfo() # call this to update UI gson
self.tapControlByPath(self.data["TopLevel"][0],"DialogPanel.OKButton")

Validation

self.updateTestInfo()
time.sleep(1)
nodialog = self.controlByPath(self.data["TopLevel"][0],"DialogPanel")
self.assertIsNone(nodialog,"Reward must be closed")