This is the method handle the keydown event in java. 
public boolean onKeyDown(int keyCode, KeyEvent event)
    {
		GiderosApplication app = GiderosApplication.getInstance();
		
		if (app != null && app.onKeyDown(keyCode, event) == true)
			return true;
		
                //I want to add my back handler here if the return value is false in lua code 
		return super.onKeyDown(keyCode, event);
    }
is app.onKeyDown always return true for back key, can I determined it in the keydown event listener of lua code?
For example, in some scene I will handle the back keydown event in lua, but in main menu scene I will just handle it in this java method.
stage:addEventListener(Event.KEY_DOWN, function() 
    if event.keyCode == KeyCode.BACK then
        return false  ---then app.onKeyDown in java will return false
    end
 end)                
                
             
        
Comments
But it depends on the purpose you want to use button in java.
For example you can do something like:
https://sites.google.com/site/xraystudiogame