uiHandle: handle of the OpenGl-based custom UI
buttonHandle: handle of the button
size: pointer to the x- and y-resolution of the texture. Can be NULL, in which case the texture is removed from the button.
textureData: rgb data of the texture. Texture data will be copied internally, so the textureData can be released directly after this call.