psx_gpu: more limit checking