UrlLoader slow on Windows

PaulH

I'm seeing a big difference in download speed using UrlLoader between platforms. Downloading a 15 MB file over the same internet connection with the same Gideros app, the Windows desktop version takes much longer than expected:

Android app: 20 seconds
App in Gideros Player on Windows: 20 seconds
WinRT app: 90 seconds

Any ideas why the Windows application would be over 4 times slower than either the player or Android version?

Could this be related to the issue where the WinRT implementation of UrlLoader doesn't trigger progress events except once at the start and once end of the download?

Paul

hgy29

@PaulH, I admit that although it works, WinRT implementation of UrlLoader may not be really efficient: buffers are way too small, hence data is appended by small chunks into a growing buffer (realloc'ed each time a new chunk of bytes is received), etc.
So I see reasons why it would be slow, indeed. And the way progress is supposed to work in WinRT is a mystery!

PaulH

Thanks for the reply.

Is this inherent in the WinRT environment, or is it possible to improve the implementation of UrlLoader in Gideros?

Of course, I'm mostly glad it works at all. When I first tested an in-app download in the WinRT app I thought it wasn't downloading, between the lack of progress events and the much longer download time. I display a progress bar saying "Downloading..." during the process, and the green bar that should be filling it never appears. I made the whole bar flash slowly to prevent giving users the impression the app has locked up.

Paul

PaulH

Another observation on the WinRT download: If the app loses focus (the user switches to a different application while the download is running) the download makes no progress. I've had a user report half an hour waiting for a download to finish, one that takes me 1.5 minutes on WinRT, and 20 seconds on Android, and I suspect that moving the focus to another app is the reason that it didn't finish in 30 minutes.

Is the WinRT code for downloads something I can look at directly? If I can help by making that more efficient, I'd be happy to do so for my own sake as well as that of the Gideros community.

Paul

hgy29

Hi @PaulH,

You'll need to fetch gideros sources from github (https://github.com/gideros/gideros) and compile WinRT libraries yourself. Luckily, this is not the most complex part of gideros build. You'll find a batch file in scripts folder to get you started building WinRT libs: scripts/buildwinrt.bat. UrlLoader implemention will be in libgid/src/winrt/ghttp_winrt.cpp

Thanks for you help

PaulH

I'm finally working on this issue again as it's starting to impact customers significantly. In trying to build the WinRT libraries, whether from the scripts using msbuild or from Visual Studio, the build fails because dx11_shaders.c is not found. I can't seem to find this file anywhere. Is it missing from the github repository, or am I just overlooking something?

Paul

hgy29

This file is generated by a script, there is a compileshaders.bat somewhere to do that

PaulH

Ah. That makes sense. Thanks.

PaulH

I see two versions of compileshaders.bat: One in master\winrt, which tries to access texture.hlsl in that folder, which isn't there. The other compileshaders.bat is in mater\2dsg\gfxbackends\dx11, which I assume is the right one. But that one uses bin2, but I don't see an executable or script called bin2c. There's a bin2c.c, so I built an executable out of that, but its command line arguments don't match the way bin2c is called in compileshaders.bat. I'm definitely missing something here.

Is there any documentation on setting up the build environment to compile the WinRT libraries? So far I've worked out that I need a command line path that includes Windows Kits 8.1 to pick up fxc.exe, and MSBuild to pick up msbuild.exe.

Paul

hgy29

Yeah, there are two ways to build gideros: the script based version (.bat/.sh) and the Makefile based one, and both versions don't always use the same tools. bin2c.c is used by the Makefile version, while .bat file used to use a prebuilt version from somewhere on the internet.
Since you've built bin2c.exe from gideros sources, here is the relevant part of /scripts/GidWinRT.mk to build the missing file:

WINRT_SHADERS=Basic Color Texture TextureAlpha TextureColor Particle Particles PathFillC PathStrokeC PathStrokeLC WINRT_SHADERS_PATH=2dsg/gfxbackends/dx11 WINRT_SHADERS_FILE=dx11_shaders.c BIN2C=$(ROOT)/scripts/bin2c   $(WINRT_SHADERS_PATH)/$(WINRT_SHADERS_FILE): $(BIN2C) $(addsuffix .hlsl,$(addprefix $(WINRT_SHADERS_PATH)/,$(WINRT_SHADERS))) rm -f $(WINRT_SHADERS_PATH)/$(WINRT_SHADERS_FILE) for s in $(WINRT_SHADERS); do $(FXC) //T vs_4_0_level_9_3 //E VShader //Fo $(WINRT_SHADERS_PATH)/v$$s.cso $(WINRT_SHADERS_PATH)/$$s.hlsl; done for s in $(WINRT_SHADERS); do $(FXC) //T ps_4_0_level_9_3 //E PShader //Fo $(WINRT_SHADERS_PATH)/p$$s.cso $(WINRT_SHADERS_PATH)/$$s.hlsl; done P=$(PWD); cd $(WINRT_SHADERS_PATH); for s in $(WINRT_SHADERS); do $$P/$(BIN2C) p$$s.cso >> $(WINRT_SHADERS_FILE); $$P/$(BIN2C) v$$s.cso >> $(WINRT_SHADERS_FILE); done rm -f $(WINRT_SHADERS_PATH)/*.cso   winrt.shaders: $(WINRT_SHADERS_PATH)/$(WINRT_SHADERS_FILE)

So basically just use your bin2c.exe with input file as single arg and redirect the output to the target file (with appending).

PaulH

That helped. I wrote a batch job to do the equivalent steps using the bin2c.exe, so dx11_shaders.c seems fine now. The WinRT libraries build with lots of warnings for use of deprecated functions, but no errors. After swapping them into my WinRT exported project, the app crashes right after the launch screen, however. The debugger shows:

ERROR FindDefaultInterface:CM_Get_Device_Interface_List_Size failed:0x00000000, InterfaceListSize = 1
ERROR SensorOpenByType:Failed to open sensor error:0x00000490

I can't find any relevant code. Any idea if this indicates a problem in the WinRT libraries I've built, or is there more that I'd need to bring over to my exported project besides the libraries themselves?

hgy29

Those two lines aren't a problem, error is caught by code and ignored. Is there something else ?

PaulH

Yes, after those errors, I see a "pad integer in precompiled chunk', then a few more libraries load, followed by some exceptions thrown:

DataEntryScreen.lua: bad integer in precompiled chunk'giderosgame.Windows.exe'
(... a few more libraries load ...)
The thread 0x2d04 has exited with code 0 (0x0).
Exception thrown at 0x761EA882 (KernelBase.dll) in giderosgame.Windows.exe: 0x40080201: WinRT originate error (parameters: 0x8000001A, 0x0000005D, 0x0361DAFC).
Exception thrown at 0x761EA882 (KernelBase.dll) in giderosgame.Windows.exe: 0x40080201: WinRT originate error (parameters: 0x8000001A, 0x0000001D, 0x0361DCC0).
The thread 0x1f44 has exited with code 0 (0x0).
Exception thrown at 0x761EA882 (KernelBase.dll) in giderosgame.Windows.exe: 0x40080201: WinRT originate error (parameters: 0x8000000B, 0x00000040, 0x0381F6D0).

hgy29

yes, that one is easy: it means that you are trying to use encrypted assets with an otherwise unencrypted gideros binary

hgy29

btw, congratulation for compiling WinRT template

PaulH

Interesting. So to use encrypted assets, I need Gideros to encrypt the library as it encrypts the assets during the export?

PaulH

Does that mean if I replaced the libraries in (program files)\Gideros\Templates\VisualStudio\WinRT Template with the libraries I built, they'd get encrypted during the export, and that would prevent the exceptions I'm seeing?

hgy29

Yes, that's it

PaulH

Cool. I just tried with non-encrypted source, and the app works. Just changing the buffer size from 1k to 100k sped up a download from almost 2 minutes to just over 15 seconds. I'll have a look at improving the efficiency of the code, but the buffer size change alone is a huge improvement.

Thanks for all your help!

hgy29

Thanks I'll enlarge buffer in official gideros code right now

PaulH

Sounds good. I tried with buffer sizes of 100KB and 1MB and the difference was minimal when downloading files around 15 MB, so from what I've seen the 100 KB seems a reasonable choice. Thanks again!