strange memory leak

keszegh

i've a strange problem, now i think it is probably not gideros-specific, and i could not find any hint about it by searching on the net. maybe somebody has an idea what's happening.

in short: basically my app runs fine in one folder and memory leaks in another.

in details:
i use windows 10. so at some point i noticed that my app freezes and crashes without an error message (when loading and processing a certain file containing a lot of data). in task manager i can see that the memory usage runs up to 1.5gb. i tried to revert changes one by one to get back to a previous version where the exported app did not have such an issue. after a lot of trial and error (as there is never an error message) i reverted to the exact same version and it also froze in the player. i exported it and it also freezes. i compared the files and it is exactly the same as the non-freezing version.
this is already magical for me, but at the end i even copied the working old export to some other folder and it freezes there too (memory leak again).

so how it is possible that the same app (exactly same files) works in one folder and memory leaks in another? some windows telemetry or what? still i cannot imagine how can the app itself leak depending on the folder.

and how can i prevent this?

also, what can one do when there is no error message? (the error dialogs see attached)

btw i tried on another windows (8) pc my newest crashing version and the old version which works only in one folder and i could not crash any of these on the other pc.

thanks for reading and any ideas.

keszegh

wow, i managed to find the difference!

in one folder i set the app to run on my dedicated nvidia gpu (works ok) while the other one ran on the intel gpu (crashes with memory leak).

perhaps this is useful info for gideros development too.
edit:
my card that causes the leak is an intel hd 4600 (laptop gpu), driver version is 20.19.15.4549 .

i found out the difference by finding an error log referring to the error being related to ig75icd32.dll .

HubertRonald

@keszegh I remember that the first time I met this was a real headache, thanks for sharing

keszegh

@HubertRonald, don't even mention, it took me 2 days to find out the problem, while i messed up my code trying to get to it, so now it's another day to revert the mess.

i attach an example lua file, where this memory leak occurs. just some meshes.
-memory and texture memory are both under 1mb (press C in the test code to print it out)
-but task manager shows 1200 mb usage when run on intel gpu.
-when i run it on nvidia gpu then it is 70mb usage.

i also checked with GPU-Z that on nvidia gpu the vga-memory used is also just 30mb, so it's not that it has it's own memory where that 1200 mb would be allocated.

also, it wouldn't be reasonable that just this many non-textured meshes would use up this much memory.

so @hgy29 or somebody else, do you have any idea how to correct this bug in gideros?

hgy29

It turns out I have a HD 4600 GPU, I should be able to reproduce then!

I'll let you know.

hgy29

@keszegh, I tried your project but can't see any memory leak: memory grows up to ~1GB but doesn't go beyond that level, which makes me think that somehow intel's driver use lots of memory for its own internal purposes (caching ?)

Has it been a leak I would have expected memory usage to grow endlessly and crash at some point...

keszegh

@hgy29, ok, i was about to correct my terminology. it's not a leak, but clearly an unnecessary excessive use of memory.

why would it be 1000mb vs 70mb just for changing the gpu?
it feels as if all meshes were stored as square-shaped bitmaps internally, while i think even on intel gpu with opengl they should be 3 points forming a triangle and that's it.

also even if i would accept this as normal, there should be a way to measure it (and control it) but getmemory and gettexturememory does not know about this 1000mb.

hgy29

It is quite hard to tell what happens inside intel's driver, it may be something that gideros doesn't release, but why would Nvidia driver perform better then ? And why things actually get released at some point since there is no leak ?

keszegh

also it is much slower on intel gpu while 3600 meshes should be easy to handle even for an integrated gpu nowadays, i guess already quake must have shown more than that.

keszegh

also there is not much memory used if you comment out the 'addChild(mesh)' line, so only when rendered on stage do these meshes take up that awful amount of memory.

also when you remove them then after a while the used up memory slowly goes back to normal ~70mb.

keszegh

i've tried with older versions. and surprise!
v 2015.03.22 works without excessive memory usage and any slow-down.
v 2016.12.1 has the memory-overusage and slow-down.
so i need to download further versions to pinpoint where the problem was introduced. it's surprising though that i noticed it only now.

talis

It seems like an intel driver version spesific problem to me.
If you have time can you please update your drivers from intel's website to the latest.
Because when i looked in Windows update catalogue to the latest version is so much old if you compare it with the one in intel's own website. (your version 20.19.15.4549 is on 11/10/2016 )

keszegh

@talis, so far i did not update because those drivers are not certified for my laptop and i don't want blue deaths etc, reinstalls, etc. this vga drivers can be hell and in any case i use the nvidia gpu for graphics-heavy software.

now i considered to update, but did not want to yet, so that i'm able to test gideros versions about possible fixes.

so should i update @hgy29?

hgy29

I think you can update and see what happens, but all in all it doesn't look like a bug, rather something that related to intel's driver housekeeping. Thing is HD 4600 has almost no dedicated GPU memory, and uses CPU memory instead. My impression is that intel's driver caches memory blocks instead of freeing them instantly in case they could be reused between frames. That would make sense especially if dealing with DMA capable memory.

It probably began when gideros was changed to use VBO instead of direct cpu memory, for performance reasons.

keszegh

i will update and let you know what i get.

meanwhile i have to note that i tried the 2015.03.22 android player and there i could not check memory usage but it runs the test project clearly much faster than the 2017.03.1 version. so i don't know what happened but
"It probably began when gideros was changed to use VBO instead of direct cpu memory, for performance reasons."
seems to be a wrong cause as whatever happened it decreased performance in multiple cases.

hgy29

seems to be a wrong cause as whatever happened it decreased performance in multiple cases.

Just to be sure, you are saying that gideros used to be more efficient before irrespective of the GPU used ?

To explain a little more: VBO are used to keep mesh data (vertices, texture coordinates and colors) in GPU memory instead of reuploading from CPU memory each time a frame is drawn. Said like this it should definitely save memory bandwidth between GPU and CPU, except is geometry is changed frequently.

keszegh

so i updated the driver to the latest one 20.19.15.4624 (had to do it manually as it is not supported on my laptop, but it seems to have gone fine).

and the issue remains the same, ~1gb of ram is eaten up by something that should eat up nothing. (and when i do this the second time it crashes the app, although i have 12gb of memory, so i should be able to go higher actually).

keszegh

@hgy29, yes, so far it seems that on intel gpu and my android tablet it became less efficient. on the intel by a high margin, leading to unusable software quickly (which a driver update does not solve). on the tablet it's harder to measure, but the speed difference is clearly visible.

hgy29

I've been reading about HD 4600 graphic card all afternoon, with people complaining about it eating up too much memory and lagging with some games... turns out this card is considered as crap by gamers. Lots of people are asking how to change the memory limit (because yes, it won't use all memory available by default).

Anyhow if gideros can be improved I'd like to know what to do. I will try to disable VBO and see if it makes a difference on memory for this particular graphic card.

hgy29

Ok, disabling VBO leave the RAM usage low and is much much much faster in that case, but it is true that your demo always recreate stuff and never reuse meshes so we're a definitely on the case were VBO are not a good idea... I can add a setting to disable VBO (even from lua), or maybe use heuristics to detect when they are beneficial or not, like checking if cached data changes often or not

keszegh

in fact in my app it's a looping animation so i reuse the same mesh all the time when it reappears, but with VBO it becames unusable so still it seems that disabling VBO would
be better for me (in which case thousands of meshes still work fine). so please at least make it an option in lua to disable it.

as always, thanks for your efforts.

now it is probably irrelevant as you've found the cause but i narrowed down where this change happens:
2015.09 is bad
2015.06.30 is good (if i'm not mistaken, i lost track which version i've tried)

keszegh

i don't know if it's feasible but a bugfix with this correction plus the gdrbridge correction would be welcome, as these (especially the gdrbridge one) make my life harder - while i've been waiting for the media plugin and ctrl-modifier features that are only in the new version.