Hi. I am very new in gideros, just started to read.. What is the best approach for storing large amounts of data? ( table with ~ 1k records ) need to save them localy and need to be able to search in them.
I recently experienced that an ordinary Lua table with ~50k records (english dictionary) consumes around 100MB memory. But most probably a table with ~1k records won't be any problem.
Currently, if you're developing for iOS only, I can recommend to use SQLite plugin (I've attached it). This plugin will be officially available with the next version.
Once android plugin support is added will we be able to use 1 plugin such as this sqlite one for both android and ios? So that we still only have to do everything once for all platforms?
Thanks for info But my main target os is android..iOS maybe later if I am correct - Lua table is t = {} , if I need to fetch needed record, I have to iterate trought all records? It is not very slow process? P.S. Sorry about my english
I've been wondering about this too. I want to build a word game so basically I have two requirements. Storing around 100k words in memory but also checking to see if any word exists very quickly.
@atilim Are Lua tables very inefficient because taking up 100MB of memory for only 50k words seems like a huge amount (over 2K per word!)
So my question is - what on earth is Lua doing that causes it to consume 100MB of memory and does that mean that using a Lua table to store a dictionary of words is a bad idea?
Also, regardless of the memory usage, would a Lua table that big be quick to search?
Lua tables are one of the best things since sliced bread, as they can offer quite a handful of tricks, persist that using JSON and file read/writes, you have a native lua based database.
For an english dictionary with 267751 words (file size is about ~3mb), Lua array (t[#t + 1] = word) takes ~21mb and Lua table (t[word] = true) takes ~33mb.
Thats ok and thanks by the way for even investigating this. I am astounded by the level of support you guys are giving in the forums and it really does add to the value of your product. The sowpods.txt file will come in handy too
This works, don't know if it's very efficient though!
local dict ={}-- populate tablefor line inio.lines("sowpods.txt")dotable.insert(dict, line)end-- search table valueslocalfunction searchTable(word)for _, v inipairs(dict)doif v == word .. "\r"then-- word with return addedreturntrueendendreturnfalseend-- check word and print resultlocalfunction checkWord(word)if searchTable(word)thenprint("Word OK", word)elseprint(word, "You're having a laugh!")endend-- TESTlocal word =string.upper("depastures")
checkWord(word)
word =string.upper("fraggle")
checkWord(word)
word =string.upper("zeITgeiSt")
checkWord(word)
I'm posting this on behalf of @scouser (as he's a little shy on occasions and he thinks I need the post count more!)
We were initially thinking that if the list (table) is sorted in alphabetical order (and referenced by numerical index, you can try do a binary search by comparing the value at the midpoint and then repeatedly halving the search space.
Then I was thinking about Lua "associative" arrays and thought that if you had a table organised like this.
words ={}
words.tom =1
words.dick =1
words.harry =1
So the idea being that the key used to access the value is actually the contents of the array we want to store and you just store a 1 (or a true if was more efficient) as a filler
Then you could probably search quite quickly by doing something like
local guess ="fred"if words[guess]==1thenprint("found "..guess.." in list")elseprint("NOT FOUND")end
It would be an interesting experiment to compare the various implementations to see which was more efficient in terms of memory used and access speed.
WhiteTree Games - Home, home on the web, where the bits and bytes they do play! #MakeABetterGame! "Never give up, Never NEVER give up!" - Winston Churchill
I'd actually write a small utility that read in the text file and then wrote out the code as lua source, or as a json file that could be read in.
Actually storing the data as t["alice"] = "Y" (or some other single character) might be even more efficient depending on what internal data type is used to store numbers and booleans.
WhiteTree Games - Home, home on the web, where the bits and bytes they do play! #MakeABetterGame! "Never give up, Never NEVER give up!" - Winston Churchill
Comments
I recently experienced that an ordinary Lua table with ~50k records (english dictionary) consumes around 100MB memory. But most probably a table with ~1k records won't be any problem.
Currently, if you're developing for iOS only, I can recommend to use SQLite plugin (I've attached it). This plugin will be officially available with the next version.
http://lua.sqlite.org/index.cgi/index
cheers
Once android plugin support is added will we be able to use 1 plugin such as this sqlite one for both android and ios? So that we still only have to do everything once for all platforms?
Sorry if that's a real newbie question.
PS. Welcome home! hope you had a good trip!
But my main target os is android..iOS maybe later
if I am correct - Lua table is t = {} , if I need to fetch needed record, I have to iterate trought all records? It is not very slow process?
P.S. Sorry about my english
regards,
Vaicis
@Vaicis, for example you create a table with
Will wait for android plugin, it will be easier for me and more mundane approach
regards,
Vaicis
@atilim Are Lua tables very inefficient because taking up 100MB of memory for only 50k words seems like a huge amount (over 2K per word!)
Having read this article http://ejohn.org/blog/javascript-trie-performance-analysis/ 112,429 words of plain text only took up 916KB.
So my question is - what on earth is Lua doing that causes it to consume 100MB of memory and does that mean that using a Lua table to store a dictionary of words is a bad idea?
Also, regardless of the memory usage, would a Lua table that big be quick to search?
and searching within a Lua table should be very fast.
Author of Learn Lua for iOS Game Development from Apress ( http://www.apress.com/9781430246626 )
Cool Vizify Profile at https://www.vizify.com/oz-apps
For an english dictionary with 267751 words (file size is about ~3mb), Lua array (t[#t + 1] = word) takes ~21mb and Lua table (t[word] = true) takes ~33mb.
You can download the dictionary here: http://scrabblehelper.googlecode.com/svn/trunk/ScrabbleHelper/src/dictionaries/sowpods.txt
what is an efficient way to load the sowpods data into a table and search on it to see if a word exists or not?
regards
This works, don't know if it's very efficient though!
evs
We were initially thinking that if the list (table) is sorted in alphabetical order (and referenced by numerical index, you can try do a binary search by comparing the value at the midpoint and then repeatedly halving the search space.
Then I was thinking about Lua "associative" arrays and thought that if you had a table organised like this.
Then you could probably search quite quickly by doing something like
Likes: atilim
#MakeABetterGame! "Never give up, Never NEVER give up!" - Winston Churchill
sounds similar to Atilim's approach
And populate it with some words:
t["alice"] = true
t["bob"] = true
t["carol"] = true
t["dave"] = true
Then if you want to know if a key exists or not, you can simply check:
if t[key] ~= nil then
end
I am not sure how I would load the word list file and add the 'true' bit?
Actually storing the data as t["alice"] = "Y" (or some other single character) might be even more efficient depending on what internal data type is used to store numbers and booleans.
#MakeABetterGame! "Never give up, Never NEVER give up!" - Winston Churchill
Dud's link is also interesting: http://ejohn.org/blog/javascript-trie-performance-analysis/