Json plugin issue

PaulH

I can't say if this is necessarily a bug, or by design, but the JSON plugin does something I find odd. In certain circumstances it can cause crashes, or a case where encoding a table and the decoding the result may give you a table that differs from the original one. If it's not a bug, it's at least a potential pitfall to be aware of for those who use the JSON plugin.

The issue applies to tables in which all keys (indexes) are positive integers, and they are not sequential starting at 1. In that case json.encode() fills in the gaps in the sequence with null values for the missing keys from 1 to the highest included key. I assume this is done for performance, taking advantage of the sequential integer indexes for faster table access. That's not necessarily a problem.

But then when decoding that text, json.decode() puts a value that prints as "00000000", of userdata type, in each key that was absent in the original table. So encoding such a table and decoding the result doesn't give you an equivalent table.

For example this table:
{ [3] = "apple" }

gets encoded to this text:
[null,null,"apple"]

Then decoding that text gives you this table:
{
[1] => userdata: 00000000
[2] => userdata: 00000000
[3] => "apple"
}

In my code I was storing a complex table by encoding it to JSON, saving to a file, and on future runs reading and decoding it. Because the table included some tables with incomplete sequence of positive integer keys, I ended up with a table with values for keys the original table didn't. Where my code did things like:
if table[key] then
-- do something with table[key]
it would often fail because the value in the table would not be anything my code put there or expected to find there.

It's also possible to crash json.encode() by giving it an array for which it would fill in too many missing positive integer keys:
json.encode({[30] = "blue"})

will crash with the error:
Cannot serialise table: excessively sparse array

In this I'd prefer to see json.encode() treat the keys as strings. String keys may be slower, but slower is better than non-functional.

I've also had json.encode() crash with less informative error messages after a series of encoding and decoding such tables, having cases where tables gained some keys of type userdata. I haven't pinned down a sequence of steps that will reproduce that.

If this is expected behavior, there are workarounds. A pure LUA implementation of JSON that removes null keys during decoding, and can serialize extremely sparse arrays (albeit by inserting a very large number of null keys, again arguably wasteful but at least functional) is available at http://regex.info/blog/lua/json.

Another workaround is to simply avoid encoding arrays that json.encode() will alter with null values for missing keys. Adding at least one key of any other type will avoid this behavior.

This table:
[3] = "apple", color = "red" }

Encodes to:
{"3":"apple","color":"red"}

And if you decode that you get the original table. There would be more coding to do to deal with resulting tables, but one might add a special key to tables before encoding, like table.dont_add_null_keys = true, and removing it after decoding, i.e. table.dont_add_null_keys = nil

One other workaround is to purge anything from a table before encoding it. A simple recursive function can remove any entries in a table or tables it contains where either they key or value is not a number, string or a table.

SinisterSoft

I think that it inserting nulls in the json is the error, if that was fixed then the problem would go away.

However, if there is a null in json data loaded from something else (eg from a web api) that returns null data in json table then the userdata issue would appear again - maybe that should also optionally skip loading null data into a table.