Problem with json.Decode and char encoding

arcadia

In my game i use scoreoid for the leaderboard feature with json response to retrive informations.
I have a problem with some player's name like "luca999"

This is the json i read from event.data call to scoreoid

[{"Player":{"username":"sara ","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"4231","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-10 11:43:27","data":null}},{"Player":{"username":"antonella","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"2292","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-11 20:52:30","data":null}},{"Player":{"username":"gilda","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"2092","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-10 18:23:13","data":null}},{"Player":{"username":"arcadia","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"1798","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-11 14:51:25","data":null}},{"Player":{"username":"Roberta","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"1643","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-13 20:17:47","data":null}},{"Player":{"username":"Andrei","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"1620","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-01 18:57:30","data":null}},{"Player":{"username":"Big boss","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"1426","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-04 19:27:08","data":null}},{"Player":{"username":"Alezero28","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"1376","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-13 14:57:24","data":null}},{"Player":{"username":"gianluca","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"1343","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-12 16:38:39","data":null}},{"Player":{"username":"Giulia","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"1169","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-10 20:50:01","data":null}},{"Player":{"username":"zerosphere","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"1098","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-03-31 20:33:21","data":null}},{"Player":{"username":"gil","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"1016","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-01 07:28:56","data":null}},{"Player":{"username":"Lucaquellovero","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"974","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-13 19:05:56","data":null}},{"Player":{"username":"Stanyx","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"861","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-13 12:45:01","data":null}},{"Player":{"username":"Flame85","email":null,"first_name":"","last_name":"","platform":null},"Score":{"score":"850","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-03-31 18:26:10","data":null}},{"Player":{"username":"Lorenzo","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"733","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-07 13:18:06","data":null}},{"Player":{"username":"Alessio","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"653","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-05 00:39:11","data":null}},{"Player":{"username":"UnderMe","email":null,"first_name":"","last_name":"","platform":null},"Score":{"score":"630","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-03-31 18:35:05","data":null}},{"Player":{"username":"Luca","email":null,"first_name":"","last_name":"","platform":null},"Score":{"score":"550","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-03-31 18:34:28","data":null}},{"Player":{"username":"Art","email":null,"first_name":"","last_name":"","platform":null},"Score":{"score":"450","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-03-31 18:26:42","data":null}},{"Player":{"username":"luca999","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"336","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-13 08:06:10","data":null}},{"Player":{"username":"Daniel","email":null,"first_name":"","last_name":"","platform":null},"Score":{"score":"320","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-03-31 18:29:05","data":null}},{"Player":{"username":"Andronello","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"200","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-06 21:54:33","data":null}},{"Player":{"username":"paolo","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"176","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-06 19:14:44","data":null}},{"Player":{"username":"Ema","email":null,"first_name":"","last_name":"","platform":"Android"},"Score":{"score":"105","platform":null,"difficulty":0,"leaderboard":null,"created":"2013-04-06 15:09:34","data":null}}]

I can see the name "luca999" but when i decode the json with Json.Decode (@ar2rsawseen module) and i print the players name i have a strange chars instead of "luca999".
This is the output:

players = Json.Decode(event.data) for x=1, table.getn(players) do print (players[x].Player.username) end

with this output

.... Luca Art l�9 Daniel Andronello ....

ar2rsawseen

Actually not mine module, I only shared it because original link where I got it was not working anymore.
But it seem that it interprets part of luca999 as a unicode symbol \uCA99.
I just don't really get why it's interpreted that, way, it should have had slashes to be interpreted as unicode, at least this is how it encodes them.

string.format("\\u%.4X", string.byte(n))

QuickFix:
If you are sure, that you will not have to deal with unicode characters, then you can open json.lua. Find the method JsonReader:ReadString()

And change this:

return string.gsub( result, "u%x%x(%x%x)", fromunicode)

to simply

return result

arcadia

mmm ok work fine with the quickFix
thx a lot

krisis

@ar2rsawseen, @arcadia - I just hit this same issue. I implemented chat in my game and anytime there is a "u" followed by 4 characters it gets decoded to a weird char. i.e. "Stuffed" becomes "St�".

It appears the proper fix is:

return string.gsub( result, "\\u%x%x(%x%x)", fromunicode)

I tested with some Korean char input and it worked, and "stuffed" no longer became malformed.

fyi, this is in the JSON Encoder and Parser for Lua 5.1 [Copyright © 2007 Shaun Brown (http://www.chipmunkav.com)] line 362.

ar2rsawseen

@krisis today I will be testing NativeJson implementation and also will test this case, hopefully it will work better and you will be able to use Native Json (which would also be 30-40 times faster )