tvOS crash simulated in real device and symbolicated crash logs...

MobAmuse

I've finally managed to simulate the app boot crash on the real tvOS device here this morning and have the following symbolicated crash report (attached) below.

MobAmuse

Just like to point out it runs fine on the simulator and real device from Xcode, but when I archive the build and then extract the .ipa and then resend that to the real device it crashes (i.e. what Apple see their end also).

hgy29

Its a pity that the interesting part, where it actually crashes within Gideros, is not symbolicated:

2 Crock O'Gold Slots TV 0x1001d5b1c 0x1000d0000 + 1071900 3 Crock O'Gold Slots TV 0x1001d55a4 0x1000d0000 + 1070500 4 Crock O'Gold Slots TV 0x1001d4ff4 0x1000d0000 + 1069044 5 Crock O'Gold Slots TV 0x1001d60a4 0x1000d0000 + 1073316 6 Crock O'Gold Slots TV 0x1000e0b88 0x1000d0000 + 68488

I wonder what prevents it from being symbolicated, that would help us a lot!

MobAmuse

@hgy29 I have no idea other than I use encrypted Lua and assets. First time I have ever seen a crash like this personally. I happens on all 3 tvOS apps in the same place.

MobAmuse

Same thing on this one...

Incident Identifier: CB096AB9-5D3A-41A9-9EA9-94DCF4272C46 CrashReporter Key: fb45bd8495e1a2641b75cd95b6025825e1695914 Hardware Model: AppleTV5,3 Process: Flappy Chooky TV [174] Path: /private/var/mobile/Containers/Bundle/Application/4C326F7F-F4BA-45C9-8E82-72AE562559AC/Flappy Chooky TV.app/Flappy Chooky TV Identifier: com.mobileamusements.FCATV Version: 4.0 (4.0) Code Type: ARM-64 (Native) Parent Process: launchd [1]   Date/Time: 2015-12-08 09:11:33.33 +0000 Launch Time: 2015-12-08 09:11:33.33 +0000 OS Version: Apple TVOS 9.0.1 (13T402) Report Version: 105   Exception Type: EXC_CRASH (SIGABRT) Exception Codes: 0x0000000000000000, 0x0000000000000000 Exception Note: EXC_CORPSE_NOTIFY Triggered by Thread: 0   Filtered syslog: None found   Last Exception Backtrace: 0 CoreFoundation 0x189480f2c __exceptionPreprocess + 124 1 libobjc.A.dylib 0x19a0a3f80 objc_exception_throw + 56 2 Flappy Chooky TV 0x10011183c 0x10000c000 + 1071164 3 Flappy Chooky TV 0x1001112c4 0x10000c000 + 1069764 4 Flappy Chooky TV 0x100110d14 0x10000c000 + 1068308 5 Flappy Chooky TV 0x100111dc4 0x10000c000 + 1072580 6 Flappy Chooky TV 0x10001be54 0x10000c000 + 65108 7 UIKit 0x18e73dc08 -[UIApplication _handleDelegateCallbacksWithOptions:isSuspended:restoreState:] + 400 8 UIKit 0x18e96c304 -[UIApplication _callInitializationDelegatesForMainScene:transitionContext:] + 2904 9 UIKit 0x18e970644 -[UIApplication _runWithMainScene:transitionContext:completion:] + 1656 10 UIKit 0x18e96d788 -[UIApplication workspaceDidEndTransaction:] + 168 11 FrontBoardServices 0x19227f7c4 -[FBSSerialQueue _performNext] + 184 12 FrontBoardServices 0x19227fb44 -[FBSSerialQueue _performNextFromRunLoopSource] + 56 13 CoreFoundation 0x189438528 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 24 14 CoreFoundation 0x189437fbc __CFRunLoopDoSources0 + 540 15 CoreFoundation 0x189435cbc __CFRunLoopRun + 724 16 CoreFoundation 0x189364ce0 CFRunLoopRunSpecific + 384 17 UIKit 0x18e736aac -[UIApplication _run] + 460 18 UIKit 0x18e731724 UIApplicationMain + 204 19 Flappy Chooky TV 0x10001d174 0x10000c000 + 70004 20 libdyld.dylib 0x19a8e6974 start + 4     Thread 0 name: Dispatch queue: com.apple.main-thread Thread 0 Crashed: 0 libsystem_kernel.dylib 0x000000019a9ff140 __pthread_kill + 8 1 libsystem_pthread.dylib 0x000000019aac8ef8 pthread_kill + 112 2 libsystem_c.dylib 0x000000019a975f44 abort + 140 3 libc++abi.dylib 0x00000001998593f4 __cxa_bad_cast + 0 4 libc++abi.dylib 0x0000000199875e98 default_unexpected_handler() + 0 5 libobjc.A.dylib 0x000000019a0a4248 _objc_terminate() + 124 6 libc++abi.dylib 0x0000000199872f44 std::__terminate(void (*)()) + 16 7 libc++abi.dylib 0x0000000199872b10 __cxa_rethrow + 144 8 libobjc.A.dylib 0x000000019a0a4120 objc_exception_rethrow + 44 9 CoreFoundation 0x0000000189364d88 CFRunLoopRunSpecific + 552 10 UIKit 0x000000018e736aac -[UIApplication _run] + 460 11 UIKit 0x000000018e731724 UIApplicationMain + 204 12 Flappy Chooky TV 0x000000010001d174 0x10000c000 + 70004 13 libdyld.dylib 0x000000019a8e6974 start + 4

SinisterSoft

So it crashes without the debug build information ?

Is it possible in xcode to use the same flags for a 'test' build as are used for the archive build?

That way we will know if it's a build difference that is causing the problem.

MobAmuse

I set the ipa file as debug when I archived it, then ran it on real device but the crash logs don't tell me anything else difference from above regards gideros.

MobAmuse

I read via Apple docs somewhere that the final archived binary is not the same animal as the the Xcode run version hence why they saw the crash and I didn't until I ran the archived ipa file directly myself. I didn't actually know that fact until last night

MobAmuse

Therefore I conclude something gets screwed over when the archive version is built.

MobAmuse

When I run debug version from Xcode on real device and analyze it, I notice this GL error which actually happens on boot also - maybe this is related perhaps maybe don't know?

ar2rsawseen

is it appletv only and ios works?
appletv binaries are build with bitcode enabled, could that be the case?

MobAmuse

Latest findings:

With newly exported code from gideros (same lua source code for both used in this test)...
iPhone version archive ipa sent to real iphone device works OK no problem (no crash).
tvOS version archive ipa to real tvOS device does not work (crashes as earlier above).

Bitcode is required always for tvOS it seems, so not sure to get around that if it's that.

Very odd this.

MobAmuse

The only difference other than that, is one has libluasocket stuff included and the other one (TV) does not in order to compile correctly if you recall.

MobAmuse

This is a major bummer, as one of the apps I have is a Christmas one - so think I'm going to possibly miss the boat with that one in reality. Oh well :P

ChrisF

I've had a near identical issue with a native app I wrote on iOS about 2 months ago, the new Xcode is crap, lets things archive that should fail.
I first tried the bitcode on/off but that did not solve it.
Turned out to be not linking a library (was flurry in my case) on the archive build but somehow not complaining.
I would place money on it being a non-linked library.

ar2rsawseen

@hgy29 do you think providing debug variables would solve the symbolication? Can we make debug variables for ios/appltetv?

hgy29

I thought we were already providing .a files with debug information. On iOS I ame able to step through gideros native code with Xcode, so surely symbols are already in. They should have made their way into the .dsym file. Was the (right) dsym file actually used for symbolication ?

MobAmuse

This is what I did to get the crash to happen and symbolicated logs myself my end like Apple...

When I create the debug version Archive, I then export the .ipa file from the Archive. I then send the exported .ipa to the real device target via Xcode Devices (the Apple TV in this case). It then runs (well crashes) ...then I check that devices crash logs back in Xcode with symbolication.

Gideros does not seem to provide symbols via that method in the symbolicated log. I presume Xcode creates the symbol file automatically when I do all this. It even offers the option to resymbolicate the crash report (which I did also) so is there any other step I am missing here?

Anyone could probably recreate this crash by running an exported archive ipa version of the binary on their own tvOS hardware setup I guess?

MobAmuse

...I also presumed that because the .dsym file usually sits inside the Archive along with the .ipa that Xcode knows where to get it from when it symbolicates the crash log.

I've never done this bit before, can you tell yet?

MobAmuse

When I export the .ipa from the Archive without bitcode it works :P

Not good because it needs bitcode for tvOS

Ugh!

SinisterSoft

@at2rsawseen maybe just a recompile with bitcode output on all libs?

SinisterSoft

Maybe just bitcode even for apple tv libs?

MobAmuse

Least i'm not alone regards missing symbolication info in crash reports generated by bitcoded apps :P

https://forums.developer.apple.com/thread/25224

MobAmuse

Gideros bug report filed today - it's knackered.

https://github.com/gideros/gideros/issues/201

hgy29

I don't have an ATV to test, but I tried to enable bitcode for iPhone and do the same steps as you: no issue!
Here is what I did:
1) I rebuilt all gideros libs with bitcode enabled (gvfs,lua,gideros and luasocket)
2) I enable bitcode in the project settings in xcode
3) I archived the app and exported it as 'development deployment' thinned for my iPhone 6, with bitcode rebuilding enabled.
4) I installed the resulting ipa from Xcode device manager.

And it runs ok... Did I miss something, or this this more of an AppleTV specific bug than a generic bitcode issue ?

MobAmuse

Thank you for trying that route which is a good test.

Everything you did was indeed correct.

The only difference is that libluasocket has to be removed from tvOS build as it does not compile with it in place either way.

I can't find anything else my end that would cause this as it works on real tv device if I dont recompile with bitcode. Soon as I export with bitcode on, then it crashes on real device as mentioned earlier. Need somebody else with real tv device to test.

hgy29

Well, you can buy me an AppleTV

MobAmuse

Hahahaaaaaaaaa

Thanks again anyway for trying.

MobAmuse

Tried same process with bird example and new xcode 7.2 with no change with bitcode on exported development ipa...

Incident Identifier: 79FB4E7B-CAA6-40BC-926C-0A8DD926A6EE CrashReporter Key: fb45bd8495e1a2641b75cd95b6025825e1695914 Hardware Model: AppleTV5,3 Process: Bird Animation AppleTV [873] Path: /private/var/mobile/Containers/Bundle/Application/26C088DA-6DF2-4D48-B185-B49FB5359BA5/Bird Animation AppleTV.app/Bird Animation AppleTV Identifier: com.mobileamusements.BirdTV Version: 1 (1.0) Code Type: ARM-64 (Native) Parent Process: launchd [1]   Date/Time: 2015-12-12 11:04:16.16 +0000 Launch Time: 2015-12-12 11:04:15.15 +0000 OS Version: Apple TVOS 9.0.1 (13T402) Report Version: 105   Exception Type: EXC_CRASH (SIGABRT) Exception Codes: 0x0000000000000000, 0x0000000000000000 Exception Note: EXC_CORPSE_NOTIFY Triggered by Thread: 0   Filtered syslog: None found   Last Exception Backtrace: 0 CoreFoundation 0x189480f2c __exceptionPreprocess + 124 1 libobjc.A.dylib 0x19a0a3f80 objc_exception_throw + 56 2 Bird Animation AppleTV 0x10011112c 0x100008000 + 1085740 3 Bird Animation AppleTV 0x100110bb4 0x100008000 + 1084340 4 Bird Animation AppleTV 0x100110604 0x100008000 + 1082884 5 Bird Animation AppleTV 0x1001116b4 0x100008000 + 1087156 6 Bird Animation AppleTV 0x100019e68 0x100008000 + 73320 7 UIKit 0x18e73dc08 -[UIApplication _handleDelegateCallbacksWithOptions:isSuspended:restoreState:] + 400 8 UIKit 0x18e96c304 -[UIApplication _callInitializationDelegatesForMainScene:transitionContext:] + 2904 9 UIKit 0x18e970644 -[UIApplication _runWithMainScene:transitionContext:completion:] + 1656 10 UIKit 0x18e96d788 -[UIApplication workspaceDidEndTransaction:] + 168 11 FrontBoardServices 0x19227f7c4 -[FBSSerialQueue _performNext] + 184 12 FrontBoardServices 0x19227fb44 -[FBSSerialQueue _performNextFromRunLoopSource] + 56 13 CoreFoundation 0x189438528 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 24 14 CoreFoundation 0x189437fbc __CFRunLoopDoSources0 + 540 15 CoreFoundation 0x189435cbc __CFRunLoopRun + 724 16 CoreFoundation 0x189364ce0 CFRunLoopRunSpecific + 384 17 UIKit 0x18e736aac -[UIApplication _run] + 460 18 UIKit 0x18e731724 UIApplicationMain + 204 19 Bird Animation AppleTV 0x10001cea4 0x100008000 + 85668 20 libdyld.dylib 0x19a8e6974 start + 4     Thread 0 name: Dispatch queue: com.apple.main-thread Thread 0 Crashed: 0 libsystem_kernel.dylib 0x000000019a9ff140 __pthread_kill + 8 1 libsystem_pthread.dylib 0x000000019aac8ef8 pthread_kill + 112 2 libsystem_c.dylib 0x000000019a975f44 abort + 140 3 libc++abi.dylib 0x00000001998593f4 __cxa_bad_cast + 0 4 libc++abi.dylib 0x0000000199875e98 default_unexpected_handler() + 0 5 libobjc.A.dylib 0x000000019a0a4248 _objc_terminate() + 124 6 libc++abi.dylib 0x0000000199872f44 std::__terminate(void (*)()) + 16 7 libc++abi.dylib 0x0000000199872b10 __cxa_rethrow + 144 8 libobjc.A.dylib 0x000000019a0a4120 objc_exception_rethrow + 44 9 CoreFoundation 0x0000000189364d88 CFRunLoopRunSpecific + 552 10 UIKit 0x000000018e736aac -[UIApplication _run] + 460 11 UIKit 0x000000018e731724 UIApplicationMain + 204 12 Bird Animation AppleTV 0x000000010001cea4 0x10000800

MobAmuse

...Works fine though if recompile bitcode is unticked on export of ipa.

Grim bug this :P

MobAmuse

@SinisterSoft Have you had chance yet to test bird example on your Apple TV device doing this?