ofonod crash in rilmode/sim.c::ril_query_sim_passwd_state()

There's a large number of instances of ofono dying and being respawned by upstart:

Sep 27 12:03:36 ubuntu-phablet ofonod[1195]: [UNSOL]< UNSOL_RIL_CONNECTED
Sep 27 12:03:36 ubuntu-phablet ofonod[1195]: GET_SIM_STATUS reques failed: 2; retries: 1
[...]
Sep 27 12:03:39 ubuntu-phablet NetworkManager[1183]: <info> (/ril_0) updated available interfaces
Sep 27 12:03:40 ubuntu-phablet ofonod[1195]: Querying PIN authentication state failed
Sep 27 12:03:40 ubuntu-phablet NetworkManager[1183]: <info> ofono disappeared
Sep 27 12:03:40 ubuntu-phablet kernel: [ 21.828231] init: ofono main process (1195) killed by SEGV signal
Sep 27 12:03:40 ubuntu-phablet kernel: [ 21.828322] init: ofono main process ended, respawning
[...]
Sep 27 12:03:46 ubuntu-phablet ofonod[1967]: oFono version 1.12

This concerns me somewhat. Is this SIM locked by a PIN?

In the instance where you succesfully connected, it's one of the cases where there were no such errors about PIN authentication. I however can't explain how this might have changed without user intervention. One other failure seems to have been where ofono failed to respond to a dbus message.

Reassigning to ofono -- I sincerely think there is a bigger issue here which causes the connection to fail. I won't be able to reproduce this without having the same SIM (and since we're not in the same country either, it might be even harder). There is definitely a need for further investigation.

Michael, could you get debug logs from ofono? You should be able to edit /etc/init/ofono.conf to add "-d" to the exec ofonod line in that file, reboot, and still be able to reproduce the issue. In this case we'll get debug information in syslog which may help figuring out the problem.

Hi Mathieu,

No, my SIM card is not pinlocked. I already had similar issues earlier with this SIM card and ofono which prevented GSM from connecting.

I'll try to get some more logs with ofono debugging enabled.

FYI, here's the backtrace of ofono crashing:

#0 0x4063c464 in strlen () from /lib/arm-linux-gnueabihf/libc.so.6
#1 0x404c0840 in _dbus_string_init_const (str=str@entry=0xbe9ba930, value=0x1 <Address 0x1 out of bounds>) at ../../dbus/dbus-string.c:196
#2 0x404b4162 in _dbus_check_is_valid_utf8 (name=<optimized out>) at ../../dbus/dbus-marshal-validate.c:1255
#3 0x404b5adc in dbus_message_iter_append_basic (iter=iter@entry=0xbe9ba980, type=type@entry=115, value=value@entry=0xbe9baa2c) at ../../dbus/dbus-message.c:2598
#4 0x00072c6c in append_variant (iter=iter@entry=0xbe9ba9d8, type=type@entry=115, value=value@entry=0xbe9baa2c) at src/dbus.c:62
#5 0x00072f48 in ofono_dbus_signal_property_changed (conn=conn@entry=0x1ff93b0, path=<optimized out>, interface=interface@entry=0xbfbd4 "org.ofono.SimManager", name=name@entry=0xc5980 "PinRequired",
    type=type@entry=115, value=value@entry=0xbe9baa2c) at src/dbus.c:219
#6 0x000866a4 in sim_pin_query_cb (error=<optimized out>, pin_type=<optimized out>, data=0x2002fb8) at src/sim.c:2746
#7 0x0001f0b6 in ril_query_passwd_state (sim=<optimized out>, cb=0x86645 <sim_pin_query_cb>, data=0x2002fb8) at drivers/rilmodem/sim.c:606
#8 0x0001fb3e in sim_status_cb (message=<optimized out>, user_data=0x2002fb8) at drivers/rilmodem/sim.c:562
#9 0x0001aa04 in handle_response (message=0x20023e0, p=0x2000200) at gril/gril.c:366
#10 dispatch (message=0x20023e0, p=<optimized out>) at gril/gril.c:509
#11 new_bytes (rbuf=0x1ff8790, user_data=0x2000200) at gril/gril.c:607
#12 0x0001b4ae in received_data (channel=0x2000090, cond=G_IO_IN, data=0x2000270) at gril/grilio.c:124
#13 0x403e4b6a in g_main_context_dispatch () from /lib/arm-linux-gnueabihf/libglib-2.0.so.0
#14 0x403e4dca in ?? () from /lib/arm-linux-gnueabihf/libglib-2.0.so.0
#15 0x403e4dca in ?? () from /lib/arm-linux-gnueabihf/libglib-2.0.so.0

@Michael

What build are you using? Has it been modified at all?

Can you reproduce the crash at will?

I have a Nexus4, and have seen no such issues or crashes with build #70. There are re-connection issues we're working on, but in general most people aren't seeing the same issues you are.

@Tony

> What build are you using? Has it been modified at all?

The bug has been reported with an image around 50. I did make it writable and installed debug symbols in order to debug something in unity.

> Can you reproduce the crash at will?

Yes, I barely can reproduce the working case. In like 90% of the tries, the cellular network connection fails.

> I have a Nexus4, and have seen no such issues or crashes with build #70. There
> are re-connection issues we're working on, but in general most people aren't
> seeing the same issues you are.

Because I filled up / with debug symbols by now I can't upgrade to the latest image, I'm going to flash the latest ubuntu-system now, enable ofono debugging logs and reproduce the issue.

Some more info:

I'm using a simyo.de SIM card. It's a mvno of eplus.de. Still I use standard settings as Simyo shares servers with E-Plus( APN: internet.eplus.de)

@Michael

Thanks for the updates. Image #50 is very old... and since you made the image read/write, this can cause issues if you were also using system updates. We've been doing lots of work on the mobile connection code, both in NetworkManager and ofono, so running the latest stable code is highly advisable.

Anyways, glad to head this is working for you now.

If you get this situation to happen again, can you please attach your syslog, and then run the following commands and attach the output to the bug?

 - /usr/share/ofono/scripts/list-modems

 - /usr/share/ofono/scripts/list-contexts

- nmcli c

- netstat -rn

Also, feel free to ping me on IRC as well!

hmm... I might have been mistaken with the image number then. It wasn't for sure "very old". I freshly flashed it like 2 weeks ago and constantly kept it up to date by updating in the system settings. Yeah... I guess the rw image + system updates isn't the best, but I have to make it rw to be able to work with the device.

Ok... Just turned WiFi off, 3G icon appears, network not working.

phablet@ubuntu-phablet:~$ ping google.de
ping: unknown host google.de
phablet@ubuntu-phablet:~$ /usr/share/ofono/scripts/list-modems
[ /ril_0 ]
    Features = sms net gprs sim
    Emergency = 0
    Powered = 1
    Lockdown = 0
    Interfaces = org.ofono.CallVolume org.ofono.MessageManager org.ofono.NetworkRegistration org.ofono.ConnectionManager org.ofono.NetworkTime org.ofono.VoiceCallManager org.ofono.SimManager
    Online = 1
    Model = Fake Modem Model
    Revision = M9615A-CEFWMAZM-2.0.1700.48
    Type = hardware
    Serial = 353918055887901
    Manufacturer = Fake Manufacturer
    [ org.ofono.CallVolume ]
        Muted = 1
        SpeakerVolume = 0
        MicrophoneVolume = 0
    [ org.ofono.MessageManager ]
    [ org.ofono.NetworkRegistration ]
        Status = registered
        Strength = 41
        Name = simyo
        LocationAreaCode = 40229
        Mode = auto
        MobileCountryCode = 262
        Technology = umts
        CellId = 146114233
        MobileNetworkCode = 03
    [ org.ofono.ConnectionManager ]
        Powered = 1
        Attached = 1
        Bearer = none
        Suspended = 0
        RoamingAllowed = 0
    [ org.ofono.NetworkTime ]
    [ org.ofono.VoiceCallManager ]
        EmergencyNumbers = 08 118 119 112 999 911 000
    [ org.ofono.SimManager ]
        Retries =
        MobileCountryCode = 262
        FixedDialing = 0
        SubscriberNumbers =
        PreferredLanguages = de en fr
        BarredDialing = 0
        CardIdentifier = 894921001695537951
        LockedPins =
        MobileNetworkCode = 03
        SubscriberIdentity = 262032761568775
        Present = 1
        PinRequired = none

phablet@ubuntu-phablet:~$ /usr/share/ofono/scripts/list-contexts
[ /ril_0 ]
    [ /ril_0/context1 ]
        Username = eplus
        Protocol = ip
        Name = Volume rate/30 Day Flatrate
        Settings = { }
        IPv6.Settings = { }
        Active = 0
        AccessPointName = internet.eplus.de
        Password = gprs
        Type = internet

phablet@ubuntu-phablet:~$ nmcli c
NAME UUID TYPE TIMESTAMP-REAL
Marble Zone 33c71c29-a2d0-457a-a14c-4ca4697924ff 802-11-wireless Fri 04 Oct 2013 08:01:07 AM UTC
Green Hill Zone 76dd2613-8db7-4b5d-90d2-6410872b2664 802-11-wireless Thu 03 Oct 2013 06:50:05 PM UTC
/262032761568775/context1 5ce25b0b-7ef9-9ee1-6e86-e6032e0968b6 gsm Fri 04 Oct 2013 03:01:18 PM UTC
phablet@ubuntu-phablet:~$ netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
phablet@ubuntu-phablet:~$

I rebooted the device and it took really long (some minutes) for 3G to actually get data through.

syslog attached

@Michael

Per our discussion earlier on IRC, there are three distinct problems described in this bug.

1. Sometimes the mobile connection is active, but the routing table is incorrect. This is a DUP of Network Manager bug #1234364.

2. The connection takes a long time to become active. I believe this a DUP of ofono bug #1226145.

3. An ofonod crash detailed in comment #4, which is the first report of this crash. Furthermore, after examining the code, the bug is pretty obvious, and apparently was also fixed in the nemomobile code as part of some re-work to the PIN handling code. So, I've changed the Status to InProgress, and will be pushing a patch shortly.

Feature branch created on github for the fix:

https://github.com/tonyespy/ofono/tree/sim-passwd-crash-fix

Thanks a bunch Tony :)

This bug was fixed in the package ofono - 1.12+bzr6837-0ubuntu1

---------------
ofono (1.12+bzr6837-0ubuntu1) saucy; urgency=low

  [ Alfonso Sanchez-Beato ]
  * Fix for LP: #1231320: GPRS provisioning is broken for old
    (non-USIM) SIM cards in Ubuntu
  * Fix for LP: #1222106: ofono is picking the wrong APN settings
    in Ubuntu

  [ Tony Espy ]
  * Fix crash in ril_query_passwd_state() (LP: #1231995).
 -- Ricardo Salveti de Araujo <email address hidden> Sun, 13 Oct 2013 18:14:48 -0300