UnicodeDecodeError in friends-dispatcher with corrupted facebook_ids.json

Bug #1189229 reported by Genadi Saltikov
44
This bug affects 6 people
Affects Status Importance Assigned to Milestone
friends (Ubuntu)
Fix Released
High
Robert Bruce Park

Bug Description

I think this is related to having Google Account, and Facebook account set up in Empathy, and also to leaving everything running, then after a while (10 mins I think), the screen locks.

Sometimes when I come back, I unlock to find the above mentioned error report.

ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: friends-dispatcher 0.1.3daily13.04.17.1~13.04-0ubuntu1
ProcVersionSignature: Ubuntu 3.8.0-23.34-generic 3.8.11
Uname: Linux 3.8.0-23-generic x86_64
NonfreeKernelModules: nvidia
ApportVersion: 2.9.2-0ubuntu8.1
Architecture: amd64
Date: Sun Jun 9 22:39:52 2013
EcryptfsInUse: Yes
InstallationDate: Installed on 2013-02-16 (113 days ago)
InstallationMedia: Ubuntu 13.04 "Raring Ringtail" - Alpha amd64 (20130215)
MarkForUpload: True
PackageArchitecture: all
SourcePackage: friends
UpgradeStatus: No upgrade log present (probably fresh install)

Related branches

Revision history for this message
Genadi Saltikov (carmageddon-2) wrote :
Revision history for this message
Robert Bruce Park (robru) wrote :

Friends and Empathy are two completely unrelated components -- they don't even talk to each other. I'm going to need some more information about this bug before we can proceed with it -- what application is mentioned in the "internal error" dialog?

Changed in friends (Ubuntu):
status: New → Incomplete
Revision history for this message
Genadi Saltikov (carmageddon-2) wrote :

Well, I only know I get this annoying report all the time.
Usually after a long period of not using the computer (screen turns off, locks) - when I unlock, I get this crash error greeting me.
This does not happen if it locks and I unlock it within an hour or so - only after a long time like a day, when I come back home from work and unlock screen.

I am attaching a screenshot
I am not sure what is the difference between 'friends' and Empathy, I just know that thing is poor replacement for Google Talk, is too often offline until I actively open it to force it to connect and search contact, and gets messages sometimes in delay of DAYS.

If you tell me what else you need of me to help reproduce and diagnose the source of the error, I'd be happy to help.
Attached is a screenshot - I didnt close them yet.

Revision history for this message
Robert Bruce Park (robru) wrote :

Friends is not a replacement for Google Talk. Friends does not connect to Google Talk at all. In fact, Friends does not connect to any Google technologies. Friends and Empathy are unrelated components that don't interact in any way. Friends downloads messages from things like Facebook and Twitter, it's a social media client. Empathy is an instant messenger client, it would connect to Google Talk or Jabber or MSN messenger or similar.

That screenshot identifies a UnicodeDecodeError happening in friends-dispatcher, but does not contain enough information for me to actually resolve the problem. What I need you to do is click on the 'Continue' button and actually submit that error report. You might also want to check the 'Ignore future errors of this type' checkbox, since it seems to be happening to you a lot.

I also vaguely recall fixing a bug like this recently, so please update your system to all the latest packages and that might resolve the issue. Or if we're unlucky it might be a new issue.

Revision history for this message
Genadi Saltikov (carmageddon-2) wrote :

I just clicked continue on all the reports, but how exactly do I submit them?
I also just did update few dozens of packages like you suggested however friends wasnt one of them.

You saying friends is not related to Google - then who is using the Online Accounts in settings? which has Facebook and Google set up...
If it downloads messages from Facebook and the likes, isnt it also downloading messages from Google Chat?

I will update within a day if/when I get another popup about crashed friends...

Revision history for this message
Robert Bruce Park (robru) wrote :

Clicking continue does submit them. It should open in a new browser tab asking you to report a new bug, and showing other bugs that might be duplicates. Please report a new bug so that I can see it.

Online Accounts settings are there for all apps to use. If any app wants to connect to your Google account, they get that information by accessing the online accounts API. Friends is not one of those apps, at least not for Google. I know this because I wrote the code (and Google doesn't provide an API that would allow us to even do that, even though we want to).

The list of services that friends is currently capable of connecting to is: Facebook, Twitter, Foursquare, Flickr, and Identica. Soon we'll have Instagram and Linkedin support. But not Google.

Revision history for this message
Genadi Saltikov (carmageddon-2) wrote :

Interesting!
It does not open a new browser window to submit bug reports;
As soong as I submit, it disappears with no follow-up.

Revision history for this message
Genadi Saltikov (carmageddon-2) wrote :

Well, after installing all packages and a reboot - I still have the issue, and have pressed the continue button with submit checked.

What else can I do to help?

Revision history for this message
Robert Bruce Park (robru) wrote :

Ok, please take the following steps:

Delete the file ~/.cache/friends/friends.log (it will have a lot of old, irrelevant info in it right now).

Enable debugging info. Open a terminal (Ctrl+Alt+T) and enter this command:

    gsettings set com.canonical.friends debug true

Then reproduce the error.

Now find the new ~/.cache/friends/friends.log file and attach it to this bug report. This file may contain your social networking usernames, so you may want to censor those out for privacy reasons, but other than that, there will not be any passwords or other personal information in there.

Revision history for this message
Genadi Saltikov (carmageddon-2) wrote :

done, I will report as soon as I got something (probably tomorrow after I come back home).
Do I need to restart some process, service?

Revision history for this message
Robert Bruce Park (robru) wrote :

Nope, friends-dispatcher will notice the change and start outputting verbose debugging messages into the log immediately.

Revision history for this message
Genadi Saltikov (carmageddon-2) wrote :

The log does not seem to contain much information as far as I see...

And yes, the crash has just happened again - I came home, turned on my screens, logged in - and after a few minutes of not doing anything (was reading an email on the phone), when I looked back at the screen, I saw the crash report again.

I clicked continue, did you receive it anywhere?
Should I attach anything else from some place?

Revision history for this message
Robert Bruce Park (robru) wrote :

Yeah, it doesn't look like you enabled the debugging info. Even without crashing there should be much more information in that log. Mine typically looks like this, even when there isn't a problem:

INFO MainThread 2013-06-13 06:09:47,256 friends.main Friends backend dispatcher starting
DEBUG MainThread 2013-06-13 06:09:47,265 friends.utils.account Facebook (9) got send_enabled: True
DEBUG MainThread 2013-06-13 06:09:47,266 friends.utils.account Twitter (12) got send_enabled: True
DEBUG MainThread 2013-06-13 06:09:47,268 friends.utils.account Identica (15) got send_enabled: True
INFO MainThread 2013-06-13 06:09:47,269 friends.utils.account Unsupported protocol: linkedin
INFO MainThread 2013-06-13 06:09:47,269 friends.utils.account Accounts found: 5
INFO MainThread 2013-06-13 06:09:47,270 friends.main Starting friends-dispatcher main loop
DEBUG MainThread 2013-06-13 06:09:47,270 friends.service.dispatcher Refresh requested
DEBUG Thread-1 2013-06-13 06:09:47,271 friends.utils.base Flickr.receive is starting in a new thread.
DEBUG Thread-1 2013-06-13 06:09:47,271 friends.utils.base Logging in to Flickr
DEBUG Thread-2 2013-06-13 06:09:47,272 friends.utils.base Facebook.receive is starting in a new thread.
DEBUG Thread-3 2013-06-13 06:09:47,273 friends.utils.base FourSquare.receive is starting in a new thread.
DEBUG Thread-4 2013-06-13 06:09:47,274 friends.utils.base Twitter.receive is starting in a new thread.
DEBUG Thread-2 2013-06-13 06:09:47,274 friends.utils.base Logging in to Facebook
DEBUG Thread-5 2013-06-13 06:09:47,275 friends.utils.base Identica.receive is starting in a new thread.
DEBUG MainThread 2013-06-13 06:09:47,275 friends.service.dispatcher Starting new shutdown timer...
DEBUG Thread-3 2013-06-13 06:09:47,275 friends.utils.base Logging in to FourSquare
DEBUG Thread-4 2013-06-13 06:09:47,276 friends.utils.base Logging in to Twitter
DEBUG Thread-5 2013-06-13 06:09:47,278 friends.utils.base Logging in to Identica
DEBUG MainThread 2013-06-13 06:09:47,778 friends.utils.model Deleted 4 rows from Dee.SharedModel.
DEBUG MainThread 2013-06-13 06:09:47,778 friends.utils.model Trying to save Dee.SharedModel with 2000 rows.
DEBUG MainThread 2013-06-13 06:09:47,779 friends.utils.model Saving Dee.SharedModel with 2000 rows.
DEBUG MainThread 2013-06-13 06:09:47,842 friends.utils.base _seen_ids: 2000
DEBUG MainThread 2013-06-13 06:09:47,848 friends.utils.authentication _login_cb completed
DEBUG MainThread 2013-06-13 06:09:47,848 friends.utils.authentication _login_cb completed
DEBUG MainThread 2013-06-13 06:09:48,015 friends.utils.authentication _login_cb completed
DEBUG MainThread 2013-06-13 06:09:48,239 friends.utils.authentication _login_cb completed
[etc...]

Please open a terminal and tell me the output of this command:

    gsettings get com.canonical.friends debug

Revision history for this message
Genadi Saltikov (carmageddon-2) wrote :

genadi@genadi-deskubuntu:~/.cache/friends$ gsettings get com.canonical.friends debug
true

Revision history for this message
Robert Bruce Park (robru) wrote :

Wow, that's really strange.

Ok, try this:

    killall friends-dispatcher
    friends-dispatcher -d -o

Then you might have to wait up to 15 minutes, and then after that, tell me what it says on the console.

Revision history for this message
Genadi Saltikov (carmageddon-2) wrote :

Actually I didnt have to wait long :)

genadi@genadi-deskubuntu:~/.cache/friends$ killall friends-dispatcher
friends-dispatcher: no process found
genadi@genadi-deskubuntu:~/.cache/friends$ su -
Password:
root@genadi-deskubuntu:~# killall friends-dispatcher
friends-dispatcher: no process found
root@genadi-deskubuntu:~# logout
genadi@genadi-deskubuntu:~/.cache/friends$ friends-dispatcher -d -o
INFO MainThread friends.main Friends backend dispatcher starting
Traceback (most recent call last):
  File "/usr/bin/friends-dispatcher", line 9, in <module>
    load_entry_point('friends==0.1', 'console_scripts', 'friends-dispatcher')()
  File "/usr/lib/python3/dist-packages/friends/main.py", line 136, in main
    Dispatcher(gsettings, loop)
  File "/usr/lib/python3/dist-packages/friends/service/dispatcher.py", line 99, in __init__
    self.account_manager = AccountManager()
  File "/usr/lib/python3/dist-packages/friends/utils/account.py", line 48, in __init__
    self._add_new_account(account_service)
  File "/usr/lib/python3/dist-packages/friends/utils/account.py", line 68, in _add_new_account
    new_account = Account(account_service)
  File "/usr/lib/python3/dist-packages/friends/utils/account.py", line 120, in __init__
    self.protocol = protocol_class(self)
  File "/usr/lib/python3/dist-packages/friends/protocols/facebook.py", line 53, in __init__
    self._timestamps = PostIdCache(self._name + '_ids')
  File "/usr/lib/python3/dist-packages/friends/utils/cache.py", line 61, in __init__
    self.update(json.loads(cache.read()))
  File "/usr/lib/python3.3/codecs.py", line 300, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte

Revision history for this message
Robert Bruce Park (robru) wrote : Re: [Bug 1189229] Re: friends-dispatcher (Empathy) crashes almost always, usually after screen lock due to inactivity, when I unlock screen I have the "Sorry, Ubuntu 13.04 has experienced an internal error"

That is really strange, can you attach ~/.cache/friends/facebook_ids.json
for me?

Note, this file will contain ids of facebook objects in your feed, you may
want to censor those if you are concerned about privacy. I don't need to
see the ids but I need to see the overall format of the file to figure out
why it's doing this.

Revision history for this message
Genadi Saltikov (carmageddon-2) wrote : Re: friends-dispatcher (Empathy) crashes almost always, usually after screen lock due to inactivity, when I unlock screen I have the "Sorry, Ubuntu 13.04 has experienced an internal error"

added

Revision history for this message
Robert Bruce Park (robru) wrote :

Wow, well that file is severely corrupted. Please delete it and friends should start working for you (don't worry, it can recreate the contents of that file on it's own; all that will happen is a minor wastage of bandwidth as friends redownloads messages you already had before).

I'll work on making the code more robust in the face of this file being corrupted.

Changed in friends (Ubuntu):
status: Incomplete → Triaged
importance: Undecided → High
assignee: nobody → Robert Bruce Park (robru)
summary: - friends-dispatcher (Empathy) crashes almost always, usually after screen
- lock due to inactivity, when I unlock screen I have the "Sorry, Ubuntu
- 13.04 has experienced an internal error"
+ friends-dispatcher crashes almost always, usually after screen lock due
+ to inactivity, when I unlock screen I have the "Sorry, Ubuntu 13.04 has
+ experienced an internal error"
summary: - friends-dispatcher crashes almost always, usually after screen lock due
- to inactivity, when I unlock screen I have the "Sorry, Ubuntu 13.04 has
- experienced an internal error"
+ UnicodeDecodeError in friends-dispatcher with corrupted
+ facebook_ids.json
Changed in friends (Ubuntu):
status: Triaged → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package friends - 0.2.0+13.10.20130627-0ubuntu1

---------------
friends (0.2.0+13.10.20130627-0ubuntu1) saucy; urgency=low

  [ Robert Bruce Park ]
  * Handle previously-uncaught exception when json cache contained
    invalid data. (LP: #1189229). (LP: #1189229)

  [ Ubuntu daily release ]
  * Automatic snapshot from revision 214
 -- Ubuntu daily release <email address hidden> Thu, 27 Jun 2013 05:02:18 +0000

Changed in friends (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Genadi Saltikov (carmageddon-2) wrote :

I am still having the friends-dispatcher generating crash reports.
After doing some checks, my version is 0.1.3daily13.04.17.1~13.04-0ubuntu1

Is there any chance that you didnt release the fix for Raring 13.04 64bit?

Revision history for this message
Robert Bruce Park (robru) wrote :

That's right, this fix only exists in version 0.2 which has only been released to Saucy.

Can you check the contents of facebook_ids.json again? After you deleted it it should have been recreated properly. I have no idea why this corruption would happen repeatedly, but only for you. Nobody else is reporting or experiencing this problem.

Revision history for this message
Genadi Saltikov (carmageddon-2) wrote :

The file looks good, but just for good measure I used the instructions doing friends-dispatcher -d -o
So far nothing suspect, I will leave it running and will update if anything weird happens.

However, that does not answer why the version does not get updated to the one you have patched.. where is the patch? its not in the Raring repo.

Revision history for this message
Robert Bruce Park (robru) wrote :

Raring is released. Updates for it don't just happen automatically. To get this patch backported into raring it has to go through a thorough and lengthy review process that we call an SRU (Stable Release Update). This process was put in place in order to prevent new bugs from getting pushed back into older releases (ie, we want to be really sure that patch we're SRUing is actually just one single fix, and doesn't introduce a bunch of new problems like new software often does).

Considering that this bug seems to only affect you, and ought to be easily fixed by simply deleting one file every now and again, I don't personally feel that this problem warrants an SRU. If we suddenly had hundreds of people reporting this, or if it wasn't so easy to just delete that json file (very few bugs have such an easy workaround), then I might be more inclined to SRU this fix.

As it stands, Saucy will be out in 3 months and Raring will EOL by January. If this problem *really* bothers you a lot, just update to Saucy when it comes out, and you'll get the fix then. Or if you're really impatient, update to saucy after the beta freeze in September and you'll be fine.

Also, I just had a second look at the patch that fixed this issue, and it turns out that it's not even really compatible with the Raring version anyway, due to other changes that have happened in the affected file since then. Not to say that backporting it is impossible, just that the patch would need to be completely re-written, which would make the SRU process even longer and more difficult than it normally is.

Revision history for this message
Genadi Saltikov (carmageddon-2) wrote :

I completely understand.
Thanks for clearing that up, and for the hardwork! :)

Will upgrade to Saucy when its out (don't have time currently to deal with beta versions, too busy with work+studies for another year).

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.