Upload of Map with German Umlaut fails

Thanks for your bugreport.

There are currently less people working for the website... so it could take a while to solve the bug, sorry.

Forgotten to add the error from server:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u0308' in position 73: ordinal not in range(128)

I deleted the uploaded map because it was continuously throwing errors in the website.

Hm... i couldn't reproduce this error locally.

From the error message it seems like an ASCII-Unicode error, but all values given by map_object_info are unicode strings. The table in DB is using utf8 as charset.

For now i have no idea whats the cause. I am tend to upload the map again and see what happens...

Ok, the map causes a server error... but at least the error handling is much better now ... e.g. the map isn't saved into the database.

hm...

Just found: https://docs.djangoproject.com/en/1.8/ref/unicode/#files saying:

"If your environment isn’t configured correctly, you’ll encounter UnicodeEncodeError exceptions when saving files with file names that contain non-ASCII characters."

In can reproduce this error now if i set my LANG environment variable to

LANG="en_US.UTF-8"

By default its de_DE.UTF-8 here at home. If i login onto the server it's 'de_DE.utf8' and that is default at all on the server. Because LANG='de_DE.UTF-8' shows NO error at home and LANG='de_DE.utf8' DO show the error on the server it's a bit curious.

The only difference between here a home and the server is the LANGUAGE environment variable:

At home: LANGUAGE=de
Server: LANGUAGE=de_DE.utf8

According to the Ubuntu help LANGUAGE should not contain character encoding parts:

"Unlike "LANG" and "LC_*", "LANGUAGE" should not be assigned a complete locale name including the encoding part (e.g. ".UTF-8"). Instead "LANGUAGE" should contain a colon separated priority list of language codes, for instance "es:de:en". A single language code is also correct."

See: https://help.ubuntu.com/community/EnvironmentVariables#The_LANGUAGE_priority_list

I currently don't know which side effect changing the LANGUAGE environment variable could have or if it is at least needed on the server.

Maybe the easiest thing would be to catch the UnicodeEncodeError in a try: except: statement and provide a error message back to the form.

Providing an error message is a good idea. Even if you manage to fix this, there might be some other weird file names that we haven't expected causing trouble in the future.

Just for information:

I have added 'charset UTF-8;' to /etc/nginx/nginx.conf and reloaded nginx and also restartet wlwebsite. No success...
There is also a page added to check some values from environment directly in productive website: https://wl.widelands.org/locale/

Meanwhile i solved the issue here at home. Setting LANG="en_US.UTF-8" here at home did not work because it was not enabled and so the default 'C' languagetype was used. Enabling and running locale-gen solved the curious thing here at home.

But this is maybe a hint on what is going wrong on the server (misconfiguration of locales?). I have tried to figure out where the LANGUAGE variable is set, but i couldn't find such a setting in a file (f.e. /etc/default/locale). I want to set it explicitly in /etc/default/locale like LANGUAGE="de:en", but i don't know exactly when this will take effect. I know it take effect when starting a new shell, but i don't know if a new shell is used when restarting the website...

We have to modify the startscrpt in /etc/init/wlwebsite.conf and add some exports there:

export LC_ALL="de_DE.utf8"
export LANG="de_DE.utf8"

Source: http://serverfault.com/questions/699543/gunicorn-behave-differently-when-run-from-terminal-and-by-service

This is tested on the alpha site and works :-) On nit anyway: The name of the image uses still some wrong characters and so it isn't shown on the maps page, but i think my next proposal will fix this.

According to that link, you need change

    export LC_ALL=en_US.UTF-8

to

   env LC_ALL="en_US.UTF-8"

I think using export or using env is a thing of the init system. But i am not sure though. Using export="xyz" works on the alpha site (which is afaik using upstart as init system)

Well, I have no clue - I just thought I'd point it out in case you missed it.

The best solution would probably to change it, test, and it it fails revert and go back to the drawing board. And test what is working on the Alpha site first.

Yes, thanks for pointing to this :-)

Of course it has to be tested well... maybe this change has also effect on date values or so...

Maybe this fixes also bug 811686 "Wiki article was last changed tomorrow" :D (just joking)

I make a branch and merge it on the alpha site, so it can be tested :-)

Evil map

Evil image

This should be solved now :-)

Finally...