Default Chinese font changed to fonts-arphic-ukai after completing language support installation for zh-* locales

This is caused by the installation of fonts-arphic-ukai and fonts-arphic-uming, while they take the precedence over default system font in fontconfig-config/language-selector.

As we are moving from WQY MicroHei to Droid Sans Fallback, it'd be good to check if this is still relevant and we probably need to check the mentioned packages.

Which Ubuntu version are you using, anywnztj?

As far as I understand, this should not be the case on an updated Ubuntu 14.04. It's true that fonts-arphic-uming and fonts-arphic-ukai are pulled, but as a result of the discussion at bug #1173571, for a few months Droid Sans Fallback has been the preferred font for sans-serif and monospace in the 69-language-selector-zh-* files. And the latter take precedence over the fontconfig files, don't they?

This problem still exists in Ubuntu Kylin 14.04 amd64 & i386 daily build, downloaded 20 feb 2014.

Hi Gunnar, I'm using an up-to-date installation of Ubuntu, and it does exist for me, :-(

Thanks jiaowen520li and Aron.

One thing that should be said about the 69-language-selector-zh-* files is that they only apply if the current display language is Chinese. So can you please let us know if Ukai is chosen over Droid Sans Fallback when e.g. zh_CN is the display language, or does it happen only when the display language is something else but Chinese?

Anyway, I decided to make an experiment, so I uploaded a slightly modified version of the fonts-droid package to my PPA at https://launchpad.net/~gunnarhj/+archive/fonts
I simply renamed 65-droid-sans-fonts.conf to 67-droid-sans-fonts.conf in an attempt to raise precedence.

Can you please test that and let us know if it makes a difference?

ping?

Hi Gunnar, I just installed the packages in your ppa, while it makes no difference.

Thanks for letting us know, Huan Peng. It was a long shot..

Can anybody answer the other question I asked too:

"Can you please let us know if Ukai is chosen over Droid Sans Fallback when e.g. zh_CN is the display language, or does it happen only when the display language is something else but Chinese?"

I think the answer to that question is important to know where to begin to fix the issue.

Hi, Gunnar, I just confirmed the problem only exist in zh_CN, zh_HK and zh_TW are all ok.

On 2014-03-06 05:14, Huan Peng wrote:
> Hi, Gunnar, I just confirmed the problem only exist in zh_CN, zh_HK
> and zh_TW are all ok.

That makes this problem even more mysterious. All of
- 69-language-selector-zh-cn.conf
- 69-language-selector-zh-hk.conf
- 69-language-selector-zh-tw.conf
have Droid Sans Fallback as the first sans-serif font.

I'm out of ideas. To make progress with respect to this bug report, somebody using the languages in question has to play around with the font configuration files to figure out what needs to be changed.

I also get a strage problem. The computer that got the problem now is ok, these days i have the system upgraded and install some packages. But i don't know why it is ok now, i just use the system do some other things.

 But I install another system and install package fonts-arphic-ukai and fonts-arphic-uming, the problem comes. And then i install zh_HK and zh_TW, the problem still only exists in zh_CN, I also installed the packages in your ppa, no use.

And i do a diff to the two system's fonts directory, there are some diffs in 65-droid-sans-fonts.conf, the two system are all installed the apckages in your ppa, here i attach the two system's fonts directory, hope can help.

Huan, since you keep addressing me, and to avoid misunderstandings, please let me say again that I'm not the person to figure out what the problem is. Somebody should do it who both uses Chinese and understands the font configuration stuff.

I'll be happy to make the necessary changes in language-selector - if that's where the problem proves to to lie - when somebody shows what needs to be changed.

Hi Huan Peng, you can use fc-match command to figure out which font is actually selected by fontconfig, for example:

 fc-match "sans-serif"

Hopes this helps!

Thanks for the tip, Pellaeon Lin! To me it's useful indeed.

$ LANG=zh_CN.UTF-8 fc-match 'sans-serif'
ukai.ttc: "AR PL UKai CN" "Book"
$ LANG=zh_HK.UTF-8 fc-match 'sans-serif'
uming.ttc: "AR PL UMing CN" "Light"
$ LANG=zh_TW.UTF-8 fc-match 'sans-serif'
DroidSansFallbackFull.ttf: "Droid Sans Fallback" "Regular"

Not good.

I made a couple of changes to language-selector:

https://launchpadlibrarian.net/170732046/language-selector_0.124_0.125~ppa.diff.gz

and uploaded to my PPA at https://launchpad.net/~gunnarhj/+archive/fonts

$ LANG=zh_CN.UTF-8 fc-match 'sans-serif'
DroidSansFallbackFull.ttf: "Droid Sans Fallback" "Regular"
$ LANG=zh_HK.UTF-8 fc-match 'sans-serif'
DroidSansFallbackFull.ttf: "Droid Sans Fallback" "Regular"
$ LANG=zh_TW.UTF-8 fc-match 'sans-serif'
DroidSansFallbackFull.ttf: "Droid Sans Fallback" "Regular"

Looks promising, doesn't it? :)

I'd like a few Chinese users to test the language-selector version in my PPA.

Then there are a few questions:

* Should this change make it to the archive, or are there drawbacks?
* Should we add 'binding="strong"' to all the
  69-language-selector-zh-*.conf files to be safe?
* Should we add 'binding="strong"' to serif and monospace as well?

language-selector in your ppa works well，i just tested in daily iso

Thanks Huan Peng!

Then there are the other questions... I tend to think that 'binding="strong"' should be added all over in the 69-language-selector-zh-*.conf files, to make the actual fontconfig behavior less random. Any objection before I proceed?

Thanks Gunnar.

I am not familiar with the language-selector.

However I check the 69-language-selector-zh-*.conf files in 12.04 as well as latest 14.04, then I find the following difference:
(1) binding="strong" is added in 69-language-selector-zh-cn.conf in Chinese version 12.04
(2) in live mode of latest 14.04, binding="strong" is added in 69-language-selector-zh-tw.conf, but not in 69-language-selector-zh-cn.conf

So I think you are right:)

Ok, I decided to go ahead, and the language-selector fix is now in the trusty upload queue. It will give us a couple of weeks with a full scale test.

This bug was fixed in the package language-selector - 0.126

---------------
language-selector (0.126) trusty; urgency=medium

  * LanguageSelector/ImConfig.py:
    Make fcitx the system default if installed (LP: #1297831).
 -- Gunnar Hjalmarsson <email address hidden> Thu, 27 Mar 2014 15:09:00 +0100

This bug is NOT fixed! Here are my outputs of fc-match:

$ LANG=zh_CN.UTF-8 fc-match 'sans-serif'
DroidSansFallbackFull.ttf: "Droid Sans Fallback" "Regular"
$ LANG=zh_HK.UTF-8 fc-match 'sans-serif'
DroidSansFallbackFull.ttf: "Droid Sans Fallback" "Regular"
$ LANG=zh_TW.UTF-8 fc-match 'sans-serif'
DroidSansFallbackFull.ttf: "Droid Sans Fallback" "Regular"

And there is binding="strong" in my 69-language-selector-zh-cn.conf but my screen Chinese font is still UKai.

I started with 14.04 Beta 2 when it came out and have been doing updates everyday.

Kuangting Liu,

Is your display language zh_CN? (69-language-selector-zh-cn.conf is only effective if it is.)

When you say that your "screen Chinese font" is Ukai, which application(s) are you talking about? Are there possibly fonts settings in those applications which override fontconfig?

My default language is English, i.e. my windows, menus, etc. are displayed in English. But when I display, for example, a text file written in Chinese, or when I enter Chinese characters in a Google search, or go to a Chinese web site, the Chinese words are displayed in UKai. This is true regardless which applications I use (firefox, text editor, terminal, thunderbird, etc.). In 13.10, this problem did not exist.

I even modified the 69-language-selector-zh-cn.conf to completely remove UKai entries but still no effect. (Yes, I've rebooted.) It's as though those .conf files are not even used. Or maybe I need to rebuild something (database, cache?)?

OK. I just changed my display language to Chinese and you were right. It's working correctly.

So...is it possible to change default Chinese font when display language is not Chinese? In other words, to make it work like 13.10 & prior?

Thank you.

On 2014-04-01 14:29, Kuangting Liu wrote:
> is it possible to change default Chinese font when display language
> is not Chinese?

I'm sure it's possible; I just made an attempt.

In 13.10 ttf-wqy-zenhei was installed if you installed a Chinese language. ttf-wqy-zenhei is not installed automatically in 14.04 (it was replaced by fonts-droid).

The ttf-wqy-zenhei package includes the fontconfig recipe
/etc/fonts/conf.avail/64-wqy-zenhei.conf, while the fonts-droid equivalent /etc/fonts/conf.avail/65-droid-sans-fonts.conf is quite different.

So I prepared a change of fonts-droid where I added a recipe with 64-wqy-zenhei.conf as a model. The modified fonts-droid package is available in my PPA at https://launchpad.net/~gunnarhj/+archive/misc

Can you please install fonts-droid from my PPA and let us know if it makes a difference?

Thank you, Gunnar. That worked. After I installed your fonts-droid my screen Chinese font changed back to Droid Sans (or something looks like Droid Sans font). Thank you very much for the fix.

You're welcome; thanks for testing!

I have proposed it to be uploaded to the Ubuntu archive.

This bug was fixed in the package fonts-android - 1:4.3-3ubuntu1

---------------
fonts-android (1:4.3-3ubuntu1) trusty; urgency=medium

  * debian/local/65-droid-sans-fallback.conf:
    - Additional fontconfig recipe similar to 64-wqy-zenhei.conf. This
      makes Droid Sans be used for rendering Chinese rather than
      AR PL UKai or AR PL UMing when the display language is something
      else but Chinese (LP: #1227034).
 -- Gunnar Hjalmarsson <email address hidden> Wed, 02 Apr 2014 03:08:00 +0200

Hi Gunnar and all,

This is regression report of this updates from *Japanese* user.

- Today, I update my 14.04 testbeds(yes, i have three testbeds), and applied this update. My desktop font are changed. Probably, generic sans-serif override by droids. In Japanese environment, that expects takao fonts. All of my testbeds are affected.

This breaks only appearance-level (non japanese glyphs), I can read it, but I have bring a feeling of strangeness. This distinction are important for user experience.

I have two questions about this updates, because this new fontconfig *breaks Japanese* font environments (probably. currently suspicious, but i delete 65-droids, problem has gone. ).

Question 1)
This updates is good approach, but that breaks non-Chinese environments. Probably, this configs are over-powered. We have to limit this configs.
Could you please add "match" section for 65-droids like 65-fonts-takao's? And, that is important,

Question 2)
Is this(question 1's addition) require UIFe? I belive original changes had not UIFe applications, this more changes does not.

# I belive that this original changes are too late for LTS development cycle, at least, we must declare UIFe. Therefore, I look sideways this changes progress, I had not tested. real root cause are my remissness. apologies.

I send screenshot of this bug affects in japanese environment.
Left side: Before / Right side: Now(affects this bug)

Hi Fumihito / Shushi!

Thanks for reporting the effect on Japanese rendering. Really no reason to apologize, Fumihito!

I don't think it's impossible to fix it for the LTS, and I'm going to try. If this would be a UIFe thing, it would be nothing but a formality, since there are no Japanese screenshots in the docs and we are not talking about changing translatable strings.

I'm going to look at it soon, and try to come up with something that you can test tomorrow morning (your time).

Fumihito / Shushi,

I wrote a patch that adds 'binding="strong"' to two of the fonts-takao fontconfig recipes. A fonts-takao build with the patch is available in my PPA at https://launchpad.net/~gunnarhj/+archive/misc

With the current archive versions of the fonts-takao-gothic and fonts-takao-pgothic binary packages I get this:

$ LANG=ja_JP.UTF-8
$ fc-match 'sans-serif'
DroidSansJapanese.ttf: "Droid Sans" "Regular"
$ fc-match 'serif'
fonts-japanese-mincho.ttf: "Takao P明朝" "Regular"
$ fc-match 'monospace'
DroidSansJapanese.ttf: "Droid Sans" "Regular"

After having installed the versions of those packages from my PPA, I get this:

$ LANG=ja_JP.UTF-8
$ fc-match 'sans-serif'
fonts-japanese-gothic.ttf: "Takao Pゴシック" "Regular"
$ fc-match 'serif'
fonts-japanese-mincho.ttf: "Takao P明朝" "Regular"
$ fc-match 'monospace'
TakaoGothic.ttf: "Takaoゴシック" "Regular"

Can you please install fonts-takao-gothic and fonts-takao-pgothic from my PPA, carry out a 'real' test, and let us know the result?

Hi Gunnar,

I test with below packages, it works perfect. Japanese fonts rendering fine as I had expected.

$ dpkg -l |grep takao
ii fonts-takao-gothic 003.02.01-9ubuntu2~ppa all Japanese TrueType font set, Takao Gothic Fonts
ii fonts-takao-mincho 003.02.01-9ubuntu2~ppa all Japanese TrueType font set, Takao Mincho Fonts
ii fonts-takao-pgothic 003.02.01-9ubuntu2~ppa all Japanese TrueType font set, Takao P Gothic Fonts

Just to make sure, I contact Shushi and some Japanese users, and re-check with this packages. Please wait a few hours.

Hello Gunnar.

I'm Japanese user.
I had tested and look that your patch is good work.

I worked that...
----------------------
1. sudo apt-get update && sudo apt-get upgrade
2. Capture screenshot refarence #30
3. sudo apt-add-repository ppa:gunnarhj/misc
4. sudo apt-get update && sudo apt-get upgrade
5. Capture screenshot

This is just a quick note.
Thank you

I test with some japanese users(included nekomatu, Shushi), that seems fine.
So, I feel confident that fix are perfect!

Uploaded fonts-takao, thanks Gunnar!

This bug was fixed in the package fonts-takao - 003.02.01-9ubuntu2

---------------
fonts-takao (003.02.01-9ubuntu2) trusty; urgency=medium

  * 65-fonts-takao-gothic.conf, 65-fonts-takao-pgothic.conf:
    Added 'binding="strong"' for sans-serif and monospace. This makes
    Takao fonts be used for Japanese despite of the new
    65-droid-sans-fallback.conf, which is included in the seeded
    fonts-droid package (LP: #1227034).
 -- Gunnar Hjalmarsson <email address hidden> Tue, 08 Apr 2014 21:06:00 +0200

Hi Gunnar,

Thanks for the good work. The zh_CN locale now works as expected.

However, I wish I could say the same thing for the en_US locale. I am attaching a screenshot showing that many Chinese characters were missing (represented by rectangles) when running the Chromium browser.

I would like to note several other observations:

1. The missing Chinese characters were properly shown in Firefox. I.e., No problem in Firefox, big problem in Chromium.

2. In Chromium, I cannot find Droid Sans Fallback fonts in the (Setting --> Select default fonts).

3. I am also encountering missing Chinese character problems in using Fcitx (Chinese input engine). This may render Ubuntu 14.04 unusable for (at least) English locale users who have a need to input Chinese characters.

4. In zh_CN.UTF-8 locale, the command ' fc-match "sans" ' returns

      DroidSansFallbackFull.ttf: "Droid Sans" "Regular"

4. In en_US.UTF-8 locale, the same command ' fc-match "sans" ' returns

     DejaVuSans.ttf: "DejaVu Sans" "Book"
language-selector fix

Add: The problems observed in UbuntuKylin, as reported above, do not appear to be duplicatable in (regular) Ubuntu 1404. Thus, those problems seem to be limited to UbuntuKylin.

Hi Ping-Wu,

I think it's better if you file a new bug report to keep track of these seemingly Kylin + Chromium + fcitx specific issues.

Hi Gunnar,

OK, will do. Thanks.

Hi Gunnar and All,

Installed most recent (140412) Kylin build. Everything seems to be working OK now (except for the need of a little tweaking in Chromium).

The original problem might have been caused by the repo sync problem (Kylin repo was not available during the previous apt-get update). Let's hope this is the case. Keep fingers crossed & look forward to the official release (as well as a stable Kylin repo).

Hi Ping-Wu and Gunnar and All, thanks all your contributions.

It seems like that this is affecting Trusty as well. I was trying to add some IME for Traditional Chinese and this was the direction I took:

- Leave desktop manager untouched (Unity)
- sudo gnome-language-selector
- Click "Install/Remove Languages"
- Check "Chinese (traditional)"
- Restart

After restarting the computer, it seems like the default font used for Traditional Chinese Unicode characters is the "AR PL UKai" family (especially for both sans-serif and sans). This font looks very unreadable, especially on web pages. Are the fixes going to cherrypick back to Trusty?

@Alex: Yes, a mysterious issue popped up as described in bug #1335482. There is a fix in trusty-proposed (soon in trusty-updates), and it would be great if you could install language-selector-common 0.129.2 and language-selector-gnome 0.129.2 from trusty-proposed and let us know if it fixes the issue for you.

Hmm... Maybe I misunderstood you. I assumed that you also changed the display language to Traditional Chinese, but I realize now that you did not say so. So, what is your display language for menus and messages?

The display language is still English (unchanged.) The weird part is that fc-match returns "AR PL UMing TW". Let me know if I should still try language-selector-common 0.129.2 and language-selector-gnome 0.129.2 and update in the other bug https://bugs.launchpad.net/bugs/1335482.

$ sudo locale-gen zh_TW.UTF-8
Generating locales...
  zh_TW.UTF-8... done
Generation complete.

$ LANG=zh_TW.UTF-8 fc-match 'sans-serif'
uming.ttc: "AR PL UMing TW" "Light"

Well, if you run fc-match with the zh_TW.UTF-8 locale (which in fact simulates the use of Traditional Chinese as the display language), it indeed should make a difference if you install version 0.129.2 of language-selector-*.

OTOH, if you actually are using English as the display language, a more adequate test is:

fc-match -a 'sans-serif'

That should result in a long list, and you ought to see a bunch of Droid Sans fonts (including DroidSansFallbackFull.ttf) before AR PL UMing TW.

"fc-match -a 'sans-serif'" shows that ukai.ttc is before uming.ttc and after Droid Sans. DroidSansFallbackFull.ttf is before ukai/uming.ttc.

ukai (AR PL UKai) should be after uming.

On 2014-07-23 17:14, Alex Yu wrote:
> "fc-match -a 'sans-serif'" shows that ukai.ttc is before uming.ttc
> and after Droid Sans. DroidSansFallbackFull.ttf is before
> ukai/uming.ttc.

Yes, that's what I see too.

> ukai (AR PL UKai) should be after uming.

Considering that DroidSansFallbackFull.ttf is installed by default on all machines, whether Chinese language support has been explicitly installed or not, why would it make a difference? The default configuration makes DroidSansFallbackFull.ttf be used for rendering Chinese contents irrespective of the order of UKai and UMing, doesn't it?

But for whatever reasons, UKai is showing as the default font in Chrome (I haven't touched the font settings in Chrome.) When I checked the Advanced Font Settings for "Traditional Han" script, standard/serif/sans-serif are using default font but the preview is using UKai font. Umm, I just checked Firefox and it is behaving differently.

Well, the default font configuration can't reasonably cover all cases. It's much easier to control which font is used when you use a related locale, and if you switch to Traditional Chinese as the display language, you'll find that they show up in the order you say is the preferred.

Actually, Chinese is a special case. As a result of this bug report, we added an extra config file to the fonts-droid package in order to give DroidSansFallbackFull.ttf precedence over other fonts that claim an ability to render Chinese, and with that make it easy to view Chinese contents also in case of a non-Chinese locale. But that kind of configuration, which affects all users, must not be too obtrusive.

You may want to file a new bug about the chrome issue.