Intermittent crashes copying files involving symbolic links

The approach used by python-distutils-extra to install icon (and perhaps other) symbolic links can proceed or crash depending upon the random order in which icon file copies are executed.

The crash happens while trying to create the debian/tmp/ file tree when apport's make processess calls python-distutils-extra through setup.py using the following command:

  python setup.py install \
  --root="/home/gruber/packages/uapport/debian/tmp/" \
  --install-purelib=/usr/lib/python2.7/site-packages/ \
  --no-compile --install-layout=deb

For this case I received the following message from pbuilder while trying to build the apport package for ubuntu:
copying data/icons/scalable/mimetypes/text-x-apport.svg -> /tmp/buildd/apport-1.20.1/debian/tmp/usr/share/icons/hicolor/scalable/mimetypes
error: /tmp/buildd/apport-1.20.1/debian/tmp/usr/share/icons/hicolor/scalable/mimetypes/text-x-apport.svg: No such file or directory
make: *** [install/apport] Error 1
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2
-----------------
There are icon files in two different directories, one a symbolic link to the other.

In particular the source icon files are:

-rw-r--r-- 1 gruber gruber 16628 2011-04-25 12:52:10.493242000 -0400 data/icons/scalable/apps/apport.svg
lrwxrwxrwx 1 gruber gruber 18 2011-04-25 12:52:10.409823944 -0400 data/icons/scalable/mimetypes/text-x-apport.svg -> ../apps/apport.svg

these are to be copied by python-distutils to:
lrwxrwxrwx 1 gruber gruber 18 2011-04-25 17:44:10.579475687 -0400 debian/tmp/usr/share/icons/hicolor/scalable/mimetypes/text-x-apport.svg -> ../apps/apport.svg (1) and
lrwxrwxrwx 1 gruber gruber 18 2011-04-25 17:44:10.579475687 -0400 debian/tmp/usr/share/icons/hicolor/scalable/apps/apport.svg (2)

python-distutils-extra makes the symbolic link itself at the very end of auto.py, making destination directory for the symbolic link (1) in the process. python-distutils-extra command/build_icons.py add both source and destination files to distribution.data_files.

python-distutils then has the responsibility to finish the work on these icons. If the above entries were added by command/build_icons.py with the entry for the target icon file first, the build process completes. When file_util.py gets to the data_file entry for the target it builds its directory and then copies apport.svg to it. Next it gets to the entry for the mimetypes directory and checks to see if it is up-to-date and one way or the other proceeds.*

If the two data_file entries are produced in the opposite order file_util.py checks for the existence of destination directory (1) and then tries to find out whether the file is up-to-date. It isn't aware of the symbolic link and doesn't check for the existence of destination directory (2), which doesn't yet exist, so the newer function call fails when the attempt to stat the destination file is made. While python-distutils has functions for producing hard and symbolic links that code is not being accessed for this operation.

On my system the mimetypes directory has always come first and the build always fails. On launchpad the apps directory (target of the symbolic link) is usually listed first and normally the build proceeds without event.

The order of the entries is determined by glob.glob() calls in command/build_icons.py. The documentation for glob.glob() (http://docs.python.org/library/glob.html) says the python library uses the os.listdir() function to produce its results, and http://docs.python.org/library/os.html says that "The list is in arbitrary order." so whether the setup.py command and the build completes or not is also arbitrary.

* There is a second difference in execution between pbuilder and launchpad's build farm. The build farm keeps file modification time in seconds and python-distutils' dep_util.py uses subsecond floating point stat.st_mtime values around line 85 while copy times are preserved in file_util.py using stat[ST_MTIME], an integer value. This doesn't matter on an ext3 file system but causes the destination file to look older than the original on an ext4 file system, which tracks subsecond times (note the file times above). The symbolic-link-unaware copy process in this case erases the symbolic link from destination directory (1) and replaces it with the actual source file (replaces the symbolic link with the actual file). Using the comparison at the precision of 1 second is the better way to solve this to avoid floating point precision problems from continuing to confuse the "newer" function in dep_util.py.