Merge lp:~brandon-jiang-p/variety/variety into lp:~variety/variety/trunk

[Brandon]

Feature: avoid duplicate image when download.

Avoid to download duplicate images which have been downloaded before. For exisiting variety user, you may also consider to hash all previous images.

Feature: avoid duplicate image when download.

Avoid to download duplicate images which have been downloaded before. For exisiting variety user, you may also consider to hash all previous images.

[James Lu (jlu5)]

I left a few comments in the diff below.

[Peter Levi (peterlevi)]

I have bigger conceptual problems with this proposal. First, md5 is a bad way to compare images as the slightest change in the metadata results in different hashes. It's as good as or worse than simply taking the download url, so why not skip the heavy computation step of hashing the whole image and just use the url. But much more importantly - total downloads size is already guarded by the download quota, there is no reason to introduce a second, stateful mechanism for the same. Also not all downloads end up as wallpapers, the user may have paused the changes. If the image source are consumed in the meantime (some sources are limited), we don't want to stagnate on the last downloaded images and never restart the sources. And also we are piling up a huge file of hashes that never shrinks and search in it. This gets slower with time.

review: Disapprove

Unmerged revisions

555. By Brandon on 2016-06-19

Reject duplicate image when download

avoid to download duplicate images which have been downloaded. For exisiting variety user, you may also consider to hash all previous images.

=== modified file 'variety/Downloader.py'
--- variety/Downloader.py	2016-06-04 09:11:18 +0000
+++ variety/Downloader.py	2016-06-19 08:44:59 +0000
@@ -82,9 +82,9 @@
             return None
 
         try:
-            r = Util.request(image_url, stream=True)
+            data = Util.fetch(image_url)
             with open(local_filename, 'wb') as f:
-                Util.request_write_to(r, f)
+                f.write(data)
         except Exception, e:
             logger.info(lambda: "Download failed from image URL: %s (source location: %s) " % (image_url, self.location))
             raise e
@@ -94,6 +94,19 @@
             os.unlink(local_filename)
             return None
 
+        #check hash modi_v
+        import hashlib
+        logger.info(lambda: self.parent.real_download_folder)
+        hash_store_location=os.path.join(self.parent.real_download_folder, "image_hash_store")
+        file_hash=hashlib.md5(open(local_filename, 'rb').read()).hexdigest()
+
+        if  os.path.exists(hash_store_location) and file_hash in open(hash_store_location).read():
+            logger.info(lambda: "find same file hash. delete file")
+            os.unlink(local_filename)
+            return None
+        with open(hash_store_location,"a") as myfile:
+            myfile.write(file_hash+"\n")
+
         metadata = {
             "sourceType": source_type,
             "sourceName": source_name,
@@ -103,7 +116,6 @@
         }
         metadata.update(extra_metadata)
         Util.write_metadata(local_filename, metadata)
-
         logger.info(lambda: "Download complete")
         return local_filename