Merge lp:~paulgear/python-oops-datedir-repo/variable-retention-period into lp:python-oops-datedir-repo

[Paul Gear]

Landscape have requested longer retention for OOPSes; this is a presently-untested change to allow that. cRT#89955

[lp:~paulgear/python-oops-datedir-repo/variable-retention-period]

59. By Paul Gear on 2016-03-23

Update for log directory used in production

60. By Paul Gear on 2016-03-23

Specify option type

[Данило Шеган (danilo)]

Looks good, other than the changed default value for filename. If there's no particular reason to change that, it seems best to leave it as-is.

review: Approve

[Paul Gear (paulgear)]

> Looks good, other than the changed default value for filename. If there's no
> particular reason to change that, it seems best to leave it as-is.

My commit message seems to indicate that that's what is used by the running copy of this code in Launchpad production. I'd have to go back and confirm, but that's the only reason I can think of that I would have changed it.

If you aren't comfortable doing that, feel free to edit out when merging and we can maintain the patch locally in production.

[Данило Шеган (danilo)]

@Paul: do you keep a patch or pass the option instead?

Are you aware of any other users of oops code that might be affected (we've got Launchpad and Landscape).

Anyway, I am fine with it, but it generally belongs in a separate branch, though it is very small.

[Paul Gear (paulgear)]

@danilo: This was requested for Landscape by @benji. I'm not aware of any other users, but this is my first exposure to this codebase, so I'm not authoritative on it.

[Данило Шеган (danilo)]

@Paul: sure, I am on the Landscape team, so as a former Launchpad developer (with review rights), I've decided to look at this to clean up the queue (or, tbh, I've been prodded by Adam Collard to do it :)).

I was wondering if IS knows where else is this code deployed so we can contain the risk of any change? Eg. I suspect at least Launchpad is using it, and I don't want it to fall over because there is no "logs" subdirectory somewhere.

As for merging, it is usually much easier and safer if you revert the change in question, so when "landing", I can simply merge your changes.

[Paul Gear (paulgear)]

@danilo: I suspect Launchpad is using this, but I couldn't say where other than by using a brute force method.
@wgrant: Any suggestions on where to start looking?

[William Grant (wgrant)]

The only place LP uses this nowadays is with the rest on neem. The hardcoded log directory is already unpleasant, but I think the change here makes it even a bit worse. I'd revert the logging change for now, and leave it as a local change on neem.

[Данило Шеган (danilo)]

@Paul: I guess we all kind of agree to drop the change to the logging filename and then we can get it merged.

@William: does this code use PQM to land stuff (yes, it's been a while :)), or should we land it in some other way (just approve for tarmac, manually...)?

[William Grant (wgrant)]

On 14/10/16 17:46, Данило Шеган wrote:
> @Paul: I guess we all kind of agree to drop the change to the logging filename and then we can get it merged.
>
> @William: does this code use PQM to land stuff (yes, it's been a while :)), or should we land it in some other way (just approve for tarmac, manually...)?

It's owned by ~canonical-launchpad-branches, of which you are a member.
Just land it manually.

[Данило Шеган (danilo)]

У пет, 14. 10 2016. у 07:27 +0000, William Grant пише:
>
> It's owned by ~canonical-launchpad-branches, of which you are a
> member.
> Just land it manually.

Great, will do that as soon as the logging change is reverted. Thanks!

[lp:~paulgear/python-oops-datedir-repo/variable-retention-period]

61. By Paul Gear on 2017-01-30

Revert default log location change

[Paul Gear (paulgear)]

Apologies for the delay on this; I've reverted that log location change as requested.

Unmerged revisions

61. By Paul Gear on 2017-01-30

Revert default log location change

60. By Paul Gear on 2016-03-23

Specify option type

59. By Paul Gear on 2016-03-23

Update for log directory used in production

58. By Paul Gear on 2016-03-23

Remove unused code

57. By Paul Gear on 2016-03-23

Update documentation for --retention

56. By Paul Gear on 2016-03-23

Allow use of different retention periods

=== modified file 'oops_datedir_repo/prune.py'
--- oops_datedir_repo/prune.py	2012-09-26 06:57:23 +0000
+++ oops_datedir_repo/prune.py	2017-01-30 04:35:30 +0000
@@ -28,7 +28,6 @@
 import sys
 
 from launchpadlib.launchpad import Launchpad
-from launchpadlib.uris import lookup_service_root
 from pytz import utc
 
 import oops_datedir_repo
@@ -106,8 +105,8 @@
         --project and --projectgroup can be supplied multiple times.
 
         When run this program will ask Launchpad for OOPS references made since
-        the last date it pruned up to, with an upper limit of one week from
-        today. It then looks in the repository for all oopses created during
+        the last date it pruned up to, using the upper limit specified with
+        --retention. It then looks in the repository for all oopses created during
         that date range, and if they are not in the set returned by Launchpad,
         deletes them. If the repository has never been pruned before, it will
         pick the earliest datedir present in the repository as the start date.
@@ -122,6 +121,9 @@
         help="Launchpad project group to find references in.")
     parser.add_option('--repo', help="Path to the repository to read from.")
     parser.add_option(
+        '--retention', help="Number of days to keep OOPSes", default=7,
+        type="int")
+    parser.add_option(
         '--lpinstance', help="Launchpad instance to use", default="production")
     options, args = parser.parse_args(argv[1:])
     def needed(*optnames):
@@ -140,10 +142,9 @@
     logging.basicConfig(
         filename='prune.log', filemode='w', level=logging.DEBUG)
     repo = oops_datedir_repo.DateDirRepo(options.repo)
-    one_week = datetime.timedelta(weeks=1)
-    one_day = datetime.timedelta(days=1)
-    # Only prune OOPS reports more than one week old.
-    prune_until = datetime.datetime.now(utc) - one_week
+    retention = datetime.timedelta(days=options.retention)
+    # Only prune OOPS reports older than the retention period
+    prune_until = datetime.datetime.now(utc) - retention
     # Ignore OOPS reports we already found references for - older than the last
     # prune date.
     try: