ppa-dev-tools

Code review comment for ppa-dev-tools:add_rdepends_argument

Revision history for this message

Bryce Harrington (bryce) wrote on 2023-04-20:

Hi Sergio, thanks for reviewing.

Indeed, this is one of the next items on the todo list after this MP is landed. This branch should be considered the proof-of-concept for the basic functionality, to be followed with some convenience improvements and optimizations.

I don't know if you remember but the last few weeks I've researched into some caching and mirroring options and tools. Particularly I was looking at caching at urllib/http level python modules, however I did also look at apt-mirror and other Debian tools. I found one option for the former I want to explore more, but the latter options looked too overkill and dependency-heavy. I didn't look at chdist though, so will try to peek just in case. But the basic idea I have as a solution is to integrate "lazy mirroring" where instead of directly loading Sources.xz and Packages.xz from disk, if what's on disk is older than N hours, it automatically re-downloads and re-caches the file (and ONLY the requested file) on the fly. From my testing so far, on a decent network connection that should be quite performant.

I was hoping to slip that feature work in before Prague (and have at least started sketching it in on another branch) but I'm realizing there's some caveats I need to account for, and time is getting short, so decided to just go with the (reliable) rsync for now and make this a next-release enhancement. There's a couple other convenience improvements I plan to make too, and maybe other folks will also have ideas to add in.

I hadn't looked at chdist, but I did look into apt-mirror and some other Debian and python tools for mirroring apt, and for general http client-side cache management. There's one cache manager module I want to look into a bit more, but the tools I played with were a bit too heavyweight for what I need, and added dependencies.

Anyway, yes I agree more automation of the metadata fetching would indeed be awesome, and plan to do so but maybe needs to wait for next release?

Hi Sergio, thanks for reviewing.

Indeed, this is one of the next items on the todo list after this MP is landed.  This branch should be considered the proof-of-concept for the basic functionality, to be followed with some convenience improvements and optimizations.

I don't know if you remember but the last few weeks I've researched into some caching and mirroring options and tools.  Particularly I was looking at caching at urllib/http level python modules, however I did also look at apt-mirror and other Debian tools.  I found one option for the former I want to explore more, but the latter options looked too overkill and dependency-heavy.  I didn't look at chdist though, so will try to peek just in case.  But the basic idea I have as a solution is to integrate "lazy mirroring" where instead of directly loading Sources.xz and Packages.xz from disk, if what's on disk is older than N hours, it automatically re-downloads and re-caches the file (and ONLY the requested file) on the fly.  From my testing so far, on a decent network connection that should be quite performant.

I was hoping to slip that feature work in before Prague (and have at least started sketching it in on another branch) but I'm realizing there's some caveats I need to account for, and time is getting short, so decided to just go with the (reliable) rsync for now and make this a next-release enhancement.  There's a couple other convenience improvements I plan to make too, and maybe other folks will also have ideas to add in.

I hadn't looked at chdist, but I did look into apt-mirror and some other Debian and python tools for mirroring apt, and for general http client-side cache management.  There's one cache manager module I want to look into a bit more, but the tools I played with were a bit too heavyweight for what I need, and added dependencies.

Anyway, yes I agree more automation of the metadata fetching would indeed be awesome, and plan to do so but maybe needs to wait for next release?

« Back to merge proposal