Comment 15 for bug 1233466

Hi guys,

We went the alternate route of trying to convince kernel folks of fixing this problem through a patch. Unfortunately, the patch was rejected. The discussion involved is quoted below. Please also note that in the above patch we can remove the CPU hotadd request command as that is extraneous:

# CPU hotadd request
 SUBSYSTEM=="cpu", ACTION=="add", DEVPATH=="/devices/system/cpu/cpu*[0-9]", TEST=="online", ATTR{online}="1"

Please include the memory hotadd patch as not including it may lead to kernel panics related to memory management.

Thanks,
Abhishek

-----Original Message-----
From: Dave Hansen [mailto:<email address hidden>]
Sent: Thursday, July 25, 2013 9:35 AM
To: Kay Sievers
Cc: KY Srinivasan; Michal Hocko; <email address hidden>; <email address hidden>; <email address hidden>; <email address hidden>; <email address hidden>; <email address hidden>; <email address hidden>; <email address hidden>; <email address hidden>; <email address hidden>; <email address hidden>; <email address hidden>
Subject: Re: [PATCH 1/1] Drivers: base: memory: Export symbols for onlining memory blocks

On 07/25/2013 08:15 AM, Kay Sievers wrote:
> Complexity, well, it's just a bit of code which belongs in the kernel.
> The mentioned unconditional hotplug loop through userspace is
> absolutely pointless. Such defaults never belong in userspace tools if
> they do not involve data that is only available in userspace and
> something would make a decision about that. Saying "hello" to
> userspace and usrspace has a hardcoded "yes" in return makes no sense
> at all. The kernel can just go ahead and do its job, like it does for
> all other devices it finds too.

Sorry, but memory is different than all other devices. You never need a mouse in order to add another mouse to the kernel.

I'll repaste something I said earlier in this thread:

> A system under memory pressure is going to have troubles doing a
> hot-add. You need memory to add memory. Of the two operations ("add"
> and "online"), "add" is the one vastly more likely to fail. It has to
> allocate several large swaths of contiguous physical memory. For that
> reason, the system was designed so that you could "add" and "online"
> separately. The intention was that you could "add" far in advance and
> then "online" under memory pressure, with the "online" having *VASTLY*
> smaller memory requirements and being much more likely to succeed.

So, no, it makes no sense to just have userspace always unconditionally online all the memory that the kernel adds. But, the way it's set up, we _have_ a method that can work under lots memory pressure, and it is available for users that want it. It was designed 10 years ago, and maybe it's outdated, or history has proved that nobody is going to use it the way it was designed.

If I had it to do over again, I'd probably set up configurable per-node sets of spare kernel metadata. That way, you could say "make sure we have enough memory reserved to add $FOO sections to node $BAR". Use that for the largest allocations, then depend on PF_MEMALLOC to get us enough for the other little bits along the way.

Also, if this is a problem, it's going to be a problem for *EVERY* user of memory hotplug, not just hyperv. So, let's see it fixed generically for *EVERY* user. Something along the lines of:

1. Come up with an interface that specifies a default policy for
   newly-added memory sections. Currently, added memory gets "added",
   but not "onlined", and the default should stay doing that.
2. Make sure that we at least WARN_ONCE() if someone tries to online an
   already-kernel-onlined memory section. That way, if someone trips
   over this new policy, we have a _chance_ of explaining to them what
   is going on.
3. Think about what we do in the failure case where we are able to
   "add", but fail to "online" in the kernel. Do we tear the
   newly-added structures down and back out of the operation, or do
   we leave the memory added, but offline (what happens in the normal
   case now)?