MAAS lists DCPMM in App Direct mode as RAM, not Storage

The above really depends on the ipmctl tool being introduced into Ubuntu which still hasn't happened yet. So the phase one above boils down to:
1: MAAS can identify the storage devices presented by pre-configured DCPMMs
2: MAAS can partition/format those devices as it does any other block device
3: MAAS can correctly display these in the system summary (currently MAAS shows them as 100% RAM even when they're configured as 100% storage in App Direct mode).

Phase 2 would see MAAS gaining the ability to configure the DCPMMs directly once ipmctl is in Ubuntu including setting them up in a mixed mode (X% RAM, Y% fsdax, Z% devdax, etc)

Phase 1 is the minimum support needed I think.

Next step: Can we get lshw output and 'lxc info --resources` using lxd from edge?

We can then look at updating initrd to include the necessary module in MAAS images

Hi Adam,

Could you explain further? get that info from where? INside the ephemeral?

Just want to be sure we're getting the data from the right place.

also, using lxd from edge where? do I need to install that inside the commissioning ephemeral?

While Jeff is retrieving the appropriate lshw output, I wanted to re-iterate Jeff's analysis from the description asking that the kernel modules:

dax_pmem, nd_pmem

...be loaded in the MAAS ephemeral.

@Jeff - snap install --edge lxd; inside the commissioning ephemeral environment please!

MAAS currently gets memory information from lshw hardware output. While we will continue to run lshw for tagging we are switching to using LXD in 2.7 to gather hardware information. I suspect both will have to be updated to support DCPMM.

I looked into the missing kernel modules dax_pmem and nd_pmem. They are not included in the current Bionic GA kernel however they are in the hwe-18.04 kernel. When lp:maas-images generates the initrd it only includes kernel modules needed in the ephemeral environment. Extra drivers are ignored to keep the initrd as small as possible. Before I add them I'd like to verify that they are needed and this isn't just a bug in lshw/LXD.

Please switch the default minimum kernel to hwe-18.04 and upload the attached commissioning script which will install all kernel modules before commissioning starts.

maas $PROFILE maas set-config name=default_min_hwe_kernel value=hwe-18.04
maas $PROFILE node-scripts create script@=00-debug-install-kmods

Is there any environment where the memory is being detected correctly? We should verify with Intel that the firmware on the machine has fully enabled DCPMM. lshw and LXD may need to incorporate Intel's Persistent Memory Development Kit[1] to properly detect how the memory is being used.

[1] https://github.com/pmem/pmdk

As I now have my own hardware to develop with, the config has changed slightly. I have 4x 128GB DCPMMs configured so that each is 30GB RAM and 96ish GB Storage.

As you can see, the ephemeral does not contain the drivers at all:
ubuntu@lenovo-sr650:~$ lsmod |grep pmem
ubuntu@lenovo-sr650:~$
ubuntu@lenovo-sr650:~$ grep -r *pmem* /lib/modules/`uname -r`/
ubuntu@lenovo-sr650:~$ grep -r mpt_sas* /lib/modules/`uname -r`/
Binary file /lib/modules/5.0.0-23-generic/kernel/drivers/message/fusion/mptspi.ko matches

Note I grepped for a known storage driver just to confirm I'm looking in the right place.

Next, this attachment is from the commissioning image with UNCONFIGURED NVDIMMs (100% memory mode) using the HWE-Edge kernel for commissioning.

This attachment is from the commissioning image with CONFIGURED NVDIMMs (roughly 23% Memory (30GB) and the rest Storage (96GB)) using the hwe-edge kernel.

Now, I then deployed bionic using the GA kernel.

As you can see, once actually deployed with the Bionic image, the modules are loaded:

ubuntu@lenovo-sr650:~$ lsmod |grep pmem
nd_pmem 20480 0
nd_btt 24576 1 nd_pmem
dax_pmem 16384 0
device_dax 20480 1 dax_pmem

And the fsdax, devdax, sector and raw AppDirect devices appear in /dev:

ubuntu@lenovo-sr650:~$ ls /dev |grep pmem
pmem0.1
pmem0.1p1
pmem1.1
pmem1.1p1
pmem1s
pmem1s1
ubuntu@lenovo-sr650:~$ ls /dev |grep dax
dax0.0

And fdisk shows them as partitioned.

ubuntu@lenovo-sr650:~$ sudo fdisk -l |grep pmem
Disk /dev/pmem1s: 95.9 GiB, 102977568768 bytes, 25141008 sectors
/dev/pmem1s1 256 23437567 23437312 89.4G 83 Linux
Disk /dev/pmem1.1: 96 GiB, 103079215104 bytes, 201326592 sectors
/dev/pmem1.1p1 2048 187500543 187498496 89.4G 83 Linux
Disk /dev/pmem0.1: 94.5 GiB, 101466505216 bytes, 198176768 sectors
/dev/pmem0.1p1 2048 187500543 187498496 89.4G 83 Linux

Here's lshw from a deployed Bionic GA system with configured DCPMMs.

> MAAS currently gets memory information from lshw hardware output. While we will continue to run
> lshw for tagging we are switching to using LXD in 2.7 to gather hardware information. I suspect
> both will have to be updated to support DCPMM.

Yes, if that's the case, both will need to be.

> I looked into the missing kernel modules dax_pmem and nd_pmem. They are not included in the
> current Bionic GA kernel however they are in the hwe-18.04 kernel. When lp:maas-images
> generates the initrd it only includes kernel modules needed in the ephemeral environment. Extra
> drivers are ignored to keep the initrd as small as possible. Before I add them I'd like to \
> verify that they are needed and this isn't just a bug in lshw/LXD.

What kernel are you looking at? the deployed GA kernel, or the ephemeral GA kernel? The Bionic GA contains them, as noted in my comment #12 above. They do not exist in the ephemeral GA, nor in the ephemeral hwe or hwe-edge kernels, I've tried all three and introspected the ephemerals and the drivers do not exist there.

> Please switch the default minimum kernel to hwe-18.04 and upload the attached commissioning
> script which will install all kernel modules before commissioning starts.
>
> maas $PROFILE maas set-config name=default_min_hwe_kernel value=hwe-18.04
> maas $PROFILE node-scripts create script@=00-debug-install-kmods

After doing this and running commissioning, this is what the ephemeral says:

ubuntu@lenovo-sr650:~$ lsmod |grep pmem
ubuntu@lenovo-sr650:~$ modprobe dax_pmem
modprobe: FATAL: Module dax_pmem not found in directory /lib/modules/5.0.0-23-generic
ubuntu@lenovo-sr650:~$ find /lib/modules/5.0.0-23-generic/ -name "*pmem*"
ubuntu@lenovo-sr650:~$ find /lib/modules/5.0.0-23-generic/ -name "*dax*"
ubuntu@lenovo-sr650:~$ find /lib/modules/5.0.0-23-generic/ -name "*btt*"
ubuntu@lenovo-sr650:~$ find /lib/modules/5.0.0-23-generic/ -name "*mptsas*"
/lib/modules/5.0.0-23-generic/kernel/drivers/message/fusion/mptsas.ko

Note, this needs to work with the GA kernel (and should, since the drivers do exist in the GA kernel once deployed).
(Note last one was just to sanity check my find command)

From the GA kernel:
ubuntu@lenovo-sr650:~$ find /lib/modules/`uname -r`/kernel/drivers/ -iname "*pmem*"
/lib/modules/4.15.0-55-generic/kernel/drivers/nvdimm/nd_pmem.ko
/lib/modules/4.15.0-55-generic/kernel/drivers/dax/dax_pmem.ko

So, to summarize:

GA and HWE kernels in Ephemeral == no drivers for DCPMMs
GA and HWE kernels in installed OS == drivers for DCPMMs

Just on a lark, I booted the 4.15 kernel ephemeral and tried installing linux-modules-4.15.0-55-generic and while they do get installed, they're not loadable due to something about missing symbols.

/var/log/apt/term.log:Setting up linux-modules-4.15.0-55-generic (4.15.0-55.60) ...
/var/log/apt/term.log:Setting up linux-modules-extra-4.15.0-55-generic (4.15.0-55.60) ...
ubuntu@lenovo-sr650:~$ find /lib/modules/4.15.0-55-generic/kernel/drivers/ -iname "*pmem*"
/lib/modules/4.15.0-55-generic/kernel/drivers/nvdimm/nd_pmem.ko
/lib/modules/4.15.0-55-generic/kernel/drivers/dax/dax_pmem.ko
ubuntu@lenovo-sr650:/lib/modules/4.15.0-55-generic/kernel/drivers/nvdimm$ uname -a
Linux lenovo-sr650 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@lenovo-sr650:/lib/modules/4.15.0-55-generic/kernel/drivers/nvdimm$ sudo insmod nd_pmem.ko
insmod: ERROR: could not insert module nd_pmem.ko: Unknown symbol in module
ubuntu@lenovo-sr650:/lib/modules/4.15.0-55-generic/kernel/drivers/nvdimm$ sudo modprobe nd_pmem.ko
modprobe: FATAL: Module nd_pmem.ko not found in directory /lib/modules/4.15.0-55-generic

The related branch adds the DCPMM and NVDIMM drivers to the ephemeral environment. However I want to confirm that adding those is all that is needed to fix this bug. Can you please run the attached script and lxc info --resources in a deployed environment in all memory modes to confirm that the expected amount of memory is given?

@bladernr any chance of running the script Lee posted as well as lxd's output?

I thought I had done this ages ago :(

So this is the current configuration:
ubuntu@lenovo-sr650-aitken:~$ sudo ipmctl show -dimm

DimmID Capacity HealthState ActionRequired LockState FWVersion
0x0020 126.4 GiB Healthy 0 Disabled 01.02.00.5375
0x0120 126.4 GiB Healthy 0 Disabled 01.02.00.5375
0x1020 126.4 GiB Healthy 0 Disabled 01.02.00.5375
0x1120 126.4 GiB Healthy 0 Disabled 01.02.00.5375

Four 128GB DCPMMs plus 8x 16GB DDR4 DIMMs:
DIMM 1 Intel Optane DCPMM 128 GB
DIMM 3 DDR4 16 GB
DIMM 5 DDR4 16 GB
DIMM 8 DDR4 16 GB
DIMM 10 DDR4 16 GB
DIMM 12 Intel Optane DCPMM 128 GB
DIMM 13 Intel Optane DCPMM 128 GB
DIMM 15 DDR4 16 GB
DIMM 17 DDR4 16 GB
DIMM 20 DDR4 16 GB
DIMM 22 DDR4 16 GB
DIMM 24 Intel Optane DCPMM 128 GB

The DCPMMs are currently configured:
ubuntu@lenovo-sr650-aitken:~$ sudo ipmctl show -region

SocketID ISetID PersistentMemoryType Capacity FreeCapacity HealthState
       0 0x17deeeb8d5482444 AppDirect 192.0 GiB 0.0 GiB Healthy
       1 0xb48aeeb88e4c2444 AppDirect 192.0 GiB 0.0 GiB Healthy

That's 192GiB per region (region == 2x NVIDMM in this config) or 384GiB allocated to Storage and reserves 121.6GiB in Memory Mode.

Note, that when NVDIMMs have any memory allocated as RAM (in Memory Mode), the NVDIMM allocation becomes addressable RAM to the OS, and any DDR4 becomes a special cache that is managed by the firmware, and is NOT visible to the OS. When the NVDIMM is in 100% storage mode, then the DDR4 DIMMs become OS addressable memory and 100% of the NVDIMM becomes storage of some type.

This is cat/proc/cpuinfo:
ubuntu@lenovo-sr650-aitken:~$ cat /proc/meminfo
MemTotal: 123380624 kB
MemFree: 119044736 kB
MemAvailable: 119120392 kB
Buffers: 303068 kB
Cached: 427696 kB
SwapCached: 0 kB
Active: 670656 kB
Inactive: 155928 kB
Active(anon): 96836 kB
Inactive(anon): 2064 kB
Active(file): 573820 kB
Inactive(file): 153864 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 8388604 kB
SwapFree: 8388604 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 95944 kB
Mapped: 71792 kB
Shmem: 3036 kB
Slab: 816736 kB
SReclaimable: 285884 kB
SUnreclaim: 530852 kB
KernelStack: 16816 kB
PageTables: 5488 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 70078916 kB
Committed_AS: 450636 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 951844 kB
DirectMap2M: 8105984 kB
DirectMap1G: 319815680 kB

This is with the system in the above mentioned config of Mixed mode.

This is the output of Lee's script from comment 17:

ubuntu@lenovo-sr650-aitken:~$ sudo python3 memory.py
Memory: 648704.0

and this is the output of lxc info --resources:
ubuntu@lenovo-sr650-aitken:~$ lxc info --resources
If this is your first time running LXD on this machine, you should also run: lxd init
To start your first container, try: lxc launch ubuntu:18.04

cpu:
  sockets:
  - cores: 30
    frequency: 3111
    name: GenuineIntel
    vendor: Intel(R) Xeon(R) Platinum 8260M CPU @ 2.30GHz
    threads: 48
  - cores: 30
    frequency: 3099
    name: GenuineIntel
    vendor: Intel(R) Xeon(R) Platinum 8260M CPU @ 2.30GHz
    threads: 48
  total: 96
memory:
  used: 3717615616
  total: 126341758976

I think the LXD team needs to look at this.

Asked Jeff for access to the system so I can investigate.

Poking around on a system, I've found that LXD appears to behave properly.

That is, it properly reports the amount of memory which can be used as RAM.
It also reports any configured PMEM block device under its storage structs.

root@lenovo-sr650-aitken:~# ipmctl show -memoryresources
 MemoryType | DDR | DCPMM | Total
========================================================
 Volatile | 0.000 GiB | 48.000 GiB | 48.000 GiB
 AppDirect | - | 456.000 GiB | 456.000 GiB
 Cache | 128.000 GiB | - | 128.000 GiB
 Inaccessible | - | 1.689 GiB | 1.689 GiB
 Physical | 128.000 GiB | 505.689 GiB | 633.689 GiB

root@lenovo-sr650-aitken:~# free -g
              total used free shared buff/cache available
Mem: 46 4 42 0 0 42
Swap: 7 0 7

root@lenovo-sr650-aitken:~# echo $(($(lxc query /1.0/resources | jq .memory.total)/1024/1024/1024))
48

root@lenovo-sr650-aitken:~# lxc query /1.0/resources | jq .storage | grep pmem-
      "device_id": "pmem-ebd1f454-6b34-4ce4-b5e0-34aca8a83a86",
      "device_id": "pmem-ec9940cc-8f87-4b39-b730-176685533112",

That's under 20.04 mind you, so I don't know if previous kernels were getting confused?

It likely differs to lshw/dmidecode as those just read the total size of the memory modules and have no idea whether they are used as RAM or storage.

There appears to be a sysfs interface under "/sys/class/nd/" to list the persistent modules and see the defined regions but that doesn't seem to expose any data about the cache allocation.

I guess the question here is what should MAAS report as memory and storage.
LXD's view is that memory should be the amount marked as "volatile" (traditional memory) and it looks like that number is currently correct.

If non-volatile block devices are defined, those do show up (though with limited information) in the storage section of LXD's resources API.

I'll look at improving the pmem block device case at least so hopefully we can report a bit more data on those.

For the rest, it feels like firmware level trickery which also requires rebooting.

It also effectively changes the memory and storage characteristics of the machine, so for example there would be no way to have MAAS deploy the OS itself onto a PDIMM backed block device.

My opinion is that this is effectively equivalent to you going to physically change your memory and storage setup in the machine, which is something that needs a manual commission run post-change.

Trying to capture data, surface it to the user and allow for config changes through MAAS may turn out to be very difficult because of:
 - No clear per-DIMM control on role (DDR used as cache, PDIMM used as RAM, mix of both, ...)
 - No real control over cache
 - Variety of configurations considered invalid (firmware dependent)
 - Linux knows about regions and about the PDIMM sticks but does not report the amount used for cache, appdirect or volatile, instead the tool reads ACPI tables for that
 - None of the sysfs interfaces expose human readable data, everything is based on obsc...

Ok, so diving into the storage side of things, the fields which aren't filled are:

 - device_path
 - firmware_version
 - model
 - serial
 - wwn

I did some poking around and came to the conclusion that it is correct for them all to be empty.
device_path was the weirdest one to find empty but it is correct as there is no by-id entries for those block devices. I've sent a change to LXD to make that field optional so it doesn't show up as empty in the raw JSON output.

The others are all assuming a physical device with a fixed vendor, model, firmware and unique identifier. None of that applies here as we're dealing with a virtual block device made from some amount of memory coming from a variety of modules. Those modules could in theory be from different models and vendors, so even if I could track them down, we wouldn't get a nice answer for those.

That concludes my investigation. At least on Ubuntu 20.04, the system behaves as I would expect.
We are properly seeing the block devices created and the memory we report matches the memory configuration available to the system (volatile).

I merged the related branch into lp:maas-images. New images will include both the dax and nvdimm kernel modules in the initrd. Images are released by CPC about every week.

Marking this closed... Intel has abandoned Optane DIMMs