{"id":51,"date":"2026-02-03T21:27:34","date_gmt":"2026-02-03T21:27:34","guid":{"rendered":"https:\/\/underwoodtechnologies.com\/?p=51"},"modified":"2026-02-03T21:27:34","modified_gmt":"2026-02-03T21:27:34","slug":"test-post-2","status":"publish","type":"post","link":"https:\/\/underwoodtechnologies.com\/?p=51","title":{"rendered":"Hard Learned victories"},"content":{"rendered":"\n

The purpose of a home lab is to learn. For me, it’s in many ways a re-visit of the work with computers I started when I was in my teenaged years. Although the systems I’ve worked on have evolved as has my knowledge, some things remain the same. First and foremost that you have to put the investment in to get the return.<\/p>\n\n\n\n

I am working through the initial iteration of this home lab. What that means is I have TrueNAS set up serving iSCSI, and I have three nested Proxmox VMs configured. Disk Pass through was configured to the TrueNAS guest and I put four 10K 1.2TB disks in to it to serve as the ZFS Pool. The high level architecture diagram is as in the diagram at the top of this post.<\/p>\n\n\n\n

The basic Proxmox VE Image was a template, and the others were cloned off of that template post install. I created multipathing on iSCSI and set up shared LVM as per the guidelines Proxmox laid out.<\/p>\n\n\n\n

I then used this logical volume to provision storage to a VM within the virtual Proxmox Instance. Because the storage was shared LVM, I can migrate as I please between them.<\/p>\n\n\n\n

This is where things fell apart. The machine was horribly slow to boot up and even mundane things like logging in were slow. I started using dd to do disk testing, and found out that I was getting writes of about 35Mb\/s and 16MB\/S reads. I was also getting horrible IOwait doing even basic actions.<\/p>\n\n\n\n

At this point, I changed as many variables as I could think of. I bumped the MTU to 9000, started looking through caching, altered NUMA settings, altered core\/socket lay outs, altered CPU architecture, basically anything I could think of. I asked other people, and didn’t get too far. I knew that the “SAN” was capable of much more performance, because when I used the host node or an external workstation to connect to iSCSI, performance was about 200MB\/s read and 100MB\/s write.<\/p>\n\n\n\n

I’d been seeing massive amounts of repetitive error messages in the console of the TrueNAS VM. I couldn’t figure out what these meant, so I left them be. This was likely a mistake, as we’ll find out momentarily.<\/p>\n\n\n\n

Iostat was showing that the test disk I wasn’t using was seeing significant IO traffic. That seemed really odd. After looking through running processes in top, I found out udev was constantly hitting the disk. As it turns out, it was probing. I found that out via:<\/p>\n\n\n\n

udevadm monitor –udev<\/em><\/p>\n\n\n\n

After a while and some searching, that cascaded into:<\/p>\n\n\n\n

iscsiadm -m session -P 3 | grep sd<\/em><\/p>\n\n\n\n

This lead me to finding out my disks were alternating between a blocked and not blocked state.<\/p>\n\n\n\n

I then forced the logout of iSCSI and a restart of the iSCSI daemon.<\/p>\n\n\n\n

iscsiadm -m session -u<\/em><\/p>\n\n\n\n

iscsiadm -m session<\/em><\/p>\n\n\n\n

systemctl restart iscsid<\/em><\/p>\n\n\n\n

For some reason performance started working, then degraded again. After that, I knew that it wasn’t an inherent architectural issue, so what was the issue? A simple visit to the journal would have let me know what was happening. These errors were scrolling by as fast as they could.<\/p>\n\n\n\n

Feb 04 20:47:56 pveV1 kernel: sd 4:0:0:1: Power-on or device reset occurred<\/em><\/p>\n\n\n\n

Feb 04 20:47:56 pveV1 iscsid[2349493]: Kernel reported iSCSI connection 3:0 error (1020 – ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)<\/em><\/p>\n\n\n\n

Feb 04 20:47:56 pveV1 iscsid[2349493]: connection3:0 is operational after recovery (1 attempts)<\/em><\/p>\n\n\n\n

Feb 04 20:47:56 pveV1 iscsid[2349493]: Kernel reported iSCSI connection 3:0 error (1020 – ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)<\/em><\/p>\n\n\n\n

What did it mean? Obviously that resets were happening, but what was the root cause?<\/p>\n\n\n\n

As it turns out, the iSCSI initiator name was the same for all three VMs. I changed two hosts, did some iSCSI logouts, daemon resets for iscsid, and resorted to rebooting one of the Proxmox VMs. I’m getting similar to the same performance I was seeing at the host. A nice boost, and my test VM that’s in the nested Proxmox system is actually performant.<\/p>\n\n\n\n

A follow up I need to work on is to determine if the ID was generated on the VMs through derivation of some other ID that’s duplicated, or was the file already in the template and therefore just copied over. Looking at it I believe that I should be able to delete the initiator.iscsi file from the template, and it will generate a new one on boot up.<\/p>\n\n\n\n

What lessons are there from this misadventure?<\/p>\n\n\n\n