Schubis Wiki und Gedankenstützen
You are here: Motivation » VMware » VSAN

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
vmware:vsan [2019/04/17 09:26] – [Netzwerk checken] mschubivmware:vsan [2025/04/02 15:59] (current) – [VSAN] mschubi
Line 5: Line 5:
 https://storagehub.vmware.com/#!/vmware-vsan/vmware-vsan-demonstrations https://storagehub.vmware.com/#!/vmware-vsan/vmware-vsan-demonstrations
  
 +TBD
 +https://via.vmw.com/tchzcoreno1848
 +https://youtu.be/nvrCShwWMWY
 +
 +===== TRIM/UNMAP =====
 +
 +https://knowledge.broadcom.com/external/article/326595/procedure-to-enable-trimunmap.html
 +===== vSAN mit CLI ohne vCenter =====
 +
 +https://blog.rylander.io/2017/01/19/configure-2-node-vsan-on-esxi-free-using-cli-without-vcenter/
 +====== vSAN Cluster Partitions beheben ======
 +
 +Im vSAN Cluster kann es dazu kommen, dass die einzelnen Nodes partitioniert sind. 
 +Meist ist nur eine einzelne Node weg, manchmal aber auch alle :-(
 +Auch wenn auf den vSAN Kernelport alle Hosts sich sehen können, finden sich die Partitions nicht zusammen.
 +
 +{{:vmware:vsan:2020-11-15_13_12_02-vsphere_-_nuc-cluster_-_skyline_health.png?500|}}
 +Im schlimmsten Fall sieht das so aus. 
 +
 +Erst mal sollte man alle vSAN Ports testen, ob die sich untereinander pingen lassen.
 +
 +<code bash>
 +[root@esx-4:~] vmkping -d -s 7000 -I vmk2 192.168.4.11
 +PING 192.168.4.11 (192.168.4.11): 7000 data bytes
 +7008 bytes from 192.168.4.11: icmp_seq=0 ttl=64 time=0.615 ms
 +7008 bytes from 192.168.4.11: icmp_seq=1 ttl=64 time=0.854 ms
 +7008 bytes from 192.168.4.11: icmp_seq=2 ttl=64 time=0.765 ms
 +
 +--- 192.168.4.11 ping statistics ---
 +3 packets transmitted, 3 packets received, 0% packet loss
 +round-trip min/avg/max = 0.615/0.745/0.854 ms
 +</code>
 +
 +Was sagt der Cluster?
 +<code bash>
 +[root@esx-4:~] esxcli vsan cluster get
 +Cluster Information
 +   Enabled: true
 +   Current Local Time: 2020-11-15T12:18:22Z
 +   Local Node UUID: 5d0cc03c-de7b-b94f-b016-54b2031c0494
 +   Local Node Type: NORMAL
 +   Local Node State: MASTER
 +   Local Node Health State: HEALTHY
 +   Sub-Cluster Master UUID: 5d0cc03c-de7b-b94f-b016-54b2031c0494
 +   Sub-Cluster Backup UUID:
 +   Sub-Cluster UUID: 52c9d6db-b533-a36c-6030-98d37f927e6a
 +   Sub-Cluster Membership Entry Revision: 2
 +   Sub-Cluster Member Count: 1
 +   Sub-Cluster Member UUIDs: 5d0cc03c-de7b-b94f-b016-54b2031c0494
 +   Sub-Cluster Member HostNames: esx-4.schubi.local
 +   Sub-Cluster Membership UUID: af16b15f-9a9b-99f7-989e-54b2031c0494
 +   Unicast Mode Enabled: true
 +   Maintenance Mode State: OFF
 +   Config Generation: 27b68356-c3e0-4c64-ab66-3b80741ab493 14 2020-10-19T20:29:54.524
 +</code>
 +Sieht auf jedem Cluster so aus. Jeweils nur ein Member. Aber sie "fühlen" sich dem gleichen Cluster zugehörig.
 +Überall ist ''Sub-Cluster UUID: 52c9d6db-b533-a36c-6030-98d37f927e6a'' gleich.
 +
 +Mal sehen wir die UniCast Agenten sich fühlen. Da gibts bei mir eine Überraschung
 +<code bash>
 +[root@esx-1:~] esxcli vsan cluster unicastagent list
 +NodeUuid                              IsWitness  Supports Unicast  IP Address      Port  Iface Name  Cert Thumbprint                                              SubClusterUuid
 +------------------------------------  ---------  ----------------  -------------  -----  ----------  -----------------------------------------------------------  --------------
 +5d0cc03c-de7b-b94f-b016-54b2031c0494          0              true  192.168.4.232  12321              EB:EF:07:00:20:7B:09:97:AD:EB:34:27:3D:B0:A7:32:8B:CF:57:CE  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cc5d7-070b-ff59-7be6-54b20319c59c          0              true  192.168.4.149  12321              09:C6:4C:7B:AD:5F:4B:7C:29:7D:BE:DC:4C:95:04:7E:02:32:35:B5  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0ccefe-e4a3-f316-ea35-54b2031bfc15          0              true  192.168.4.151  12321              B6:89:41:DC:9B:20:72:1B:19:E3:8A:C6:F8:49:AC:6D:93:8C:A3:76  52c9d6db-b533-a36c-6030-98d37f927e6a
 +
 +[root@esx-2:~] esxcli vsan cluster unicastagent list
 +NodeUuid                              IsWitness  Supports Unicast  IP Address      Port  Iface Name  Cert Thumbprint                                              SubClusterUuid
 +------------------------------------  ---------  ----------------  -------------  -----  ----------  -----------------------------------------------------------  --------------
 +5d0cc03c-de7b-b94f-b016-54b2031c0494          0              true  192.168.4.232  12321              EB:EF:07:00:20:7B:09:97:AD:EB:34:27:3D:B0:A7:32:8B:CF:57:CE  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cca61-c2bb-4270-a20e-54b2031c044c          0              true  192.168.4.219  12321              59:E9:AA:76:F8:14:55:F2:5D:DE:B8:AA:0E:0D:D0:30:4C:39:77:21  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0ccefe-e4a3-f316-ea35-54b2031bfc15          0              true  192.168.4.151  12321              B6:89:41:DC:9B:20:72:1B:19:E3:8A:C6:F8:49:AC:6D:93:8C:A3:76  52c9d6db-b533-a36c-6030-98d37f927e6a
 +
 +[root@esx-3:~] esxcli vsan cluster unicastagent list
 +NodeUuid                              IsWitness  Supports Unicast  IP Address      Port  Iface Name  Cert Thumbprint                                              SubClusterUuid
 +------------------------------------  ---------  ----------------  -------------  -----  ----------  -----------------------------------------------------------  --------------
 +5d0cc03c-de7b-b94f-b016-54b2031c0494          0              true  192.168.4.232  12321              EB:EF:07:00:20:7B:09:97:AD:EB:34:27:3D:B0:A7:32:8B:CF:57:CE  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cc5d7-070b-ff59-7be6-54b20319c59c          0              true  192.168.4.149  12321              09:C6:4C:7B:AD:5F:4B:7C:29:7D:BE:DC:4C:95:04:7E:02:32:35:B5  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cca61-c2bb-4270-a20e-54b2031c044c          0              true  192.168.4.219  12321              59:E9:AA:76:F8:14:55:F2:5D:DE:B8:AA:0E:0D:D0:30:4C:39:77:21  52c9d6db-b533-a36c-6030-98d37f927e6a
 +
 +[root@esx-4:~] esxcli vsan cluster unicastagent list
 +NodeUuid                              IsWitness  Supports Unicast  IP Address      Port  Iface Name  Cert Thumbprint                                              SubClusterUuid
 +------------------------------------  ---------  ----------------  -------------  -----  ----------  -----------------------------------------------------------  --------------
 +5d0cc5d7-070b-ff59-7be6-54b20319c59c          0              true  192.168.4.149  12321              09:C6:4C:7B:AD:5F:4B:7C:29:7D:BE:DC:4C:95:04:7E:02:32:35:B5  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cca61-c2bb-4270-a20e-54b2031c044c          0              true  192.168.4.219  12321              59:E9:AA:76:F8:14:55:F2:5D:DE:B8:AA:0E:0D:D0:30:4C:39:77:21  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0ccefe-e4a3-f316-ea35-54b2031bfc15          0              true  192.168.4.151  12321              B6:89:41:DC:9B:20:72:1B:19:E3:8A:C6:F8:49:AC:6D:93:8C:A3:76  52c9d6db-b533-a36c-6030-98d37f927e6a
 +
 +</code>
 +Die haben die aktuellen IOs *.4.11 bis *.4.14 gar nicht "gefressen".
 +Zum Thema UniCast Agent gibt es einen VMware KB https://kb.vmware.com/s/article/2150303.
 +
 +Gehen wir mal die Liste durch.
 +<code bash>
 +[root@esx-1:~] esxcli vsan network list
 +Interface
 +   VmkNic Name: vmk2
 +   IP Protocol: IP
 +   Interface UUID: fbf7215f-49cd-05cd-f9d8-54b2031c044c
 +   Agent Group Multicast Address: 224.2.3.4
 +   Agent Group IPv6 Multicast Address: ff19::2:3:4
 +   Agent Group Multicast Port: 23451
 +   Master Group Multicast Address: 224.1.2.3
 +   Master Group IPv6 Multicast Address: ff19::1:2:3
 +   Master Group Multicast Port: 12345
 +   Host Unicast Channel Bound Port: 12321
 +   Data-in-Transit Encryption Key Exchange Port: 0
 +   Multicast TTL: 5
 +   Traffic Type: vsan
 +</code>
 +Überall gut. Die Kernelport IP Config sieht auch gut aus.
 +<code bash>
 +[root@esx-1:~]  esxcli network ip interface ipv4 get | grep vmk2
 +vmk2  192.168.4.11   255.255.255.0  192.168.4.255   STATIC        192.168.4.1      false
 +</code>
 +
 +Für das weitere Vorgehen benötigen wir die UUID.
 +<code bash>
 +[root@esx-1:~]  cmmds-tool whoami
 +5d0cca61-c2bb-4270-a20e-54b2031c044c
 +[root@esx-2:~]  cmmds-tool whoami
 +5d0cc5d7-070b-ff59-7be6-54b20319c59c
 +[root@esx-3:~]  cmmds-tool whoami
 +5d0ccefe-e4a3-f316-ea35-54b2031bfc15
 +[root@esx-4:~] cmmds-tool whoami
 +5d0cc03c-de7b-b94f-b016-54b2031c0494
 +</code>
 +
 +Damit unsere folgenden manuellen Änderungen nicht durch ein Update des vCenters gestört wird, ignorieren wir die vCenter Settings temporär.
 +<code bash>
 +[root@esx-1:~] esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListupdates
 +Value of IgnoreClusterMemberListUpdates is 1
 +</code>
 +**Wichtig! Niemals auf einem Host die eigene Agent IP hinzufügen!**
 +VMware: //"When an ESXi host has its own IP address in its Unicast agent list, many networking problems can arise and potentially lead to the host encountering a PSOD."// 
 +Also "Augen auf" bei den folgenden Schritten!
 +
 +Für den esx1 Host müssen 2, 3 und 4 Hinzugefügt werden
 +<code bash>
 +esxcli vsan cluster unicastagent add -t node -u 5d0cc5d7-070b-ff59-7be6-54b20319c59c -U true -a 192.168.4.12 -p 12321
 +esxcli vsan cluster unicastagent add -t node -u 5d0ccefe-e4a3-f316-ea35-54b2031bfc15 -U true -a 192.168.4.13 -p 12321
 +esxcli vsan cluster unicastagent add -t node -u 5d0cc03c-de7b-b94f-b016-54b2031c0494 -U true -a 192.168.4.14 -p 12321
 +</code>
 +Ein Check des Hosts bringt noch nicht ganz die 100%ige Überzeugung.
 +<code bash>
 +[root@esx-1:~] esxcli vsan cluster unicastagent list
 +NodeUuid                              IsWitness  Supports Unicast  IP Address      Port  Iface Name  Cert Thumbprint                                              SubClusterUuid
 +------------------------------------  ---------  ----------------  -------------  -----  ----------  -----------------------------------------------------------  --------------
 +5d0cc03c-de7b-b94f-b016-54b2031c0494          0              true  192.168.4.232  12321              EB:EF:07:00:20:7B:09:97:AD:EB:34:27:3D:B0:A7:32:8B:CF:57:CE  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cc03c-de7b-b94f-b016-54b2031c0494          0              true  192.168.4.14   12321                                                                           52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cc5d7-070b-ff59-7be6-54b20319c59c          0              true  192.168.4.149  12321              09:C6:4C:7B:AD:5F:4B:7C:29:7D:BE:DC:4C:95:04:7E:02:32:35:B5  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cc5d7-070b-ff59-7be6-54b20319c59c          0              true  192.168.4.12   12321                                                                           52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0ccefe-e4a3-f316-ea35-54b2031bfc15          0              true  192.168.4.151  12321              B6:89:41:DC:9B:20:72:1B:19:E3:8A:C6:F8:49:AC:6D:93:8C:A3:76  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0ccefe-e4a3-f316-ea35-54b2031bfc15          0              true  192.168.4.13   12321                                                                           52c9d6db-b533-a36c-6030-98d37f927e6a
 +</code>
 +
 +Aber zum Glück kann man "überflüssige" Einträge löschen...
 +<code bash>
 +esxcli vsan cluster unicastagent remove -a 192.168.4.232
 +esxcli vsan cluster unicastagent remove -a 192.168.4.149
 +esxcli vsan cluster unicastagent remove -a 192.168.4.151
 +</code>
 +So sieht es nun gut aus.
 +<code bash>
 +[root@esx-1:~]  esxcli vsan cluster unicastagent list
 +NodeUuid                              IsWitness  Supports Unicast  IP Address     Port  Iface Name  Cert Thumbprint  SubClusterUuid
 +------------------------------------  ---------  ----------------  ------------  -----  ----------  ---------------  --------------
 +5d0cc03c-de7b-b94f-b016-54b2031c0494          0              true  192.168.4.14  12321                               52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cc5d7-070b-ff59-7be6-54b20319c59c          0              true  192.168.4.12  12321                               52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0ccefe-e4a3-f316-ea35-54b2031bfc15          0              true  192.168.4.13  12321                               52c9d6db-b533-a36c-6030-98d37f927e6a
 +</code>
 +
 +Exemplarisch noch für die weiteren Hosts
 +
 +ESX2:
 +<code bash>
 +esxcli vsan cluster unicastagent add -t node -u 5d0cca61-c2bb-4270-a20e-54b2031c044c -U true -a 192.168.4.11 -p 12321
 +esxcli vsan cluster unicastagent add -t node -u 5d0ccefe-e4a3-f316-ea35-54b2031bfc15 -U true -a 192.168.4.13 -p 12321
 +esxcli vsan cluster unicastagent add -t node -u 5d0cc03c-de7b-b94f-b016-54b2031c0494 -U true -a 192.168.4.14 -p 12321
 +</code>
 +
 +ESX3:
 +<code bash>
 +esxcli vsan cluster unicastagent add -t node -u 5d0cca61-c2bb-4270-a20e-54b2031c044c -U true -a 192.168.4.11 -p 12321
 +esxcli vsan cluster unicastagent add -t node -u 5d0cc5d7-070b-ff59-7be6-54b20319c59c -U true -a 192.168.4.12 -p 12321
 +esxcli vsan cluster unicastagent add -t node -u 5d0cc03c-de7b-b94f-b016-54b2031c0494 -U true -a 192.168.4.14 -p 12321
 +</code>
 +
 +ESX4:
 +<code bash>
 +esxcli vsan cluster unicastagent add -t node -u 5d0cca61-c2bb-4270-a20e-54b2031c044c -U true -a 192.168.4.11 -p 12321
 +esxcli vsan cluster unicastagent add -t node -u 5d0cc5d7-070b-ff59-7be6-54b20319c59c -U true -a 192.168.4.12 -p 12321
 +esxcli vsan cluster unicastagent add -t node -u 5d0ccefe-e4a3-f316-ea35-54b2031bfc15 -U true -a 192.168.4.13 -p 12321
 +</code>
 +
 +Danach natürlich wieder Kontrolle mit 
 +<code bash>
 + esxcli vsan cluster unicastagent list
 +</code>
 +und ggf. Löschen mit
 +<code bash>
 +esxcli vsan cluster unicastagent remove -a 
 +</code>
 +
 +Kaum macht man es richtig, schon geht's. Ich habe einen Master, einen Backup, der Rest Agent. Hier mal ein Auszug.
 +<code bash>
 +[root@esx-1:~]  esxcli vsan cluster get
 +Cluster Information
 +   Enabled: true
 +   Current Local Time: 2020-11-15T13:10:16Z
 +   Local Node UUID: 5d0cca61-c2bb-4270-a20e-54b2031c044c
 +   Local Node Type: NORMAL
 +   Local Node State: BACKUP
 +   Local Node Health State: HEALTHY
 +   Sub-Cluster Master UUID: 5d0cc5d7-070b-ff59-7be6-54b20319c59c
 +   Sub-Cluster Backup UUID: 5d0cca61-c2bb-4270-a20e-54b2031c044c
 +   Sub-Cluster UUID: 52c9d6db-b533-a36c-6030-98d37f927e6a
 +   Sub-Cluster Membership Entry Revision: 4
 +   Sub-Cluster Member Count: 4
 +   Sub-Cluster Member UUIDs: 5d0cc5d7-070b-ff59-7be6-54b20319c59c, 5d0cca61-c2bb-4270-a20e-54b2031c044c, 5d0ccefe-e4a3-f316-ea35-54b2031bfc15, 5d0cc03c-de7b-b94f-b016-54b2031c0494
 +   Sub-Cluster Member HostNames: esx-2.schubi.local, esx-1.schubi.local, esx-3.schubi.local, esx-4.schubi.local
 +   Sub-Cluster Membership UUID: b216b15f-40f2-2395-1900-54b20319c59c
 +   Unicast Mode Enabled: true
 +   Maintenance Mode State: OFF
 +   Config Generation: 27b68356-c3e0-4c64-ab66-3b80741ab493 20 2020-11-15T12:57:11.0
 +</code>
 +
 +Zum Schluss nicht vergessen, den Advanced Parameter zu ersetzen:
 +<code bash>
 +esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListupdates
 +</code>
 +
 +
 +===== vSAN Services checken und starten =====
 +
 +<code bash>
 +/etc/init.d/cmmdsd status && /etc/init.d/epd status && /etc/init.d/clomd status
 +
 +/etc/init.d/cmmdsd restart && /etc/init.d/epd restart && /etc/init.d/clomd restart
 +</code>
 +====== ESXi Infos in der RVC ======
 +
 +<code bash>
 +/localhost/DC/computers/vSAN-Cluster> table -f name -s name -f state.connection:state.maintenancemode:build:uptime:num.vms:num.poweredonvms:cpuusage:memusage hosts/*
 +</code>
 +
 +===== Samsung 970 EVO PlusFirmware=====
 +
 +Bei den 970er Evos wird bei neueren vSAN Releases evtl. die Firmware angemeckert.
 +{{:vmware:vsan:2019-12-15_19_35_01-vsphere_-_vsan-nuc_-_skyline-integritaet.png?400|}}
 +
 +<code bash>
 +[root@esx-1:~] esxcli nvme device get -A vmhba1 | egrep "Serial Number|Model Number|Firmware Revision"
 +   Serial Number: S4EUNG0M219966P
 +   Model Number: Samsung SSD 970 EVO Plus 250GB
 +   Firmware Revision: 1B2QEXM7
 +</code>
 +
 +Richtig wäre hier aber FW EDA7402Q.
 +
 +Die Firmware findet man unter https://www.samsung.com/semiconductor/minisite/ssd/download/tools/
 +Aber da ist keine EDA7402Q zu finden :-( 
 +Die höchste Version ist 2B2QEXM7.
 +Weitere Infos für den firmware Download können auch unter https://tinkertry.com/ssd-magician-and-firmware-download-limits-exceeded-samsung-seriously gefunden werden. Das ISO findet man unter http://ssd.samsungsemi.com/ecomobile/ssd/update14.do?fname=/Samsung_SSD_960_PRO_2B6QCXP7.iso
 +
 +
 +Also wird der Healthcheck nach https://www.virten.net/2017/04/how-to-silence-vsan-health-checks/ abgeschaltet.
 +RVC:
 +<code bash>
 +vsan.health.silent_health_check_configure -a controllerfirmware .
 +</code>
 +{{:vmware:vsan:2019-12-15_19_57_24-mremoteng_-_confcons.xml.png?600|}}
 ===== Netzwerk checken ===== ===== Netzwerk checken =====
  
- esxcli network ip interface set  -m 9000 -i vmk3+<code bash> 
 +esxcli network ip interface set  -m 9000 -i vmk3
  
 esxcli vsan health cluster list esxcli vsan health cluster list
Line 14: Line 287:
  
 esxcli vsan network ip add -i vmk0 -T=witness esxcli vsan network ip add -i vmk0 -T=witness
 +</code>
 +
 +Durchsatz mit iperf checken:
 +
 +   * Firewall ausschalten auf dem Ziel - <code bash>esxcli network firewall set --enabled false</code>
 +   * iperf im Listen Modus unter Angabe der horchenden IP Angeben - <code bash>/usr/lib/vmware/vsan/bin/iperf3.copy -s -B <Kernelport IP></code>
 +   * auf der Quelle die Firewall ausschalten - <code bash>esxcli network firewall set --enabled false</code>
 +   * Auf der Quelle iperf unter Angabe der Ziel-IP starten - <code bash>/usr/lib/vmware/vsan/bin/iperf3.copy -c <Kernelport IP></code>
 +   * nach dem Test nicht vergessen, die Firewall wieder einzuschalten -  <code bash>esxcli network firewall set --enabled true</code>
 +
  
  
Line 40: Line 323:
 https://blogs.vmware.com/performance/2018/12/hcibench-specific-issues-recommendations-vsan.html https://blogs.vmware.com/performance/2018/12/hcibench-specific-issues-recommendations-vsan.html
  
 +https://storagehub.vmware.com/section-assets/powercli-cookbook-for-vsan
 +
 +https://storagehub.vmware.com/t/vmware-vsan/vmworld/vmworld-vsan-sessions/1
  
  
Line 211: Line 497:
 Value of SwapThickProvisionDisabled is 1 Value of SwapThickProvisionDisabled is 1
 </code> </code>
 +
 +====== Cluster Shutdown ======
 +
 +https://lifeofbrianoc.com/2017/09/19/shutdownpower-up-a-vsan-cluster-with-powercli/
 +
 +https://www.isjw.uk/post/vmware/vsan/vsan-driver-not-certified-copy/
 +
 +https://4sysops.com/archives/startup-and-shutdown-a-vmware-cluster-with-powercli-and-powershell/
 +
 +
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International