Schubis Wiki und Gedankenstützen
You are here: Motivation » VMware » VSAN

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
Last revisionBoth sides next revision
vmware:vsan [2019/12/15 19:59] – [Samsung 970 EVO PlusFirmware] mschubivmware:vsan [2021/11/07 21:15] – [VSAN] mschubi
Line 4: Line 4:
  
 https://storagehub.vmware.com/#!/vmware-vsan/vmware-vsan-demonstrations https://storagehub.vmware.com/#!/vmware-vsan/vmware-vsan-demonstrations
 +
 +TBD
 +https://via.vmw.com/tchzcoreno1848
 +
 +===== vSAN mit CLI ohne vCenter =====
 +
 +https://blog.rylander.io/2017/01/19/configure-2-node-vsan-on-esxi-free-using-cli-without-vcenter/
 +====== vSAN Cluster Partitions beheben ======
 +
 +Im vSAN Cluster kann es dazu kommen, dass die einzelnen Nodes partitioniert sind. 
 +Meist ist nur eine einzelne Node weg, manchmal aber auch alle :-(
 +Auch wenn auf den vSAN Kernelport alle Hosts sich sehen können, finden sich die Partitions nicht zusammen.
 +
 +{{:vmware:vsan:2020-11-15_13_12_02-vsphere_-_nuc-cluster_-_skyline_health.png?500|}}
 +Im schlimmsten Fall sieht das so aus. 
 +
 +Erst mal sollte man alle vSAN Ports testen, ob die sich untereinander pingen lassen.
 +
 +<code bash>
 +[root@esx-4:~] vmkping -d -s 7000 -I vmk2 192.168.4.11
 +PING 192.168.4.11 (192.168.4.11): 7000 data bytes
 +7008 bytes from 192.168.4.11: icmp_seq=0 ttl=64 time=0.615 ms
 +7008 bytes from 192.168.4.11: icmp_seq=1 ttl=64 time=0.854 ms
 +7008 bytes from 192.168.4.11: icmp_seq=2 ttl=64 time=0.765 ms
 +
 +--- 192.168.4.11 ping statistics ---
 +3 packets transmitted, 3 packets received, 0% packet loss
 +round-trip min/avg/max = 0.615/0.745/0.854 ms
 +</code>
 +
 +Was sagt der Cluster?
 +<code bash>
 +[root@esx-4:~] esxcli vsan cluster get
 +Cluster Information
 +   Enabled: true
 +   Current Local Time: 2020-11-15T12:18:22Z
 +   Local Node UUID: 5d0cc03c-de7b-b94f-b016-54b2031c0494
 +   Local Node Type: NORMAL
 +   Local Node State: MASTER
 +   Local Node Health State: HEALTHY
 +   Sub-Cluster Master UUID: 5d0cc03c-de7b-b94f-b016-54b2031c0494
 +   Sub-Cluster Backup UUID:
 +   Sub-Cluster UUID: 52c9d6db-b533-a36c-6030-98d37f927e6a
 +   Sub-Cluster Membership Entry Revision: 2
 +   Sub-Cluster Member Count: 1
 +   Sub-Cluster Member UUIDs: 5d0cc03c-de7b-b94f-b016-54b2031c0494
 +   Sub-Cluster Member HostNames: esx-4.schubi.local
 +   Sub-Cluster Membership UUID: af16b15f-9a9b-99f7-989e-54b2031c0494
 +   Unicast Mode Enabled: true
 +   Maintenance Mode State: OFF
 +   Config Generation: 27b68356-c3e0-4c64-ab66-3b80741ab493 14 2020-10-19T20:29:54.524
 +</code>
 +Sieht auf jedem Cluster so aus. Jeweils nur ein Member. Aber sie "fühlen" sich dem gleichen Cluster zugehörig.
 +Überall ist ''Sub-Cluster UUID: 52c9d6db-b533-a36c-6030-98d37f927e6a'' gleich.
 +
 +Mal sehen wir die UniCast Agenten sich fühlen. Da gibts bei mir eine Überraschung
 +<code bash>
 +[root@esx-1:~] esxcli vsan cluster unicastagent list
 +NodeUuid                              IsWitness  Supports Unicast  IP Address      Port  Iface Name  Cert Thumbprint                                              SubClusterUuid
 +------------------------------------  ---------  ----------------  -------------  -----  ----------  -----------------------------------------------------------  --------------
 +5d0cc03c-de7b-b94f-b016-54b2031c0494          0              true  192.168.4.232  12321              EB:EF:07:00:20:7B:09:97:AD:EB:34:27:3D:B0:A7:32:8B:CF:57:CE  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cc5d7-070b-ff59-7be6-54b20319c59c          0              true  192.168.4.149  12321              09:C6:4C:7B:AD:5F:4B:7C:29:7D:BE:DC:4C:95:04:7E:02:32:35:B5  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0ccefe-e4a3-f316-ea35-54b2031bfc15          0              true  192.168.4.151  12321              B6:89:41:DC:9B:20:72:1B:19:E3:8A:C6:F8:49:AC:6D:93:8C:A3:76  52c9d6db-b533-a36c-6030-98d37f927e6a
 +
 +[root@esx-2:~] esxcli vsan cluster unicastagent list
 +NodeUuid                              IsWitness  Supports Unicast  IP Address      Port  Iface Name  Cert Thumbprint                                              SubClusterUuid
 +------------------------------------  ---------  ----------------  -------------  -----  ----------  -----------------------------------------------------------  --------------
 +5d0cc03c-de7b-b94f-b016-54b2031c0494          0              true  192.168.4.232  12321              EB:EF:07:00:20:7B:09:97:AD:EB:34:27:3D:B0:A7:32:8B:CF:57:CE  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cca61-c2bb-4270-a20e-54b2031c044c          0              true  192.168.4.219  12321              59:E9:AA:76:F8:14:55:F2:5D:DE:B8:AA:0E:0D:D0:30:4C:39:77:21  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0ccefe-e4a3-f316-ea35-54b2031bfc15          0              true  192.168.4.151  12321              B6:89:41:DC:9B:20:72:1B:19:E3:8A:C6:F8:49:AC:6D:93:8C:A3:76  52c9d6db-b533-a36c-6030-98d37f927e6a
 +
 +[root@esx-3:~] esxcli vsan cluster unicastagent list
 +NodeUuid                              IsWitness  Supports Unicast  IP Address      Port  Iface Name  Cert Thumbprint                                              SubClusterUuid
 +------------------------------------  ---------  ----------------  -------------  -----  ----------  -----------------------------------------------------------  --------------
 +5d0cc03c-de7b-b94f-b016-54b2031c0494          0              true  192.168.4.232  12321              EB:EF:07:00:20:7B:09:97:AD:EB:34:27:3D:B0:A7:32:8B:CF:57:CE  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cc5d7-070b-ff59-7be6-54b20319c59c          0              true  192.168.4.149  12321              09:C6:4C:7B:AD:5F:4B:7C:29:7D:BE:DC:4C:95:04:7E:02:32:35:B5  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cca61-c2bb-4270-a20e-54b2031c044c          0              true  192.168.4.219  12321              59:E9:AA:76:F8:14:55:F2:5D:DE:B8:AA:0E:0D:D0:30:4C:39:77:21  52c9d6db-b533-a36c-6030-98d37f927e6a
 +
 +[root@esx-4:~] esxcli vsan cluster unicastagent list
 +NodeUuid                              IsWitness  Supports Unicast  IP Address      Port  Iface Name  Cert Thumbprint                                              SubClusterUuid
 +------------------------------------  ---------  ----------------  -------------  -----  ----------  -----------------------------------------------------------  --------------
 +5d0cc5d7-070b-ff59-7be6-54b20319c59c          0              true  192.168.4.149  12321              09:C6:4C:7B:AD:5F:4B:7C:29:7D:BE:DC:4C:95:04:7E:02:32:35:B5  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cca61-c2bb-4270-a20e-54b2031c044c          0              true  192.168.4.219  12321              59:E9:AA:76:F8:14:55:F2:5D:DE:B8:AA:0E:0D:D0:30:4C:39:77:21  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0ccefe-e4a3-f316-ea35-54b2031bfc15          0              true  192.168.4.151  12321              B6:89:41:DC:9B:20:72:1B:19:E3:8A:C6:F8:49:AC:6D:93:8C:A3:76  52c9d6db-b533-a36c-6030-98d37f927e6a
 +
 +</code>
 +Die haben die aktuellen IOs *.4.11 bis *.4.14 gar nicht "gefressen".
 +Zum Thema UniCast Agent gibt es einen VMware KB https://kb.vmware.com/s/article/2150303.
 +
 +Gehen wir mal die Liste durch.
 +<code bash>
 +[root@esx-1:~] esxcli vsan network list
 +Interface
 +   VmkNic Name: vmk2
 +   IP Protocol: IP
 +   Interface UUID: fbf7215f-49cd-05cd-f9d8-54b2031c044c
 +   Agent Group Multicast Address: 224.2.3.4
 +   Agent Group IPv6 Multicast Address: ff19::2:3:4
 +   Agent Group Multicast Port: 23451
 +   Master Group Multicast Address: 224.1.2.3
 +   Master Group IPv6 Multicast Address: ff19::1:2:3
 +   Master Group Multicast Port: 12345
 +   Host Unicast Channel Bound Port: 12321
 +   Data-in-Transit Encryption Key Exchange Port: 0
 +   Multicast TTL: 5
 +   Traffic Type: vsan
 +</code>
 +Überall gut. Die Kernelport IP Config sieht auch gut aus.
 +<code bash>
 +[root@esx-1:~]  esxcli network ip interface ipv4 get | grep vmk2
 +vmk2  192.168.4.11   255.255.255.0  192.168.4.255   STATIC        192.168.4.1      false
 +</code>
 +
 +Für das weitere Vorgehen benötigen wir die UUID.
 +<code bash>
 +[root@esx-1:~]  cmmds-tool whoami
 +5d0cca61-c2bb-4270-a20e-54b2031c044c
 +[root@esx-2:~]  cmmds-tool whoami
 +5d0cc5d7-070b-ff59-7be6-54b20319c59c
 +[root@esx-3:~]  cmmds-tool whoami
 +5d0ccefe-e4a3-f316-ea35-54b2031bfc15
 +[root@esx-4:~] cmmds-tool whoami
 +5d0cc03c-de7b-b94f-b016-54b2031c0494
 +</code>
 +
 +Damit unsere folgenden manuellen Änderungen nicht durch ein Update des vCenters gestört wird, ignorieren wir die vCenter Settings temporär.
 +<code bash>
 +[root@esx-1:~] esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListupdates
 +Value of IgnoreClusterMemberListUpdates is 1
 +</code>
 +**Wichtig! Niemals auf einem Host die eigene Agent IP hinzufügen!**
 +VMware: //"When an ESXi host has its own IP address in its Unicast agent list, many networking problems can arise and potentially lead to the host encountering a PSOD."// 
 +Also "Augen auf" bei den folgenden Schritten!
 +
 +Für den esx1 Host müssen 2, 3 und 4 Hinzugefügt werden
 +<code bash>
 +esxcli vsan cluster unicastagent add -t node -u 5d0cc5d7-070b-ff59-7be6-54b20319c59c -U true -a 192.168.4.12 -p 12321
 +esxcli vsan cluster unicastagent add -t node -u 5d0ccefe-e4a3-f316-ea35-54b2031bfc15 -U true -a 192.168.4.13 -p 12321
 +esxcli vsan cluster unicastagent add -t node -u 5d0cc03c-de7b-b94f-b016-54b2031c0494 -U true -a 192.168.4.14 -p 12321
 +</code>
 +Ein Check des Hosts bringt noch nicht ganz die 100%ige Überzeugung.
 +<code bash>
 +[root@esx-1:~] esxcli vsan cluster unicastagent list
 +NodeUuid                              IsWitness  Supports Unicast  IP Address      Port  Iface Name  Cert Thumbprint                                              SubClusterUuid
 +------------------------------------  ---------  ----------------  -------------  -----  ----------  -----------------------------------------------------------  --------------
 +5d0cc03c-de7b-b94f-b016-54b2031c0494          0              true  192.168.4.232  12321              EB:EF:07:00:20:7B:09:97:AD:EB:34:27:3D:B0:A7:32:8B:CF:57:CE  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cc03c-de7b-b94f-b016-54b2031c0494          0              true  192.168.4.14   12321                                                                           52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cc5d7-070b-ff59-7be6-54b20319c59c          0              true  192.168.4.149  12321              09:C6:4C:7B:AD:5F:4B:7C:29:7D:BE:DC:4C:95:04:7E:02:32:35:B5  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cc5d7-070b-ff59-7be6-54b20319c59c          0              true  192.168.4.12   12321                                                                           52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0ccefe-e4a3-f316-ea35-54b2031bfc15          0              true  192.168.4.151  12321              B6:89:41:DC:9B:20:72:1B:19:E3:8A:C6:F8:49:AC:6D:93:8C:A3:76  52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0ccefe-e4a3-f316-ea35-54b2031bfc15          0              true  192.168.4.13   12321                                                                           52c9d6db-b533-a36c-6030-98d37f927e6a
 +</code>
 +
 +Aber zum Glück kann man "überflüssige" Einträge löschen...
 +<code bash>
 +esxcli vsan cluster unicastagent remove -a 192.168.4.232
 +esxcli vsan cluster unicastagent remove -a 192.168.4.149
 +esxcli vsan cluster unicastagent remove -a 192.168.4.151
 +</code>
 +So sieht es nun gut aus.
 +<code bash>
 +[root@esx-1:~]  esxcli vsan cluster unicastagent list
 +NodeUuid                              IsWitness  Supports Unicast  IP Address     Port  Iface Name  Cert Thumbprint  SubClusterUuid
 +------------------------------------  ---------  ----------------  ------------  -----  ----------  ---------------  --------------
 +5d0cc03c-de7b-b94f-b016-54b2031c0494          0              true  192.168.4.14  12321                               52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0cc5d7-070b-ff59-7be6-54b20319c59c          0              true  192.168.4.12  12321                               52c9d6db-b533-a36c-6030-98d37f927e6a
 +5d0ccefe-e4a3-f316-ea35-54b2031bfc15          0              true  192.168.4.13  12321                               52c9d6db-b533-a36c-6030-98d37f927e6a
 +</code>
 +
 +Exemplarisch noch für die weiteren Hosts
 +
 +ESX2:
 +<code bash>
 +esxcli vsan cluster unicastagent add -t node -u 5d0cca61-c2bb-4270-a20e-54b2031c044c -U true -a 192.168.4.11 -p 12321
 +esxcli vsan cluster unicastagent add -t node -u 5d0ccefe-e4a3-f316-ea35-54b2031bfc15 -U true -a 192.168.4.13 -p 12321
 +esxcli vsan cluster unicastagent add -t node -u 5d0cc03c-de7b-b94f-b016-54b2031c0494 -U true -a 192.168.4.14 -p 12321
 +</code>
 +
 +ESX3:
 +<code bash>
 +esxcli vsan cluster unicastagent add -t node -u 5d0cca61-c2bb-4270-a20e-54b2031c044c -U true -a 192.168.4.11 -p 12321
 +esxcli vsan cluster unicastagent add -t node -u 5d0cc5d7-070b-ff59-7be6-54b20319c59c -U true -a 192.168.4.12 -p 12321
 +esxcli vsan cluster unicastagent add -t node -u 5d0cc03c-de7b-b94f-b016-54b2031c0494 -U true -a 192.168.4.14 -p 12321
 +</code>
 +
 +ESX4:
 +<code bash>
 +esxcli vsan cluster unicastagent add -t node -u 5d0cca61-c2bb-4270-a20e-54b2031c044c -U true -a 192.168.4.11 -p 12321
 +esxcli vsan cluster unicastagent add -t node -u 5d0cc5d7-070b-ff59-7be6-54b20319c59c -U true -a 192.168.4.12 -p 12321
 +esxcli vsan cluster unicastagent add -t node -u 5d0ccefe-e4a3-f316-ea35-54b2031bfc15 -U true -a 192.168.4.13 -p 12321
 +</code>
 +
 +Danach natürlich wieder Kontrolle mit 
 +<code bash>
 + esxcli vsan cluster unicastagent list
 +</code>
 +und ggf. Löschen mit
 +<code bash>
 +esxcli vsan cluster unicastagent remove -a 
 +</code>
 +
 +Kaum macht man es richtig, schon geht's. Ich habe einen Master, einen Backup, der Rest Agent. Hier mal ein Auszug.
 +<code bash>
 +[root@esx-1:~]  esxcli vsan cluster get
 +Cluster Information
 +   Enabled: true
 +   Current Local Time: 2020-11-15T13:10:16Z
 +   Local Node UUID: 5d0cca61-c2bb-4270-a20e-54b2031c044c
 +   Local Node Type: NORMAL
 +   Local Node State: BACKUP
 +   Local Node Health State: HEALTHY
 +   Sub-Cluster Master UUID: 5d0cc5d7-070b-ff59-7be6-54b20319c59c
 +   Sub-Cluster Backup UUID: 5d0cca61-c2bb-4270-a20e-54b2031c044c
 +   Sub-Cluster UUID: 52c9d6db-b533-a36c-6030-98d37f927e6a
 +   Sub-Cluster Membership Entry Revision: 4
 +   Sub-Cluster Member Count: 4
 +   Sub-Cluster Member UUIDs: 5d0cc5d7-070b-ff59-7be6-54b20319c59c, 5d0cca61-c2bb-4270-a20e-54b2031c044c, 5d0ccefe-e4a3-f316-ea35-54b2031bfc15, 5d0cc03c-de7b-b94f-b016-54b2031c0494
 +   Sub-Cluster Member HostNames: esx-2.schubi.local, esx-1.schubi.local, esx-3.schubi.local, esx-4.schubi.local
 +   Sub-Cluster Membership UUID: b216b15f-40f2-2395-1900-54b20319c59c
 +   Unicast Mode Enabled: true
 +   Maintenance Mode State: OFF
 +   Config Generation: 27b68356-c3e0-4c64-ab66-3b80741ab493 20 2020-11-15T12:57:11.0
 +</code>
 +
 +Zum Schluss nicht vergessen, den Advanced Parameter zu ersetzen:
 +<code bash>
 +esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListupdates
 +</code>
 +
 +
 +===== vSAN Services checken und starten =====
 +
 +<code bash>
 +/etc/init.d/cmmdsd status && /etc/init.d/epd status && /etc/init.d/clomd status
 +
 +/etc/init.d/cmmdsd restart && /etc/init.d/epd restart && /etc/init.d/clomd restart
 +</code>
 +====== ESXi Infos in der RVC ======
 +
 +<code bash>
 +/localhost/DC/computers/vSAN-Cluster> table -f name -s name -f state.connection:state.maintenancemode:build:uptime:num.vms:num.poweredonvms:cpuusage:memusage hosts/*
 +</code>
  
 ===== Samsung 970 EVO PlusFirmware===== ===== Samsung 970 EVO PlusFirmware=====
Line 78: Line 320:
  
 https://storagehub.vmware.com/section-assets/powercli-cookbook-for-vsan https://storagehub.vmware.com/section-assets/powercli-cookbook-for-vsan
 +
 +https://storagehub.vmware.com/t/vmware-vsan/vmworld/vmworld-vsan-sessions/1
  
  
Line 249: Line 493:
 Value of SwapThickProvisionDisabled is 1 Value of SwapThickProvisionDisabled is 1
 </code> </code>
 +
 +====== Cluster Shutdown ======
 +
 +https://lifeofbrianoc.com/2017/09/19/shutdownpower-up-a-vsan-cluster-with-powercli/
 +
 +https://www.isjw.uk/post/vmware/vsan/vsan-driver-not-certified-copy/
 +
 +https://4sysops.com/archives/startup-and-shutdown-a-vmware-cluster-with-powercli-and-powershell/
 +
 +
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International