Hyperflex Notes

  1. BOM check error
During create cluster I am running into a bom check error
—————————————-
FIRMWARE-Check UCSC-SAS-M5HD : Required: 00.00.00.29,00.00.00.32,00.00.00.35,00.00.00.50,00.00.00.58, Found: 09.00.00.06; Action Needed: Update the Controller Firmware to Required Version
—————————————-
Can you please update the firmware on these 4 nodes. I am trying to install 3.5.2e on it
Solution :
A . Update Firmware on FI
B. Disable the bom check
go to each ctl
then open file /bin/bom-check.sh
comment out gen_Version_check from above code
hba_fw_check()
{
        CTLR="$1"
        FW_VER="$2"
        TYPE="FIRMWARE"
        ACTION_MSG="Update the Controller Firmware to Required Version"

        KEY="storage.controllers.${CTLR}.fw_version"
# Comment the below one to get pass the error
        gen_version_check "$CTLR" "$TYPE" "$KEY" "$FW_VER" "$ACTION_MSG"
}

 

2. Find FI from the ESX 

If you dont know the FI – You can check it from ESX with below command :

smbiosDump | grep -C5 “OEM Strings”

[root@ucs1367:~] smbiosDump | grep -C5 “OEM Strings”

    Data 10: 04 00 00 00 00 00 00 00 00 00 00 00 00 00 03 01

    Data 20: 00 00 00 00

  Type 204 Record: #99

    Data 00: cc 1c 63 00 03 00 01 02 00 00 08 00 00 00 00 00

    Data 10: 00 00 00 00 00 00 00 00 00 00 00 00

  OEM Strings: #102

    Cisco

    $SPI:rack-unit-14

    $SPT:hx-nodes-m5

$SYS:eng-fi78

 

3. Check the network speed for the FI connected to ucs610

ssh admin@eng-fi13.eng.storvisor.com

pwd : admin

run: Connect nxos

Show interface eth x/x detail

 

To find the Eth for the server node you are looking for :

Find the Eth with respect to the serial Number or Can find it easy from FI

eng-fi94-B(nx-os)# show interface chassis-info
    Chassis    Chassis    Remote      Chassis                   Module
 ID Port      Port State  Uplink  Model      Serial       Model       Serial
-------------------------------------------------------------------------------
  1 Eth1/1      Active    2    HXAF220C-M5SN WZP230319H3  UCSC-MLOM-C25Q-04 FCH23197EEJ
  2 Eth1/2      Active    2    HXAF220C-M5SN WZP230319GH  UCSC-MLOM-C25Q-04 FCH23197F1Z
  3 Eth1/3      Active    2    HXAF220C-M5SN WZP22500EDE  UCSC-MLOM-C25Q-04 FCH23197F4S
  4 Eth1/4      Active    2    HXAF220C-M5SN WZP230319HK  UCSC-MLOM-C25Q-04 FCH23197EZS
  5 Eth1/5      Active    2    HXAF240C-M5SX WZP23090WYA  UCSC-MLOM-C25Q-04 FCH23197F0Q
  6 Eth1/6      Active    2    HXAF240C-M5SX WZP23130HQM  UCSC-MLOM-C25Q-04 FCH23197EF5
  7 Eth1/7      Active    2    HXAF240C-M5SX WZP23090WXJ  UCSC-MLOM-C25Q-04 FCH23197EY7
  8 Eth1/8      Active    2    HXAF240C-M5SX WZP23090WYG  UCSC-MLOM-C25Q-04 FCH23197EWW
  9 Eth1/13      Active    3    HXAF220C-M5SX WZP23360FV9  UCSC-MLOM-C25Q-04 FCH23447F2G
 10 Eth1/14      Active    3    HXAF220C-M5SX WZP23360FT6  UCSC-MLOM-C25Q-04 FCH23447FBQ
 11 Eth1/15      Active    3    HXAF220C-M5SX WZP23360FMT  UCSC-MLOM-C25Q-04 FCH23447EKW
 12 Eth1/16      Active    3    HXAF220C-M5SX WZP23360FX8  UCSC-MLOM-C25Q-04 FCH23447E74

Or You can find from FI too

Screen Shot 2020-05-28 at 11.48.07

Once you get the Eth – You can find the speed under “Speed” column : 10GbE

eng-fi94-B(nx-os)# sh int eth 1/1 bri

--------------------------------------------------------------------------------
Ethernet      VLAN    Type Mode   Status  Reason                   Speed     Port
Interface                                                                    Ch #
--------------------------------------------------------------------------------
Eth1/1        1       eth  vntag  up      none                        10G(D) 1284

 

4. Installer fails with ESXI reachability check – Check if the VLAN is right. I used wrong VLAN : 210 , the right VLAN is 211 .Once i fixed it – it worked.

ESXi reachability check
IP address: 10.104.32.126at default port: 902 is not reachable.

Reset VC root password

I followed these KBs and went ahead fixed it:

1) reset the password – https://www.opvizor.com/vcsa-6-5-reset-root-password

2) fix the 503 error – http://www.brigm.com/2017/10/recover-from-vcsa-6-5-503-service-unavailable/

 

Collecting a support bundle from vCenter 6.0 or later Server Appliance or external Platform Service Controller using the command line

  1. Open a console session to the vCenter Server Appliance.
  2. Log in as an administrative user, such as root.
  3. Type shell.set –enabled true and press Enter.
  4. Type shell and press Enter.
  5. Run this command to export the logs to /storage/log/:

    vc-support -l

ntp issue

Deploying Storage Controller VM on ESXi Host

failed with ‘NTP server synchronization failed. Click Edit Configuration, enter a reliable NTP server, and retry.’

 

The issue is because the esx was not able to ping ntp1.eng.storvisor.com and ntp4.eng.storvisor.com

Checked with the working setup and fixed the vmk1 netmask : 255.255.248.0

 

[root@ucs847:~] ping ntp1.eng.storvisor.com

[root@ucs847:~] ping ntp4.eng.storvisor.com

[root@ucs847:~] ping 10.64.1.48

PING 10.64.1.48 (10.64.1.48): 56 data bytes

— 10.64.1.48 ping statistics —

3 packets transmitted, 0 packets received, 100% packet loss

[root@ucs847:~] vmkping -I vmk0 10.64.1.48

PING 10.64.1.48 (10.64.1.48): 56 data bytes

64 bytes from 10.64.1.48: icmp_seq=0 ttl=63 time=0.151 ms

64 bytes from 10.64.1.48: icmp_seq=1 ttl=63 time=0.104 ms

64 bytes from 10.64.1.48: icmp_seq=2 ttl=63 time=0.126 ms

— 10.64.1.48 ping statistics —

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.104/0.127/0.151 ms

[root@ucs847:~] vmkping -I vmk1 10.64.1.48

PING 10.64.1.48 (10.64.1.48): 56 data bytes

— 10.64.1.48 ping statistics —

3 packets transmitted, 0 packets received, 100% packet loss

[root@ucs847:~] esxcfg-route

VMkernel default gateway is 10.64.64.1

[root@ucs847:~] esxcfg-route -l

VMkernel Routes:

Network          Netmask          Gateway          Interface      

10.64.64.0       255.255.248.0    Local Subnet     vmk0           

10.64.0.0        255.240.0.0      Local Subnet     vmk1           

default          0.0.0.0          10.64.64.1       vmk0           

[root@ucs847:~] esxcli network interface ip standard list

Error: Unknown command or namespace network interface ip standard list

[root@ucs847:~] esxcli network

Usage: esxcli network {cmd} [cmd options]

Available Namespaces:

  firewall              A set of commands for firewall related operations

  ip                    Operations that can be performed on vmknics

  nic                   Operations having to do with the configuration of Network Interface Card and getting and

                        updating the NIC settings.

  port                  Commands to get information about a port

  sriovnic              Operations having to do with the configuration of SRIOV enabled Network Interface Card and

                        getting and updating the NIC settings.

  vm                    A set of commands for VM related operations

  vswitch               Commands to list and manipulate Virtual Switches on an ESX host.

  diag                  Operations pertaining to network diagnostics

[root@ucs847:~] esxcli network ip

Usage: esxcli network ip {cmd} [cmd options]

Available Namespaces:

  dns                   Operations pertaining to Domain Name Server configuration.

  interface             Operations having to do with the creation, management and deletion of VMkernel network

                        interfaces (vmknic).

  ipsec                 Operations on Internet Protocol Security

  route                 Operations pertaining to route configuration

  connection            List active tcpip connections

  neighbor              Operations that can be performed on arp tables

  netstack              Operations having to do with the creation, management and deletion of VMkernel Netstack

                        Instances.

Available Commands:

  get                   Get global IP settings

  set                   Update global IP settings

[root@ucs847:~] esxcli network ip route

Usage: esxcli network ip route {cmd} [cmd options]

Available Namespaces:

  ipv4                  IPv4 route operations

  ipv6                  IPv6 route operations

[root@ucs847:~] esxcli network ip route ipv4

Usage: esxcli network ip route ipv4 {cmd} [cmd options]

Available Commands:

  add                   Add IPv4 route to the VMkernel.

  list                  List configured IPv4 routes

  remove                Remove IPv4 route

[root@ucs847:~] esxcli network ip route ipv4 list

Network     Netmask        Gateway     Interface  Source

———-  ————-  ———-  ———  ——

default     0.0.0.0        10.64.64.1  vmk0       MANUAL

10.64.0.0   255.240.0.0    0.0.0.0     vmk1       MANUAL

10.64.64.0  255.255.248.0  0.0.0.0     vmk0       MANUAL

[root@ucs847:~] esxcfg-route -l

VMkernel Routes:

Network          Netmask          Gateway          Interface      

10.64.64.0       255.255.248.0    Local Subnet     vmk0           

10.64.0.0        255.240.0.0      Local Subnet     vmk1           

default          0.0.0.0          10.64.64.1       vmk0           

[root@ucs847:~] esxcli network ip

Usage: esxcli network ip {cmd} [cmd options]

Available Namespaces:

  dns                   Operations pertaining to Domain Name Server configuration.

  interface             Operations having to do with the creation, management and deletion of VMkernel network

                        interfaces (vmknic).

  ipsec                 Operations on Internet Protocol Security

  route                 Operations pertaining to route configuration

  connection            List active tcpip connections

  neighbor              Operations that can be performed on arp tables

  netstack              Operations having to do with the creation, management and deletion of VMkernel Netstack

                        Instances.

Available Commands:

  get                   Get global IP settings

  set                   Update global IP settings

[root@ucs847:~] esxcli network ip interface

Usage: esxcli network ip interface {cmd} [cmd options]

Available Namespaces:

  ipv6                  Commands to get and set IPv6 settings for vmknic.

  ipv4                  Commands to get and set IPv4 settings for vmknic.

  tag                   Commands to add and remove tags on vmknics.

Available Commands:

  add                   Add a new VMkernel network interface.

  list                  This command will list the VMkernel network interfaces currently known to the system.

  remove                Remove a VMkernel network interface from the ESXi host. A VMKernel network interface can

                        be uniquely specified by –interface-name or –portgroup-name or –dvs-name/–dvport-id.

                        i.e. Providing its name or its connection point are two ways to uniquely specify a

                        VMKernel network interface.

  set                   This command sets the enabled status and MTU size of a given IP interface

[root@ucs847:~] esxcli network ip interface ipv4

Usage: esxcli network ip interface ipv4 {cmd} [cmd options]

Available Commands:

  get                   Get IPv4 settings for VMkernel network interfaces.

  set                   Configure IPv4 setting for a given VMkernel network interface.

[root@ucs847:~] esxcli network ip interface ipv4 list

Error: Unknown command or namespace network ip interface ipv4 list

[root@ucs847:~] esxcli network ip interface ipv4 get

Name  IPv4 Address  IPv4 Netmask   IPv4 Broadcast  Address Type  DHCP DNS

—-  ————  ————-  ————–  ————  ——–

vmk0  10.64.64.232  255.255.248.0  10.64.71.255    STATIC           false

vmk1  10.64.72.227  255.240.0.0    10.79.255.255   STATIC           false

[root@ucs847:~] esxcli network ip interface ipv4 set

Error: Missing required parameter -t|–type

       Missing required parameter -i|–interface-name

Usage: esxcli network ip interface ipv4 set [cmd options]

Description:

  set                   Configure IPv4 setting for a given VMkernel network interface.

Cmd options:

  -i|–interface-name=<str>

                        The name of the VMkernel network interface to set IPv4 settings for. This name must be an

                        interface listed in the interface list command. (required)

  -I|–ipv4=<str>       The static IPv4 address for this interface.

  -N|–netmask=<str>    The static IPv4 netmask for this interface.

  -P|–peer-dns         A boolean value to indicate if the system should use the DNS settings published via DHCP

                        for this interface.

  -t|–type=<str>       IPv4 Address type :

                            dhcp: Use DHCP to aquire IPv4 setting for this interface.

                            none: Remove IPv4 settings form this interface.

                            static: Set Static IPv4 information for this interface. Requires –ipv4 and –netmask

                        options.

                         (required)

[root@ucs847:~] esxcli network ip interface ipv4 set -i vmk1 -N 255.255.248.0 -t static

The –type=static requires the –ipv4 and –netmask options.

Invalid –ipv4 option provided. This address is not a valid IPv4 address

[root@ucs847:~] esxcli network ip interface ipv4 set -i vmk1 -N 255.255.248.0 -t static –ipv4 10.64.72.227

[root@ucs847:~] esxcli network ip interface ipv4 get

Name  IPv4 Address  IPv4 Netmask   IPv4 Broadcast  Address Type  DHCP DNS

—-  ————  ————-  ————–  ————  ——–

vmk0  10.64.64.232  255.255.248.0  10.64.71.255    STATIC           false

vmk1  10.64.72.227  255.255.248.0  10.64.79.255    STATIC           false

[root@ucs847:~] vmkping -I vmk1 10.64.1.48

PING 10.64.1.48 (10.64.1.48): 56 data bytes

— 10.64.1.48 ping statistics —

3 packets transmitted, 0 packets received, 100% packet loss

[root@ucs847:~] /etc/init.d/n

nfcd     nfsgssd  nscd     ntpd

[root@ucs847:~] /etc/init.d/n

nfcd     nfsgssd  nscd     ntpd

[root@ucs847:~] esxcfg-n

esxcfg-nas   esxcfg-nics

[root@ucs847:~] esxcfg-n

esxcfg-nas   esxcfg-nics

[root@ucs847:~] esxcfg-nics -l

Name    PCI          Driver      Link Speed     Duplex MAC Address       MTU    Description                   

vmnic0  0000:1c:00.0 enic        Up   40000Mbps Full   00:25:b5:fe:a1:10 1500   Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic1  0000:1c:00.1 enic        Up   40000Mbps Full   00:25:b5:fe:b2:10 1500   Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic2  0000:1c:00.2 enic        Up   40000Mbps Full   00:25:b5:fe:a3:10 9000   Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic3  0000:1c:00.3 enic        Up   40000Mbps Full   00:25:b5:fe:b4:10 9000   Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic4  0000:1c:00.4 enic        Up   40000Mbps Full   00:25:b5:fe:a5:10 1500   Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic5  0000:1c:00.5 enic        Up   40000Mbps Full   00:25:b5:fe:b6:10 1500   Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic6  0000:1c:00.6 enic        Up   40000Mbps Full   00:25:b5:fe:a7:10 9000   Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic7  0000:1c:00.7 enic        Up   40000Mbps Full   00:25:b5:fe:b8:10 9000   Cisco Systems Inc Cisco VIC Ethernet NIC

[root@ucs847:~] esxcli network ip interface ipv4 get

Name  IPv4 Address  IPv4 Netmask   IPv4 Broadcast  Address Type  DHCP DNS

—-  ————  ————-  ————–  ————  ——–

vmk0  10.64.64.232  255.255.248.0  10.64.71.255    STATIC           false

vmk1  10.64.72.227  255.255.248.0  10.64.79.255    STATIC           false

[root@ucs847:~] esxcfg-route -l

VMkernel Routes:

Network          Netmask          Gateway          Interface      

10.64.64.0       255.255.248.0    Local Subnet     vmk0           

10.64.72.0       255.255.248.0    Local Subnet     vmk1           

default          0.0.0.0          10.64.64.1       vmk0           

[root@ucs847:~] ping ntp4.eng.storvisor.com

PING ntp4.eng.storvisor.com (10.64.1.48): 56 data bytes

64 bytes from 10.64.1.48: icmp_seq=0 ttl=63 time=0.108 ms

64 bytes from 10.64.1.48: icmp_seq=1 ttl=63 time=0.102 ms

64 bytes from 10.64.1.48: icmp_seq=2 ttl=63 time=0.087 ms

— ntp4.eng.storvisor.com ping statistics —

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.087/0.099/0.108 ms

[root@ucs847:~] ping ntp1.eng.storvisor.com

PING ntp1.eng.storvisor.com (10.64.1.48): 56 data bytes

64 bytes from 10.64.1.48: icmp_seq=0 ttl=63 time=0.138 ms

64 bytes from 10.64.1.48: icmp_seq=1 ttl=63 time=0.128 ms

64 bytes from 10.64.1.48: icmp_seq=2 ttl=63 time=0.109 ms

— ntp1.eng.storvisor.com ping statistics —

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.109/0.125/0.138 ms

[root@ucs847:~]

[root@ucs847:~]

Workload

Longevity workload :

[ubu2-fio-106-1-clone4] Start workload “/usr/local/bin/fio –ioengine=libaio –filename=/dev/sdc –output=/root/fio/fio.log –time_based=1 –rw=randrw –verify_dump=1 –name=longevity_vsi –thread=1 –verify=md5 –verify_fatal=1 –scramble_buffers=1 –bs=4k –direct=1 –buffer_compress_percentage=50 –do_verify=1 –randrepeat=0 –rwmixread=70 –buffer_compress_chunk=8192 –numjobs=5 –bssplit=4K/30:8k/25:32k/20:64k/15:128k/10 –blockalign=4k –iodepth=8 –verify_interval=4096 –runtime=168h”…

 

Explaination :

Fio :fio – flexible I/O tester

/usr/local/bin/fio

Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. The typical use of fio is to write a job file matching the I/O load one wants to simulate.

Fio which stands for Flexible I/O Tester is a free and open source disk I/O tool used both for benchmark and stress/hardware verification developed by Jens Axboe.

It has support for 19 different types of I/O engines (sync, mmap, libaio, posixaio, SG v3, splice, null, network, syslet, guasi, solarisaio, and more), I/O priorities (for newer Linux kernels), rate I/O, forked or threaded jobs, and much more. It can work on block devices as well as files.

Performing a Random Write Test

Let’s start by running the following command. This command will write a total 4GB file [4 jobs x 512 MB = 2GB] running 2 processes at a time:

 fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randwrite
--bs=4k --direct=0 --size=512M --numjobs=2 --runtime=240 --group_reporting

 

–ioengine=libaio

Linux Asynchronous I/O Explained		    
*******************************************************************************
				
Asynchronoes I/O (AIO) is a method for performing I/O operations so
that the process that issued an I/O request is not blocked till the 
data is available. Instead, after an I/O request is submitted, the 
process continues to execute its code and can later check the status 
of the submitted request.

There are two main libraries in Linux that facilitate AIO, we will 
refer to them as *libaio* and *librt*

 

filename=/dev/sdc

–outputfilename : Write output to filename.

References :
1.https://www.fsl.cs.sunysb.edu/~vass/linux-aio.txt
2.https://linux.die.net/man/1/fio
3.https://dotlayer.com/how-to-use-fio-to-measure-disk-performance-in-linux/

Find if the disk is SAMSUNG in Linux

 

 

root@ucs856scvm:~# sudo lshw -businfo | grep -i samsung

Note :240 represents 240GB size of the disk

scsi@2:0.0.0 /dev/sdb disk 240GB SAMSUNG MZ7LM240
scsi@2:0.1.0 /dev/sdc disk 960GB SAMSUNG MZ7LM960
scsi@2:0.2.0 /dev/sdd disk 960GB SAMSUNG MZ7LM960
scsi@2:0.3.0 /dev/sde disk 960GB SAMSUNG MZ7LM960
scsi@2:0.4.0 /dev/sdf disk 960GB SAMSUNG MZ7LM960
scsi@2:0.5.0 /dev/sdg disk 960GB SAMSUNG MZ7LM960
scsi@2:0.6.0 /dev/sdh disk 960GB SAMSUNG MZ7LM960
scsi@2:0.7.0 /dev/sdi disk 960GB SAMSUNG MZ7LM960

 

The below command will give you the complete Product ID of the ssd disk used

root@ucs856scvm:~# hdparm -I /dev/sdb | grep -i samsung

Model Number: SAMSUNG MZ7LM240HMHQ-00003

Change ESXi “root” password

Changing the ESXi Host Password

Tip: Although it is not required for the root user to have the same password on all hosts, doing so will make cluster management and support much easier. If you do select a different password for one or more hosts, make sure to note the password for each host.

Perform these steps on every ESXi host in the cluster.

  1. Log on to the ESXi host with SSH.
  2. Change the root password.
    root@esx# passwd root
  3. Respond to the prompts, providing the current and new root password.
    Changing password for root.
    Old Password:
    New password:
    Retype new password:
    Password changed.

    The minimum password length is 8 characters.

vSphere Administration Guide for Acropolis

Deploy installer vm

ovftool –diskMode=thin –noSSLVerify –acceptAllEulas=true –skipManifestCheck –X:injectOvfEnv –datastore=dest_ds1 –name=installer    /lc_test/Cisco-HX-Data-Platform-Installer-v3.5.1a-30983-esx.ova  vi://administrator@vsphere.local@qa075-vc65.eng.storvisor.com/qa062_dc/host/qa062_cl/qa-062d.eng.storvisor.com

The above gives some password error . Do it directly from VC . Go to VC –> select vc cluster –> right click –> Deploy OVF template –> In the last but one page : give the password (Cisco!1234) and create it there directly.

ESX upgrade

steps for esxi upgrade