DavidRing.ie

Troubleshooting a VxRail environment often starts with knowing where to look. VxRail 8.0 builds on the existing architecture but introduces a few changes in file locations, log collection practices, and the way system events are reported.

The following log files can be accessed through either the console or by using SSH to connect to the VxRail Manager appliance.

• Check the SLES operating system version used by VxRail Manager:
  cat /etc/os-release
• Check the currently installed VxRail version:
  rpm -qa | grep vxrail

Monitor VxRail Bring-Up Logs

The bring-up process logs key stages of initial configuration and microservice initialization. The most relevant files include:

• dayone.log – Provides detailed information about the initial first-run configuration.
  Location: /var/log/microservice_log/dayone.log
• short.term.log – Contains operational details about the VxRail microservices.
  Location: /var/log/microservice_log/short.term.log
• firstboot.log – Captures microservice boot-up details during the initial VxRail configuration sequence.
  Location: /var/log/firstboot.log

Example: Querying dayone.log

To review and monitor the dayone.log file, navigate to the microservice log directory:

cd /var/log/microservice_log/

Common commands for working with dayone.log:

# cat dayone.log
# tail -f dayone.log
# tail -f dayone.log | grep ERROR
# tail -n 100 dayone.log
# more dayone.log

Note: short.term.log is an excellent starting point when investigating microservices-related activity. To drill down into a specific microservice, simply filter the log by the microservice name. For example:

• cat short.term.log | grep microservice.do-vxrail
• cat short.term.log | grep microservice.do-host
• cat short.term.log | grep microservice.do-cluster
• cat short.term.log | grep microservice.do-network
• cat short.term.log | grep microservice.do-vm
• cat short.term.log | grep microservice.do-ecosystem
• cat short.term.log | grep microservice.do-eservices
• cat short.term.log | grep microservice.do-storage

List all microservices and check their status:

General VxRail Manager Log Files

Below is a list of commonly referenced VxRail log files and what they are used for:

Useful Log Analysis Commands

Below are commonly used commands for reviewing and analyzing VxRail log files. These examples use marvin.log, but the same commands can be applied to any log file:

/var is Running Low on Space: If the /var partition on the VxRail Manager appliance begins to run low on available space, you can identify usage and safely clear unnecessary log files. Check available space on VxRail manager with command “df -h”. Clear files under /var/log/microservice_log as needed:

Find Log or Configuration Files:

Lookup Event Codes:

psql -U postgres vxrail -c "SELECT * FROM serviceability.event_code ;"

psql -U postgres vxrail -c "SELECT * FROM serviceability.event_code ;" | grep MYSTIC010011

JSON: cat /var/lib/vmware-marvin/config-initial.json | json_pp

Manifest File: cat /var/lib/vmware-marvin/manifest.xml

Application Properties: cat /usr/lib/vmware-marvin/marvind/webapps/ROOT/WEB-INF/classes/application.properties

Confirm DNS entries in resolv.conf & DB: cat /etc/resolv.conf

NTP Validation (Returns 0): timeout 2 bash -c "</dev/udp/10.104.0.254/123"; echo $?

In terms of viewing a “Configuration complete!” equivalent message in the logs the string is :
cd /var/log/microservice_log

# cat dayone.log | grep "notify {'level': 'workflow', 'state': 'COMPLETED', 'progress': 100}"

RADAR VERSION: /mystic/radar/radar --version

Transfer Log Files via SCP (example):
scp ./marvin.log.log ringd@10.10.0.10:/users/ringd/logs/

Using DU cmd to search for specific log files:

du -a -h / |grep dayone |grep log

du -a /var/log/ | sort -n -r | head -n 5

Collecting VxRail Manager logs using command line:

/mystic/generateLogBundle.py -v

The path of the log collection is printed on the screen:

ESXi Useful commands

Below are commonly used commands for checking ESXi host status, logs, networking, storage, and vSAN configuration within a VxRail environment.

vCenter Log Monitoring

If VxRail Deployment Wizard fails to load or Vmware marvin service fails to start:

journalctl -xeu vmware-marvin.service

On the vxrail manager in /var/log/messages there are events indicating a duplicate IPv6 address is detected & kubernates fails to deploy:

cat /var/log/messages | grep "IPv6 duplicate address"

systemctl status runjars.service

systemctl status vmware-marvin.service

Modify the IPV6 address:

vi /etc/sysconfig/network/ifcfg-eth0

IPADDR0=’fd39:3939:3939:3939::180‘

Save & exit :wq!

systemctl restart network

Reboot VXRM.

After ~10 minutes check that kubernates “helium” pods havd been deployed:

• kubectl get pods
• kubectl get pods -A
• kubectl get pods -o wide
• kubectl get deployments

Confirm services are active or start if not:

systemctl status runjars.service
systemctl status vmware-marvin.service
systemctl start vmware-marvin
systemctl start runjars