I had a problem activating TM DS Agent with symptoms similar to the ones described in Trend Micro Knowledge Base / Solution ID 1095437 “Anti-malware module of the Deep Security Agent (DSA) shows “Driver offline / Not installed” in the Deep Security Notifier“.

After numerous attempts to uninstall/clean up/reinstall Trend Micro Deep Security Agent the issue still was not resolved.

So, here is the ULTIMATE guide to uninstalling TM DSA that worked for me on several servers:

N.B. If you have network teaming configured, this procedure may break the team or wipe the team’s network stack. Just re-create the team and it should work OK.

1. Uninstall the DSA from the server
2. Run the tbclean.exe utility. See [Solution ID 1054528]
3. Clean up the registry
   HKEY_LOCAL_MACHINE\SOFTWARE\TrendMicro\AEGIS
   HKEY_LOCAL_MACHINE\SOFTWARE\TrendMicro\AMSP
   HKEY_LOCAL_MACHINE\SOFTWARE\TrendMicro\AMSPStatus
   HKEY_LOCAL_MACHINE\SOFTWARE\TrendMicro\Deep Security Agent
   HKEY_LOCAL_MACHINE\SOFTWARE\TrendMicro\WL
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Amsp
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ds_agent
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ds_notifier
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\tbimdsa
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\tmactmon
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\tmcomm
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\tmevtmgr
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\eventlog\Application\Deep Security Agent
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\eventlog\Application\Deep Security Relay
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\eventlog\System\tbimdsa\
   HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Installer\Features\C4AF20E48325C454BBBE163E418FCEA9\
   HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Installer\Products\C4AF20E48325C454BBBE163E418FCEA9\
   HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Installer\UpgradeCodes\689D08D76B5A47A4FB59D97D2C4B9308\
   HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UpgradeCodes\689D08D76B5A47A4FB59D97D2C4B9308\
   HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{4E02FA4C-5238-454C-BBEB-61E314F8EC9A}\
4. Restart the server
5. Check if any of these files or folders are present and delete them if found:
   C:\WINDOWS\System32\Drivers\tbimdsa.sys
   C:\WINDOWS\System32\Drivers\tmactmon.sys
   C:\WINDOWS\System32\Drivers\tmcomm.sys
   C:\WINDOWS\System32\Drivers\tmevtmgr.sys
   C:\WINDOWS\System32\LogFiles\ds_agent\
   C:\Program Files\Trend Micro\AMSP\
   C:\Program Files\Trend Micro\Deep Security Agent\Agent
   C:\Program Files\Trend Micro\Deep Relay of Security Settings\Local (Relay)
   C:\Program Files\Trend Micro\Deep Notifier of Security Settings\Local (Notifier)
   C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Trend Micro\Deep Security\Trend Micro Deep Security Notifier (for Windows 2008)
   C:\Documents and Settings\All Users\Start menu\programs\Trend Micro\ Deep Security\Trend Micro Deep Security Notifier (for Windows 2003)
   C:\Windows\Installer\{4E02FA4C-5238-454C-BBEB-61E314F8EC9A}/Agent 64-bit
6. Reviewing the file C:\Windows\inf\setupapi.dev.log.
   Look for entries containing tmcomm.sys, tmevtmgr.sys and tmactmon.sys On this entries you will be able to identify if there are any remains of a previous installation, look for lines like “Installing catalog (any of the three drivers above).cat as:” note the dates of the installation and the oemXX.inf files used to install these drivers.
7. Uninstall the existing tmcomm.sys, tmevtmgr.sys and tmactmon.sys using pnputil -d oemfile.inf (on this particular computer oem26.inf, oem27.inf and oem28.inf)
   Identify which oemXX.inf files you need to uninstall by reviewing the setupapi.dev.log
8. Delete any catalog files for AMSP drivers present in C:\Windows\system32\catroot
   (on this particular computer oem9.cat, oem10.cat and oem11.cat) that are leftovers from previous installations and that tbclean and pnputil did not remove
9. Delete old driver files present in windows driver store
   C:\Windows\system32\DriverStore\FileRepository\tmxxxx (folders)
   N.B. You might need to take ownership of those folders
10. Install all the comodo certificates following the KB: http://esupport.trendmicro.com/solution/en-US/1104241.aspx
    Remember to place them in the appropriate store.
11. Reinstall the DSA using the freshly downloaded installation package. [Trend Micro Software Download Center]
12. Restart the server
13. Verify that the drivers are present in the device manager (using view non P&P devices), you should see the following drivers tmcomm.sys, tmevtmgr.sys and tmactmon.sys
14. Deactivate the agent on the DSM (to remove the old association)
15. Activate the agent from the DSM.
    If you prefer using the agent initiated activation use this command: “dsa_control /a dsm://<host or IP>:<port>/” (default port 4120)

Hope this will help

We discovered another bug in EMC UIM/P 4.x where VMFS volumes are not syncing with vCenter.

Here are the error messages:

• In the vCenter Sync Status:
  

The operation failed.

* Overall status of X successful hosts X failed disks

•  All those “failed disks” are LUNs that are configured as VMFS datastores.
  

The operation failed.

* System error Validation Error(s) occurred during save():
– Field error in object ‘viu.Host’ on field ‘disks.uuid': rejected value [null]; codes [viu.Disk.uuid.nullable.error.viu.Host.disks.uuid,viu.Disk.uuid.nullable.error.disks.uuid,viu.Disk.uuid.nullable.error.uuid,viu.Disk.uuid.nullable.error,disk.uuid.nullable.error.viu.Host.disks.uuid,disk.uuid.nullable.error.disks.uuid,disk.uuid.nullable.error.uuid,disk.uuid.nullable.error,viu.Disk.uuid.nullable.viu.Host.disks.uuid,viu.Disk.uuid.nullable.disks.uuid,viu.Disk.uuid.nullable.uuid,viu.Disk.uuid.nullable,disk.uuid.nullable.viu.Host.disks.uuid,disk.uuid.nullable.disks.uuid,disk.uuid.nullable.uuid,disk.uuid.nullable,nullable.viu.Host.disks.uuid,nullable.disks.uuid,nullable.uuid,nullable]; arguments [uuid,class viu.Disk]; default message [Property [{0}] of class [{1}] cannot be null]

Engineering team confirmed it’s a bug and it will be fixed in EMC UIM/P 4.1 P3. We don’t have the release date yet.

Stay tuned!

In my ‘EMC UIM Service Adoption Utility (SAU)‘ article I talked about a potential need to amend the Service Offering template that will be used for multiple Services during adoption through UMC SAU. See Step 13 for details.

Some services could not be adopted using an Existing Service Offering and the adoption failed with the following error message:

The selected Service Offering does not match for these reasons:

• There was no network adapter that satisfies this network constraint…
• Network adapter XXX does not match any network constraint…

The screenshot above shows the error message in EMC UIM/P 4.1 where VLAN IDs is used to configure vNICs, rather than VLAN names like in previous version. Side Note: This is really annoying and I complained about this approach to the development team. I hope they will listen to their customers. This issue is not specific to UIM/P 4.1, it existed in previous versions as well.

These errors do not make any sense as the existing Service Offering we try to use in the EXACT copy of the Service we try to adopt – see ‘EMC UIM Service Adoption Utility (SAU)‘ article, steps 14 to 20.

No, we are not going mad, it is just a bug!

The workaround is to allow the SAU to create a new Service Offering until the fix is released in UIM 4.1 P3 (no release date currently available).

I hope this will help.

EMC KB 193094 : ESA-2014-113: EMC Unified Infrastructure Manager Operations and Unified Infrastructure Manager Provisioning Security Update for Multiple Vulnerabilities in Bash
https://support.emc.com/kb/193094

Affected Products
EMC Software: EMC Unified Infrastructure Manager/Operations (UIMO) All versions
EMC Software: EMC Unified Infrastructure Manager/Provisioning (UIMP) All versions

Resolution
Remediate bash by following the steps below. These steps work the same for UIM/O and UIM/P.

Link To Remedies:
Registered EMC Online Support customers can download shell_shock_fix.zip from support.emc.com at: https://support.emc.com/downloads/28497_Unified-Infrastructure-Manager

• Download shell_shock_fix.zip and copy it to the system to be updated
  Use WinSCP or similar product to upload to /tmp
• Unzip shell_shock_fix.zip
uim001:/tmp # unzip shell_shock_fix.zip
Archive:  shell_shock_fix.zip
creating: shell_shock/
inflating: shell_shock/bash-3.2-147.22.1.x86_64.rpm
inflating: shell_shock/shell-shock-vulnerability-fix.pl
inflating: shell_shock/bash-doc-3.2-147.22.1.x86_64.rpm
inflating: shell_shock/libreadline5-5.2-147.22.1.x86_64.rpm
inflating: shell_shock/readline-doc-5.2-147.22.1.x86_64.rpm
• Change directory to shell_shock
  uim001:/tmp # cd shell_shock/
uim001:/tmp/shell_shock #
• Run ./shell-shock-vulnerability-fix.pl
  uim001:/tmp/shell_shock # ./shell-shock-vulnerability-fix.pl
This system is vulnerable to shell shock.
Do you want to install the update? (y/n) y
Preparing...             ########################################### [100%]
1:libreadline5           ########################################### [ 25%]
2:bash                   ########################################### [ 50%]
3:bash-doc               ########################################### [ 75%]
4:readline-doc           ########################################### [100%]
Done

I hope this will help.

I had to re-visit a vCloud Connector environment I built quite a while ago but could not login to neither VMware vCloud Connector Server nor Node with the admin account. I was sure that I did not change the default password (vmware) so was the customer.  I also checked that admin account was not locked out – see “HOW TO: Unlock admin account in vCloud Connector Server / Node” blog post for details.

Here is the error message:

After a bit of goggling I found Marek Zdrojewski’s post “How to upgrade vCloud Connector 2.5 to version 2.6” that mentions that VMware vCloud Connector Server / Node admin password may have expired.

When I tried to login to the vCC Server / Node console as admin, there was a confirmation that the admin password has indeed expired and needs to be changed:

login as: admin
vCloud Connector Node Appliance
admin@vcn001's password:
Your password has expired. Choose a new password.
WARNING: Your password has expired.
You must change your password now and login again!
Changing password for admin.
Old Password:
etc etc

I did not want to change the default password and therefore reconfigured admin account to stop it expiring.

Login to vCCS/vCCN as root and use chage command check current admin account aging.

login as: root
vCloud Connector Node Appliance
root@vcn001's password:
vcn001:~ # chage -l admin
Minimum:        0
Maximum:        365
Warning:        7
Inactive:       -1
Last Change:            Jun 15, 2013
Password Expires:       Jun 15, 2014
Password Inactive:      Never
Account Expires:        Never

To disable password aging / expiration for user admin, type command as follows and set:
Maximum Password Age to 99999
Account Expiration Date to -1

vcn001:~ # chage -M 99999 -E -1 admin
Aging information changed.
vcn001:~ # chage -l admin
Minimum:        0
Maximum:        99999
Warning:        7
Inactive:       -1
Last Change:            Jun 15, 2013
Password Expires:       Never
Password Inactive:      Never
Account Expires:        Never

Hope this helps.

I would like to thank Marek Zdrojewski (@MarekDotZ) for his prompt replies and his help to get me on the right track in troubleshooting this issue. Please check out his blog http://defaultreasoning.com

Today one of our Solution Architects asked me how to reclaim Cisco UCS Chassis ID.

Someone has decommissioned a previous chassis and removed it physically! It was done weeks/months ago and chassis is gone.

If you try to recommission chassis with original ID you’ll get error:
Error: Update failed: [Chassis 3 has already been allocated (N20-C6508 FOX1448GT9G).]

You need to delete old chassis from CLI and then you can use ID again

FI-A# show config all | grep "recommission chassis"
recommission chassis "Cisco Systems Inc" N20-C6508 FOX1326GC2R 4
recommission chassis "Cisco Systems Inc" N20-C6508 FOX1437G8KE 1
recommission chassis "Cisco Systems Inc" N20-C6508 FOX1448GT9G 3 <-- this one was gone
recommission chassis "Cisco Systems Inc" N20-C6508 FOX1452G718 2

If the old chassis# was 3, then do this :

FI-A# delete decommissioned chassis "Cisco Systems Inc" N20-C6508 FOX1448GT9G
FI-A* # commit-buffer

(c) Zbigniew ‘Ziggy’ Misiak

I tried to upgrade VMware vCloud Connector Server but the upgrade got stuck at “Checking for available updates…” and Check Updates button was greyed out even after restarting the vCC Server.

In order to stop vCC from continuing in this state perform the following steps:

1. In the vCC Server web interface go to the Update section where we see the ‘Checking for available updates…’ message being displayed. Please then choose the Settings tab within the Update section;
2. Under the Update Repository section please set the update to point to ‘Use CDROM Updates’. Hence you cannot manually initiate vCC Server upgrade, you also need to set vCC Server to automatically check for new updates by selecting “Automatic check for updates” and schedule the check to run at the next full hour. Unfortunately, there is no option to set the time more granularly, you may need to wait for the schedule to kick in.  Click Save Settings under Actions;
   
3. Shut down the vCC Server (System Tab, Shutdown) and attach CDROM .ISO file (as vCC does not come with CDROM drive per-configured, you need to add it to the VA). Any .ISO file will do;
4. Restart the vCC Server and wait for the update process to check for update, it should then fail due to invalid .ISO;
   
5. Reconfigure Update Settings back to ‘No automatic updates’ and ‘Use Default Repository’, click Save Settings.
6. Go back to the Status tab, click on Check Updates and then on Install Updates.
   

Hope this helps.

I recently upgraded VMware vCloud Connector from version 2.6.0 to 2.6.1 (check out THIS POST for details) and noticed a cool new feature!

In the previous versions, including v 2.6.0, the pre copy validation did not check the size of the vCC Node staging area – the local disk storage used for the OVF export and therefore if you try to copy a VM bigger than the staging area, the copy may fail.  In the new version 2.6.1 it warns you that this may happen.

One thing we need to keep in mind – during VM export to OVF template, the disk files are stored in a compressed, sparse format.  In the example above, VM being migrated is bigger than the size of the staging area on the vCC Node BUT it is not a problem, I know the disks are in TPLZ format and only 30GB is actually used by the guest OS.  The VM Copy completed successfully.

You may ask what can we do if VM is definitely larger that the staging area on the vCC Node?  No problem!  You can resize the vCC Node data disk.  The procedure is documented in the Installing and Configuring vCloud Connector (PDF version) document, search for “Configure vCloud Connector Node Allocated Storage in vSphere”.

…and for your convenience… :)

Configure vCloud Connector Node Allocated Storage in vSphere

To successfully copy resources to or from a vSphere cloud, you must configure and resize the data disk associated with the vCloud Connector node for that vSphere cloud.

Prerequisites

• You are a vSphere administrator.
• You have taken a snapshot of the virtual appliance.

Procedure

1. Log in to the vSphere Client.
2. In the hierarchy tree, select the vCloud Connector node virtual appliance.
3. Right-click and select Edit Settings.
   The Virtual Machine Properties window opens to the Hardware tab.
4. Select Hard disk 2 in the Hardware column.
5. Modify the size, based on the size of the resources you are going to be transferring, and click OK.
6. Open the console for the vCloud Connector node, ALT-F2, login as admin. Alternatively, SSH to the vCC Node.
7. Run the following command to resize the disk:

    sudo /opt/vmware/hcagent/scripts/resize_disk.sh

   In the example below I extended the disk from default 40GB to 100GB:

   login as: root
   vCloud Connector Node Appliance
   root@10.XXX.XXX.XXX's password:
   Last login: Fri Oct 31 16:59:41 2014 from desktop.vStrong.info
   vcn001:~ # sudo /opt/vmware/hcagent/scripts/resize_disk.sh
     Physical volume "/dev/sdb" changed
     1 physical volume(s) resized / 0 physical volume(s) not resized
     Extending logical volume data to 100.00 GiB
     Logical volume data successfully resized
   Filesystem at /dev/data_vg/data is mounted on /data; on-line resizing required
   old desc_blocks = 3, new_desc_blocks = 7
   Performing an on-line resize of /dev/data_vg/data to 26213376 (4k) blocks.
   The filesystem on /dev/data_vg/data is now 26213376 blocks long.

Hope this helps.

I have been playing with EMC UIM/P 4.1 recently and discovered a couple new features that were not available in the previous versions.

If you need to add new vNICs to an existing Service, you can now add them as a pair! Selected VLANs, MTU, Native VLAN, QoS and other Policies will be configured identically for both vNICs AND they will be places on different switches, one on switch A, one on switch B! How cool is that!!! Unfortunately, in the best tradition of EMC UIM development, this feature is not available for new Service Offering configuration…

… and here is another cool new feature – before you hit the Apply button, you can check what changes are pending. No more guessing or double checking the changes you are about to apply!

Click on Changes Pending and confirm the difference between the current and the target configuration:

Have you spotted any more interested and useful features?  Please share them in the comments below.

In this guide I will take you through the vCloud Networking and Security / vShield Manager and components upgrade to version 5.5.3.1. This release addresses Shellshock vulnerability (VMware KB 2091218).

Before you proceed with the upgrade, please confirm the new version is compatible with the existing VMware and other security products in your environment (Trend Micro Deep Security, for example):

• VMware Product Interoperability Matrixes;
• Trend Micro Deep Security & VMware compatibility matrixes

Download VMware vShield Manager Upgrade bundle from my.VMware.com :

• vShield Manager Upgrade Bundle 5.5.3.1
  https://my.vmware.com/group/vmware/info/slug/security_products/vmware_vcloud_networking_and_security/5_5
  Click on Read More to reveal the release date, build number and simple reference to the upgrade path:
  File name: VMware-vShield-Manager-upgrade-bundle-5.5.3-2175697.tar.gz
  Upgrade to vCNS 5.5.3.1 from: vCNS 5.1.1, 5.1.2, 5.1.3, 5.1.4.x, 5.5.0, 5.5.1, 5.5.2, 5.5.3.

You must first upgrade the vShield Manager, then update the other components.

vShield Manager Upgrade

1. First, take a snapshot of vShield Manager virtual appliance!
2. Login to vShield Manager, navigate to inventory panel, View: > Host & Clusters, click Settings & Reports.
3. Click the Updates tab.
4. Click Upload Upgrade Bundle.
   
5. Click Browse and select the VMware-vShield-Manager-upgrade_bundle-buildNumber.tar.gz file.
6. Click Open.
7. Click Upload File.
8. After the file is uploaded, click Update Status.
   
9. Click Install to begin the upgrade process.
10. Click Confirm Install.
    
    The upgrade process reboots vShield Manager, so you might lose connectivity to the vShield Manager user interface. None of the other vShield components are rebooted.
    
11. After the reboot, log back in to the vShield Manager.
12. OPTIONAL: Login to Trend Micro Deep Security Manager and make sure it still can connect to vSM.
    

Although this upgrade looks straightforwards and, in the majority of cases, runs OK, I have a couple of tips for you:

• ALWAYS take a snap of vShield Manager!
• vShield Manager upgrade from 5.0.x to 5.1.x is different as you need to upgrade its virtual hardware. See VMware KB 2044458 for detailed instructions.
• Make sure vShield Manager VA hardware spec is correct (I experienced an issue with the upgrade when someone manually changed vSM VA configuration)
• A minimum of 2.5 GB free disk space is available in the /common partition. Login to vSM console and run show filesystems:
  FAIL:
  
  OK:
  

vShield App and vShield Endpoint Upgrade

You must upgrade vShield App and vShield Endpoint on each host in your datacenter.  Both can can be upgraded either through vSphere Client or vShield Manager interface.  I personally prefer the later.

1. Login vShield Manager, navigate to inventory panel, select View: > Hosts & Clusters.
2. Select Inventory > Hosts and Clusters.
3. Under Datacenters / Your_Datacenter / Your_Cluster click the host on which you want to upgrade vShield App / vShield Endpoint.
4. The Summary tab displays each vShield component that is installed on the selected host and the available upgrade version.
5. Select Update next to vShield App or vShield Endpoint.
   
6. Select the vShield App or vShield Endpoint checkbox. You can also select both and upgrade them in one go.
   
7. Click Install.  During vShield App upgrade, the ESXi host is placed into Maintenance Mode and rebooted. Ensure that virtual machines on the ESXi host are migrated (using DRS or vMotion), or that they are powered off to allow the host to be placed into Maintenance Mode.
   
   
8. All done:
   
9. OPTIONAL: Again, it may be a good idea to check the host in Trend Micro Deep Security Manager…

 vShield Edge Upgrade

1. Log in to the vShield Manager, navigate to View: > Networks.
2. Select the Datacenter, Network Vistualization, Edges.
3. Select the vShield Edge. Notice the up arrow – Upgrade available.
   
4. Click the Actions and select Upgrade.
   
5. Click Yes to continue.
   
6. This is where it gets interesting! vShield Manager will NOT upgrade vShield Edge virtual appliance, instead it will deploy a new appliance, copy the config and then deleted the old appliance.
   

I hope this helps.

This simple PowerShell/PowerCLI script calculates the average CPU, Memory, Network and Disk usage for powered on virtual machines over the last 30 days, 5 minutes interval.

Amend ‘AddDays(-N)‘ and ‘-IntervalMins 5‘ if needed.

Get-VM | Where {$_.PowerState -eq "PoweredOn"} | Select Name, Host, NumCpu, MemoryMB, `
@{N="CPU Usage (Average), Mhz" ; E={[Math]::Round((($_ | Get-Stat -Stat cpu.usagemhz.average -Start (Get-Date).AddDays(-30) -IntervalMins 5 | Measure-Object Value -Average).Average),2)}}, `
@{N="Memory Usage (Average), %" ; E={[Math]::Round((($_ | Get-Stat -Stat mem.usage.average -Start (Get-Date).AddDays(-30) -IntervalMins 5 | Measure-Object Value -Average).Average),2)}} , `
@{N="Network Usage (Average), KBps" ; E={[Math]::Round((($_ | Get-Stat -Stat net.usage.average -Start (Get-Date).AddDays(-30) -IntervalMins 5 | Measure-Object Value -Average).Average),2)}} , `
@{N="Disk Usage (Average), KBps" ; E={[Math]::Round((($_ | Get-Stat -Stat disk.usage.average -Start (Get-Date).AddDays(-30) -IntervalMins 5 | Measure-Object Value -Average).Average),2)}} |`
Export-Csv -Path d:\AverageUsage.csv

Example of the script output:

I hope you find this useful.

VMware vCenter Support Assistant Appliance installation step 1 – Configure “Lookup Service address” failed with the error message: “Cannot connect to the lookup service at the specified URL. Check if the URL is correct”.

I tested the URL and DNS configuration on the VA – all OK.  Tried the FQDN of the vCenter Server server name but it failed. I also tried to configure the lookup service address with the IP address rather than a server name and it worked OK!  The documentation refers to the address (the IP address ?!), but what about the server name:

Provide Lookup Service Address and Review Certificate

To establish connection to the lookup service server and review the lookup service certificate details, you
must provide the lookup service address.

vCenter Support Assistant must know the location of the vCenter Server instances that it monitors. To get
the list of the vCenter Server instances in your environment you must provide the vCenter Server lookup
service address.

Procedure

1. In the Lookup Service Address text box, type the vCenter Lookup Service address.
   You must provide the lookup service address to register vCenter Support Assistant components with vSphere.
   For example, https://vCenter_Server_IP:7444.
   vCenter Support Assistant establishes a connection to the server.
2. (Optional) Click Show Details to view the full certificate details and click OK.
3. Click Next.

Yes, you can use the server name, BUT you must use the FQDN! And it is not just that I afraid! The service address is CASE SENSITIVE! I.e. if your vCenter Server is using self-signed certificate and during the VM deployment computer name was typed in capital letters, then the certificate would be issued to something like SERVER.domain.com.

So, I typed the vCenter Server name in the “correct” way and Support Assistant VA accepted it and allowed me to proceed to the next step:

I hope this will help.

When using third-party management products for vCenter Server, these plug-ins appear in the Available Plug-ins list. Some of these products may not have an uninstaller to remove the entry under Available Plug-ins or have been deleted/uninstalled before the plug-in was unregistered from vCenter.

For example, we ran a PoC for vFabric Data Director in the test environment some time ago but somehow managed to delete the vFDD vApp before unconfiguring and uninstalling it properly.  It would not be an issue but it appeared that ALL VMs in that test environment fell under the vFDD radar management (My guess is that the whole cluster was selected during System Resource Bundle configuration…).

VMware KB article 1025360 “Removing unwanted plug-ins from vCenter Server” describes this procedure in details.  For the purpose of this blog post I will be removing vFabric Data Director and EMC VSI (Virtual Storage Integrator) Plug-Ins.

To remove unwanted plug-ins from the Available Plug-ins list:

1. In a web browser, navigate to http://vCenter_Server_name_or_IP/mob.
   Where vCenter_Server_name_or_IP is the name of your vCenter Server or its IP address.
2. Click Content.
   
3. Click ExtensionManager.
   
4. Select and copy the name of the plug-in you want to remove from the list of values under Properties. For a list of default plug-ins, see the Additional Information section of VMware KB article 1025360 “Removing unwanted plug-ins from vCenter Server”.
   • vFabric Data Director Plug-In: com.vmware.aurora.vcext.instance-XXXX
     The vFDD’s codename is “Project Aurora“!
   • EMC VSI (Virtual Storage Integrator) Plug-Ins:
     • com.emc.vsi.plugin
     • com.emc.EMC.VSI.VSphere4.Features.SPO.PathManagement
     • com.emc.StoragePoolManagement
     • com.emc.VSI-RP-Plugin-Net-Win

   
   WOW! Someone’s been busy!
   

   BTW, if you click on the extension link, you will get Plug-In properties. The ‘lastHeartbeatTime‘ tells you when it was used last time. The ‘server‘ shows the connection URL details – very useful for troubleshooting.
   

5. Scroll down the page and click UnregisterExtension. A new window appears.
6. Paste the name of the plug-in and click Invoke Method. This removes the plug-in.
   
7. Close the window.
8. Refresh the Managed Object Type: ManagedObjectReference:ExtensionManager window to verify that the plug-in is removed successfully.
9. Restart vCenter Web Client service.

Note: If the plug-in still appears, you may have to restart the vSphere Client.

If you have vCenter Server in linked mode, perform this procedure on the vCenter Server that is used to install the plug-in initially, then restart the vCenter Server services on the linked vCenter Server.

OK, let’s get back on the subject of removing all references to vFabric Data Director! We removed the plug-in but it did not remove the fact that vCenter still thinks that vFDD manages some VMs – the “Managed by / Solution: com.vmware.aurora.vcext.instance-544f” is still there:

There are two ways how you can amend this:

1. Change the ‘MANAGED_BY_EXT_KEY‘ and ‘MANAGED_BY_TYPE‘ columns for all ‘affected’ VMs in the dbo.VPX_VM table of the vCenter database to ‘NULL‘:
   Run this query to list all VMs managed by vFDD:

   SELECT [DNS_NAME],[MANAGED_BY_EXT_KEY],[MANAGED_BY_TYPE]
   FROM [vcenter].[dbo].[VPX_VM]
   WHERE MANAGED_BY_EXT_KEY = 'com.vmware.aurora.vcext.instance-544f'

   DNS_NAME    MANAGED_BY_EXT_KEY                       MANAGED_BY_TYPE
   TST001      com.vmware.aurora.vcext.instance-544f    dbvm
   SQL001      com.vmware.aurora.vcext.instance-544f    dbvm
   NULL        com.vmware.aurora.vcext.instance-544f    dbvm

2. Power VM off, remove it from vCenter Inventory and add it back again.

I hope you will find this useful.

In April, VCE unveiled the VCE Certification Program, a brand-new technology curriculum to train IT professionals in designing, deploying and managing converged infrastructures.

On the 1st of December 2014 VCE advanced the certification program even further, focusing on IT pros who manage the operations of their organization’s converged infrastructure.  This new “Manage Track” includes three certifications:

1. VCE Certified Converged Infrastructure Associate (VCE-CIA) – Confirms basic skills and knowledge associated with Vblock Systems
   This certification is open and available to all professionals who would like to validate their understanding of what differentiates VCE converged infrastructure environment. Get to know the basics of VCE Vblock® Systems components and solutions, and the underlying investor technologies used to create VCE Vblock Systems.

   1. Exam: Included in the VCE Vblock Systems Foundations eLearning course. Course registration is required in order to access the exam. However, completing the eLearning course is not a prerequisite. A candidate can by-pass the course and go directly to the exam.
   Exam available now

2. VCE Certified Converged Infrastructure Administration Engineer (VCE-CIAE) – Validates day-to-day skills in operating Vblock Systems for administration
   This level is open and available to administration and management professionals who would like to validate their understanding of the important knowledge, skills, and abilities necessary to administer and manage VCE Vblock Systems on a day-to-day basis. Knowledge and skills include concepts, such as utilizing service profiles and service catalogues to more efficiently deploy resources to meet SLAs, manage pooled resources to effectively meet capacity and performance requirements, and use system management software to validate configurations and perform common management tasks.The ideal candidate has earned the VCE-CIA credential and has cross-domain product experience that spans servers, networking, virtualization, and storage.Certification Requirement(s)
   Prerequisites

   a. VCE Certified Converged Infrastructure Associate (VCE-CIA)

   1. Exam

   a. VCE Vblock Systems Administration Exam (220-010)
   Exam available now

3. VCE Certified Converged Infrastructure Master Administration Engineer (VCE-CIMAE) – Certifies highly advanced knowledge and skills in administering Vblock Systems
   This level is open and available to administration and management professionals who have broad, multidisciplinary skills needed to expertly perform administration and management of Vblock Systems. Candidates have demonstrated significant competencies in areas related to Vblock Systems management and can substantiate the expert skills needed to manage Vblock Systems at a master level with best practices and proven methodology to guarantee optimal performance.The ideal candidate has earned the VCE-CIAE credential and has demonstrated cross-domain product integration and customization experience that spans all areas: compute, storage, virtualization, and network.Certification Requirement(s)
   Prerequisites

   a. VCE Certified Converged Infrastructure Associate (VCE-CIA)

   b. VCE Certified Converged Infrastructure Administration Engineer (VCE-CIAE)

   1. Exam

   a. VCE Vblock Systems Master Administration Exam (320-010) Available 2015

   Exam expected in early 2015

Use discount code VCEAM2014 to get 50% off VCE Vblock Systems Administration Exam, 220-010.
The code is valid between 01/12/2014 – 28/02/2015.

Good luck!

Today I am wearing my “Keen Upgrader” hat!

I have been upgrading my lab environment and got stuck with VMware vCenter Operations Manager upgrade.  The procedure is quite straight forward and is well documented.

The procedure consists of two steps (see here):

1. Upgrade vCenter Operations Manager vApp;
2. Upgrade the OS to SUSE Linux Enterprise Server 11 to SPx.

I tried to upgrade vCenter Operations Manager from v. 5.8.3 to 5.8.4.  Yes…, I know,… Hence the “Keen Upgrader” badge… :)

The first step of the upgrade went well, no issues whatsoever but when I tried to upgrade the OS, it stopped straight after the launch without any errors or messages:

firstvm-external:/data # /usr/lib/vmware-vcops/user/conf/upgrade/va_sles11_spx_init.sh /data/VMware-vcops-SP3-2191616.pak
Started applying upgrade..
firstvm-external:/data #

That was odd!  Upon close inspection of the scripts, the issues has been identified!  vCenter Operations Manager v. 5.8.3 comes with SUSE Linux Enterprise Server 11 Patch 3 already installed and the OS upgrade is not required!

You can actually check current version of the OS by running the following command:

firstvm-external:~ # cat /etc/SuSE-release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 3

Here is what actually happened during the upgrade:

1. When you apply vCOPs vApp patch (as per upgrade procedure) it changes scripts:
   Old ones:

   rwxrwxrwx  1 root  root  1729 Nov 27  2013 va_sles11_sp2_common.sh
   -rwxrwxrwx  1 root  root  1091 Nov 27  2013 va_sles11_sp2_common_defs.sh
   -rwxrwxrwx  1 root  root  4174 Nov 27  2013 va_sles11_sp2_init.sh
   -rwxrwxrwx  1 root  root  1001 Nov 27  2013 va_sles11_sp2_postupgrade.sh
   -rwxrwxrwx  1 root  root  3504 Nov 27  2013 va_sles11_sp2_upgrade.sh
   -rwxrwxrwx  1 root  root   247 Nov 27  2013 va_sles11_sp2_upgrade_required.sh

   Replaced with:

   -rwxrwxrwx  1 admin admin 1956 Nov 13 14:02 va_sles11_spx_common.sh
   -rwx------  1 root  root  1947 Nov 13 13:51 va_sles11_spx_common.sh.old
   -rwxrwxrwx  1 admin admin  991 Oct 10 06:12 va_sles11_spx_common_defs.sh
   -rwxrwxrwx  1 admin admin 4337 Oct 10 06:12 va_sles11_spx_init.sh
   -rwxrwxrwx  1 admin admin 1425 Oct 10 06:12 va_sles11_spx_postupgrade.sh
   -rwxrwxrwx  1 admin admin 3695 Oct 10 06:12 va_sles11_spx_upgrade.sh
   -rwxrwxrwx  1 admin admin  247 Oct 10 06:12 va_sles11_spx_upgrade_required.sh

2. For the OS upgrade you launch: ‘./va_sles11_spx_init.sh /data/VMware-vcops-SP3-2191616.pak‘
   Then ‘va_sles11_spx_init.sh' calls ‘va_sles11_spx_common.sh':

   #Checks the SLES version to see if the upgrade is needed
   checkSLESversionAsString()
   {
       OS_MAJOR_VERSION=`awk '/^VERSION = /{print $NF}' /etc/SuSE-release`
       OS_SP_VERSION=`awk '/^PATCHLEVEL = /{print $NF}' /etc/SuSE-release`
       if expr match "$OS_MAJOR_VERSION" "11" > /dev/null; then
           if ( expr match "$OS_SP_VERSION" "1" || expr match "$OS_SP_VERSION" "2" ) > /dev/null; then
               return $SPx_UPGRADE_REQ
           else
               return $SPx_UPGRADE_NOT_REQ
           fi
       else
           return $SPx_UPGRADE_NOT_REQ
       fi
   }

3. And there is a procedure:

   log_and_echo()
   {
           log_info $1
   }

   in the old script the procedure is slightly different:

   log_and_echo()
   {
           log_info $1
           echo $1
   }

   The “echo $1” is missing!!!

4. When you add it back and run upgrade again you get:

   firstvm-external:/usr/lib/vmware-vcops/user/conf/upgrade # ./va_sles11_spx_init.sh /data/VMware-vcops-SP3-2191616.pak
   Started applying upgrade..
   SLES upgrade not required on Analytics VM hence skipping..

I hope you will find this entertaining! :)

I would like to thank Zbigniew ‘Ziggy’ Misiak for his help with the upgrade scripts!

The following script helps you to:

1. Import Active Directory groups and their descriptions from a CSV file;
2. Create Active Directory Groups and add Description
3. If the Group is flagged (Yes or No), it will be added to another group (e.g. User groups –> Application provisioning group)

Excel view:

CSV file:

AD Group Name,AD Group Description,Add to Group
ROLE-G-APPLICATION-Users,Application Users,NO
ROLE-G-APPLICATION-PowerUsers,Application Power Users,YES
ROLE-G-APPLICATION-Administrators,Application Administrators,YES

The script:

# Prompts you for Domain Admin like credentials
$LoginPassword = Get-Credential

# Imports groups and description list from CSV file
$groups = Import-CSV "D:\groups.csv"    

foreach ($item in $groups) {
# Map variables from CSV
$group = $item.'AD Group Name'
$description = $item.'AD Group Description'
# Creates Active Directory Group
New-ADGroup –name $group –groupscope Global -Description $description –path “OU=Application,OU=Groups,DC=internal,DC=vstrong,DC=info” -Credential $LoginPassword
# Adds newly created group to existing group
if ($item.'Add to Group' -eq "YES")
{Add-ADGroupMember APP-APPLICATION-Publish -Members $group -Credential $LoginPassword}
}

I hope you will find this useful.

In my previous “PowerShell: Create new Active Directory groups, change group membership” blog post I published a PowerShell script that automates create Active Directory group creation.  Today I needed to create multiple test user accounts and, and there were 12 account created already, the TestUser account number should start from 13.
All New-ADUser cmdlet parameters can be found <here>.

Let’s look into different ways user account password can be configured. Mode details on -AccountPassword:

AccountPassword

Specifies a new password value for an account. This value is stored as an encrypted string.

The following conditions apply based on the manner in which the password parameter is used:

$null password is specified – No password is set and the account is disabled unless it is requested to be enabled
No password is specified – No password is set and the account is disabled unless it is requested to be enabled
User password is specified – Password is set and the account is disabled unless it is requested to be enabled
Notes:
User accounts, by default, are created without a password. If you provide a password, an attempt will be made to set that password however, this can fail due to password policy restrictions. The user account will still be created and you may use Set-ADAccountPassword to set the password on that account. In order to ensure that accounts remain secure, user accounts will never be enabled unless a valid password is set or PasswordNotRequired is set to true.
The account is created if the password fails for any reason.

The following example shows one method to set this parameter. This command will prompt you to enter the password.
-AccountPassword (Read-Host -AsSecureString "AccountPassword")

I need all test users passwords to be the same. Before a chosen password can be used with -AccountPassword, you need to convert plain text password into Secure String:

$password = "Passw0rd"
$SecurePassword = ConvertTo-SecureString -String $password -AsPlainText -Force

…and, of course, I would like to have them enabled:

-Enabled $true

The script:

# If you need to use another account that has permissions to create AD users
# $LoginPassword = Get-Credential - prompts you for credentials.
# -Credential $LoginPassword - add this to New-ADUser and Add-ADGroupMember

$password = "Passw0rd"
$SecurePassword = ConvertTo-SecureString -String $password -AsPlainText -Force

$prefix = "TestUser"
foreach ($number in (13..23)) {
$user = $prefix+$number
	New-ADUser -SamAccountName $user -Name $user -EmailAddress "$user@test.vStrong.info" -AccountPassword $SecurePassword -Path “OU=TestUsers,DC=test,DC=vStrong,DC=info” -HomeDrive "H:" -HomeDirectory "\\file001\home$\$user" -Enabled $true
 	Add-ADGroupMember TestUsersGroup -Members $user # OPTIONAL - Add user to a group
} 

I hope you will find this useful.

I recently upgraded VMware environment from vSphere 5.0 to vSphere 5.5 and thought I should document the host upgrade process.

For detailed instructions please refer to the “Upgrading and Migrating Your Hosts” topic of the vSphere Upgrade Guide.

1. Connect the vSphere Client to a vCenter Server with which Update Manager is registered;
2. Open Update Manager, Admin View;
3. On the ESXi Image tab click on Import ESXi Image:
   
4. Browse and Select the ISO image that you want to upload.
   This can either be:
   • a base VMware vSphere ESXi image that you can download from my.VMware.com (please select correct version);
   • an ISO image that was customised by the hardware vendor. Customised images include vendor specific NIC, HBA and other device drivers, management utilities etc:
     • HP customised image for VMware vSphere ESXi
     • Cisco customised image for VMware vSphere ESXi
     • Hitachi customised image for VMware vSphere ESXi
     • Fujitsu customised image for VMware vSphere ESXi
     • NEC customised image for VMware vSphere ESXi
     • IBM customised image for VMware vSphere ESXi
     • DELL customised image for VMware vSphere ESXi 5.5 & ESXi 5.5 Update 2
     • VCE Vblock customised image for VMware vSphere ESXi (Log in to support.vce.com, open VCE Download Center / Select Vblock type / Select RCM (Release Certification Matrix) – Software Download for Release Certification Matrix / Select VMware vSphere).
   • Your own customised ISO image. See “HOW TO: Create custom vSphere ESXi ISO with VMware vSphere ESXi Image Builder” for details.

     For the purpose of this guide I will be using VCE Vblock customised image from RCM 4.8.0.

5. Import the ISO image
   
6. Upload successful
   
7. Create ESXi host upgrade baseline
   
8. New baseline:
   
9. Select Datacenter / Cluster or individual host and attach the baseline:
   
10. Select baseline and click Attach.
    
11. In the Hosts and Cluster view right-click on the Cluster you attache the baseline to, select Scan for Updates, select Upgrades and click Scan.
    
12. After Scan is completed, Update Manager will display the Host Compliance status:
    
13. In the majority of cases the status will be Non-Compliant.  The reason these two hosts are Incompatible is that there are some outstanding software configuration changes that require a reboot.
    
14. Reboot the host and re-Scan it for Upgrades:
    
15. To upgrade a host, put it in Maintenance Mode, click on Remediate…, select Upgrade Baseline and click Next;
    
16. Accept the License Agreement, click Next;
    
17. The ISO image that I am using for host upgrade includes all third-party software and drivers required for the new version of the ESXi (Thank you VCE! :) ) and therefore I will allow the removal of the incompatible software and drivers.
    
18. Schedule the upgrade or run it immediately:
    
19. Change Maintenance Mode options or accept defaults. Click Next;
    
20. Change Cluster Remediation options or accept defaults. Click Next;
    
21. Confirm the upgrade options and click Finish.
    
22. In the Recent Tasks pane, the remediation task is displayed and will remain at about 22 percent for most of the process. The process is still running and will take approximately 15 minutes to complete;
23. After the upgrade the Host compliance status will change to Compliant.

EMC UIM/P Shared Storage Service is the new type of service that allows you to create and share storage between multiple Standard Services.
For more in-depth information on the Shared Storage Service please refer to “EMC UIM/P (Nimbus) review” and “UIM/P 4.0 Release” posts.

ATTENTION! The initial release of UIM/P 4.0 had a serious bug! When performing a full decommission on a shared storage service, VMs and datastores were deleted from any attached standard services. This bug has been fixed in UIM/P 4.0 Patch 2. Make sure to upgrade UIM/P to the latest version.

In my environment I have two SSS (Shared Storage Services):

• Shared LUN to facilitate live VM migration (between pre vSphere 5.1 clusters)
• Shared Templates LUN

These shared datastores are attached to two clusters:

I would like to remove both datastores from PDC2VEXC05 cluster:

1. Make sure the Storage I/O Control is disabled on these datastores:
   
   BTW, this is required for decomissioning of datastores attached to the Standard services as well.
2. Click Decommission…
   Let’s just stop here for a second and discuss the options:
   • Detach will remove the Datastore from Standard Service(s). The SS’ hosts will not be able to see it. The LUN will be removed from the hosts’s Storage Group. This process does not delete the data and, if the datastore is Attached to any other Standard Services, it will still be usable.
   • Decommission will remove the shared datastore from Standard Services and delete the LUN.
3. I would like to keep the shared datastores attached to the second Standard Service, therefore I will select Decommission/Detach Specific Service Components and select the Standard Service I want to detach the shared datastores from:
   
   Click Decommission.
4. This is where it gets interesting!  The following message appears:
   
   It is not clear if UIM/P is going to Detach or Decomission the datastore! The message is the same for both actions… You may need to go back and double check what option you selected!
5. OK, everything is checked and we type Confirm…
6. As there are several VMs and Templates on this datastore (obviously, as it is still in use by other clusters) and ever though we are not trying to Decommission the datastore, we get another misleading warning message:
   
7. Although this does not give us any comfort, we type Override and pray go to the Service History and observe the configuration changes:
   
   As you can see, UIM did what we ask it to do… Both datastores have been removed from the cluster hosts.  Neither the Shared nor the Templates datastore were deleted.

Although this process worked OK, I can see that the lack of clarity of the warning messages and dialog windows may discourage you from using UIM/P to manage Shared Storage.  I hope this will be addressed in the future releases.

As an alternative, you can use UIM/P Service Adoption Utility to manage Shared Storage Services…

Hope you will find this useful.

The WordPress.com stats helper monkeys have been busy putting together a personalised report detailing how www.vStrong.info did in 2014!

The Louvre Museum has 8.5 million visitors per year. This blog was viewed about 136,000 times in 2014. If it were an exhibit at the Louvre Museum, it would take about 6 days for that many people to see it.

The busiest day of the year was December 30th with 798 views. The most popular post that day was Default Passwords.  I hope that saved you a lot of time! :)

Click here to view the full report.

Thank you for your continues support!