Lately, it has been brought to my attention that on some particular WVD deployments, the GPU spiked to a 100% resulting in a very bad user experience. In these particular cases, we are using AMD GPU drivers on NVv4-series machines.
The drivers for these AMD GPUs are being installed via the AMD GPU driver extension during WVD host deployment.
One strange factor was that not all systems did experience high GPU usage.
The following checklist was performed to pinpoint the difference in behavior.
- Are all hosts based upon the same image? => Yes
- Are all AMD GPU extensions based upon the same version? => Yes
- Is there any other software interfering with the hosts which are not included in the image? So MSIX or other extensions. => No
- Are all AMD GPU Drivers at the same version? => No
Our 4th check led us to some more in-depth troubleshooting. Why is there a version difference at the AMD GPU Driver level if all our AMD GPU VM extensions are at the same level?
Looking at the different hosts, we noticed that some hosts had other deployment times or received a VM update somewhere along the road.
When looking at the scripts used by the AMD GPU extension, when applying for the extension, it will automatically download and install the latest version of the drivers. Hence the difference in driver versions.
The “update” script
In the scripts folder (C:\Packages\Plugins\Microsoft.HpcCompute.AmdGpuDriverWindows\1.0.0.0\scripts) you can find all of the scripts used by the extension.
Curious thing is that there is an “update” script, however, it seems that some parts are still missing. Looks like the driver update part will be something for the next version of the extension :).
The “enable” script
When taking a closer look at the “enable” script, this is the script that is run during the deployment of the AMD GPU extension, the “enable” script takes care of the installation. There is a check mechanism included to see if a driver is already installed.
The “Drive Update” script
To keep current with your AMD GPU driver version control the following paragraph will describe how you can achieve this by taking the above scripts into account, and provide a best of both worlds joint solution to ease up the versioning of your AMD GPU drivers.
What I’ve done, I took the update.ps1 as the base and added the install parts from the enable.ps1 script.
In order to run the script, the driverupdate.ps1 file needs to be located in the scripts folder of the extension.
Below you can find an outline of the script. Here is a link to my GitHub Repo. Place the script in the scripts folder of the extension before running it.
$ErrorActionPreference = 'Stop'
if (-not $PSScriptRoot)
{
$PSScriptRoot = Split-Path $MyInvocation.InvocationName
}
if ( -not $env:Path.Contains( "$PSScriptRoot;") )
{
$env:Path = "$PSScriptRoot;$env:Path"
}
$Status = New-PackageStatus -Message 'Update Extension' -Status 'transitioning' -Name 'AmdGpuDriverWindows' -Operation 'Update'
# Read the environment, config settings and get status target
$Env = Get-Environment
$StatusFile = $Env.StatusFile
# Update AMD Driver
try
{
$FatalError = @()
$Status = New-PackageStatus -Message 'Enable Extension' -Status 'transitioning' -Name 'AmdGpuDriverWindows' -Operation 'Enable'
# Read the environment, config settings and get status target
$Env = Get-Environment
$StatusFile = $Env.StatusFile
# Check if the driver is installed
$Driver = Set-DriverInfo
$DriverVersion = Check-Driver $Driver
if ( $Null -ne $DriverVersion )
{
$Message = "AMD GPU driver version $DriverVersion detected. Already installed. Trying to upgrade"
$Status | Add-PackageSubStatus -Message $Message -Status 'success' | Out-Null
#region new driver download and update
# Select the driver to install
if ($Env.DriverUrl -eq $null)
{
Get-DriverDownloadInfo $Driver
}
else
{
$Driver.Url = $Env.DriverUrl
}
# Download Driver archive
$source = $Driver.Url
$Driver.ArchiveFile = $Driver.Url.Substring($Driver.Url.LastIndexOf("/") + 1)
$dest = "$($Driver.InstallFolder)\$($Driver.ArchiveFile)"
. download.ps1
Get-DriverFile $source $dest
#install the driver from the downloaded package
$argumentList = "/S /D=" + $Driver.ExpandFolder
$p = Start-Process -FilePath $dest -ArgumentList $argumentList -PassThru -Wait
# check if installation process was successful
if ($p.ExitCode -eq 0)
{
$Status | Add-PackageSubStatus -Message "Installation process complete." -Status 'success' -Code $p.ExitCode | Out-Null
$Status | Set-PackageStatus -Status 'success'
Write-Host "Installation process complete. Code: $($p.ExitCode)"
}
else
{
$Status | Add-PackageSubStatus -Message "Installation of $dest failed!" -Status 'error' -Code $p.ExitCode | Out-Null
$Status | Set-PackageStatus -Status 'error'
Write-Host "Installation of the driver package failed! Code: $($p.ExitCode)"
}
#endregion
Write-Host $Message
$Status | Set-PackageStatus -Status 'success'
}
else
{
$Status | Add-PackageSubStatus -Message "AMD GPU driver not detected. Attempting to install." -Status 'success' | Out-Null
# Select the driver to install
if ($Env.DriverUrl -eq $null)
{
Get-DriverDownloadInfo $Driver
}
else
{
$Driver.Url = $Env.DriverUrl
}
# Download Driver archive
$source = $Driver.Url
$Driver.ArchiveFile = $Driver.Url.Substring($Driver.Url.LastIndexOf("/") + 1)
$dest = "$($Driver.InstallFolder)\$($Driver.ArchiveFile)"
. download.ps1
Get-DriverFile $source $dest
#install the driver from the downloaded package
$argumentList = "/S /D=" + $Driver.ExpandFolder
$p = Start-Process -FilePath $dest -ArgumentList $argumentList -PassThru -Wait
# check if installation process was successful
if ($p.ExitCode -eq 0)
{
$Status | Add-PackageSubStatus -Message "Installation process complete." -Status 'success' -Code $p.ExitCode | Out-Null
$Status | Set-PackageStatus -Status 'success'
Write-Host "Installation process complete. Code: $($p.ExitCode)"
}
else
{
$Status | Add-PackageSubStatus -Message "Installation of $dest failed!" -Status 'error' -Code $p.ExitCode | Out-Null
$Status | Set-PackageStatus -Status 'error'
Write-Host "Installation of the driver package failed! Code: $($p.ExitCode)"
}
}
}
catch [NotSupportedException]
{
$Message = $_.ToString()
Write-Host $Message
$Status | Add-PackageSubStatus -Message $Message -Status 'error' -Code 100 | Out-Null
$Status | Set-PackageStatus -Status 'error'
$Status | Save-PackageStatus $StatusFile
exit 0
}
catch
{
$FatalError += $_
Write-Host -ForegroundColor Red ($_ | Out-String)
exit $FatalError.Count
}
finally
{
$Status | Save-PackageStatus $StatusFile
}
Solutions
The update management process of the AMD GPU drivers has a couple of possible solutions.
- Use the script provided “DriverUpdate.ps1” => No reboot required
- Remove the VM extension and re-add it to the VM => reboot required
- Microsoft to release a new version of the extension which includes a built-in update mechanism
Food for thought
During troubleshooting, there were a couple of things that crossed my mind.
- The current AMD GPU driver is download as a self-extracting Exe file, with close to none “exe. /” switches, which leads us to install/uninstall every time we want to extract the package (unless you are using 3rd party products like 7zip) which makes it difficult to compare driver versions at that time in the update process.
- Alter the GPU AMD Azure extension based on driver version (changing the extension version number based on driver updates)
- Don’t use the GPU AMD Azure extension and perform the installation & updates via your software packaging and management tool
Warning!
The scripts are provided as-is, please be very careful and test run the scripts on a “test” environment or an environment that allows you to perform some quick checks and tests. Updating drivers, in general, can provide performance benefits, but there are rare cases that things just stop working for Windows Virtual Desktop users.
Thank you!
Thank you for reading through this blog post, I hope I have been able to assist in updating the AMD GPU drivers on your Windows Virtual Desktop farm.
If you encounter any new insights, feel free to drop me a comment or contact me via mail or other social media channels
2 Comments on “How to update AMD GPU drivers on N-series VMs running Windows (Windows Virtual Desktop)”