HTTP 502: Handling backend health in Azure Application Gateway

I’ve been working with a client recently on a secure web platform design and implementation using a bunch of cool Azure tech, including Azure Service Fabric and App Service Isolated (ASEv2). This design involves all the infrastructure bits being deployed to a regional VNET, therefore effectively air-gapped from inbound internet traffic.

Among the requirements was that external customers should be able to access web apps on the secure platform, and both internal and external 3rd party developers should be able to work with web apps hosted within App Service Isolated environment. To provide this access, we decided to use Application Gateway. However, during implementation, we ran into a few issues.

Publishing the public facing web app was relatively easy. Ahmed Sabbour wrote a great post on How to run an App Service behind a WAF-enabled Application Gateway. However, the problems came when attempting to publish the Kudu, ‘Advanced Tools’s site. WebDeploy, the component used for deploying code to an Azure Web App is a component for the Kudu site, (i.e. *.scm.yourdomain.com). Therefore, we needed additional listeners, HTTP Settings and rules to present these sites via Application Gateway. All good I thought. However, having done this, when browsing to the site we received a 502 error, stating that something went wrong whilst trying to proxy traffic to the backend (i.e. our Kudu site running on our web app.) I also checked out the backend health in the Azure Portal, which was showing ‘Unhealthy’.

The issue here was to do with the fact that the Kudu site requires authenticated. Now this actually differs depending on whether you are using App Service (Free/Basic/Standard/Premium as hosted by Microsoft) or whether you are using App Service Environment or App Service Isolated (both dedicated offerings). App Service is natively integrated with your Azure AD tenant for authentication. When you try and access an App Service Kudu site, you are redirected to https://login.microsoftonline.com and asked to log in with your AAD credentials. This redirection would ordinarily result in some sort of 300 HTTP status code. For App Service Environment or App Service Isolated, these are not integrated with Azure AD, and instead use basic authentication – you are merely presented with a browser pop-up to enter a username and password, which produces a 401 HTTP status code. Now, herein lies the problem…

Application Gateway health probes are configured by default to probe the backend resource and expect a HTTP status code of something between a 200 (a normal, ‘OK’ response) and 399 (some sort of redirect) HTTP response. Since the Kudu site uses basic authentication, and responds with a 401, the Application Gateway believes the backend is not working correctly, and prevents traffic from going to the backend. You can read more about this in the Application Gateway health monitoring overview

After searching for alternatives to make this all work with Application Gateway, I eventually opened a ticket with Microsoft Premier Support. After a couple of weeks of back and forth trying things out a solution was found! It seems that a recent Azure PowerShell update has exposed a new configuration option in the Application Gateway provider. This functionality allows a Match array to be added to the health probe, so we can now customise what constitutes a healthy HTTP status code. Hurrah!

So, to get started with this new functionality the first thing you will need to do is update your Azure PowerShell module. The best place to get these updates is on GitHub. If you look down the release notes you can see that the functionality we need was added in v4.2.0 of the module, but I recommend updating to the latest anyway.TTP response

As you can see from the portal, I have already configured a listener, with certificate, a health probe (with an authentication certificate), Some HTTP settings (using the custom health probe) and a rule to pull the whole lot together:

You can also see that with this configuration the backend health of the published Kudu site is Unhealthy

The backend health status is being flagged as unhealthy due to the HTTP 401 response from the Kudu web app. Now, to rectify this.

Open a PowerShelll session, and login to Azure using Login-AzureRmAccount. Then switch to the subscription your Application Gateway is located in using Set-AzureRmContext. Next, we need to get the Application Gateway running config using the following cmdlet:

$agw = Get-AzureRmApplicationGateway -Name <appGatewayName> -ResourceGroupName <resourceGroupName>

 

If we just output the contents of the $agw object to the console and have a look for the ProbesText element of the configuration, we will be able to see the following:

See, the Match array just there? Now, to set the content of the Match array. First, we need to set the status codes that we want to accept. This can be a range, and is done using the following cmdlet:

$match = New-AzureRmApplicationGatewayProbeHealthResponseMatch -StatusCode "200-401"

Then we want to get the configuration of the current health probe that we are going to modify. We’ll use this to set the correct configuration on the health probe:

$probeConfig = Get-AzureRmApplicationGatewayProbeConfig -Name <healthProbeName> -ApplicationGateway $agw


Next, we need to update the probe config stings. We’ll use most of the settings from those gathered by the previous cmdlet, plus we’ll include the Match object:

Set-AzureRmApplicationGatewayProbeConfig -ApplicationGateway $agw -Name $probeConfig.Name -Potocol $probeConfig.Protocol -HostName $probeConfig.Host -Path $probeConfig.Path -Interval $probeConfig.Interval -Timeout $probeConfig.Timeout -UnhealthyThreshold $probeConfig.UnhealthyThreshold -MinServers $probeConfig.MinServers -Match $match

If we output the content of the $agw object again, you’ll now see that the Match array is populated with our new, and expanded range of accepted response codes:

Finally, we need to update the running config of the Application Gateway by using the following cmdlet:

Set-AzureRmApplicationGateway -ApplicationGateway $agw

If we check in the portal now, we will see that the Application Gateway is being updated:

Once this is completed, we’ll see that the backend health is now in a Healthy for this probe:

You should now be able to access the Kudu site using the username and password from your web app’s publishing profile, and your developers should be able to deploy their code from anywhere. (or wherever your Network Security Group permits, as in my case )

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

search previous next tag category expand menu location phone mail time cart zoom edit close