App Service Plan – Outbound Network Connection Limit

Few days back I ran into a problem where our production azure web apps were throwing below error:

[SocketException (0x271d): An attempt was made to access a socket in a way forbidden by its access permissions x.x.x.x:80] System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) +208 System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception) +464

We opened a case with Microsoft and upon investigation they told us that your App Service Plan (running on Standard S1 2 Instances) are hitting the outbound connection limit. What? How the heck we know that? As of when i am writing, below were the connection limits given my MS.

App Service Plan	Connection Limit
Free F1	250
Shared D1	250
Basic B1 1 Instance	1920
Basic B2 1 Instance	3968
Basic B3 1 Instance	8064
Standard S1 1 Instance	1920
Standard S1 2 Instances	1920 per instance
Standard S2 1 Instance	3968
Standard S3 1 Instance	8064
Premium P1 1 Instance (Preview)	1920

On further request, MS gave us a table of apps under the app service place and their open socket connection count. It clearly indicates that Web App 1 worker process is not reusing the connection pool and creating new connections hitting the overall limit of the app service plan.

WebApp Name	Process Name	Open Socket Count
App2	<WebJob>.WebJob.exe	4
App2	<WebJob>.WebJob.exe	4
App2	w3wp.exe	2
App1	<WebJob>.WebJob.exe	8
App1	<WebJob>.WebJob.exe	4
App1	<WebJob>.WebJob.exe	6
App1	w3wp.exe	2
App1	w3wp.exe	1870
App1	<WebJob>.WebJob.exe	6
App1	w3wp.exe	2
App1	<WebJob>.WebJob.exe	6
App3	w3wp.exe	4
App3	w3wp.exe	2
Total		1920

With the above data from MS at least you would be able to know where the problem lies and can review the app again.

For your web apps, you can at least review the code (ensuring it doesn't happen to your azure web apps) where you are handing the connection with external entities. Some of the common external dependencies in modern cloud world are:

1. SQL - https://azure.microsoft.com/en-us/documentation/articles/sql-database-develop-dotnet-simple/
2. Redis - https://azure.microsoft.com/en-us/documentation/articles/cache-dotnet-how-to-use-azure-redis-cache/
3. Service Bus - https://azure.microsoft.com/en-us/documentation/articles/service-bus-performance-improvements/

Thanks to the blog http://www.freekpaans.nl/2015/08/starving-outgoing-connections-on-windows-azure-web-sites/ which explains about the same problem.

However the fact is with no monitoring tool available which monitors the open socket count, you will never be able to know the number of open socket connections for your app service plan unless requested from Microsoft.