Few days back I ran into a problem where our production azure web apps were throwing below error:
[SocketException (0x271d): An attempt was made to access a socket in a way forbidden by its access permissions x.x.x.x:80] System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) +208 System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception) +464
We opened a case with Microsoft and upon investigation they told us that your App Service Plan (running on Standard S1 2 Instances) are hitting the outbound connection limit. What? How the heck we know that? As of when i am writing, below were the connection limits given my MS.
App Service Plan | Connection Limit |
Free F1 | 250 |
Shared D1 | 250 |
Basic B1 1 Instance | 1920 |
Basic B2 1 Instance | 3968 |
Basic B3 1 Instance | 8064 |
Standard S1 1 Instance | 1920 |
Standard S1 2 Instances | 1920 per instance |
Standard S2 1 Instance | 3968 |
Standard S3 1 Instance | 8064 |
Premium P1 1 Instance (Preview) | 1920 |
On further request, MS gave us a table of apps under the app service place and their open socket connection count. It clearly indicates that Web App 1 worker process is not reusing the connection pool and creating new connections hitting the overall limit of the app service plan.
WebApp Name | Process Name | Open Socket Count |
App2 | <WebJob>.WebJob.exe | 4 |
App2 | <WebJob>.WebJob.exe | 4 |
App2 | w3wp.exe | 2 |
App1 | <WebJob>.WebJob.exe | 8 |
App1 | <WebJob>.WebJob.exe | 4 |
App1 | <WebJob>.WebJob.exe | 6 |
App1 | w3wp.exe | 2 |
App1 | w3wp.exe | 1870 |
App1 | <WebJob>.WebJob.exe | 6 |
App1 | w3wp.exe | 2 |
App1 | <WebJob>.WebJob.exe | 6 |
App3 | w3wp.exe | 4 |
App3 | w3wp.exe | 2 |
Total | 1920 |
With the above data from MS at least you would be able to know where the problem lies and can review the app again.
For your web apps, you can at least review the code (ensuring it doesn't happen to your azure web apps) where you are handing the connection with external entities. Some of the common external dependencies in modern cloud world are:
- SQL - https://azure.microsoft.com/en-us/documentation/articles/sql-database-develop-dotnet-simple/
- Redis - https://azure.microsoft.com/en-us/documentation/articles/cache-dotnet-how-to-use-azure-redis-cache/
- Service Bus - https://azure.microsoft.com/en-us/documentation/articles/service-bus-performance-improvements/
Thanks to the blog http://www.freekpaans.nl/2015/08/starving-outgoing-connections-on-windows-azure-web-sites/ which explains about the same problem.
However the fact is with no monitoring tool available which monitors the open socket count, you will never be able to know the number of open socket connections for your app service plan unless requested from Microsoft.