Lately I was working on production issue where the web jobs hosted on azure were automatically restarting by itself or moving in stopped state. There was no signs of user manually restarting it (you can watch those via Activity Logs on your website). So the question is why was it happening? The answer lies in the web job logs. In order to understand it better, I have laid out mostly all reasons of automatic restart with sample logs (you can get the log from either Kudu web job dashboard or directly from storage account).
Reason 1: Due to website shutdown/restart
[10/24/2016 06:53:36 > b3e7a2: SYS INFO] WebJob is stopping due to website shutting down [10/24/2016 06:53:36 > b3e7a2: SYS INFO] Status changed to Stopping [10/24/2016 06:53:39 > b3e7a2: INFO] Job host stopped [10/24/2016 06:53:41 > b3e7a2: ERR ] Thread was being aborted. [10/24/2016 06:53:42 > b3e7a2: SYS INFO] WebJob process was aborted [10/24/2016 06:53:42 > b3e7a2: SYS INFO] Status changed to Stopped [10/24/2016 06:59:17 > 521cd3: SYS INFO] Status changed to Starting [10/24/2016 06:59:20 > 521cd3: SYS INFO] Run script '<YourWebJobName>.WebJob.exe' with script host - 'WindowsScriptHost' [10/24/2016 06:59:20 > 521cd3: SYS INFO] Status changed to Running
Reason 2: Due to changes in the azure web job directory files or file content (D:\home\site\wwwroot\app_data\jobs\continuous\<WebJobName>)
[10/29/2016 00:00:43 > 521cd3: SYS INFO] Detected WebJob file/s were updated, refreshing WebJob [10/29/2016 00:00:43 > 521cd3: SYS INFO] Status changed to Stopping [10/29/2016 00:00:47 > 8d7eea: SYS INFO] Detected WebJob file/s were updated, refreshing WebJob [10/29/2016 00:00:47 > 8d7eea: SYS INFO] Status changed to Stopping [10/29/2016 00:00:48 > 521cd3: INFO] Job host stopped [10/29/2016 00:00:49 > 521cd3: ERR ] Thread was being aborted. [10/29/2016 00:00:51 > 521cd3: SYS INFO] Status changed to Stopped [10/29/2016 00:00:51 > 521cd3: SYS INFO] Status changed to Starting [10/29/2016 00:00:52 > 521cd3: SYS INFO] WebJob process was aborted [10/29/2016 00:00:52 > 521cd3: SYS INFO] Status changed to Stopped [10/29/2016 00:00:52 > 521cd3: SYS INFO] Job directory change detected: Job file 'ApplicationInsights.config' timestamp differs between source and working directories. [10/29/2016 00:00:51 > 8d7eea: INFO] Job host stopped [10/29/2016 00:00:52 > 8d7eea: ERR ] Thread was being aborted. [10/29/2016 00:00:52 > 8d7eea: SYS INFO] WebJob process was aborted [10/29/2016 00:00:52 > 8d7eea: SYS INFO] Status changed to Stopped [10/29/2016 00:00:53 > 8d7eea: SYS INFO] Status changed to Starting [10/29/2016 00:00:54 > 8d7eea: SYS INFO] Job directory change detected: Job file 'ApplicationInsights.config' timestamp differs between source and working directories.[10/29/2016 00:01:01 > 521cd3: SYS INFO] Run script '<YourWebJobName>.WebJob.exe' with script host - 'WindowsScriptHost' [10/29/2016 00:01:01 > 521cd3: SYS INFO] Status changed to Running
Reason 3: Due to a web app and/or web job deployment
Easy to understand since it will trigger Scenario 2
Reason 4: Due to an azure outage or maintenance
Assuming you can rule out reason 1, 3 and 4 (straightforward too) by using standard azure web app monitoring tools (like Failure History and others), 2 would still require further analysis.
Basically Reason 2 indicates that if there is a change in the web job directory i.e. a new file/folder is added or removed automatically or manually, web jobs will restart. Or if the content of the file within the directory is changes, that would also initiate trigger in web job restart.
In that case need to dig further what triggered the directory and/or file content changes within web job directory. You need to ask below question:
- Was there any changes to web app settings via azure portal?
- Was there any runtime SDK upgrades? Like upgrading App Insights or installing an extension
- Are you creating any temporary files at runtime with the web job directory?
You might want to review the application design if 3 is correct.
This helped me fixing my production issue. Hope same for you.