Azure App Services and robots933456.txt
What is this robots933456.txt I see in my logs all the time? Let me tell you.
When you use Azure App Services you might see lines in your logging like:
2019-04-08T14:07:56.641002476Z "-" - - [08/Apr/2019:14:07:56 +0000] "GET /robots933456.txt HTTP/1.1" 404 415 "-" "-"
At first glance you might think this is some search engine bot scraping your site for robot files, but if you look at the details of the request:
{
"TraceId": "xxxxx:00000001",
"HttpMethod": "GET",
"RequestUrl": "http://x.x.x.x:8080/robots933456.txt",
"IpAddress": "x.x.x.x",
"ResponseStatus": "Not Found",
"ResponseStatusCode": 404,
"RequestBody": {},
"Headers": {
"Host": "x.x.x.x:8080",
"User-Agent": "HealthCheck/1.0"
}
}
You can see the User-Agent: "Healthcheck/1.0" header which indicates this might be some Azure thing.
On the azure-docs repo you can find some additional clarification:
robots933456 in logs
You may see the following message in the container logs:
2019-04-08T14:07:56.641002476Z "-" - - [08/Apr/2019:14:07:56 +0000] "GET /robots933456.txt HTTP/1.1" 404 415 "-" "-"
You can safely ignore this message. /robots933456.txt is a dummy URL path that App Service uses to check if the container is capable of serving requests. A 404 response simply indicates that the path doesn't exist, but it lets App Service know that the container is healthy and ready to respond to requests.
So now you can update all your existing logging to exclude this as it will clutter up your logs for sure.