Lots of errors being printed to stderr

[AGWA]

Thanks for the explanation. Unfortunately the log file has become very unwieldy, as even in non-verbose mode it is spammed by the above messages at the rate of a few per second. Is it possible to reduce logging to just the important messages? Fwiw, I'm using the -stdout option (because Docker).

Originally posted by @schildbach in #49

@schildbach what do you consider to be an important message?

Also I wonder if it's really necessary to log the response body of the HTTP errors? Wouldn't the error code plus message not be enough?

Sometimes the body of the response contains useful information, e.g. the body of a "400 Bad Request" error might explain why the request is bad. It might be reasonable to truncate the body to 100 bytes or so.

[schildbach]

@schildbach what do you consider to be an important message?

• Spotted certificates
• Significant backlogs
• Generally, all events that you currently create emails for
• Non-transient errors, e.g. if the mail sending process fails

All those things are currently buried in the log, even in non-verbose mode.

Sometimes the body of the response contains useful information.

Maybe print the body only in -verbose mode, and/or add more log levels?

[AGWA]

Spotted certificates
Significant backlogs
Generally, all events that you currently create emails for

These are printed to stdout, allowing you to route them separately from stderr messages.

Non-transient errors, e.g. if the mail sending process fails

These are indeed printed to stderr. They are also accompanied by a non-zero exit, but I can see how that might not always be easy to detect.

As of 15e35ab, log errors will only be printed to stderr if -verbose is specified. This means that by default stderr is reserved for fatal errors only. It also means that if someone wants to debug why a health check is failing, they'll need to enable -verbose mode, but #106 will provide a better solution.

[schildbach]

Fyi Docker doesn't differentiate between stdout and stderr. Both go to stdout of the host, and/or the container's log file.

So I'm hoping for the solution proposed in #106.

[AGWA]

OK, but in the interim 15e35ab improves the situation right?

[schildbach]

Yes, I think so. Thanks.

[Sebbito]

Think this is the right place for this:
I'm running certspotter on my hetzner vm. It looks like I get errors ONLY:

$ ./certspotter -start_at_end -stdout -watchlist watchlist
2025/06/13 14:09:46 https://sabre.ct.comodo.com/: Get "https://sabre.ct.comodo.com/ct/v1/get-sth": dial tcp: lookup sabre.ct.comodo.com on 127.0.0.53:53: no such host
2025/06/13 14:09:46 https://tiger2025h2.ct.sectigo.com/: Get "https://tiger2025h2.ct.sectigo.com/ct/v1/get-sth": 429 Too Many Requests ("<html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body>\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n")
2025/06/13 14:09:46 https://elephant2025h2.ct.sectigo.com/: Get "https://elephant2025h2.ct.sectigo.com/ct/v1/get-sth": 429 Too Many Requests ("<html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body>\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n")
2025/06/13 14:09:46 https://mammoth2025h2.ct.sectigo.com/: Get "https://mammoth2025h2.ct.sectigo.com/ct/v1/get-entries?start=351207424&end=351207679": 429 Too Many Requests ("<html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body>\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n")
2025/06/13 14:09:46 https://mammoth2025h2.ct.sectigo.com/: Get "https://mammoth2025h2.ct.sectigo.com/ct/v1/get-sth": 429 Too Many Requests ("<html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body>\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n")
2025/06/13 14:09:46 https://sabre2025h2.ct.sectigo.com/: Get "https://sabre2025h2.ct.sectigo.com/ct/v1/get-sth": 429 Too Many Requests ("<html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body>\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n")
2025/06/13 14:09:47 https://mammoth2025h2.ct.sectigo.com/: Get "https://mammoth2025h2.ct.sectigo.com/ct/v1/get-entries?start=351207424&end=351207679": 429 Too Many Requests ("<html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body>\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n")
2025/06/13 14:09:47 https://sabre2025h2.ct.sectigo.com/: Get "https://sabre2025h2.ct.sectigo.com/ct/v1/get-sth": 429 Too Many Requests ("<html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body>\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n")
2025/06/13 14:09:47 https://sabre.ct.comodo.com/: Get "https://sabre.ct.comodo.com/ct/v1/get-sth": dial tcp: lookup sabre.ct.comodo.com on 127.0.0.53:53: no such host
...

Not sure what to do about it. Any hints on what could work to resolve or further troubleshoot this issue?

[AGWA]

In general you don't need to worry about this type of error, which is generally transient, unless you receive a failed health check notification. Since this has been causing confusion, I just released certspotter v0.20.0 which only prints these errors if -verbose is used.

[Sebbito]

Good to know but why shouldn't I worry about these errors.
Do i get rate limited and will it thus take longer to fetch the ct logs?
Also unrelated: there seems to be a dead (or unreachable) service (https://sabre.ct.comodo.com/ct/v1/get-sth). It seems to resolve to some localhost address that the program tries to reach repeatedly. Why does it try to query that one even though it couldn't reach it the first time?

[AGWA]

Because most of the time retrying a failed request causes it to succeed, which doesn't cause a material delay in monitoring the log. If certspotter falls behind monitoring a log, the daily health check will notify you.

sabre.ct.comodo.com was decommissioned very recently and there was a delay before your certspotter instance downloaded a new copy of the log list without this log. At this point, your certspotter instance should have an up-to-date log list and won't be trying to contact this log anymore.

[AGWA]

v0.20.0 will only print these errors to stderr if -verbose is specified. This should suffice until #106 is implemented. Closing as resolved.