So far I couldn’t find anything really, but is it true that curl doesn’t really time out at all?
<a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="e693958394a68e899592">[email protected]</a>:~# curl http://localhost/testdir/image.jpg
I’m asking because I’m redirecting any request for images in testdir to a separate Apache module which generates those pictures on the fly. It can take up to 15 minutes before the picture is actually ready and delivered to the requesting client.
Will curl always wait (or is it depending on configuration) or is there any sort of timeout?
Answers:
Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.
Method 1
Yes.
Timeout parameters
curl has two options: --connect-timeout and --max-time.
Quoting from the manpage:
--connect-timeout <seconds>
Maximum time in seconds that you allow the connection to the
server to take. This only limits the connection phase, once
curl has connected this option is of no more use. Since 7.32.0,
this option accepts decimal values, but the actual timeout will
decrease in accuracy as the specified timeout increases in deci‐
mal precision. See also the -m, --max-time option.
If this option is used several times, the last one will be used.
and:
-m, --max-time <seconds>
Maximum time in seconds that you allow the whole operation to
take. This is useful for preventing your batch jobs from hang‐
ing for hours due to slow networks or links going down. Since
7.32.0, this option accepts decimal values, but the actual time‐
out will decrease in accuracy as the specified timeout increases
in decimal precision. See also the --connect-timeout option.
If this option is used several times, the last one will be used.
Defaults
Here (on Debian) it stops trying to connect after 2 minutes, regardless of the time specified with --connect-timeout and although the default connect timeout value seems to be 5 minutes according to the DEFAULT_CONNECT_TIMEOUT macro in lib/connect.h.
A default value for --max-time doesn’t seem to exist, making curl wait forever for a response if the initial connect succeeds.
What to use?
You are probably interested in the latter option, --max-time. For your case set it to 900 (15 minutes).
Specifying option --connect-timeout to something like 60 (one minute) might also be a good idea. Otherwise curl will try to connect again and again, apparently using some backoff algorithm.
Method 2
There is
timelimit: /usr/bin/timelimit
– effectively limit the absolute execution time of a process
Options:
-p If the child process is terminated by a signal, timelimit
propagates this condition, i.e. sends the same signal to itself.
This allows the program executing timelimit to determine
whether the child process was terminated by a signal or
actually exited with an exit code larger than 128.
-q Quiet operation - timelimit does not output diagnostic
messages about signals sent to the child process.
-S killsig
Specify the number of the signal to be sent to the
process killtime seconds after warntime has expired.
Defaults to 9 (SIGKILL).
-s warnsig
Specify the number of the signal to be sent to the
process warntime seconds after it has been started.
Defaults to 15 (SIGTERM).
-T killtime
Specify the maximum execution time of the process before
sending killsig after warnsig has been sent. Defaults to 120 seconds.
-t warntime
Specify the maximum execution time of the process in
seconds before sending warnsig. Defaults to 3600 seconds.
On systems that support the setitimer(2) system call, the
warntime and killtime values may be specified in fractional
seconds with microsecond precision.
Method 3
Better than --max-time are the --speed-limit and --speed-time options. In short, --speed-limit specifies the minimum average speed which you are willing to accept, and --speed-time specifies how long the transfer speed can remain below that limit before the transfer times out and is aborted.
Method 4
If you have coreutils installed on MacOS you can use the GNU timeout command which is included with that package. The GNU tools are all prefixed with a g so the CLI would be gtimeout.
gtimeout --help Usage: gtimeout [OPTION] DURATION COMMAND [ARG]... or: gtimeout [OPTION] Start COMMAND, and kill it if still running after DURATION.
Example
$ gtimeout 1s curl -I http://www.google.com/ HTTP/1.1 200 OK Date: Wed, 31 Oct 2018 03:36:08 GMT Expires: -1 Cache-Control: private, max-age=0 Content-Type: text/html; charset=ISO-8859-1 P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info." Server: gws X-XSS-Protection: 1; mode=block X-Frame-Options: SAMEORIGIN Set-Cookie: 1P_JAR=2018-10-31-03; expires=Fri, 30-Nov-2018 03:36:08 GMT; path=/; domain=.google.com HttpOnly Transfer-Encoding: chunked Accept-Ranges: none Vary: Accept-Encoding
Method 5
Couple of solutions in BASH4+
# -- server available to check via port xxx ? --
function isServerAvailableNC() {
max_secs_run="${3}"
if timeout $max_secs_run nc -z ${1} ${2} 2>/dev/null >/dev/null; then
#echo "${1} ✓"
true
else
#echo "${1} ✗"
return
fi
}
# -- server available to check via port xxx ? --
# -- supported protocols (HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP or FILE) --
#/usr/bin/curl -sSf --max-time 3 https://ifwewanted.to.confirm.https.com/ --insecure
function isServerAvailableCURL() {
max_secs_run="${3}"
proto="http://"
if [ ! -z ${2} ] || [ ${2} -gt 80 ] ;then
proto="https://"
fi
if /usr/bin/curl -sSf --max-time "${max_secs_run}" "${1}" --insecure 2>/dev/null >/dev/null; then
#echo "${1} ✓"
true
else
#echo "${1} ✗"
false
fi
}
Sample use:
RECOMMEND THAT NC used if we need a specific port
host="1.2.3.4"
if isServerAvailableCURL "$host" "80" "3";then
check_remote_domain_cert "$host"
fi
host="1.2.3.4"
if isServerAvailableNC "$host" "80" "3";then
check_remote_domain_cert "$host"
fi
All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0