序
本文主要研究一下jdk httpclient的retry参数
DEFAULT_MAX_ATTEMPTS
java.net.http/jdk/internal/net/http/MultiExchange.java
class MultiExchange<T> {
static final Logger debug =
Utils.getDebugLogger("MultiExchange"::toString, Utils.DEBUG);
private final HttpRequest userRequest; // the user request
private final HttpRequestImpl request; // a copy of the user request
final AccessControlContext acc;
final HttpClientImpl client;
final HttpResponse.BodyHandler<T> responseHandler;
final HttpClientImpl.DelegatingExecutor executor;
final AtomicInteger attempts = new AtomicInteger();
HttpRequestImpl currentreq; // used for retries & redirect
HttpRequestImpl previousreq; // used for retries & redirect
Exchange<T> exchange; // the current exchange
Exchange<T> previous;
volatile Throwable retryCause;
volatile boolean expiredOnce;
volatile HttpResponse<T> response = null;
// Maximum number of times a request will be retried/redirected
// for any reason
static final int DEFAULT_MAX_ATTEMPTS = 5;
static final int max_attempts = Utils.getIntegerNetProperty(
"jdk.httpclient.redirects.retrylimit", DEFAULT_MAX_ATTEMPTS
);
//......
}
- 这里有一个AtomicInteger类型的attempts变量，用来记录请求次数
- 另外还有一个max_attempts，读取的是jdk.httpclient.redirects.retrylimit值，读取不到默认取DEFAULT_MAX_ATTEMPTS，为5
MultiExchange.responseAsyncImpl
java.net.http/jdk/internal/net/http/MultiExchange.java
private CompletableFuture<Response> responseAsyncImpl() {
CompletableFuture<Response> cf;
if (attempts.incrementAndGet() > max_attempts) {
cf = failedFuture(new IOException("Too many retries", retryCause));
} else {
if (currentreq.timeout().isPresent()) {
responseTimerEvent = ResponseTimerEvent.of(this);
client.registerTimer(responseTimerEvent);
}
try {
// 1. apply request filters
// if currentreq == previousreq the filters have already
// been applied once. Applying them a second time might
// cause some headers values to be added twice: for
// instance, the same cookie might be added again.
if (currentreq != previousreq) {
requestFilters(currentreq);
}
} catch (IOException e) {
return failedFuture(e);
}
Exchange<T> exch = getExchange();
// 2. get response
cf = exch.responseAsync()
.thenCompose((Response response) -> {
HttpRequestImpl newrequest;
try {
// 3. apply response filters
newrequest = responseFilters(response);
} catch (IOException e) {
return failedFuture(e);
}
// 4. check filter result and repeat or continue
if (newrequest == null) {
if (attempts.get() > 1) {
Log.logError("Succeeded on attempt: " + attempts);
}
return completedFuture(response);
} else {
this.response =
new HttpResponseImpl<>(currentreq, response, this.response, null, exch);
Exchange<T> oldExch = exch;
return exch.ignoreBody().handle((r,t) -> {
previousreq = currentreq;
currentreq = newrequest;
expiredOnce = false;
setExchange(new Exchange<>(currentreq, this, acc));
return responseAsyncImpl();
}).thenCompose(Function.identity());
} })
.handle((response, ex) -> {
// 5. handle errors and cancel any timer set
cancelTimer();
if (ex == null) {
assert response != null;
return completedFuture(response);
}
// all exceptions thrown are handled here
CompletableFuture<Response> errorCF = getExceptionalCF(ex);
if (errorCF == null) {
return responseAsyncImpl();
} else {
return errorCF;
} })
.thenCompose(Function.identity());
}
return cf;
}
- 进入该方法的时候，调用attempts.incrementAndGet()，递增请求次数，然后判断有无超出限制，有则返回带有new IOException("Too many retries", retryCause)异常的failedFuture，即通过CompletableFuture.completeExceptionally返回
- 如果没有超出限制，但是执行请求失败，则调用getExceptionalCF来判断是否应该重试，如果返回null，则重试，通过再次调用responseAsyncImpl，通过这种递归调用完成重试逻辑
MultiExchange.getExceptionalCF
java.net.http/jdk/internal/net/http/MultiExchange.java
/**
* Takes a Throwable and returns a suitable CompletableFuture that is
* completed exceptionally, or null.
*/
private CompletableFuture<Response> getExceptionalCF(Throwable t) {
if ((t instanceof CompletionException) || (t instanceof ExecutionException)) {
if (t.getCause() != null) {
t = t.getCause();
}
}
if (cancelled && t instanceof IOException) {
if (!(t instanceof HttpTimeoutException)) {
t = toTimeoutException((IOException)t);
}
} else if (retryOnFailure(t)) {
Throwable cause = retryCause(t);
if (!(t instanceof ConnectException)) {
if (!canRetryRequest(currentreq)) {
return failedFuture(cause); // fails with original cause
}
}
// allow the retry mechanism to do its work
retryCause = cause;
if (!expiredOnce) {
if (debug.on())
debug.log(t.getClass().getSimpleName() + " (async): retrying...", t);
expiredOnce = true;
// The connection was abruptly closed.
// We return null to retry the same request a second time.
// The request filters have already been applied to the
// currentreq, so we set previousreq = currentreq to
// prevent them from being applied again.
previousreq = currentreq;
return null;
} else {
if (debug.on()) {
debug.log(t.getClass().getSimpleName()
+ " (async): already retried once.", t);
}
t = cause;
}
}
return failedFuture(t);
}
private boolean retryOnFailure(Throwable t) {
return t instanceof ConnectionExpiredException
|| (RETRY_CONNECT && (t instanceof ConnectException));
}
/** Returns true if the given request can be automatically retried. */
private static boolean canRetryRequest(HttpRequest request) {
if (RETRY_ALWAYS)
return true;
if (isIdempotentRequest(request))
return true;
return false;
}
/** Returns true is given request has an idempotent method. */
private static boolean isIdempotentRequest(HttpRequest request) {
String method = request.method();
switch (method) {
case "GET" :
case "HEAD" :
return true;
default :
return false;
}
}
private Throwable retryCause(Throwable t) {
Throwable cause = t instanceof ConnectionExpiredException ? t.getCause() : t;
return cause == null ? t : cause;
}
/** True if ALL ( even non-idempotent ) requests can be automatic retried. */
private static final boolean RETRY_ALWAYS = retryPostValue();
/** True if ConnectException should cause a retry. Enabled by default */
private static final boolean RETRY_CONNECT = retryConnect();
private static boolean retryPostValue() {
String s = Utils.getNetProperty("jdk.httpclient.enableAllMethodRetry");
if (s == null)
return false;
return s.isEmpty() ? true : Boolean.parseBoolean(s);
}
private static boolean retryConnect() {
String s = Utils.getNetProperty("jdk.httpclient.disableRetryConnect");
if (s == null)
return false;
return s.isEmpty() ? true : Boolean.parseBoolean(s);
}
如果cancelled为true且是IOException则直接返回，否则先判断retryOnFailure再判断canRetryRequest(
如果不是ConnectException才走canRetryRequest这个判断)
- retryOnFailure方法判断如果是ConnectionExpiredException或者是ConnectException且开启retryConnect，则返回true
- RETRY_CONNECT读取的是jdk.httpclient.disableRetryConnect参数，如果值为null，则方法返回false，即不进行retryConnect
- canRetryRequest首先判断RETRY_ALWAYS，在判断isIdempotentRequest(
GET、HEAD方法才重试)，都不是则返回false - RETRY_ALWAYS读取的是jdk.httpclient.enableAllMethodRetry，如果值为null，则方法返回false，即不进行retryPostValue
- 如果该重试的话，则返回null，responseAsyncImpl里头在getExceptionalCF返回null的时候，重新调用了一次responseAsyncImpl，通过递归调用来完成重试逻辑
NetProperties
java.base/sun/net/NetProperties.java
public class NetProperties {
private static Properties props = new Properties();
static {
AccessController.doPrivileged(
new PrivilegedAction<Void>() {
public Void run() {
loadDefaultProperties();
return null;
}});
}
private NetProperties() { };
/*
* Loads the default networking system properties
* the file is in jre/lib/net.properties
*/
private static void loadDefaultProperties() {
String fname = StaticProperty.javaHome();
if (fname == null) {
throw new Error("Can't find java.home ??");
}
try {
File f = new File(fname, "conf");
f = new File(f, "net.properties");
fname = f.getCanonicalPath();
InputStream in = new FileInputStream(fname);
BufferedInputStream bin = new BufferedInputStream(in);
props.load(bin);
bin.close();
} catch (Exception e) {
// Do nothing. We couldn't find or access the file
// so we won't have default properties...
}
}
/**
* Get a networking system property. If no system property was defined
* returns the default value, if it exists, otherwise returns
* <code>null</code>.
* @param key the property name.
* @throws SecurityException if a security manager exists and its
* <code>checkPropertiesAccess</code> method doesn't allow access
* to the system properties.
* @return the <code>String</code> value for the property,
* or <code>null</code>
*/
public static String get(String key) {
String def = props.getProperty(key);
try {
return System.getProperty(key, def);
} catch (IllegalArgumentException e) {
} catch (NullPointerException e) {
}
return null;
}
/**
* Get an Integer networking system property. If no system property was
* defined returns the default value, if it exists, otherwise returns
* <code>null</code>.
* @param key the property name.
* @param defval the default value to use if the property is not found
* @throws SecurityException if a security manager exists and its
* <code>checkPropertiesAccess</code> method doesn't allow access
* to the system properties.
* @return the <code>Integer</code> value for the property,
* or <code>null</code>
*/
public static Integer getInteger(String key, int defval) {
String val = null;
try {
val = System.getProperty(key, props.getProperty(key));
} catch (IllegalArgumentException e) {
} catch (NullPointerException e) {
}
if (val != null) {
try {
return Integer.decode(val);
} catch (NumberFormatException ex) {
}
}
return defval;
}
/**
* Get a Boolean networking system property. If no system property was
* defined returns the default value, if it exists, otherwise returns
* <code>null</code>.
* @param key the property name.
* @throws SecurityException if a security manager exists and its
* <code>checkPropertiesAccess</code> method doesn't allow access
* to the system properties.
* @return the <code>Boolean</code> value for the property,
* or <code>null</code>
*/
public static Boolean getBoolean(String key) {
String val = null;
try {
val = System.getProperty(key, props.getProperty(key));
} catch (IllegalArgumentException e) {
} catch (NullPointerException e) {
}
if (val != null) {
try {
return Boolean.valueOf(val);
} catch (NumberFormatException ex) {
}
}
return null;
}
}
- 这里通过loadDefaultProperties先加载默认配置，读取的是JAVA_HOME/conf/net.properties文件
- 然后getString、getInteger、getBoolean方法采用的是System.getProperty来读取，而net.properties值仅仅作为System.getProperty的defaultValue
- 因此要设置httpclient相关参数，只需要通过System.setProperty或者-D来设置即可
- net.properties
/Library/java/JavaVirtualMachines/jdk-11.jdk/Contents/home/conf/net.properties
java.net.useSystemProxies=false
http.nonProxyHosts=localhost|127.*|[::1]
ftp.nonProxyHosts=localhost|127.*|[::1]
jdk.http.auth.tunneling.disabledSchemes=Basic
net.properties文件默认设置了如上四个参数
相关异常
HttpTimeoutException
java.net.http/java/net/http/HttpTimeoutException.java
/**
* Thrown when a response is not received within a specified time period.
*
* @since 11
*/
public class HttpTimeoutException extends IOException {
private static final long serialVersionUID = 981344271622632951L;
/**
* Constructs an {@code HttpTimeoutException} with the given detail message.
*
* @param message
* The detail message; can be {@code null}
*/
public HttpTimeoutException(String message) {
super(message);
}
}
- 属于java.net.http包，继承至IOException
- 如果设置了request的timeout，则注册ResponseTimerEvent，在超时时抛出HttpTimeoutException: request timed out，同时设置MultiExchange的cancelled为true
- 这类由于客户端设置超时引起的HttpTimeoutException，不会进行重试，即使开启相关重试参数
- 如果这个时间设置得太短，则在connect的时候就超时了，这个时候会抛出HttpConnectTimeoutException，而非HttpTimeoutException: request timed out
HttpConnectTimeoutException
java.net.http/java/net/http/HttpConnectTimeoutException.java
/**
* Thrown when a connection, over which an {@code HttpRequest} is intended to be
* sent, is not successfully established within a specified time period.
*
* @since 11
*/
public class HttpConnectTimeoutException extends HttpTimeoutException {
private static final long serialVersionUID = 321L + 11L;
/**
* Constructs an {@code HttpConnectTimeoutException} with the given detail
* message.
*
* @param message
* The detail message; can be {@code null}
*/
public HttpConnectTimeoutException(String message) {
super(message);
}
}
- 属于java.net.http包，继承至HttpTimeoutException
- 如果设置了client的connectTimeout，则会注册ConnectTimerEvent，在超时时抛出ConnectException("HTTP connect timed out")，同时设置MultiExchange的cancelled为true，这个在MultiExchange.getExceptionalCF方法里头会被包装为HttpConnectTimeoutException
ConnectionExpiredException
java.net.http/jdk/internal/net/http/common/ConnectionExpiredException.java
/**
* Signals that an end of file or end of stream has been reached
* unexpectedly before any protocol specific data has been received.
*/
public final class ConnectionExpiredException extends IOException {
private static final long serialVersionUID = 0;
/**
* Constructs a {@code ConnectionExpiredException} with a detail message of
* "subscription is finished" and the given cause.
*
* @param cause the throwable cause
*/
public ConnectionExpiredException(Throwable cause) {
super("subscription is finished", cause);
}
}
- 一般是在read error的时候触发，比如EOFException，IOException("connection reset by peer)，或者SSLHandshakeException
小结
jdk httpclient的retry参数涉及到的参数如下：
- jdk.httpclient.redirects.retrylimit(
默认为5，用来控制重试次数，不过实际上还有expiredOnce参数，看代码貌似顶多重试一次) - jdk.httpclient.disableRetryConnect(
默认为null，即RETRY_CONNECT为false，不在ConnectException的时候retry) - jdk.httpclient.enableAllMethodRetry(
默认为null，即RETRY_ALWAYS为false，即需要判断请求方法是否幂等来决定是否重试)
是否重试的判断逻辑如下：
- 如果重试次数超过限制，则返回失败，否则往下
- 如果cancelled为true(
这里如果request设置了timeout，触发时cancelled设置为true)且是IOException(例如设置了连接超时抛出的HttpConnectTimeoutException)，则不走重试逻辑；否则往下 - 如果retryOnFailure(
ConnectionExpiredException，或者ConnectException且开启retryConnect)，则往下 - 如果是异常不是ConnectException，则还额外判断canRetryRequest(
判断该请求类型是否允许重试)，满足则往下 - 如果expiredOnce为false，则返回null，即满足重试条件，走递归重试
