一、前言
初看OkHttp源码,由于对Address、Route、Proxy、ProxySelector、RouteSelector等理解不够,读源码非常吃力,看了几遍依然对于寻找复用连接、创建连接、连接服务器、连接代理服务器、创建隧道连接等逻辑似懂非懂,本篇决定梳理一遍相关的概念及基本原理。
二、基础知识
(1)HTTP网络协议:
● HTTP/1.1(HTTPS)
● HTTP/2
● SPDY
(2)HTTP请求流程
一个http请求的流程(直连):
1、输入url及参数;
2、如果是url是域名则解析ip地址,可能对应多个ip,如果没有指定端口,则用默认端口,http请求用80;
3、创建socket,根据ip和端口连接服务器(socket内部会完成3次TCP握手);
4、socket成功连接后,发送http报文数据。
一个https请求的流程(直连):
1、输入url及参数;
2、如果是url是域名则解析ip地址,可能对应多个ip,如果没有指定端口,则用默认端口,https请求用443;
3、创建socket,根据ip和端口连接服务器(socket内部会完成3次TCP握手);
4、socket成功连接后进行TLS握手,可通过java标准款提供的SSLSocket完成;
5、握手成功后,发送https报文数据。
(3)代理相关说明
1、分类
● HTTP代理:普通代理、隧道代理
● SOCKS代理:SOCKS4、SOCKS5
2、HTTP代理分类及说明
普通代理
HTTP/1.1 协议的第一部分。其代理过程为:
● client 请求 proxy
● proxy 解析请求获取 origin server 地址
● proxy 向 origin server 转发请求
● proxy 接收 origin server 的响应
● proxy 向 client 转发响应
其中proxy获取目的服务器地址的标准方法是解析 request line 里的 request-URL。因为proxy需要解析报文,因此普通代理无法适用于https,因为报文都是加密的。
隧道代理
通过 Web 代理服务器用隧道方式传输基于 TCP 的协议。
请求包括两个阶段,一是连接(隧道)建立阶段,二是数据通信(请求响应)阶段,数据通信是基于 TCP packet ,代理服务器不会对请求及响应的报文作任何的处理,都是原封不动的转发,因此可以代理 HTTPS请求和响应。
代理过程为:
● client 向 proxy 发送 CONNET 请求(包含了 origin server 的地址)
● proxy 与 origin server 建立 TCP 连接
● proxy 向 client 发送响应
● client 向 proxy 发送请求,proxy 原封不动向 origin server 转发请求,请求数据不做任何封装,为原生 TCP packet.
3、SOCKS代理分类及说明
● SOCKS4:只支持TCP协议(即传输控制协议)
● SOCKS5: 既支持TCP协议又支持UDP协议(即用户数据包协议),还支持各种身份验证机制、服务器端域名解析等。
SOCK4能做到的SOCKS5都可得到,但反过来却不行,比如我们常用的聊天工具QQ在使用代理时就要求用SOCKS5代理,因为它需要使用UDP协议来传输数据。
三、OkHttp路由
有了上面的基础知识,下面分析结合源码分析OkHttp路由相关的逻辑。OkHttp用Address来描述与目标服务器建立连接的配置信息,但请求输入的可能是域名,一个域名可能对于多个ip,真正建立连接是其中一个ip,另外,如果设置了代理,客户端是与代理服务器建立直接连接,而不是目标服务器,代理又可能是域名,可能对应多个ip。因此,这里用Route来描述最终选择的路由,即客户端与哪个ip建立连接,是代理还是直连。下面对比下Address及Route的属性,及路由选择器RouteSelector。
(1)Address解析
描述与目标服务器建立连接所需要的配置信息,包括目标主机名、端口、dns,SocketFactory,如果是https请求,包括TLS相关的SSLSocketFactory 、HostnameVerifier 、CertificatePinner,代理服务器信息Proxy 、ProxySelector 。
public final class Address {
final HttpUrl url; //与目标服务器连接的配置,包括协议、主机名、端口等
final Dns dns; //dns解析器
final SocketFactory socketFactory; //原始Socket创建工厂
final Authenticator proxyAuthenticator;
final List<Protocol> protocols;
final List<ConnectionSpec> connectionSpecs;
final ProxySelector proxySelector; //代理选择器,默认用系统的
final @Nullable Proxy proxy;//设置的代理,可为空
final @Nullable SSLSocketFactory sslSocketFactory; //SSLSocket创建工厂
final @Nullable HostnameVerifier hostnameVerifier; //用于https证书检验
final @Nullable CertificatePinner certificatePinner;//用于https证书检验
public Address(String uriHost, int uriPort, Dns dns, SocketFactory socketFactory,
@Nullable SSLSocketFactory sslSocketFactory, @Nullable HostnameVerifier hostnameVerifier,
@Nullable CertificatePinner certificatePinner, Authenticator proxyAuthenticator,
@Nullable Proxy proxy, List<Protocol> protocols, List<ConnectionSpec> connectionSpecs,
ProxySelector proxySelector) {
}
(2)Route解析
Route提供了真正连接服务器所需要的动态信息,明确需要连接的服务器IP地址及代理服务器,一个Address可能会有很多个路由Route供选择(一个DNS对应对个IP)。
public final class Route {
final Address address;
final Proxy proxy;
final InetSocketAddress inetSocketAddress;//需要连接的服务器地址
public Route(Address address, Proxy proxy, InetSocketAddress inetSocketAddress) {
if (address == null) {
throw new NullPointerException("address == null");
}
if (proxy == null) {
throw new NullPointerException("proxy == null");
}
if (inetSocketAddress == null) {
throw new NullPointerException("inetSocketAddress == null");
}
this.address = address;
this.proxy = proxy;
this.inetSocketAddress = inetSocketAddress;
}
(3)RouteSelector解析
Address和Route都是数据对象,没有提供操作方法,OkHttp另外定义了RouteSelector来完成选择的路由的操作。
public final class RouteSelector {
private final Address address;
//用于记录连接失败的路由的黑名单,如果在黑名单,不用尝试
private final RouteDatabase routeDatabase;
private final Call call;
private final EventListener eventListener;
private List<Proxy> proxies = Collections.emptyList(); //代理列表
private int nextProxyIndex;//下一个尝试的代理地址
private List<InetSocketAddress> inetSocketAddresses = Collections.emptyList(); //sockect地址列表
private final List<Route> postponedRoutes = new ArrayList<>();//失败的路由列表
public RouteSelector(Address address, RouteDatabase routeDatabase, Call call,
EventListener eventListener) {
...
resetNextProxy(address.url(), address.proxy());
}
/*
* 获取下个需要尝试的Route集
*/
public Selection next() throws IOException {}
/*
* 记录失败路由
*/
public void connectFailed(Route failedRoute, IOException failure) {}
/*
* 读取代理服务器信息
*/
private void resetNextProxy(HttpUrl url, Proxy proxy) {}
/*
* 准备需要尝试的socket地址(目标服务器或者代理服务器)
private void resetNextInetSocketAddress(Proxy proxy) throws IOException {}
}
1、读取代理配置信息:resetNextProxy()
private void resetNextProxy(HttpUrl url, Proxy proxy) {
if (proxy != null) {
// If the user specifies a proxy, try that and only that.
proxies = Collections.singletonList(proxy);
} else {
// Try each of the ProxySelector choices until one connection succeeds.
List<Proxy> proxiesOrNull = address.proxySelector().select(url.uri());
proxies = proxiesOrNull != null && !proxiesOrNull.isEmpty()
? Util.immutableList(proxiesOrNull)
: Util.immutableList(Proxy.NO_PROXY);
}
nextProxyIndex = 0;
}
}
读取代理配置:
● 如果有指定代理(不读取系统配置,在OkHttpClient实例中指定),则只用1个该指定代理;
● 如果没有指定,则读取系统配置的,可能有多个。
2、获取需要尝试的socket地址(目标服务器或者代理服务器):resetNextInetSocketAddress()
private void resetNextInetSocketAddress(Proxy proxy) throws IOException {
// Clear the addresses. Necessary if getAllByName() below throws!
inetSocketAddresses = new ArrayList<>();
String socketHost;
int socketPort;
if (proxy.type() == Proxy.Type.DIRECT || proxy.type() == Proxy.Type.SOCKS) {
socketHost = address.url().host();
socketPort = address.url().port();
} else {
SocketAddress proxyAddress = proxy.address();
if (!(proxyAddress instanceof InetSocketAddress)) {
throw new IllegalArgumentException(
"Proxy.address() is not an " + "InetSocketAddress: " + proxyAddress.getClass());
}
InetSocketAddress proxySocketAddress = (InetSocketAddress) proxyAddress;
socketHost = getHostString(proxySocketAddress);
socketPort = proxySocketAddress.getPort();
}
if (socketPort < 1 || socketPort > 65535) {
throw new SocketException("No route to " + socketHost + ":" + socketPort
+ "; port is out of range");
}
if (proxy.type() == Proxy.Type.SOCKS) {
inetSocketAddresses.add(InetSocketAddress.createUnresolved(socketHost, socketPort));
} else {
eventListener.dnsStart(call, socketHost);
// Try each address for best behavior in mixed IPv4/IPv6 environments.
List<InetAddress> addresses = address.dns().lookup(socketHost);
if (addresses.isEmpty()) {
throw new UnknownHostException(address.dns() + " returned no addresses for " + socketHost);
}
eventListener.dnsEnd(call, socketHost, addresses);
for (int i = 0, size = addresses.size(); i < size; i++) {
InetAddress inetAddress = addresses.get(i);
inetSocketAddresses.add(new InetSocketAddress(inetAddress, socketPort));
}
}
}
结合Address的host和代理,解析要尝试的套接字地址(ip+端口)列表:
● 直连或者SOCK代理, 则用目标服务器的主机名和端口,如果是HTTP代理,则用代理服务器的主机名和端口;
● 如果是SOCK代理,根据目标服务器主机名和端口号创建未解析的套接字地址,列表只有1个地址;
● 如果是直连或HTTP代理,先DNS解析,得到InetAddress列表(没有端口),再创建InetSocketAddress列表(带上端口),InetSocketAddress与InetAddress的区别是前者带端口信息。
3、获取路由列表:next()
/**
* Returns true if there's another set of routes to attempt. Every address has at least one route.
*/
public boolean hasNext() {
return hasNextProxy() || !postponedRoutes.isEmpty();
}
public Selection next() throws IOException {
if (!hasNext()) {
throw new NoSuchElementException();
}
// Compute the next set of routes to attempt.
List<Route> routes = new ArrayList<>();
while (hasNextProxy()) {
// Postponed routes are always tried last. For example, if we have 2 proxies and all the
// routes for proxy1 should be postponed, we'll move to proxy2. Only after we've exhausted
// all the good routes will we attempt the postponed routes.
Proxy proxy = nextProxy();
for (int i = 0, size = inetSocketAddresses.size(); i < size; i++) {
Route route = new Route(address, proxy, inetSocketAddresses.get(i));
if (routeDatabase.shouldPostpone(route)) {
postponedRoutes.add(route);
} else {
routes.add(route);
}
}
if (!routes.isEmpty()) {
break;
}
}
if (routes.isEmpty()) {
// We've exhausted all Proxies so fallback to the postponed routes.
routes.addAll(postponedRoutes);
postponedRoutes.clear();
}
return new Selection(routes);
}
/** Returns true if there's another proxy to try. */
private boolean hasNextProxy() {
return nextProxyIndex < proxies.size();
}
/** Returns the next proxy to try. May be PROXY.NO_PROXY but never null. */
private Proxy nextProxy() throws IOException {
if (!hasNextProxy()) {
throw new SocketException("No route to " + address.url().host()
+ "; exhausted proxy configurations: " + proxies);
}
Proxy result = proxies.get(nextProxyIndex++);
resetNextInetSocketAddress(result);
return result;
}
选择路由的流程解析:
● 遍历每个代理对象,可能多个,直连的代理对象为Proxy.DIRECT(实际是没有中间代理的);
● 对每个代理获取套接字地址列表;
● 遍历地址列表,创建Route,判断Route如果在路由黑名单中,则添加到失败路由列表,不在黑名单中则添加到待返回的Route列表;
● 如果最后待返回的Route列表为空,即可能所有路由都在黑名单中,实在没有新路由了,则将失败的路由集合返回;
● 传入Route列表创建Selection对象,对象比较简单,就是一个目标路由集合,及读取方法。
(4)RouteDatabase 解析
为了避免不必要的尝试,OkHttp会把连接失败的路由加入到黑名单中,由RouteDatabase管理,该类比较简单,就是一个失败路由集合。
public final class RouteDatabase {
private final Set<Route> failedRoutes = new LinkedHashSet<>();
/** Records a failure connecting to {@code failedRoute}. */
public synchronized void failed(Route failedRoute) {
failedRoutes.add(failedRoute);
}
/** Records success connecting to {@code route}. */
public synchronized void connected(Route route) {
failedRoutes.remove(route);
}
/** Returns true if {@code route} has failed recently and should be avoided. */
public synchronized boolean shouldPostpone(Route route) {
return failedRoutes.contains(route);
}
}
(5)选择路由的流程分析
1、创建Address
Address的创建在RetryAndFollowUpInteceptor里,每次请求会声明一个新的Address及StreamAllocation对象,而StreamAllocation使用Address创建RouteSelector对象,在连接时RouteSelector确定请求的路由。
public final class RetryAndFollowUpInterceptor implements Interceptor {
@Override public Response intercept(Chain chain) throws IOException {
...
//原始请求
StreamAllocation streamAllocation = new StreamAllocation(client.connectionPool(),
createAddress(request.url()), call, eventListener, callStackTrace);
...
while (true) {
...
//原始请求返回
response = realChain.proceed(request, streamAllocation, null, null);
...
//创建重定向请求
Request followUp = followUpRequest(response, streamAllocation.route());
...
//重定向StreamAllocation
streamAllocation = new StreamAllocation(client.connectionPool(),createAddress(followUp.url()), call,
eventListener, callStackTrace);
}
}
private Address createAddress(HttpUrl url) {
SSLSocketFactory sslSocketFactory = null;
HostnameVerifier hostnameVerifier = null;
CertificatePinner certificatePinner = null;
if (url.isHttps()) {
sslSocketFactory = client.sslSocketFactory();
hostnameVerifier = client.hostnameVerifier();
certificatePinner = client.certificatePinner();
}
return new Address(url.host(), url.port(), client.dns(), client.socketFactory(),
sslSocketFactory, hostnameVerifier, certificatePinner, client.proxyAuthenticator(),
client.proxy(), client.protocols(), client.connectionSpecs(), client.proxySelector());
}
每个Requst都会构造一个Address对象,构造好了Address对象只是有了与服务器连接的配置信息,但没有确定最终服务器的ip,也没有确定连接的路由。
2、创建RouteSelector
在StreamAllocation声明的同时会声明路由选择器RouteSelector,为一次请求寻找路由。
public final class StreamAllocation {
public StreamAllocation(ConnectionPool connectionPool, Address address, Call call,
EventListener eventListener, Object callStackTrace) {
...
//每个StreamAllocation会声明一个RouteSelector
this.routeSelector = new RouteSelector(address, routeDatabase(), call, eventListener);
}
}
3、选择可用的路由Route
public final class StreamAllocation {
private RealConnection findConnection(int connectTimeout, int readTimeout, int writeTimeout,
int pingIntervalMillis, boolean connectionRetryEnabled) throws IOException {
...
//根据address判断ConnectionPool中是否有可重用的连接
Internal.instance.get(connectionPool, address, this, null);
...
//ConnectionPool没有可用的,根据Address和proxy获取路由集合
if (selectedRoute == null && (routeSelection == null || !routeSelection.hasNext())) {
newRouteSelection = true;
routeSelection = routeSelector.next();
}
synchronized (connectionPool) {
if (canceled) throw new IOException("Canceled");
if (newRouteSelection) {
// Now that we have a set of IP addresses, make another attempt at getting a connection from
// the pool. This could match due to connection coalescing.
//遍历每个路由(IP地址),根据Route判断线程池是否有符合重用的连接
List<Route> routes = routeSelection.getAll();
for (int i = 0, size = routes.size(); i < size; i++) {
Route route = routes.get(i);
Internal.instance.get(connectionPool, address, this, route);
if (connection != null) {
foundPooledConnection = true;
result = connection;
this.route = route;
break;
}
}
}
//线程池没有匹配的,从集合中选择一个路由
if (!foundPooledConnection) {
if (selectedRoute == null) {
selectedRoute = routeSelection.next();
}
// Create a connection and assign it to this allocation immediately. This makes it possible
// for an asynchronous cancel() to interrupt the handshake we're about to do.
// 根据刚选择的路由创建新连接RealConnection,并分配给当前流
route = selectedRoute;
refusedStreamCount = 0;
result = new RealConnection(connectionPool, selectedRoute);
acquire(result, false);
}
}
}
}
四、实例分析
下面在测试过程跟踪实例对象来理解,分别测试直连和HTTP代理HTTP2请求路由的选择过程:
● 直连请求流程
● HTTP代理HTTPS流程
请求url:https://www.jianshu.com/p/63ba15d8877a
(1)直连请求
1、构造address对象
scheme:https
host:www.jianshu.com
端口:默认443
proxy:null
2、读取代理配置:resetNextProxy
没有设置代理服务器,返回的代理对象为DIRECT。
3、解析目标服务器套接字地址:resetNextInetSocketAddress
域名:www.jianshu.com
IP地址:8个
端口:默认443
4、选择Route创建RealConnection
选中了路由列表中第一个,IP为183.240.216.228
5、确定协议
通过SSLSockect完成ssl握手后,判断协议版本,获得的协议是h2,可以看出简书用了HTTP/2.0.
(2)HTTP代理
测试方法:
● 在PC端打开Charles,设置端口,如何设置代理,网上有教程,比较简单;
● 手机打开WIFI,选择连接的WIFI修改网络,在高级选项中设置中指定了代理服务器,ip为PC的ip,端口是Charles刚设置的端口;
● OkHttpClient不指定代理,发起请求。
1、构造address对象
address对象与直连一样。
scheme:https
host:www.jianshu.com
端口:默认443
proxy:null
2、读取代理配置:resetNextProxy
可以看出通过默认的代理选择器读取到一个代理地址。
3、解析目标服务器套接字地址:resetNextInetSocketAddress
可以看到目标地址就是代理服务器地址,只有1个,可以HTTP代理时客户端与代理服务器建立socket连接。
4、选择Route创建RealConnection
创建与代理服务器的连接对象。
5、创建隧道
由于是代理https请求,需要用到隧道代理。
从图可以看出,建立隧道其实是发送CONNECT请求,header包括字段Proxy-Connection,目标主机名,请求内容类似:
CONNECT www.jianshu.com:443 HTTP/1.1
6、确定协议,SSL握手
隧道代理,客户端需要与代理服务器建立安全连接。
五、总结
1、代理可分为HTTP代理和SOCK代理;
2、HTTP代理又分为普通代理和隧道代理;普通代理适合明文传输,即http请求;隧道代理仅转发TCP包,适合加密传输,即https/http2;
3、SOCK代理又分为SOCK4和SOCK5,区别是后者支持UDP传输,适合代理聊天工具如QQ;
4、没有设置代理(OkHttpClient没有指定同时系统也没有设置),客户端直接与目标服务器建立TCP连接;
5、设置了代理,代理http请求时,客户端与代理服务器建立TCP连接,如果代理服务器是域名,则解释代理服务器域名,而目标服务器的域名由代理服务器解析;
6、设置了代理,代理https/http2请求时,客户端与代理服务器建立TCP连接,发送CONNECT请求与代理服务器建立隧道,并进行SSL握手,代理服务器不解析数据,仅转发TCP数据包。