Taking a Long Look at QUIC: An Approach for Rigorous Evaluation of Rapidly Evolving Transport Protocols Arash Molavi Kakhki1, Samuel Jero2, David Choffnes1, Cristina Nita-Rotaru1, and Alan Mislove1 1 Northeastern University 2 Purdue University IMC 2017 QUIC (Quick UDP Internet Connections) Improve performance for HTTP traffic Reduce handshake time (0-RTT) Prevent head-of-line blocking Improve loss recovery Modular congestion control Connection migration across IPs Avoid middlebox meddling Protect user data Facilitate rapid deployment HTTP/2 Cubic TLS 5.6.7.8
IMC 2017 QUIC BBR 1.2.3.4 TCP Samuel Jero HTTP/2 API UDP No ACK ambiguity QUIC Better RTT/BW IP estimation 0111 0101 user space TCP kernel space 2 QUIC (Quick UDP Internet Connections) Significant deployment today Google, Akamai >85% of requests from Chrome to Google use QUIC >7% of Total Internet Traffic
IETF standardization ongoing -07 drafts 20+ implementations in progress Google Data, from: https://www.ietf.org/proceedings/96/slides/slides-96-quic-3.pdf Samuel Jero IMC 2017 3 Existing Evaluations of QUIC Googles evaluation 3% page load time improvement searchstatistics on AllGoogle aggregated 8% latency reduction on Google search Limited controlled tests 18% reduction in buffer time on YouTube Not reproducible Other QUIC evaluations Limited environments/networks Limited tests Old versions of QUIC
Untuned implementations Results not statically sound No root cause analysis Samuel Jero IMC 2017 4 Our Work Goal: Provide a rigorous evaluation of QUICs performance compared to TCP Extensive calibration to ensure fair comparisons between QUIC and TCP Methodology for automatic head-to-head comparisons, including root-cause-analysis Statistical analysis of results Evaluation of QUICs performance compared to TCP Variety of emulated network conditions (loss, delay, jitter, ) Variety of network types (fixed-line, wireless, cellular, ) Variety of environments (desktop, mobile, ) Root cause analysis Samuel Jero IMC 2017 5 Outline Motivation Methodology
Calibration QUICs performance for HTTP compared with TCP Fairness of QUIC with TCP Summary Samuel Jero IMC 2017 6 Methodology Compare QUIC with TCP+TLS+HTTP/2 for HTTP TCP Network Server upports QUIC and TCP) Ensures fair comparison, Provides similar Router features QUIC Primary use case for QUIC today TCP (OpenWRT+TC/NETEM) Client
Root-Cause Analysis Simple HTML pages isolate workload from performance Samuel Jero QUIC Instrument (and TCP) QUIC haveserver manytocomponents, capture congestion features, and optimizations control statethat machine maketraces identifying to look theforcauses differences of differences in internal very challenging behavior Compare PLTs (Page Load Times) Determine whether results are statistically significant using Welchs t-test
IMC 2017 7 Calibration QUIC Network Server QUIC and TCP) upports QUIC TCP Router (OpenWRT+TC/NETEM) Client Google Servers No control Toy QUIC server in Chromium Not performant Tuned toy server by: Matching parameters with Googles Fixing a bug regarding slow start Downloading a 10MB obj at 100Mbps Samuel Jero IMC 2017 8 HTTP Performance: QUIC vs TCP
RTT = 36ms, loss = 0% QUIC is faster RTT = 36ms, loss = 1% Bandwidth 44% 45% QUIC 44% Network TCP is faster Object Size Our server on EC2 Support QUIC and TCP) RTT = 112ms, loss = 0% TCP Router (OpenWRT+TC/NETEM) Client RTT = 112ms with re-ordering, loss = 0% RTT = 112ms with re-ordering, loss = 0%
QUIC performs better than TCP, except under reordering, but improvements diminish with transfer size Samuel Jero IMC 2017 9 HTTP Performance: QUIC vs TCP % of time spent in each state RTT = 36ms, loss = 0% WIFI Slow Start Cong. Avoid. App. Limit. Recovery 1.7% 91.5% 7.1% 0% 0.42%
40.6% 58.8% 0.2% RTT = 36ms, loss = 0% Resource constraints limit mobile performance Samuel Jero IMC 2017 10 HTTP Performance: QUIC vs TCP Reordering (%) Cellular Loss (%) Sprint - 3G Sprint - 3G Verizon - 3G Verizon - 3G Sprint - LTE Sprint - LTE Verizon - LTE Verizon - LTE
0 1 2 3 4 5 6 7 8 9 10 0 0.01 0.02 0.03 0.04 0.05 QUIC for 3G: + Lower bandwidth, higher loss
- Higher reordering QUIC performs better than TCP in cellular networks, but limited by reordering Samuel Jero IMC 2017 11 0.06 Fairness: QUIC vs TCP A property of transport protocols that ensures that a flow does not consume more than its fair share of bottleneck bandwidth Flow 5Mbps bottleneck link, RTT=36ms, buffer=30 KB Avg. Tput 2.71 QUIC vs. QUIC Not Responsible: TCP TCP 1.62 TCP Segment Offload Delayed ACKs QUIC RTTvs.estimation QUIC 2.8 TCPx2
avg(TCP) 0.83 Investigation still in progress QUIC is very unfair to TCP, despite both using CUBIC Samuel Jero IMC 2017 QUIC vs. TCPx4 QUIC 2.75 avg(TCP) 0.41 12 Summary Evaluated the performance of QUIC, an application-layer transport protocol that is rapidly evolving and deployed at scale Focus on the performance and fairness of QUIC for HTTP in comparison to TCP+TLS+HTTP/2 Under a variety of network conditions In desktop, mobile, and cellular environments With careful calibration Ensuring statistical significance
Root-cause analysis to understand our findings by instrumenting the QUIC implementation Approach can be applied to future versions and protocols Samuel Jero IMC 2017 13 Questions? Samuel Jero [email protected] Check out the code! http://quic.ccs.neu.edu Samuel Jero IMC 2017 14
