Jonathan Mace

PhD Candidate
Brown University
Department of Computer Science

About Me

I am graduating in May 2018 and I am looking for a faculty position at an academic research institution in: .

CV

Research Statement

Teaching Statement

Please feel free to contact me if you think I would be a good fit for your institution.

At Brown, I am advised by Professor Rodrigo Fonseca. Currently I am collaborating with researchers in Facebook's tracing and performance groups.

My research focuses on monitoring, understanding, and enforcing distributed system behaviors. In my PhD work I adapted techniques from end-to-end request tracing to new applications in multi-tenant resource management and dynamic causal profiling. My work on Pivot Tracing introduced baggage, a concept for generic cross-system metadata now widely used by tracing systems like Zipkin and OpenTracing.

I am currently working on large-scale end-to-end performance analysis, ranging from data collection, aggregation, and storage, to deriving high-level insights using machine learning and statistical analysis.

Interests

  • Distributed Systems
  • Networking & Operating Systems
  • Multi-Tenant Cloud Systems
  • Multi-Resource Scheduling
  • End-to-End Request Tracing
  • Data-Driven Performance Analysis

Recent News

  • 2017-11
    Our SOSP paper, Canopy, was featured on the Morning Paper blog today :)
  • 2017-10
    I'm graduating next May! Looking for academic jobs in US, EU, Australia, and New Zealand.
  • 2017-08
    Canopy, our paper on end-to-end tracing at Facebook, is accepted to appear at SOSP 2017.
  • 2017-05
    Prototype of our Baggage Definition Language (BDL) compiler and libraries for Java and Go released on GitHub with a paper soon to come
  • 2016-04
    2DFQ is accepted to appear in SIGCOMM 2016! This is joint work with my colleagues from Microsoft.
  • 2016-01
    I'm very fortunate to be awarded a Facebook Graduate Fellowship!
  • 2015-10
    Pivot Tracing received a Best Paper Award at this year's SOSP

Publications

SOSP 2017

Canopy: An End-to-End Performance Tracing And Analysis System

Jonathan Kaldor, Jonathan Mace, Michał Bejda, Edison Gao, Wiktor Kuropatwa, Joe O'Neill, Kian Win Ong, Bill Schaller, Pingjia Shan, Brendan Viscomi, Vinod Venkataraman, Kaushik Veeraraghavan, Yee Jiun Song
In Proceedings of the 26th ACM Symposium on Operating Systems Principles (SOSP '17)
-----
Also featured in The Morning Paper

Abstract

This paper presents Canopy, Facebook’s end-to-end performance tracing infrastructure. Canopy records causally related performance data across the end-to-end execution path of requests, including from browsers, mobile applications, and backend services. Canopy processes traces in near real-time, derives user-specified features, and outputs to performance datasets that aggregate across billions ofrequests. Using Canopy, Facebook engineers can query and analyze performance data in real-time. Canopy addresses three challenges we have encountered in scaling performance analysis: supporting the range of execution and performance models used by different components of the Facebook stack; supporting interactive ad-hoc analysis of performance data; and enabling deep customization by users, from sampling traces to extracting and visualizing features. Canopy currently records and processes over1 billion traces per day. We discuss how Canopy has evolved to apply to a wide range of scenarios, and present case studies of its use in solving various performance challenges.

Canopy: An End-to-End Performance Tracing And Analysis System

SIGCOMM 2016

2DFQ: Two-Dimensional Fair Queuing for Multi-Tenant Cloud Services

Jonathan Mace, Peter Bodik, Madanlal Musuvathi, Rodrigo Fonseca, Krishnan Varadarajan
In Proceedings of the 2016 ACM SIGCOMM Conference

Abstract

In many important cloud services, different tenants execute their requests in the thread pool of the same process, requiring fair sharing of resources. However, using fair queue schedulers to provide fairness in this context is difficult because of high execution concurrency, and because request costs are unknown and have high variance. Using fair schedulers like WFQ and WF²Q in such settings leads to bursty schedules, where large requests block small ones for long periods of time. In this paper, we propose Two-Dimensional Fair Queuing (2DFQ), which spreads requests of different costs across di erent threads and minimizes the impact of tenants with unpredictable requests. In evaluation on production workloads from Azure Storage, a large-scale cloud system at Microsoft, we show that 2DFQ reduces the burstiness of service by 1-2 orders of magnitude. On workloads where many large requests compete with small ones, 2DFQ improves 99th percentile latencies by up to 2 orders of magnitude.

2DFQ: Two-Dimensional Fair Queuing for Multi-Tenant Cloud Services

SOCC 2016

Principled Workflow-Centric Tracing of Distributed Systems

Raja R. Sambasivan, Ilari Shafer, Jonathan Mace, Benjamin H. Sigelman, Rodrigo Fonseca, Gregory R. Ganger
In Proceedings of the 7th ACM Symposium on Cloud Computing (SOCC '16)

Abstract

Workflow-centric tracing captures the workflow of causally-related events (e.g., work done to process a request) within and among the components of a distributed system. As distributed systems grow in scale and complexity, such tracing is becoming a critical tool for understanding distributed system behavior. Yet, there is a fundamental lack of clarity about how such infrastructures should be designed to provide maximum benefit for important management tasks, such as resource accounting and diagnosis. Without research into this important issue, there is a danger that workflow-centric tracing will not reach its full potential. To help, this paper distills the design space of workflow-centric tracing and describes key design choices that can help or hinder a tracing infrastructure’s utility for important tasks. Our design space and options for them are based on our experiences developing several previous workflow-tracing infrastructures.

Principled Workflow-Centric Tracing of Distributed Systems

SOSP 2015 Best Paper Award

Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems

Jonathan Mace, Ryan Roelke, Rodrigo Fonseca
In Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP '15)
In ACM Transactions on Computer Systems (TOCS, forthcoming 2017)
In Communications of the ACM (CACM, forthcoming 2017)
-----
Also featured in The Morning Paper.

Abstract

Monitoring and troubleshooting distributed systems is notoriously difficult; potential problems are complex, varied, and unpredictable. The monitoring and diagnosis tools commonly used today – logs, counters, and metrics – have two important limitations: what gets recorded is defined a priori, and the information is recorded in a component- or machine-centric way, making it extremely hard to correlate events that cross these boundaries. This paper presents Pivot Tracing, a monitoring framework for distributed systems that addresses both limitations by combining dynamic instrumentation with a novel relational operator: the happened-before join. Pivot Tracing gives users, at runtime, the ability to define arbitrary metrics at one point of the system, while being able to select, filter, and group by events meaningful at other parts of the system, even when crossing component or machine boundaries. We have implemented a prototype of Pivot Tracing for Java-based systems and evaluate it on a heterogeneous Hadoop cluster comprising HDFS, HBase, MapReduce, and YARN. We show that Pivot Tracing can effectively identify a diverse range of root causes such as software bugs, misconfiguration, and limping hardware. We show that Pivot Tracing is dynamic, extensible, and enables cross-tier analysis between inter-operating applications, with low execution overhead.

Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems

NSDI 2015

Retro: Targeted Resource Management in Multi-tenant Distributed Systems

Jonathan Mace, Peter Bodik, Rodrigo Fonseca, Madanlal Musuvathi
In Proceedings of the 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI '15)

Abstract

In distributed systems shared by multiple tenants, effective resource management is an important pre-requisite to providing quality of service guarantees. Many systems deployed today lack performance isolation and experience contention, slowdown, and even outages caused by aggressive workloads or by improperly throttled maintenance tasks such as data replication. In this work we present Retro, a resource management framework for shared distributed systems. Retro monitors per-tenant resource usage both within and across distributed systems, and exposes this information to centralized resource management policies through a high-level API. A policy can shape the resources consumed by a tenant using Retro’s control points, which enforce sharing and rate-limiting decisions. We demonstrate Retro through three policies providing bottleneck resource fairness, dominant resource fairness, and latency guarantees to high-priority tenants, and evaluate the system across five distributed systems: HBase, Yarn, MapReduce, HDFS, and Zookeeper. Our evaluation shows that Retro has low overhead, and achieves the policies’ goals, accurately detecting contended resources, throttling tenants responsible for slowdown and overload, and fairly distributing the remaining cluster capacity.

Retro: Targeted Resource Management in Multi-tenant Distributed Systems

HPTS 2015

We are Losing Track: a Case for Causal Metadata in Distributed Systems

Rodrigo Fonseca, Jonathan Mace
In Proceedings of the 15th International Workshop on High Performance Transaction Systems (HPTS '15)

As our systems move to more concurrent and distributed execution patterns, the tools and abstractions we have to understand, monitor, schedule, and enforce their behavior become progressively less effective or adequate. We argue that systems should be built with causal propagation of generic metadata as a first class primitive, to serve as the narrow waist upon which many debugging and troubleshooting tools could be built, in an analogy to the role of the IP layer in networking

We are Losing Track: a Case for Causal Metadata in Distributed Systems

HotDep 2014

Towards General-Purpose Resource Management in Shared Cloud Services

Jonathan Mace, Peter Bodik, Rodrigo Fonseca, Madanlal Musuvathi
In Proceedings of the 10th Workshop on Hot Topics in Dependability (HotDep '14)

Abstract

In distributed services shared by multiple tenants, managing resource allocation is an important pre-requisite to providing dependability and quality of service guarantees. Many systems deployed today experience contention, slowdown, and even system outages due to aggressive tenants and a lack of resource management. Improperly throttled background tasks, such as data replication, can overwhelm a system; conversely, high-priority background tasks, such as heartbeats, can be subject to resource starvation. In this paper, we outline ve design principles necessary for ešffective and e›fficient resource management policies that could provide guaranteed performance, fairness, or isolation. We present Retro, a resource instrumentation framework that is guided by these principles. Retro instruments all system resources and exposes detailed, real-time statistics of pertenant resource consumption, and could serve as a base for the implementation of such policies.

Towards General-Purpose Resource Management in Shared Cloud Services

Other Documents

Survey 2017

End-to-End Tracing: Adoption and Use Cases

Jonathan Mace
Survey, Brown University, March 2017

Abstract

This document summarizes information about end-to-end tracing for 26 companies. The information was gathered from documents shared to the Distributed Tracing Workgroup and through in-person conversations at tracing workshops.

End-to-End Tracing: Adoption and Use Cases

;login: 2016

Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems

Jonathan Mace, Ryan Roelke, Rodrigo Fonseca
USENIX ;login: Magazine, Spring 2016

Abstract

Pivot Tracing is a monitoring framework for distributed systems that can seamlessly correlate statistics across applications, components, and machines at runtime without needing to change or redeploy system code. Users can define and install monitoring queries on-the-fly to collect arbitrary statistics from one point in the system while being able to select, filter, and group by events meaningful at other points in the system. Pivot Tracing does not correlate cross-component events using expensive global aggregations, nor does it perform offline analysis. Instead, Pivot Tracing directly correlates events as they happen by piggybacking metadata alongside requests as they execute—even across component and machine boundaries. This gives Pivot Tracing a very low runtime overhead—less than 1% for many cross-component monitoring queries.

Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems

M.Sc. Project 2013

Revisiting End-to-End Trace Comparison with Graph Kernels

Jonathan Mace, Rodrigo Fonseca
Master's Project, Brown University, May 2013

Abstract

End-to-end tracing has emerged recently as a valuable tool to improve the dependability of distributed systems by performing dynamic verification and diagnosing correctness and performance problems. End-to-end traces are commonly represented as richly annotated directed acyclic graphs, with events as nodes and their causal dependencies as edges. Being able to automatically compare these graphs at scale is a key primitive for tasks such as clustering, classification, and anomaly detection. In this paper we explore recent developments in the theory of graph kernels, and investigate the feasibility of using a family of kernels based on the Weisfeiler-Lehman graph isomorphism test as an efficient and robust graph comparison primitive. We find that graph kernels provide a good formulation of the execution graph comparison problem, and present preliminary but encouraging results on their ability to distinguish high-level differences between execution graphs.

Revisiting End-to-End Trace Comparison with Graph Kernels

Experience

  • 2011 - 2018 (expected)

    Ph.D. Candidate

    Department of Computer Science

    Brown University, USA

  • Summer 2016

    Research Intern

    Facebook, New York

  • Summer 2013 Summer 2015

    Research Intern

    Microsoft Research, Redmond

  • 2011 - 2014

    MSc Computer Science

    Brown University, USA

  • 2009 - 2011

    Software Developer

    IBM UK

  • 2005 - 2009

    Mathematics & Computer Science

    MMathComp, 1st Class

    Oxford University, UK

Awards

  • 2017

    SIGCOMM Student Scholar

    50 Years of the ACM Turing Award Celebration

  • 2016

    Facebook Graduate Fellowship

    Pervasive Monitoring, Diagnostics, and Analytics of Distributed Systems through Dynamic Causal Tracing

  • 2016

    USENIX ATC "Best of the Rest"

    Invited speaker for Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems

  • 2015

    Best Paper Award

    Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems, SOSP '15

  • 2015

    Great TA Award

    Nominated by students of Brown CS138: Distributed Systems, Spring Semester

  • 2015

    Student Scholar

    3rd Heidelberg Laureate Forum

  • 2011

    Graduate School Fellowship

    Brown University

  • 2006

    Hertford College Scholarship

    Oxford University, Hertford College

Demos

Interactive Hadoop Visualization

Execution Graph Comparison

Execution Graph Clustering