Abstraction Merchants Revisited
A Response from Scott Shenker
Four weeks ago Larry commented on an ongoing debate in the networking research community about how to improve SIGCOMM’s conferences. His post, entitled Abstraction Merchants, specifically referred to Scott Shenker’s CCR article Rethinking SIGCOMM's Conferences: Making Form Follow Function. When Scott approached us with his response, we thought it was a good time to try the first-ever guest post in Systems Approach. You can read his reply below with some concluding thoughts from Larry at the end.
I appreciated Larry’s generous and thoughtful commentary on my CCR editorial, and I agree with much of what he wrote. However, I was surprised by his last paragraph where he suggests we should first reach consensus on how we should review nontraditional papers and then trust that our collective actions will follow suit.
My takeaway is that focusing on the “mismatch between… SIGCOMM's goals… and what our current practices achieve” will not be fruitful until we are certain we understand our goals. Crafting a policy about how reviewers should evaluate papers describing new abstractions, new architectures, and new systems will not make a difference unless the community truly values that work, even when (1) it’s difficult to identify an immediate path to deployment or imagine the associated business model, and (2) it falls outside the traditional boundaries of what is considered to be in-scope. If we can reach consensus on this, then I believe the form (practice) will follow.
At the outset, let me be clear about where Larry and I agree and disagree. We agree on the need for change in SIGCOMM’s conferences. We also agree on the need to engage in an ongoing discussion about our goals and how to review the kinds of papers Larry refers to above. Where we disagree is that I don’t see that conversation ever reaching consensus, nor do I think consensus is needed for achieving significant change. In the next few paragraphs I delve more deeply into why I believe in this non-consensus path for change, because I think this exchange between us raises the crucial question of how to achieve change in large and diverse communities such as SIGCOMM. My reasoning is built on four basic points.
(1) Inclusion, consensus, and change: pick two. As individuals and as a community, we in SIGCOMM believe that (i) it is important to include a wide range of voices in SIGCOMM’s deliberations and (ii) policy decisions should be driven by community-wide consensus. We also, at least in the past decade or so, have come to believe that we need to change our conferences so that they better support SIGCOMM’s intellectual goals. Unfortunately, achieving consensus in any large organization is hard, especially in a community as diverse and loosely coupled as SIGCOMM. For instance, several people strongly objected to the statement that the research goal of SIGCOMM was to “lay the intellectual foundation” of our field. I thus do not think that achieving consensus should be our initial goal, because we are almost certainly doomed to fail. Instead, we should focus on inclusion and change, sidestepping the need for community-wide consensus; I will return to this in my fourth point.
(2) Our reviewing practices are shaped more by our experience and our sense of community norms than by vague exhortations about how we should review. It has been widely observed that when you ask graduate students to review papers, they are often sharply if not brutally critical. This is understandable because seeing the overly-negative reviews of their own papers is their only contact with what the community expects from reviewers, and they are merely following what they perceive as the community’s norm. SIGCOMM has provided guidance for reviewers, but this guidance has been widely ignored for over a decade. While I think we, as a community, should emphasize this guidance more than we do, and be engaged in an ongoing dialog about the nature of such guidance, we should not fool ourselves that such vague directives by themselves will have the impact we hope for.
(3) Change is hard, and the alternative is the status quo. Creating change requires hard work, and merely registering a disagreement while remaining on the sideline is essentially a vote for the status quo. Over the past year, my three intrepid collaborators – Fabian E. Bustamante, Nate Foster, and Aurojit Panda – and I have talked with dozens of people in SIGCOMM (actively seeking out those who disagree with us), surveyed the community for input, and written thousands of words articulating our thinking. Our writings have described both some general principles and some specific incremental steps forward. You can see these efforts documented here and on the SIGCOMM Slack conference channel (which also contains valuable input from many others, including this report). Our process started with a fairly radical set of principles and ended with (i) a specific incremental proposal (described and later clarified) and (ii) a general process for ongoing change (which was developed in collaboration with the SIGCOMM Executive Committee). While our goals remain ambitious and far-reaching, we settled on a concrete first step and a process that could support ongoing progress, because we thought the former was the most we could achieve in the short-term, and the latter provided a solid foundation for future evolution. We greatly appreciate those who, while not agreeing completely with our ideas, were willing to lend their support to these efforts. From this I have learned the following lesson, modeled on Jon Postel’s famous advice: Be conservative in what changes you propose, but liberal in what changes you’ll support.
(4) Empowering positive voices may be sufficient. Larry and I agree that we need to value papers that describe new abstractions, new architectures, and new systems even if they are hard to evaluate and inconsistent with current practice. You might naturally ask: How might our proposed set of actions cause the community to value such papers? Our community norms of how to review have been strongly shaped by how others review, and by what papers are accepted by the conference. Seeing the negative reviews of papers that fall outside the current norms, and their high rejection rate, has led many to believe that they shouldn’t be accepted, and I don’t think we can change people’s minds by telling them otherwise. Instead, we want to empower the current minority of reviewers who value such papers via our proposal (alluded to above) to accept papers that have a single explicit champion. This will change the kinds of papers that people see at SIGCOMM conferences, thereby demonstrating that there are those in the community who value this kind of work; this will hopefully lead to more people writing such papers, and more reviewers willing to advocate for such papers. I believe this gradual process of changing perceived norms, without reaching explicit consensus, is the path to change in organizations like SIGCOMM. Note that this approach is not restricted to accepting a particular set of “nontraditional” papers, but instead leads to a greater diversity of papers; any set of papers that has at least a minority of voices in favor will receive a more favorable hearing from program committees. This is crucial for SIGCOMM, because I think our conferences should broaden our perspectives, not just reinforce our current preconceptions. On that, I think we can all agree.
Larry’s Closing Thoughts
I like Scott’s focus on how to make change happen. Talk is cheap, and his attention to the process is something I can get behind. If the consensus I called for is manifest in concrete proposals for how SIGCOMM improves its conferences, then I’m happy. I do think this process is complicated by three things I talked about, which we need to acknowledge and address head-on. One is the breadth of methodologies we apply, which, to oversimplify a bit, range from theory to practice with plenty of ground in the middle. A second is the range of opinions about how we measure impact, which (to oversimplify again) range from changing perspectives to changing protocol specs. A third is the diversity of values we bring to the table, which is especially problematic when we conflate value with quality. As I wrote, I place a lot of value on the synthesis of complex systems that lead to new abstractions. But that’s just me. Others put their emphasis on policy questions, economic factors, and societal impact (to name a few). I’m not sure how to best reconcile all of that, but I’m encouraged by the fact that SIGCOMM is willing to try.
As we continue to follow the AI/ML space, an article on the poor performance of ChatGPT on an Algorithms exam caught our attention (and frankly didn’t surprise us). Thanks to user solstice333 on Github, we fixed a bug in our main textbook that had lurked undetected for 28 years–score one for open source publishing. Thanks also to Richard Clegg who sent us this video about the history of putting compass points other than North at the top of maps. Finally, don’t forget to do your bit for Internet decentralization by joining us on Mastodon or Peertube.