Organizing a Workshop
1 TL;DR
2 Basic Rules
3 Planning by Survey
3.1 Generic Survey
3.2 Specific Surveys
4 Talk Advice
5 The Icebreaker
6 Day Schedules (Dagstuhl)
7 Breakouts
8 Publications
9 Miscellany
10 Nurturing the Seedcorn
Wednesday, July 17th, 2019 10:27:11am

Organizing a Workshop

Shriram Krishnamurthi

    1 TL;DR

    2 Basic Rules

    3 Planning by Survey

      3.1 Generic Survey

      3.2 Specific Surveys

    4 Talk Advice

    5 The Icebreaker

    6 Day Schedules (Dagstuhl)

    7 Breakouts

    8 Publications

    9 Miscellany

    10 Nurturing the Seedcorn

Versions: 2019-07-17. 2019-02-11. 2015-09-21. 2015-02-20 (with extensive feedback from Michael Hicks). 2015-02-07. 2014-11-08. I’ve been fortunate to jointly run a few nice workshops (and a few weak ones, too). I’ve also attended about thirty ad hoc events whose organizers were charged with preparing a one-off schedule. I’m writing down some of what I’ve learned from all this. These instructions are especially designed for venues like Dagstuhl: small, focused, and enclosed, with high quality-control on who participates.

1 TL;DR

There is one overarching principle that covers everything else below: Take Responsibility. Perform the following calculation. Say 40 people are attending your event, at an average cost of $1000 per person (almost certainly a very weak lower bound). All told, a 5-day Dagstuhl is roughly a 4-day event. So the workshop is costing (probably mainly taxpayers somewhere) $10,000 per day. Does each day deliver $10,000 worth of value? When you can say “Yes”, you’re done planning; until then, you’re not.I perform a similar calculation every semester for my courses. It’s sobering.

It’s exciting to submit a workshop request. It’s even more exciting to get an acceptance. However, unlike a conference paper—where your work is mostly done at that point—the point of acceptance is when your work actually begins. Don’t bask in the acceptance for more than a day (if even that long); get cracking.

Good luck. I hope your event is a huge success. If you have comments about the following content, please contact me.

2 Basic Rules

Here are some important rules to keep in mind:
  • Remind people they are there to participate, not necessarily to speak. There’s an important difference. If you make the latter the focus, some people will simply show up, give their talk, and not get involved in anything else. Too many of these people and you end up with a room full of people reading email most of the time. If you make clear that they may not get to give a talk—but can still participate, usefully—it not only sets the right tone, it weeds out attendees who really had no interest in the event being a two-way street. Hopefully they won’t come at all.

  • At the same time, recognize that people want a chance to be heard. They may even need to formally be able to show this to get their trip reimbursed, etc. There are ways (The Icebreaker) to get most people to speak a little early on. Once they’ve had a bit of a say, they will be more attentive to what others are saying. Otherwise they will obsessively fine-tune their own slides, shutting out everyone else.

  • Control the WiFi. Tell everyone that you have the ability to turn it off, and will do so if you find that too many people are spending too much of their time reading email. However, don’t turn it off by default. Not only are they adults, they may have perfectly legitimate reasons (e.g., downloading a paper corresponding to a talk they just heard); besides, if sufficiently many of them are bored, you need to get your event back in shape first.

  • Don’t operate out of a mistaken sense of “fairness”. Not everyone needs to get the exact same amount of time to speak (with zero being the limiting case of this). Fit their topics to the audience. Within limits—that is, without abusing power or going overboard—remember that you are more like a creative director or composer, not a mere bureaucrat dividing out minutes with emotionless arithmetic. It is perfectly fine to have everything from five minute to one hour talks, in various distributions. In fact, mixing things up a bit can keep an event more lively.

  • Take into account travel and time differences. We schedule events as if our bodies can magically take the abuse of a ten hour flight, jet lag, dehydration, and so on, and function exactly as they would at home—and then ourselves suffer from the consequences. If you have significant populations crossing many time zones, think about when they are likely to “crash” and how the schedule can accommodate that. For instance, don’t put an hour-long talk at that time; have lots of short talks, or a poster session (which lets people walk around), or even a group activity.

  • Give people time to exercise or seek other outlets. At Dagstuhl, in particular, there is a tendency to run events all the way right up to meal times. For people who want to do something physical, doing so after a meal may not be easy. At Dagstuhl, stop the day at 4:45pm so people have time to return to their rooms, change, exercise, go back, shower, and still make it to dinner on time.

  • Leave some time for people to be on their own. It’s a sad fact of our lives that we’re flooded with email and other interruptions that seem to demand constant nurturing. If you leave some room in the schedule, you can convince people to put off their emailing to those times. If you don’t, they will eventually feel this intense pressure to ignore the event and switch to their mailbox.

3 Planning by Survey

My style is to run a few quick surveys to plan the event. Keep the surveys brief to maximize participation. If you run them too early, nobody pays attention; if you run them too late, you give yourself no room to make adjustments or plan creative events (e.g., inviting certain people to prepare talks that the audience wants to hear).However, don’t put too much faith in survey results; use your judgment too. I have sometimes scheduled sessions that I thought would be dreadful but that got the most votes, and by half-way in the whole room wished the session would end. Starting about three months out seems just right, because by then your attendee list will also have settled.

To run a survey, you may find it convenient to use various free on-line tools rather than doing it all by email. I have found SurveyMonkey to be perfectly good for most purposes.

3.1 Generic Survey

This survey is the most important component of designing your workshop. Keep the message focused so the recipients don’t lose sight of its purpose: you want as many people as possible to respond. Remind them of the purpose of the workshop, and then ask:

  1. Your name:

  2. What would you like to hear about for 5-10 minutes? (use generic topics)

  3. What would you like to hear about for 30 minutes? (use generic topics)

  4. What topics do you think you can speak about for 5-10 minutes?

  5. What topics do you think you can speak about for 30 minutes?

  6. Do you have suggestions for things we should do at the workshop (besides keeping time free for people to mingle and to organize spontaneously)?

  7. Do you have suggestions for things we should not do at the workshop (besides not cramming it full of talks)?

  8. Any other thoughts/comments/suggestions?

You will need to cluster their responses to find both affinities and outliers. (Neither is more important than the other; in particular, the outliers may be very specific people and interests that are not more widely known, and maybe you should give them a little more time than you do those that are already central to the area.) Naturally, people won’t use the same generic terms, so you’ll have to use your judgment and expertise as you classify and cluster. Knowing who they are not only helps you figure out whom to call on for what purpose, it also lets you email them to figure out their intent behind some of these answers.

Michael Hicks suggested another way to think about these questions. You could instead ask questions like:
  1. What are things you feel prepared/comfortable talking about?

  2. What are topics on which you would like to get feedback?

This is an orthogonal dimension of questions to the one above. The important division the above two questions get at is: the former is about the audience learning from the speaker, while the latter is about the speaker learning from the audience.

You will spend a lot of time processing the output. The quality of your event will depend heavily on how much attention you pay to these responses.

3.2 Specific Surveys

Keep in mind that there are hidden interests not revealed by the above survey. For instance, there may be many people interested in the topic of exploding pumpkins, but few thought to mention it. But when you put it on a menu of options, it may percolate to very near the top. Therefore, you should view the clusters you formed above a prompts: now you present them all to the group and gauge what their interest level is in each one.

There are various other specific, topical questions you might wish to ask. For instance:
  • You might want to organize some debates; for each debate topic, find out whether there are people with strong views on it, people willing to take a lead in a discussion, and whether people even want to hear a debate on it.

  • You may have more topics than time, in which case a survey can help you identify which events you can most afford to have run simultaneously.

  • You might invite people to submit specific puzzles or challenge problems. These are often best done as two brief sessions with some time in-between: a brief presentation early in the week to present the problem, and a longer session later in the workshop to discuss solutions, approaches, etc.

  • You could even envision an entirely different workshop structure. David Eppstein says:

    “The workshops that I’ve been most satisfied with have […] been focused around doing research: collecting problems near the start of the workshop (maybe with a small number of overview talks) and then spending the rest of the time in dynamically changing working groups trying to solve these problems (with periodic progress reports and the usual social activities).”

    You might be able to kickstart such a process with a preliminary survey (or at least use a survey to determine whether this style of workshop is feasible with the attendees you have).

And so on. So long as you keep your subsequent surveys easy to take (few questions, all can be answered with a check or click, comment boxes are present but optional), people will respond.

4 Talk Advice

Inevitably, a fair bit of time at your workshop will probably be spent hearing talks. This isn’t inherently a bad thing. At a one-off event like a Dagstuhl, this is a rare opportunity for a group that might never congregate again to hear ideas from outside their ambit. So don’t excessively undervalue talks in favor of other snazzy activities.

That said, it’s important to get good talks. Therefore, the most important thing you can do is to encourage teaching. Many of your participants are likely academics. They teach, and some probably teach very well. When they teach, they likely often do it by writing on a board.I believe slides are the enemy of good teaching, but that’s a topic for another day. Put them in a “conference stage” setting and they become all stiff, formal, and one-way; put them in a “front of the classroom” setting and they become loose, casual, and bidirectional. (You, Dear Reader, may even recognize yourself in this remark.) Encourage them to skip slides entirely and use the board. Chances are it may have simply never occurred to them to do this, and even if they don’t take you up on your suggestion literally, it will get them to think harder about how to present their material.

The next most important characteristic of quality is to avoid canned talks. The speaker’s most recent (or old) conference talk left completely unchanged is usually a pretty bad candidate. You have to get them to spend a little time customizing their talk for the venue.

In particular, tell speakers to edit out the obvious material. (You’d be surprised how many need to be told this.) For instance, don’t come to a Web security event in 2014 and begin with a whole slide on the importance of JavaScript. Don’t come to a programming languages event in 2014 and begin with a whole slide on the rise of Python. Don’t quote Gartner reports. Assume that your audience has basic knowledge and smarts.

As a stick,There is some disagreement about the precise meaning of carrot and stick: are they two different things (an incentive and a punishment) or two parts of a single system (a carrot dangling at the end of a stick before a donkey: a seeming incentive that can never be attained)? I think of it as the latter, but the former use seems to have won out in practice. threaten that the entire room will hoot and holler if they put up this kind of “throat-clearing” material. As a carrot, offer to work with speakers to figure out what needs to be said and what doesn’t, and really do spend some time with them on this. Nobody should even notice the missing content; that’s when you’ve done your job.

Be on time. It’s so obvious it barely needs stating, but it’s important. Few things agitate people as much as a sense of unfairness. An early speaker in a session who takes too long is being unfair by putting the squeeze on the later speaker. It ruins the latter speaker’s talk, which may have been carefully tuned.

Marc Herbstritt pointed me to a Dagstuhl timer. This could be useful, but I warn against being too rigid. Sense the temperature in the room and, if a reallly good discussion with broad participation is going on, it’s okay to let things run over a bit. This is, after all, a workshop; the whole point is to explore things. Therefore, build slack into the schedule (if you don’t need it, well, nobody ever objects to finishing a few minutes early). Remember that you’re a conductor, not a metronome.

Tell all your speakers to have their talks ready before the event begins: that they may need to be able to go on very short notice. (If they’re still working on their talks there, then they can’t be paying very much attention to other speakers, can they?) This way, you can accommodate scheduling changes better by moving people around dynamically: e.g., if your last speaker of the session had 15 minutes but earlier talks ran over (with your permission) by 8-9, grab a 5-minute speaker from later in the event and move them forward; after the session finishes, rearrange things.

There are two corollaries to being on time. One is to know how you’re going to keep track of time. Make sure all session chairs have some means of timing. The other is to select session chairs whom you trust to appreciate and enforce this matter.

5 The Icebreaker

One of the problems that vexes (or ought to vex) organizers is how members of the group introduce themselves to one another. Since not everyone will necessarily give a talk, and even if they do it might not be until late in the week, that’s not a very good option. What else to do?

A staple is to start with a half hour when everyone stands up and introduces themselves, a ritual for which there are no known fans. I can’t make out half the names of the people. Second, some people go on for too long, and you can sense the tension in the room as this happens. Third, if you’re actually paying attention, about ten people in it’s pretty hard to keep track of the endless stream of content. Finally, the introductions are either perfunctory or they sap energy. (If you are going to do this anyway, ask people to pronounce their names and affiliations slowly and to keep their self-descriptions focused. This is a time when the anxious will try to enumerate every single thing they have ever done.)

Fortunately, there are at least two good alternatives.

  • Michael Hicks suggested giving everyone a chance to stand up and introduce themselves accompanied by a single slide. This is akin to the brief presentation of a day’s talks that is done by many conferences. This does require the organizer to do the work of assembling a slideshow. However, it addresses many of the problems that plague the traditional introduction: you can see their names, you can see in words things that may be hard to understand spoken (which especially helps people who are non-native speakers of the workshop’s language), and the organizer gets to control time.

    A useful technology solution is a shared slide infrastructure like Google Slides. While it doesn’t produce the most beautiful artifacts, it has lots of other useful characteristics: everyone can edit simultaneously, no need to worry about merging, it can be run off several machines easily, etc. Do tell people to minimize special effects, since they may not be standing at the machine during presentation and hence may find it hard to control (though people also instinctively line up in order to go in front of the room when their turn comes).

    The main innovation we used in Seminar 19281 is pre-preparing a note sheet that everyone could use, which looked like this (credit: Jan Vahrenhold). This was distributed both on paper and electronically, and the names were organized in the same order as the slides. This made it possible to take quick notes on each person without spending time writing down their name. The only change I’d make is to add a “name” box to the top of the sheet: that way, if you lose your sheet, someone who finds it can return it to you.

  • The organizers of Seminar 15062 (and possibly of other events, too) did something better. They put aside the last chunk of Monday afternoon for a poster session. People either brought posters or made up ones on the spot (including hand-written ones). Everyone was asked to put their photograph on the poster, in case they weren’t there when someone else came around. I liked this.

    More recently, I’ve run a workshop devoting multiple sessions of the workshop just to posters—in lieu of any individual talks. Attendees were split into groups of 10-15 each, and each group got one session. Therefore, each 80 minute session would have 10-15 presenters and 20-30 audience members. This gives everyone more time to interact, gets everyone to learn about (almost) everyone else early on, and reduces the amount of poster space needed (and hence crowding). I generally heard nothing but good things about how this went.

    Of course, posters are clunky to travel with. There are several alternatives. One option is to just make slides with the essential content and print those and pin them up: it’s lightweight, and you could even print at Dagstuhl itself. Even better, take a sheet off one of the easel pads available at Dagstuhl, and hand write/draw your content. Or you could instead use a laptop to run a demo. Whatever: It’s your space, be creative!

Note that you can use both ideas. Seminar 19281 had a first session with slide-based quick self-presentations (80 seconds/person, strictly enforced so that we didn’t run over). We also had poster sessions. To avoid having too many people in each session, we had multiple poster sessions, and each one was preceded by a short tutorial talk (20 minute talk, 10 minute q&a, 45 minute poster session).

6 Day Schedules (Dagstuhl)

Some of this material is just common sense, so you should be able to adapt it to other venues than Dagstuhl.

If you’re running a five-day Dagstuhl event, you have sixteen 1h30m sessions at your disposal: four each on Mon, Tue, Thu; two on Wed; and two on Fri. That’s the theory; now let’s look at the practice.

First, you don’t have 1h30m; you actually have about 1h15m per session. If you get the arithmetic wrong, that’s a lot of minutes to make up.

You should assume that people will start drifting away on Friday: at Dagstuhl this could be Europeans who can drive home, Asians and Americans who want to take a midday flight, etc. In general, it makes sense to wind things down by around 11am on Friday, let people socialize for an hour, get to lunch in leisure, and leave.

The evenings are a tricky time. Sometimes it makes sense to organize an evening event (such as a software demo), but make sure it’s optional and labeled explicitly as such. Younger attendees, especially, may not realize what is okay to skip, and as a result may both deprive themselves of great social interactions and end up exhausting themselves before the week is over.

Try to have a mix of events. Some will be sessions of talks, some might be debates, some may be brainstorming activities. Try to avoid the generic “breakout session” without a clear plan of what you’re breaking out about. (Read more about Breakouts.) This doesn’t have to be a goal—the main purpose should be to talk, not to produce output—but you should know what the subject is going in (even though it may not be the subject coming out).

Don’t over-schedule the week. I suggest having a schedule that runs at most through lunch on Thursday, but with the understanding that everything from Wednesday onward is flexible. By Thursday if you haven’t found things people want to talk about more, something’s wrong with the event. If you have, then you have plenty of time to figure out how to structure Thursday afternoon and Friday morning. Use the earlier part of the week to accumulate topics that people might want to talk about later. Don’t forget to schedule the The Icebreaker.

At the same time, don’t under-schedule the week either. The very worst kind of event is one where organizers say they will figure out the schedule “at the workshop”. To me this is just a euphemism for “Despite being called organizers, we couldn’t actually be bothered to organize.” These events are often terrible unless the organizer has some brilliant plan for how this will all come together (which hasn’t yet happened on my watch).

If you’ve done your job, you have various other organizational tools available. For one thing, you could introduce people by clusters. Another option is to have just those who aren’t speaking in the first two days give a very brief summary about themselves, but let their talks speak for the others. I suggest doing this not at the very beginning of the first day. Instead, get straight to something with content and interaction.

7 Breakouts

A staple of these events is the dreaded “breakout” session. Breakouts are great, but they can also be terrible. At their best, they let people with shared interests dive deep into a topic. At their worst, they split the event, cause people to just congregate with their community and ignore the mixing that may have been the point of the event, and can be lengthy rambles without a point.

Here are a few things that greatly improve breakouts, making them a joy rather than a pain:
  • Appoint a leader, or ask each session to begin by appointing a leader. They need to keep the conversation moving, recognize people who haven’t spoken much, and keep the session focused on its topic.

  • Ask each session to vote at the end on whether it wants to continue or not. People have a tendency to go to Abilene; an explicit vote forces the question that should have been asked.

  • Do not report back immediately. The bane of breakouts is the “report” that is a hastily-assembled set of slides—almost always bullet liststhat simply describe what happened. (I’ve been guilty of this too.) Everyone nods through it, remembers nothing, and learns nothing.

    Why does this happen? Because there’s not enough time. Reporting requires coming up for air, reflecting, distilling, asking “What did we learn that we want to pass on?”. So make sure there’s enough time between when the breakout ends and reporting begins. At Seminar 19281 we made sure there was at least a 30-minute break, ideally a lunch or even evening, between session and report. The reports were thoughtful, memorable, educational, and even visually attractive.

8 Publications

You will probably be expected to write a report. A useful technique is to ask the graduate students and other young scholars to serve as scribes. (Of course, don’t over-burden them so they can’t participate themselves.) It has sometimes been effective to assign a pair of students—a master scribe, and a reader—to each activity (session, breakout, etc.), with the scribe responsible for producing the drafts and the reader reading drafts to help improve them before they percolate up to the organizers.

Beyond such a report, unless you have to (or that is your express purposeAn obvious exception would be a Dagstuhl Perspectives type of event.), don’t make the event be about producing a document, such as an LNCS issue. If you are going to, make that amply clear up front—at invitation time—so those who are not interested in such an activity know not to attend. And then, organize your entire week around the production of these documents.

It’s tempting to ignore this last bit of advice and try to have it all. Remember that it’s very easy for people in the heat of the moment to salivate at the idea of yet another publication and say yes to just about anything you ask for; the odds of actually getting something out of them once they return to their routines may be close to zero.

It’s okay to not have a written record. It’s okay. It really is. Science is a social activity, and we should be honest about the value of that socialization instead of having to always cover it up with reports that might be read fewer than one time.

9 Miscellany

Choose the location of sub-events well. For instance, it appearsThanks, Barbara Ericson. coffee rooms are bad for breakouts because people keep coming through wanting coffee, which is distracting in several ways.

10 Nurturing the Seedcorn

Pay attention to younger members. Know who they are; know who’s vouching for them. Make that person do some work to prepare and nurture this person. (If they’re someone else’s advisee and the advisor won’t be present, that mentor may very well be you. As the old line in a slightly different context goes, if you’re looking around the room to figure out who it is, you’re it.)