Everyone knows what acceptable SIGGRAPH papers look like: just look in the proceedings. When one only sees the accepted papers and not the rejected ones, it is easy to get the wrong impression of what it is that SIGGRAPH likes and doesn't like.
I've submitted a lot of papers that SIGGRAPH didn't like, as well as a few they did. Also, I've been on the papers committee a few times and know what it is they look for. This note tells you something about what happens to your paper as it goes through the reviewing process as well as what people discuss when they're trying to decide whether to accept or reject your paper. I'll try to tell you everything I've learned about the SIGGRAPH secret: What SIGGRAPH wants, and how you can give it to them so they'll accept your paper. I'll also talk about some of the flaws in the reviewing process and how you can protect yourself against them. Finally, I want to share some thoughts on the present course and the future of technical papers for SIGGRAPH.
Before we do this, I would like to say why SIGGRAPH reviews are done the way they are. There are two reasons.
The first reason is the principal feature of the SIGGRAPH conference publication that makes it very attractive: speed. SIGGRAPH is one of the few high-quality publications that can publish a paper in less than a year. In 10 weeks, SIGGRAPH can do what other major publications take 10 months to do. In a fast-moving field like computer graphics, this is crucial.
The second reason is that SIGGRAPH has chosen a very different quality strategy than most other conferences. While other conferences will accept papers of incomplete work in progress, SIGGRAPH has chosen to shoot for the highest quality papers of complete results. Because of this, 80% of submitted papers are rejected. The MacArthur Foundation is more generous with its "genius" awards than SIGGRAPH is with its papers. There are more MacArthur awards each year than SIGGRAPH technical papers.
The emphasis on both speed and quality makes the reviewing process for SIGGRAPH very different from of a journal or another conference. The speed and quality emphasis also puts severe strains on the reviewing process. In a journal, the reviewer and authors can have a dialog where shortcomings and misunderstandings can be resolved over a leisurely pace. Also, even if there are significant flaws in a paper for another conference, the chances are that strengths will overcome the weaknesses in the judging. In SIGGRAPH, if the reviewers misunderstand your paper, or if some flaw in your paper is found, you're dead.
The reviewing process for SIGGRAPH is far from perfect, although most
everyone is giving it their best effort. The very nature of the process is
such that many reviewers will not be able to spend nearly enough time
weighing the nuances of your paper. This is something for which you must
compensate in order to be successful. But I'll get to that later. First,
let's talk about what happens to your paper.
First, you work for months, slaving away at equations, hacking code, and feeding slides to your local photofinisher. SIGGRAPH fever rises to absurd heights at the last week: "Let's see I only have 105 hours until the deadline....". You put everything together, accomplishing superhuman tasks to make the Federal Express deadline at the very last minute. Your six copies are taken by the courier safely to the papers chair, then you-and everyone around you-collapse.
The next day, as you and hundreds of other morlocks around the world come out of sub-basements to blink at the first natural sunlight you've seen in weeks, is deadline day. Fully 85% of the 200 or so SIGGRAPH submissions arrive at the papers chair doorstep. Everyone else has worked until the last possible minute, too. The papers chair and several dedicated assistants then spend their long all-nighter giving your paper a number, entering it into the database, and typing and mailing a letter acknowledging receipt of your paper.
Immediately after this, the papers chair, along with two or three others on the papers committee, sorts through all the papers and assigns your paper to the pile for a particular senior reviewer. The papers program committee is made up of 25 or so of these senior reviewers. With the large number of papers, this partitioning process takes a full day.
One copy of your paper is retained by the papers chair. One copy is mailed to the secondary reviewer, and four copies are mailed to the senior reviewer. Thus each reviewer receives a large Federal Express box of your papers and video tapes. This usually happens a week after the deadline.
The senior reviewers receive a set of 14-18 papers. For half of these they act as secondary reviewer and for half as senior reviewer. As senior reviewers, they look at your paper and choose three additional reviewers-at least two of whom are external to his or her institution. The senior reviewer sends a list of these reviewers to the chair within two weeks.
The reviewers then each receive a copy of your paper, slides, and video. The reviewer reads your paper, evaluates it, and fills out the review form that eventually makes its way back to you. He or she may fill out the hard copy or may email the review back to the senior reviewer. The reviewer has four weeks to do this.
After the senior reviewer gets each review of your paper, a review summary is made and a score is computed. Copies are made of the summaries and reviews. The originals are then Federal Expressed to the chair.
The chair tabulates all the scores, sorts your paper according to score, records it in a database, and prints out a set of custom lists for each senior reviewer summarizing all the papers.
The following week, the paper selection meeting occurs. This meeting, where the fate of your paper is determined, lasts for two full days. If your paper is on the very bottom or very top of the list, very little discussion is given to your paper (unless the senior reviewer wants a short discussion by full committee). This no-discussion acceptance/rejection eats away at the top and bottom of the list until the density of discussion slows the process.
Then a "triage" session occurs. During this time, the senior and secondary reviewers, as well as others who might share expertise in the subject area, discuss your paper. They then decide to accept, reject, or discuss your paper. If they decide to accept or reject, your paper will receive a short summary in full committee session. But let's say they opt for discussion.
Toward the latter part of the first day, the triage session is over and the real work begins. About 60% of the papers could not be judged easily one way or the other. Yours is among them. So the entire committee discusses each paper and decides its fate. Often the discussion is postponed while more people read your paper and discuss it with the other senior reviewers. These papers are then discussed, often over dinner.
The second day is taken up with full committee discussions of your paper. I've been in sessions when some papers have been discussed and then postponed and then discussed again for five or six times. There's a lot of argument, some shouting, photos are passed around, and the slides are peered at. Usually the videotapes are viewed during the breaks. For difficult cases, summary letters are written to you that described the final opinion of the committee. At the end of the day, consensus has been achieved on all submissions, and your paper is either accepted or rejected.
After that, the disposition of each paper is double checked by the entire committee. All materials that go back to you are collected, and all copies to be destroyed are collected. People say good bye and rush off to the airport. Some stay to help the chair to group accepted papers into sessions for the conference, try to make up some sort of silly theme for each session, and to assign session chairs among the senior reviewers.
The chair then takes the database and generates acceptance or rejection letters and packages it up with any additional material to be sent back to you. You find out whether your paper was accepted or not in about 10 weeks' time.
If all this sounds like a scheme to exercise Federal Express, you're right.
SIGGRAPH's Federal Express bills for this process run over $3,000. That
doesn't count your Federal Express bill, which in toto probably matches this.
The SIGGRAPH paper selection meeting is an intense experience that only a few dozen people have ever encountered. It is not coincidental that the same people who sit on the selection committee will author many papers that appear in SIGGRAPH year after year. This is not because they're part of the "in" crowd whose papers are given favorable treatment-I haven't seen anything like that the times I've been on the committee. There are two real reasons. The first is that the program committee members are all accomplished authorities in their respective fields-they tend to do good stuff. The second reason, though, is due to their experience as a papers committee member. In this, they do have an advantage over you, an ordinary author, who hasn't been among the chosen few.
The advantage these people have is that they know what it takes to get a
SIGGRAPH paper accepted. They know what the reviewers like and don't like.
They know what kinds of things get discussed in the selection meeting. In
short, they know the secret of what SIGGRAPH is looking for.
They discuss what the reviewers said in their answers and whether they believed the reviewers. They talk about their personal answers to the review form questions concerning your paper. They sometimes are absolutely positive, or other times may admit they're unsure. Often times they want other committee members to read your paper and form an opinion. Several people who are intrigued may volunteer and enter into a small separate discussion on the various points in your paper.
The questions on the review form change slightly from year to year, but the basic thrust remains the same. If you know the questions asked on this form, you'll be able to predict what the discussion topics will be in the committee meeting. Let's look at the questions and see what kind of discussion goes with each.
Your paper will get rejected unless you make it very clear, up front, what you think your paper has contributed. If you don't explicitly state the problem you're solving, the context of your problem and solution, and how your paper differs (and improves upon) previous work, you're trusting that the reviewers will figure it out. Don't try to make the reviewers dig it out from inside your paper. Maybe they will, or maybe they won't.
Again, stating the problem and its context is important. But what you want to do here is to state the "implications" of your solution. Sure it's obvious....to you. But you run the risk of misunderstanding and rejection if you don't spell it out explicitly in your introduction.
Well, to get rejected, pick a subject no one cares about. But, if your subject has less than obvious application to a wide range of graphics problems, you'd better figure out how to say it convincingly in your introduction.
On the other hand, a paper might have a really neat new idea. That idea may open up a whole new line of work. But the paper is badly written, and it doesn't really explain things enough so that someone without a Ph.D. in mathematical physics would be able to do anything with it. Because of this, all the reviews are bad. Someone says that one of the authors is a responsible person and will probably rewrite the paper into something decent. Someone else says that there's no guarantee that anything at all will be changed, then the proceedings will have this horrible paper in it: why not reject and wait till next year? Finally the committee votes, it passes by a narrow margin. Thus the committee has decided to gamble on the authors to fix the problems once they're pointed out. Sometimes the gamble pays off; sometimes it doesn't.
All this brings up a phenomenon that happens inside the paper selection meeting. Often a committee member may take up the cause of getting your paper "in" and argue for acceptance of your paper. Tom Sederberg, the SIGGRAPH 91 chair, has called the people who can ferret out the good features of your paper "paper champions." On the other hand, there may be a committee member who is very articulate, forceful, and negative, who argues against your paper. They look for and find flaws in your paper, they sway the committee to reject your paper. Ed Catmull, the SIGGRAPH 92 chair, has called these people "paper killers."
One job of the papers chair is to see that the committee is staffed with people who are paper champions. We want to avoid paper killers.
So that's it. That's what goes on in the discussion. I must admit that as a paper author I've been guilty of screwing up on almost all the points mentioned in the review criteria. My long string of paper rejects have been due to repeated deficiencies in not stating the problem or its context, not explaining why the subject is interesting, writing disorganized papers, and leaving out key points that I thought were obvious. And just writing stuff that was plain hard to read, so that some of the reviewers just missed my point.
How can you protect yourself against these mistakes? You must make your paper easy to read. You've got to make it easy for anyone to tell what your paper is about, what problem it solves, why the problem is interesting, what is really new in your paper (and what isn't), why it's so neat. And you must do it up front. In other words, you must write a dynamite introduction. In your introduction you can address most of the points we talked about in the last section. If you do it clearly and succinctly, you set the proper context for understanding the rest of your paper. Only then should you go about describing what you've done.
Another point is why rendering papers have an advantage in SIGGRAPH. If you have good-looking pictures, you've got your foot in the door. SIGGRAPH reviewers are like everyone else. They first look at the pictures in your paper. If your pictures are really good looking, they're going to go to some effort to find out how you did them.
You can use those pictures in another way. Ivan Sutherland once told me that Scientific American articles are constructed so that you can get the point of the article just by reading the captions to the illustrations. Now, I'm not suggesting that you write a technical comic book; but you should take a look at those SIGGRAPH papers you were initially attracted to and see how they went about getting their point across.
Unless you write about a very limited subject, or unless your results are technically incorrect, rejection has very little to do with the subject of your paper. It has a great deal to do with how you wrote your paper. After all, if everyone misunderstood your paper, you might consider that it might not be quite as clear as you thought. Reviewers are in a hurry: you have to get your paper just right or you will suffer rejection. Rejection doesn't come from the subject area, it really just comes from an imperfect understanding on both sides.
But on the whole, it's a very noisy process. The SIGGRAPH review is done
quickly, by the best people the chair knows, and by the best people they
know, with everyone earnestly committed to put out the highest quality
proceedings possible. Mistakes are sometimes made.
I fear that this trend is all too real. I'm very worried about it. I believe that the papers sessions at SIGGRAPH are in trouble. Only about 10% of the technical program registrants go to the papers sessions. Sometimes fewer than 200 people are in attendance at a paper session. This tells me that very few people find the SIGGRAPH papers interesting anymore.
For some years, people thought of the papers sessions as almost exclusively about rendering - SIGGRAPH as "SIGRay" or "SIGRadiosity." Or people have viewed the papers sessions as valid only for those papers that have been about "pure" graphics. Almost everyone agrees that the papers are the exclusive domain of the academics, exploring esoteric and obscure corners of graphics.
I believe that the reason for this alarming narrowing of SIGGRAPH papers is a dangerous positive feedback loop. You see, people can't see what papers are rejected. They can only see the papers that are accepted. Thus when you look at a proceedings you see a certain set of papers and you say, "Ahh,...THAT'S the kind of thing that SIGGRAPH wants." So, if you have an idea for a paper that isn't like the kind that have been appearing in SIGGRAPH for the last ten years or so, you wouldn't send it in to SIGGRAPH. You say, THIS is not really what they want at SIGGRAPH anymore, they want THAT. If you are brave, do submit to SIGGRAPH, and your paper becomes a casualty of the 80% rejection rate, you feel that SIGGRAPH really doesn't want your type of paper anymore. Thus you don't send anything in to SIGGRAPH about that subject again.
Well, the papers committee and the papers chair don't really determine what
SIGGRAPH publishes. The authors who brave the SIGGRAPH review process are
the real controllers of what appears in SIGGRAPH. The committee can only
select among the papers that are submitted. Consider this: if there are 150
rendering papers submitted, only two systems papers, one interactive
techniques paper, and no applications papers submitted, what will the
proceedings look like? Then everyone will say, "See, SIGGRAPH only wants
rendering." But what really happened is that SIGGRAPH "rejected" 127
rendering papers, and rejected only one systems paper, and didn't reject a
single application paper!
It is foolish to capriciously tinker with the speed and the quality of SIGGRAPH in the hopes one might fix the serious positive feedback focus problem. Frankly, I'm afraid to make big, sweeping changes in a process that works well a lot of the time. However, you'll note that this year there is a new class of papers: long papers. Only systems and applications category papers will be admitted in the long class. Andy van Dam has pointed out that the page limit for regular papers favors research papers. A research paper can usually state the problem, its context, and the solution in a short space. A system paper needs more pages to do this and it must also describe the experience that the builders have had with the system. Eight pages was just not enough room to write a decent system or application paper.
The root cause of the positive feedback loop, however, remains. It is self- censure. People just won't send in papers on subjects they think SIGGRAPH doesn't want. I can understand this: even after all my SIGGRAPH rejections, it still hurts.
The entire reason I've written this document is to try to break the loop. I want to communicate to you, and I want you to communicate to your colleagues, that SIGGRAPH is in the business of publishing good technical work in graphics of "all" flavors.
SIGGRAPH really does want papers about user interfaces, visualization, graphics hardware, graphics software systems, interactive techniques, displays, innovative applications, video games, combined graphics and sound, hypermedia, virtual reality, typesetting, color, paint systems, image and video compression, image and video processing, and how to make pictures that aren't just pretty but say something too.
Sure it's true that they've rejected papers in all these areas over and over again. But, it's also true that they've rejected 10 times as many papers on ray tracing. The narrowness of the technical focus of the papers can be fixed only if you and your colleagues send in quality papers about a wider range of subjects. My earnest hope is that the SIGGRAPH 93 technical papers program will not just be about modeling, rendering, and animation.