Sunday, May 20, 2007

Open Access Computational Linguistics

Hal's opening shot has many interesting responses in the comments, and my blog reply also yielded interesting comments. Instead of commenting on the comments individually, here are some additional points:

  • I understand that Computational Linguistics and the other ACL publications are distinct, but the ACL owns Computational Linguistics and could change its operation if it so wished.
  • Open access Computational Linguistics seems a good idea, but...
  • Open access by itself does not solve the critical problem with our field, which is the poor quality of reviewing for conferences, and the slow reviewing for journals.
  • These two problems are linked. Reviewing resources tied up in conference reviewing are not available for journal reviewing. Journals get even slower, authors give up on journals, submit more to conferences, and conference reviewing gets even worse as a result of the increased load.
  • Conference reviewing can never be as good as journal reviewing because it is done under time pressure, because it does not allow for careful editorial consideration, and most of all because it generates a surge of demand over a short period that cannot be met by competent reviewers.
  • Journals spread their load evenly all year round, and revisions or resubmissions with detailed answers to previous reviews are possible (and desirable).
  • Journals have an institutional memory, including a memory of reviewer quality and timeliness that conferences do not have with their annual change in program chairs.

An open access Computational Linguistics would also to be a fast turn-around journal with quick and effective editorial control. After getting a paper through PLoS Computational Biology in eight months from initial submission to publication of a revised version, with two rounds of extraordinarily helpful reviews and editorial comments, I cannot accept anything less for our field. However, I don't see that this can happen if current reviewers move their reviewing efforts from conferences to journals. It is evident that we are scrapping the bottom of the reviewing barrel at the moment, from the many experiences we have all had with uninformed, sloppy, and cursory conference reviews. High-quality journal reviews take time, easily several times longer than typical conference paper reviews. Something has to give.

However, if we had an efficient open access journal, we could easily create an excellent conference without extra work. All of the papers accepted for publication in a given year would be eligible for presentation at the conference. Of those, the editorial board would select a subset of nuggets for plenary talk presentation. The rest could be presented as short talks or posters. We would have all of the social advantages of a conference with the higher quality of a well-edited journal.

I hear that some faculty oppose ideas like this because it removes the conference deadline pressure from their graduate students. I say to those faculty that if deadlines is all you have to motivate your students, you are in deep trouble. Conference deadlines are terrible for quality. All of my worst published mistakes are all in hastily written conference publications, and they were not caught by the deadline-driven reviewers.

5 comments:

Julia Hockenmaier said...

Interesting model for a conference! Has anybody actually implemented something this? I'm just aware of the reverse concept, where the accepted full papers get published in a special issue of the society's journal (I'm thinking of ISMB. Looking at their stylefile, they seem to require longer papers for conference submission than ACL, which might in itself be a good deterrent for many of the weaker papers).
But since many people can only justify their travel expenses if they are actually presenting, it might be necessary to have something like a poster session (this could also be done without published papers, like ISMB), where people could present their work in progress, or smaller projects. Otherwise, the conference might turn into too much of an in-crowd event.

chris brew said...

It is the availability of "revise and resubmit" which encourages authors and reviewers to spend effort on making good work better. This is the main deficiency of the ACL style, since an article is accepted or rejected irrespective of whether the author makes the suggested revisions. So only the more conscientious reviewers spend time on making suggestions. People inevitably complain about dubious or uninformed rejections more than they complain about this, but it is at least as important.

I wonder if there is a place for a conference in which nothing is accepted outright, everything is either a reject or a revise and resubmit. We could keep it small, and make the deadline September. Does this have any advantages over the OA journal?

Fernando Pereira said...

@JH: I don't know of any conference on the model I proposed, but I know of discussions on such proposals in other areas of computer science, although none has been implemented so far.
@CB: I don't see the need for a deadline. In my proposal, everything that has been accepted for the OA journal between certain dates is accepted for presentation in the conference as a talk or poster. NIPS has this format, and everything there is reviewed, although the reviewing is the traditional deadline+rush process instead of the more spread-out process I proposed.

hal said...

I this this model makes sense, but something concerns me: "I don't see the need for a deadline. In my proposal, everything that has been accepted for the OA journal between certain dates is accepted for presentation in the conference..." -- it seems like this might encourage exactly the same deadline-hounding as the current system. Unless there's significant variance in the OA acceptance, which is of course not good :).

Fernando Pereira said...

@HD: There is no connection between submission date and appearance in my proposal, because papers may have to go around the reviewing loop several times, as they do typically in reputable journals.