Welcome! Please contribute your ideas for what challenges we might aspire to solve, changes in our community that can improve machine learning impact, and examples of machine learning projects that have had tangible impact.
Lacking context for this site? Read the original paper: Machine Learning that Matters (ICML 2012). You can also review the slides.
Restructure Journal/Conference Roles
  • A number of people have griped about ICML and about CS conferences in general.  I won't repeat the whole argument here, but the gist is that the review cycle penalizes certain kinds of work and reinforces certain idea sets at the cost of others, and the conference length/venue size imposes a rather artificial ceiling on the number of accepted papers.  I would like to call for ML (and CS in general) to move to the journal-centric system like all other academic disciplines.  (They have already wrestled with these problems, and solved them, so we should follow their lead, rather than reinventing that wheel.)

    As a specific suggestion on how to get there, I propose:

    - Keep ICML (and other ML/KDD venues) as dual presentation/poster track
    - Accept presentations from a selection of the accepted papers to the corresponding journal(s) over the past year.  (E.g., pick presentations for ICML from JMLR and MLJ.)  Responsibility for reviewing and curating the journal articles remains with the journal editorial board; responsibility for selecting for presentation at the conference remains with the conference program committee.
    - Direct submissions to the conference become extended abstract (1-2 pages) for poster presentation, and almost all are accepted (low bar).
    - Journals should be prepared to accept "short" papers (on the order of a current ICML/KDD conference size), without penalty of comparison to the current 30-50 page mega-tombs that they often accept.

    This would move the prominence of publications to journals, where it belongs (and where proper reviewing can be done), and still keep the role of a conference for early dissemination of preliminary results and a place to meet/network/discuss.
  • 11 Comments sorted by
  • I have to disagree with Marie's position.  If there ever was a baby in this bathwater, it has long since grown up and moved out of the house; only the cold, scummy bathwater is left.

    Yes, we've spent decades trying to convince colleagues in other disciplines of the value of our conference papers, and we've made some good inroads.  But it's a continual fight -- we have to keep re-convincing them.  (I seem to have this conversation with some other scientist at least once a semester.)  The effort spent on that persuasion is now sunk cost -- we should not fall into the economic trap of continuing down a bad path, simply because we've invested in it.

    Kiri's experience is hardly unique.  I have personally experienced this in two different disciplines (biology and neuroscience) -- I've submitted papers and had them reviewed, responded to reviews, had the acceptance, and had the online version appear, all in a shorter time than from ICML or AAAI paper submission to conference.  I think that a lot of fields are aware of the problems of long lag times, and are working hard to correct them.  I don't think that we in CS are any less capable of driving our journal review times down than they are.  (Indeed, they're using a lot of the information processing tools that *we* invented to make their cycle times short!)

    I agree that we need to fix the journal review cycle times.  But I don't think that many CS researchers are really going to do what's necessary to make that happen until we shift emphasis away from conferences.  As long as conferences are perceived as being the preeminent venues, that's (naturally) where people will put their effort.  It's a bit of a chicken and egg problem.  But I don't think that that's a reason to not take action on it.

    With respect to the issue of applications research: I agree that a separate submission track, with clear criteria and identified reviewers, is probably the way to go for applications.  However, I do think that the conference system has systematically hurt applications papers in ML.  Application papers almost inevitably involve educating the reviewers about something outside the field of ML, which (a) cuts into page count and (b) is unfamiliar and, therefore, tends to arouse "alien, therefore suspicious, therefore reject" kinds of gut reactions.  The first can be helped, somewhat, by using the power of electronic proceedings to allow longer papers or more background or something.  But the second can only be helped, I think, by providing for more back-and-forth discusion between authors and reviewers.  Further, the alien-therefore-reject instinct is exacerbated by the zero-sum mentality that conference acceptance ceilings encourage.  Again, a journal, with no upper bound on the *number* of papers that are accepted, doesn't have that factor in play -- my voting to accept a paper into the journal does not penalize my own paper's chances of being accepted.

    A rolling deadlines option for conferences is an interesting idea, but it feels like an attempt to patch more and more journal-like behaviors onto conferences.  (Just as the current "author response phases" that a number of ML and CS conferences have adopted are a poor approximation to the full discussion cycle that you can have between authors and reviewers in a journal process.)  Journals have already *solved* these problems -- why should we keep re-inventing these wheels?

    The goal, I think, is to improve the quality of science that gets published and to make sure that good science has an outlet.  I'm not convinced that the deadline-oriented mindset that conferences tend to encourage really promote the first goal, or that the artificial quantity limits that they enforce encourage the second.
  • Ultimately, what purpose do we want our conferences to serve?  The primary strength of conferences (over, say, journals) is that you bring people face-to-face and enable the kind of discussions and collaborative sparks journals will never foster.  Conference talks are (at least right now) so short that what's conveyed is almost never a deep technical contribution -- they function instead as a teaser advertisement to interest you in reading the paper, talking to the author at their poster, following up by email, etc.  Note that these purposes can ALL be served even in the absence of an actual paper associated with the conference talk.  Presenters could refer to their (more substantive) journal publications for deeper details.  

    Thinking along these lines makes me wonder if we've tried to force conferences to serve an unnatural purpose, especially given the review time and paper space constraints.  (I agree however that the latter is vestigial given electronic proceedings.)

    All that said, Marie's also right that this doesn't directly address the "publishing about ML impact" issue.  But thinking broadly about different formats and organizations of how we share results within the field might spark some relevant ideas!
  • Yes!  This would be a radical change for the community, but I think the end result would be higher-quality work for the field as a whole, and less "deadline-junkie" behavior that we all (sigh) manifest.  The current push towards including author-response phases with conference submissions has the right motivation, but is trying to squeeze too much into a very constrained timeframe, and doesn't really allow for the kind of back-and-forth refinement that leads to truly polished and substantial contributions.

    Another benefit from accepting shorter papers to the journals is also that we should expect review turnaround times to go down.  I've submitted papers to the Astrophysical Journal and had a review response within *a week* (and apparently that's typical).  Can you imagine?
  • YES!  How do we accomplish this?  I publish in meteorology now too and this is exactly how it works there.  I far prefer it!
  • I actually don't agree with this.

    First, I genuinely don't understand how this would address the fundamental problem posed by this forum, that of lack of acceptance of applications-related papers.  It's not like you can get an applications paper into MLJ or JMLR.

    Second, while Kiri may have had a short-review-time experience with a journal in another field, from what I hear that is by no means the norm.  Most people I know in journal-focused disciplines expect it to take up to TWO YEARS to get a paper published.  With the conference cycle, you can go from submission to appearance in six months.  That just isn't going to happen in the journal world.  (Particularly lower-tier journals:  sure, JAIR and AAMAS have pretty good turnaround times, but I've tried publishing in Computational Intelligence, JETAI, and other places, and it's more typical for it to take a year or two to get something out there that way.  Granted this is partly because CS folks tend to be more reluctant to review for journals, because they are busy reviewing for conferences, but the culture shift required to get people to review more frequently and quickly for journals would be really difficult to establish and would take a long time, I think.)

    Third, we have fought for years with our colleagues in other disciplines to recognize conference papers as "counting" for tenure.  You are proposing that we completely throw that away -- because a conference paper (especially one presented as a poster) counts for exactly zero in the tenure process in other disciplines.  And you're proposing to throw it away without having established a significant infrastructure for reducing the reviewing cycle in journals. 

    Fourth, even with all of the reviewing we have now, there isn't enough room in the main conferences for the people who want to publish there, and the papers are of pretty mixed quality, at least in my view.  So we are likely to end up with one (maybe both) of two situations:  a deluge of low-quality short papers without enough space to present them, or a sharp reduction in the number of papers submitted (which is also likely to mean lower quality).  Which would lead to a bimodal conference -- a few good papers pulled from the journals, and a lot of crappy papers that you have to somehow wade through because there is no bar at all for acceptance.

    I think you are throwing the baby out with the bathwater.  A lot of people I know in other disciplines view CS as very fortunate to have the short-cycle high-quality conference papers that we have.  It's much more worthwhile going to a CS conference than, say, a biology conference, because in addition to schmoozing with your colleagues, you also get to see a lot more high-quality work presented.

    I do think there are two problems -- one that's highly relevant to this forum specifically, and one that's less so.  The less relevant problem is that conference papers (a) aren't long enough and (b) still aren't accepted as "counting" by some colleagues.  I think a better way to solve that is a post-conference journal option, where some or all of the accepted conference papers are invited to submit a longer version (or the short paper if they want to) for publishing in an affiliated journal -- maybe a new journal that's established for this purpose, maybe an existing journal.  You could combine this with a "rolling deadline" review option (Michael and I have talked about this for AAAI, though obviously not for 2013...) where there are quarterly submission deadlines, with the review cycle spread out over the year, and the possibility of being accepted for next year's conference in, say, September.  (so you'd have four batches of accepted papers per conference, and if you get accepted just preceding the conference -- too late for camera-ready and scheduling -- you'd have a slot in the following year's conference, and perhaps some publicity at this year's conference.)

    The more relevant issue for this forum is that lack of acceptance of applications paper.  But as I started off by saying, that's a problem with the journals too.  My solution would be to establish a parallel set of criteria for applications papers, and then asking authors to self-identify their papers to this "track."  Reviewers would be recruited for that track, and most importantly, senior PC members would be chosen who actually believe that there are useful contributions from applications papers.  At AAAI, this has been somewhat solved by the IAAI parallel-conference model. I don't see why ICML couldn't have an analog.

    I also think that ML as a discipline has gotten caught in the "more equations and more proofs are better" mentality in general.  That leads, in my opinion, to even the non-applications papers being biased towards certain types of approaches (manifold support vector kernel methods by far dominate the conference these days) and certain types of problems (pure classification for the most part).  I've found it very difficult to have papers on really important but nonstandard topics accepted to the main conferences (active learning, confidence estimation, model understanding, cost-based learning -- anything that isn't "UCI dataset ready" and compared to the state-of-the-art manifold support vector kernel method).  This is an issue that Kiri's paper doesn't directly address, but I think is pervasive.  I also think it's quite a lot harder to solve than the "applications don't matter" problem.  I believe we can get past the disdain for applications that obviously matter and are directly tied to real-world domains.  I think it is harder to get past the disdain for interesting problems that are less immediately applicable to real-world domains but have much broader relevance for the long-term success of real-world machine learning.  I guess that could be a new thread at some point but I'm tired of typing. :-)
  • I agree with mariedj's comment.

    Anyone who battles with the university re conferences and tenure should be aware of this famous piece by some extremely eminent computer scientists. I used this to substantial effect at the ANU. I think we have a great system of rapid turn around with long papers published. And the fact is that people will always respond better to deadlines.

    The page limit is no longer an issue (with online publication). Check out the COLT2012 proceedings that I just finished editing for a concrete example!

    And by all means make subsections in journals. There will be resistance. When this was submitted some of the referees were aghast at making an opinion piece at all. Luckily JMLR relented, it was published, and now there is a healthy stream of open source ML software.

    Finally, there remains in ML (as every other field) a strong physics envy that is behind the fascination with complex mathematics. Maths itself is not bad  (look at my papers!). But technique driven mathematical complication for its own sake is. It is up to reviewers to call such paper's bluff. I do this regularly. And I know others do too. Of course proponents of new stuff need to still be rigorous (in what ever way they can)

    Bob
  • The page limit is no longer an issue (with online publication).

    Bob, I should clarify that by space limits, I was thinking of the limit on the number of papers, rather than of the length of individual papers.  (The latter has an effect, too, but, as you say, there are ways around it.)  But conferences have an upper bound on the number of papers they're willing to accept, given venue size, number of tracks, presentation time available, etc.

    I have always found this limit artificial and limiting.  If, say, a conference can practically present 90 papers, but there are 120 genuinely good papers submitted in a given year, why should only 90 of them appear?  Conversely, if there are only 50 high enough quality papers submitted, why should there be incentive to accept 40 sub-par papers that year?

    I prefer the journal model because, now that we have electronic publication, we have no limits (nor expectations) on the number of papers that are published.  That seems better aligned with the goal of publishing all good science (and only good science).
  • You could combine this with a "rolling deadline" review option (Michael and I have talked about this for AAAI, though obviously not for 2013...) where there are quarterly submission deadlines, with the review cycle spread out over the year, and the possibility of being accepted for next year's conference in, say, September.

    I just learned that they are actually trying this for next year's ECML/PKDD (not this year's).  It should be an interesting experiment!
  • I have been arguing for years (in the NLP community) that conferences should allocate some of their talk slots to recent journal papers.  Not every talk has to be on work that was submitted and reviewed via the conference.  Already there are invited talks, and some ML and AI conferences have been allocating talk slots to reprise work that was first published in other communities.

    The reason to want talks for journal papers is so that the work actually gets read.  There are many things to like about journals, including longer papers and a more interactive review cycle.  But they are "pull" venues that need a "push" mechanism. I am reluctant to publish in a journal unless I have a chance to advertise the paper with a talk.

    The NLP community has now started down this road with the new TACL journal edited by Michael Collins and Dekang Lin.  (Note that Michael is also an ML researcher.)  There's a paper deadline on the first of every month, and they aim for quick turnaround, but there's a revise-and-resubmit cycle.  Accepted papers are guaranteed a talk slot at the next ACL.  So this may gradually replace the usual ACL review mechanism. 

    The TACL papers are restricted to be almost as short as conference
    papers, to prevent an arms race on paper length, which would arguably be bad for authors, reviewers, and readers.  However, ACL and its chapter conferences are keeping open the idea, in future, of allocating talk slots to long-form papers published in other journals. 

    Full disclosure: I'm an action editor for TACL.  See transacl.org for policy details.

    Note: Supplemental material is another reasonable way to deal with the page limit issue.  Several ML, AI, and NLP conferences have been doing this.  Supplemental material clearly divides the paper into "must review/must read" and "available for reference if needed." 
  • Interesting idea.  I hear that some other conferences (VLDB?) are also trying out the monthly deadline idea. 

    I'm curious about some of the implications.  If you submit a paper to this deadline and it is accepted, and then you are guaranteed a slot in the next ACL, but that is 10 months away, what do you talk about when the conference finally arrives?  Your 10-month-old result, or some new topic?  I'd expect many would prefer to talk about their current work, but then providing a talk slot for unknown-but-related topic Y based on a paper on topic X seems a bit weird.  How is this expected to work out?
  • I hear that some other conferences (VLDB?) are also trying out the monthly deadline idea.
    Yes, I think VLDB were the pioneers.
    I'm
    curious about some of the implications.  If you submit a paper to this
    deadline and it is accepted, and then you are guaranteed a slot in the
    next ACL, but that is 10 months away, what do you talk about when the
    conference finally arrives?  Your 10-month-old result, or some new
    topic? 
    The paper is what you're selling and it's why you got the talk slot, so it's still what you should talk about when you give the talk.  (Though I'm sure no one would mind if you included extra results.)

    If there were multiple conferences throughout the year with similar
    policies, then you could just choose to present it at the next one that
    will take you.  I think that will ultimately reduce the delay between publication and talk.

    The delay won't be 10 months unless there is only one conference a year.  And any field with only one conference a year is a buffer that would include some 10-month-old work.  The question here is only whether such work also got released right away upon journal acceptance.

Welcome!

To post or add a comment, please sign in or register.

Tip: click the star icon to bookmark (follow) a discussion. You will receive email notifications of subsequent activity.
If search doesn't work, try putting a + in front of your search term.