Welcome! Please contribute your ideas for what challenges we might aspire to solve, changes in our community that can improve machine learning impact, and examples of machine learning projects that have had tangible impact.
Lacking context for this site? Read the original paper: Machine Learning that Matters (ICML 2012). You can also review the slides.
Accept application papers
  • I think probably the #1 most important thing for this to change is that the ML community has to start valuing applications/"lessons learned" papers.  The ICML rejection list is littered with great applications papers that didn't incrementally improve performance on any UCI benchmark dataset, introduce any confusing new mathematical notation, or prove any theorems, and were therefore deemed not worthy of acceptance.  Having a separate "applications track" or review criteria for papers that are identified as applications papers might help.  Maybe ICML needs a parallel conference along the lines of IAAI?
  • 9 Comments sorted by
  • ICML has wrestled with this subject for a few years now, trying different strategies (ranging from appointing area chairs who were instructed to solicit good applications papers to inviting authors of good papers from other venues to share their results at ICML), but I don't think we've found the best solution yet.  Having separate review criteria for applications papers has pros and cons; more papers might get in, but they might also be viewed as "less deserving" (the "affirmative action" effect).

    I was recently pointed to this conference: ICMLA, the International Conference on Machine Learning and Applications, but I know little about it. Last year's program does cover a range of applications: medicine, robotics, NLP, music, etc.  Does anyone have a sense of the quality of papers at this venue?
  • I've been aware of ICMLA for a few years as well, but, like you, I don't really know anything about it.  Not only have I never been, I've never *talked* to anyone who's been, nor do I know any of the PC.  It feels like it's an orthogonal community?
  • I've also known of its existence since last year, where one of the presenters at the MLSS, SIngapore mentioned it. 
    It seems to have many submissions form Asia, which already point to a more application centered rather than theory centered conference.

    For what I've heard, it poses the same problems as many of the IEEE minor conferences (even the largest ones). Half full sessions, (even with no-shows), a lot more importance on publishing something rather than the academic merit. Half-baked reviews, etc
  • FYI at a recent ICAPS (Planning & Scheduling Conference), it was noted that by and large the conference had been in-effective in soliciting applications papers.  One comment made was "why are there no applications papers, except a few in Space applications".

    Doesnt KDD have a gazillion ML applications papers, they are all over the place like in the multi agent systems conf, they dont even have to really have an applications track?
  • Also, in order to get more applications papers, in the ICAPS community we've tken to getting SPC members who are more likely to be able to solicit applied papers, as well as have mechanisms to work with authors to improve papers, either within the same year, or for the next year.  One big win for IAAI was to allow conditional acceptance of papers, so long as we could find a PC member advocate to work with
    the submitters kind of like a journal, but on a compressed timescale.
  • Re ICMLA:  I don't know what to make of ICMLA.  I don't know the organizers, don't recognize any names on the program committee and don't know anyone who's had a paper there.  I'd expect at least some overlap. Orthogonal is one thing but disjoint is another.

    @Steve: KDD has a healthy number of applications papers in its research track, and it has a separate industrial track that's nothing but applications.  My feeling is that there's a different culture at KDD, an attitude that applications are an important part of the field, which is missing at ICML.  My fear is that without such an attitude introducing an applications track at ICML might not work; it might result in an "applications ghetto" track that researchers don't attend.
  • I share Tom's fear associated with creating an applications-specific track, for the same "applications affirmative action" reverse discrimination problem mentioned above.  We may already be seeing this right now, with ICML's inclusion of "invited applications papers" that don't experience the regular review process.  Good, yet bad?  I really don't know how they've been perceived by the ICML audience as a whole.  
  • As the ICML review process is currently wrapping up, I have a thought about this (although I see that discussion on this site has gone stale). A paper award for "Machine Learning that Matters" could be awarded at ICML in addition to the typical Best Paper. This would give special recognition to at least one paper that appears to have direct impact in application areas, without creating a separate track.

    The review process itself clearly has not changed to include more emphasis on impact. Perhaps reviewers' attitudes are changing, but that is extremely difficult to gauge. One problem with truly application-specific venues is that there may be machine learning innovations that address practical issues in many application domains. ICML would be a better place to disseminate such information than attempting to present the innovation separately in disparate applications areas.
  • Diane, that's an interesting suggestion.  I wonder if next year's ICML chairs (or even this year's chairs) would be interested?

Welcome!

To post or add a comment, please sign in or register.

Tip: click the star icon to bookmark (follow) a discussion. You will receive email notifications of subsequent activity.
If search doesn't work, try putting a + in front of your search term.