Machine Learning for Science and Society

A Special Issue of Machine Learning

Cynthia Rudin and Kiri L. Wagstaff, guest editors

Reviewer Guidelines

Accepted papers must: What we are not looking for:

Review Form

  1. Please summarize the paper's claims about impact achieved from a machine learning advance.
  2. Impact: In your judgment, what is the significance of the impact described? Major or minor? Whom does it (or can it) affect? If the paper only discusses performance (e.g., accuracy on a task) but not the impact or potential impact of a system with that performance, it is not responsive to this call.
  3. Novelty: Is this a novel application of ML, or is this a topic with an existing, established mechanism for success using ML? Is this a problem that truly can benefit from a new application of ML proposed by the authors?
  4. Problem description: What is the problem domain? Is the problem described sufficiently to be understandable to those outside the problem domain?
  5. Data preparation: Do the data preparation steps taken appear to be reproducible, given access to the data? (Note some data sources may be proprietary, and we do not expect authors to make data public.) Are they appropriate for the motivating problem and data available?
  6. Machine learning: Is the machine learning component described in enough detail to understand what was done and how? Note that the machine learning technique need not be a novel advance for the field of machine learning, if it is applied in a novel way, or to a problem of unprecedented scale.
  7. Results: Is the methodology clearly described and appropriate for the problem? Were problem-specific baselines and metrics employed? Are there additional experiments or metrics that should be conducted to evaluate impact? Are the results discussed and interpreted, including a discussion of the implications for the problem domain? Taken as a whole, do the results support the claims of impact?
  8. Domain expert: Does the paper provide evidence from domain experts that the machine learning advance has (or can have) significant impact? Does the work described in the paper result in new knowledge or insights for the problem domain? Are most of the results written in such a way as to be interpretable to experts in that domain?
  9. Infusion: Is there a clear description of how the machine learning advance was (or will be) incorporated into a deployed system? Are domain experts currently using the system? Have the authors provided a description of the steps needed for the technology to be adopted, if it isn’t already?
  10. Lessons: Does the paper provide a summary of lessons learned from this experience that can benefit future machine learning researchers? What are they?
  11. Overall judgment: Accept with minor revisions, Reject with encouragement to resubmit, or Reject.
Evaluation (from 0 to 10, where highest = best): Any other comments: