Beki Grinter

Posts Tagged ‘journals’

Snowbird: Peer Reviewing

In academia, computer science, discipline, research on July 20, 2010 at 1:17 pm

The third session I attended at CRA was on peer review, it was a panel organized by Moshe Vardi.

Computer Science is very unique. We rely heavily on conferences as the means of publication. More so than other fields. Additionally we have a model of specialized conferences, unlike other sciences that have an Annual Meeting, the last ACM Annual Meeting was in 1984.

Someone quipped that “a computer science conference is just a journal that meets at a hotel.”

So recently there have been concerned about the number of conferences, the quality of those conferences, and what it means to be driven by conference deadlines. Jeannette Wing also pointed out that this applies to funding deadlines. Another concern raised by her was how this taxes the community of reviewers. She also said something I liked which was a reminder, but well put, that conferences and journals are a means of documenting the discovery of scientific truth by building on past knowledge in order to share it with others. Finally, it was observed that conferences cost time and money.

Perhaps the most troubling concern was the implications of the profusion of conferences for the field of Computer Science. The concerns raised included a tendency towards incrementalism, conservatism (in submission and review I believe), that the field would splinter, and it would miss big ideas. Computer Science would lose it’s vibrancy and excitement.

But why does this happen, why do we continue to submit to conferences? That led to a discussion of how we understand impact. Not surprisingly given that this is largely a crowd of department heads and Deans, it led to a discussion of how impact is measured on the academic vita at those crucial points: admission into graduate programs, faculty hire, tenure and promotion to Full.

So this raises a two questions for me.

First, how do we change this, if we think we should? The scale of the change required seems vast to me, requiring both procedural and cultural changes. It requires changing behaviors of the 1000s of people collectively involved in Computer Science. It also requires convincing those at the earliest steps (the undergraduates who are considering graduate school and working on publications) that they still have a chance of participating in those later steps. Someones just mentioned that it’s going to involve ensuring that every single review letter changes in accordance…

Second, what about considering the production process? We spent our time focused on the outputs, but what about looking at the inputs into the system, i.e. the number of people we’ve trained. Specifically a focus on PhD production. If a faculty member can produce 14 students in 20 years, who are all trained in the process and seek to continue to publish, well that seems like a scaling up.

Publications, Publications, Publications

In academia, computer science, discipline, HCI, research on June 24, 2010 at 4:14 pm

Recently I was asked to provide a few thoughts on publications to share with some students. Specifically, I was asked what are the differences among the many genres that you can select from and how are they evaluated when you look for jobs. I thought it would be interesting to post what I sent out, just to get the feedback.

Caveat, this is written with a Human Computer Interaction/Human Centered Computing spin. That matters for at least two reasons. First, my examples are drawn from these disciplines. Second, I think there’s something different about the role of writing when it may not be supported by two other types of expression of research (programming or mathematics — perhaps there are others but these two genres of demonstrating evidence and making argument are common in Computer Science). When you have multiple forms of argumentation, after all what are pieces of code or proofs if not argumentation), then you have multiple means to express your research contributions, which I think creates a different balance and role for written English. In HCI you may have nothing but writing, and certainly in the parts I’ve practiced, that’s very true.

— Message

So, caveat, the answers to these questions depend on a) your area, b) your career aspirations. So, it’s imperative that you have these sorts of conversations with your advisors, and since this is a somewhat interdisciplinary area, to meet people who are working towards different goals because they’re in different areas, have different career aspirations etc…

The rest is my own thoughts, but I hope that it’ll give you the tools to ask your advisors these questions.

So rather than rank the publication perse lets ask a different question: what work does a particular publication do for you?

Archival conference/journal papers. Peer reviewed and are the way to report the results of your work. I think it’s fair to say that for the majority of areas in HCC, conference papers are a primary if not the primary mechanism of scholarly communication and so we hope that we submit them to the place where the audience is most likely to be interested in and find our work useful to them and their work. There’s a decision space behind this which involves juggling things like highly ranked conference in area, specialized conference where I will be able to have more time to communicate to a smaller but more centrally relevant audience, speculation on a new conference in which if I participate early I might benefit from visibility in later years…

Journal papers can be opportunities to go into more depth than possible in a single conference paper (add new data, provide a deeper theoretical treatment). Others create insight from synthesis of multiple papers. Some journals have taken to asking for an essay about the novelty of the piece over and above any existing publication, so some degree of something beyond just a rewrite is essential.

Short versus long papers, Notes versus Papers. This is still a relatively new experiment (I think CSCW 2004 may have been the first experiment…). I think it’s more pervasive in the HCI + related areas than other areas of HCC, apologies if I’m wrong. Beki’s opinion, I think that a Note is harder to write than a Paper. To do justice to novelty, set up the argument, and cover enough related work to be compelling, it a tall order. As far as I can tell they are treated identically, as novel peer-reviewed contributions.

Then there are a whole class of non-refereed to lightly refereed papers: workshops, professional society magazine pieces etc…

What work can these do? Well they can all get your message out in different ways. Workshops, for example, can help you spend a day interacting with people who are of interest to you because they are in a particular field. They may also help you present an idea and get feedback on it while it’s still in its early stages. Some of you may organize workshops, and that’s an opportunity to get a collection of people together and mine their collective intelligence for an idea that you’re interested in. Some of what you want, and what you might get is going to depend on who is there. But, I think that things like workshops, especially those genres that have a physical component, are a way to start a conversation. Panels are also another forum to talk about your research, they tend to be more focused on your expertise, so they suggest (at least to me) I think discussions where you’re on familiar and comfortable ground. Professional magazines are a way to reach a different audience, you might have something you want to say not just to academics but to practitioners. Magazines tend to be written differently (some would say more engagingly ;-)…

So going back to the original question, how are all these evaluated. Each genre of publication has their appropriate use, and the collection of them that you have together is how ultimately anyone is evaluated, for the area that you’re in and the career goal that is triggering the evaluation.

Perhaps one way to think about this is that you make choices at two levels. First, there’s the choice you make each time you write. And each time you write you can ask yourself, what is this submission I’m preparing doing for me? Is it the right genre for what I want to accomplish. Then there’s a holistic reasoning process. If you think of your CV as a portfolio of activities that collectively express your accomplishments, what is the prevailing balance? Each time you write you’re actively changing that portfolio… But a publication of any form is just a means to an end… the question is what is the end you’re pursuing and why, and how does it interact with all the others you already have, and that you would like to do (thinking forward is the way to make sure that you parcel out your work in appropriate pieces — I’m not referring to the “MPU — minimum publishable unit” more the thought that you want to roll out your results probably in ways that build on each other, rather than collide.