Beki Grinter

Archive for the ‘research’ Category

Another Facebook Study: Reflections on Writing about Corporations

In research on May 28, 2015 at 3:21 pm

Facebook recently published another study, this time about what news individuals who list their political affiliations on the platform see on their News Feed. Its received a lot of attention, particularly from academics who have questions about the methods and the way in which the results were written. I won’t reiterate these critiques, instead I point to the very eloquent Christian Sandvig who writes about his concerns and has kindly included links to others.

I started to wonder about how Facebook decides what can be published by their researchers. I assume that proprietary information such as how News Feed works is not publishable. Facebook has never published anything about people who leave the platform as far as I know. I wonder whether that’s off-limits.

I started to think about my own experiences of publishing about corporations while working for them. At Bell Labs, anything I wrote had to be read and approved by three non-researchers in the corporation, as well as a lawyer. The lawyer was always looking for intellectual property possibilities. The others were meant to review it for completeness and anything else they felt was appropriate.

If there were explicit rules or areas that we couldn’t publish about I didn’t know what they were. But I did choose to omit two types of information from papers. I didn’t describe intellectual property. As I said the lawyer was always looking for new revenue streams (I was a sad disappointment) but I am quite sure that he would have removed anything that was describing a competitive advantage. (And since I often wrote about things going wrong, I was actually usually more focused on what one might describe as competitive disadvantage, more on this later).

I also didn’t provide explicit financial information. This I am a bit more troubled by. In any organization the importance of money, how much, to whom it flows, how often, authorized by, etc… shape many software outcomes. I referred to that, but I never put dollar figures to the flows. But even now I do still wonder whether I should have written more about money. Its a statement of the obvious to say that money has incredible influence on what happens, but how money flows inside a corporation is also fascinating. For although the corporation was participating in a market economy, its internal finances bore no resemblance to a marketplace.

I want to return to the competitive disadvantage. I remember one paper in particular. It’s one of my favorites. Its a horror story about a corporate metrics program. Lucent wasn’t named, but it was clear that it was the corporation. We’d been asked to look into this by the CTO of the corporation at the time. So no pressure. When we conducted our empirical research we quickly found a lot of problems. We wrote them up. I wondered as we submitted the paper for internal review whether it would be allowed to be published. We wrote about cynicism and apathy, about how metrics competed with “real” work, and how the view from the top can overly black box the corporation. I wondered whether our reviewers would feel we were painting the corporation in a negative light. But it was allowed and it appeared in ICSE.

I’ve thought a lot about this experience over the years. I am glad that I worked for a corporation that allowed me to publish this work because I think that the most important lesson that comes from it is that initiatives that look deceptively simple from the top of the corporation can be surprisingly complex for an organization to implement. That was an important conversation to have with an intellectual community that I felt at the time was very invested in process engineering without having too much real-world experience in rolling any processes out.

But today I’m thinking about this as an example of writing about things that might appear to be against the corporate interest. Lucent took a chance on us. I don’t think they got damaged by it. Maybe their competitors laughed (although I’d bet that they too struggled with the same problems). We got no press attention for it, either positive or negative. I wonder whether people thought positively of the corporation for letting its researchers write about things that didn’t go so well (or horribly).

I will probably never know what Facebook encourages and prohibits its researchers from writing about. But there’s a sense that the study had a framing, and one that was at odds with the results. And that’s unfortunate for both the researchers and Facebook.

Advertisements

538, the World Cup, and Facebook: Telling Stories about Data

In computer science, discipline, empirical, research, social media on July 15, 2014 at 6:49 am

As many of you already know, I’ve been following the World Cup. My team, Germany, won. Watching the World Cup has always involved reading news reports and commentary about the matches. This year I decided to include 538 in my reading.

538 is Nate Silver’s website. Nate Silver became famous predicting US elections. He is a master of analyzing big data to make predictions. It works well for elections. But it doesn’t work so well for the World Cup, at least not for me. First, the site predicted Brazil to win for a long time.

But it’s not just that 538 did not accurately predict the winners. I think that 538 misses the point of a World Cup. Crunching data about the teams doesn’t tell the whole story. And the World Cup is stories. Many stories. As a fan you learn the stories of your team and its history. You might start with world history—this is very salient as a Germany fan. England versus Argentina similarly (1984). It also involves stories about the teams previous encounters. Germany versus Argentina has happened before, even in Finals. And those stories are recounted, and reflected on, in the build up to a game. You might tell stories about strategy. Certainly the Germans have been telling those, about a decade long commitment to raising German players. How you structure a league to encourage more domestic players that can also play for the national side. How you balance the demands of a national league and a national team.

In a nutshell, context matters. These stories of world politics, former World Cups, and the arc of time turn statistics about the players into something richer. 538 tells none of those stories. And I suppose that’s exactly what it wants to be, a “science” of the World Cup. But my World Cup isn’t statistics, it’s larger, more discursive and has a multi-decade narrative arc.

Reflecting on this caused me to revisit the Facebook study. Yes, that Facebook study. The study reported data. But it was data about people. However, at the same time I think some of the response could be interpreted as people feeling that there was more to the story than just statistical reporting of the outcomes. Is it a similar type of human-dimension, an infusion of humanity? This is the question I’ve kept wondering since reflecting on the problems of both of these data-driven reports. 538 reduces football to data. In so doing it loses the human dimension. The Facebook study started as data and the public raised human concerns and considerations. If I have a take away it is that fields like social computing, or any data science of humans, need to seriously pay attention to the stories that we tell about people. How we frame or potentially reduce people is something that the public will care about, for it is their humanity, their stories that we seek to tell.

That Facebook Study

In academia, computer science, discipline, empirical, European Union, research, social media on July 8, 2014 at 8:07 am

Following Michael Bernstein’s suggestion that Social Computing researchers join the conversation.

Facebook and colleagues at Cornell and the University of California, San Francisco published a study in which it was revealed that ~600,000 people had their Newsfeed curated to see either positive or negative posts. The goal was to see how seeing happy or sad posts influenced the users. Unless you’ve been without Internet connectivity you likely have heard about the uproar its generated.

Much has been said, Michael links to a list and some more essays that he’s found. Some people have expressed concerns about the role that corporations play in shaping our views of the world (via their online curation of it). Of course they do that everyday, but this study focused attention on that curation process by telling us, at least for a week how it was done for the subjects of the study. Others have expressed concern about the ethics of this study.

What do I think?

I’ve been dwelling on the ethical concerns. It helps that I’m teaching a course on Ethics and Computing. And that I’m doing it in Oxford, England. So I’m going to start from here.

First, this study has caused me to reflect on the peculiar situation that exists in the United States with regards to ethical review of science, and the lack of protection for individuals that participate in it.

In the United States, only institutions that take Federal Government research dollars are required to have Institutional Review Boards (IRBs). The purpose of an IRB is to review any study involving human subjects to ensure that it meets certain ethical standards. The IRB process has its origin in the appalling abuses conducted in the name of science like the Tuskegee Experiment. Facebook does not take Federal research money, and is therefore not required to have an IRB. The institutions by which research gets published are also not required to perform ethical reviews of work that they receive.

I find myself asking whether individuals who participate in a research study, irrespective of who funds that work, have the right to be protected? Currently there’s an inconsistency, in some research the answer is yes, and in others it is no. It seems very peculiar to me that who funds the work determines whether the research is subject to ethical review and whether the people who participate have protection.

Second, most of the responses I’ve read have been framed in American terms. But social computing, including this study, aspires to be a global science. What I mean is that nowhere did I read that these results only apply to a particular group of people from a particular place. And with the implication of being global comes a deeper and broader responsibility: to respect the values of the citizens that it touches in its research.

The focus on the IRB is uniquely American. Meanwhile I am in Europe. I’ve been learning more about European privacy laws, and my understanding is that they provide a broader protection for individuals (for example, not distinguishing based on who pays for the research), and also place a greater burden on those who collect data about people to inform them, and to explicitly seek consent in many cases. I interpret these laws as reflecting the values that the 505 million European Union citizens have about their rights.

I’ve not been able to tell whether European citizens were a part of the 600,000 people in the study. The PNAS report said that it was focused on English speakers, which perhaps explains why the UK was the first country to launch an inquiry. If Europeans citizens were involved we might get more insight into how the EU and its member nations view ethical conduct in research. If they were not, there is still some possibility that we will learn more about what the EU means when it asks “data controllers” (i.e. those collecting, holding, and manipulating data about individuals) to be transparent in their processes.

I’ve read a number of pieces that express concern about what it means to ask people to consent to a research study. Will we lose enough people that we can’t study network effects? How do we embed it into systems? These are really good questions. But, at the same time I don’t think we can or should ignore citizen’s rights and this will mean being knowledgable about systems that do not just begin and end with the IRB. Its not just because its the law, but because without it I think we demonstrate a lack of respect for other’s values. And I often think that’s quite the point of an ethical review, to get beyond our own perspective and think about those we are studying.

MOOC Participation: Diversity and Assumptions of Development

In computer science, discipline, empirical, research, social media on February 12, 2013 at 11:30 am

Continuing my series of posts about MOOCs. Today’s is about a type of open/development rhetoric I keep hearing associated with MOOCs. It’s well meant I am quite sure, but I’ve heard the following sentiment: MOOCs will allow anyone from any continent to access content. And that in turn leads to increased education, skills for all.

I have a number of problems with this argument.

Starting with the obvious, this sentiment makes important assumptions about access. That access to the Internet and its content is uniform across the world. But it’s not. The Internet is a very different experience if you have a smartphone as your only means of access, versus if you have a laptop. Behind the hardware, there are questions of corporate policies and pricing mechanisms that influence access. Bandwidth caps, bandwidth pricing can influence how people use their phones, and in many parts of the world also how they use the wired network.

Behind these crucial practical questions of access lurk other assumptions, which warrant questioning. Is the content we create relevant or useful for everyone? What assumptions do the producers of content make about, say, what has been previously taught? What assumptions are made about the types of hardware and software the students have access too? And most critically, what assumptions get made about why the person is taking the course and whether that content will ultimately be most useful?

Although its not used too much, I have heard the word “Africa” used to describe diversity. I do think its well meant but it has the danger to collapse all of these questions into a stereotype of a person. Africa is not a person, nor is it a country, it’s a continent of great diversity in all senses. A person from Africa may well contribute to diversity in a MOOC setting, but so might a person from America.

Like others, I see this as being part of understanding the participation divide that shapes the Internet today. Some of that divide is the question of access, its costs, modalities, and so forth. But that’s not all that shapes the participation divide. When we overly simplify an entire continent we close down the question of what shapes participation in very problematic ways. If we are really committed to understanding how online education might help more people learn, the participation divide is precisely the question we ought to open up, to really take account of the highly diverse population of people that have some reach to the Internet. Because it’s only when we actually take diversity seriously that we have any shot at getting to something better than more education for the already well educated.

Program Chair: Reasons to Say Yes…

In academia, academic management, discipline, research on January 27, 2013 at 4:14 pm

I’ve chaired the papers track of a couple conferences now. I could write about the process itself, but instead I want to write about the learning experience of doing this. The first conference I ever co-Papers Chaired was CHI 2006 (with Tom Rodden). I owe Tom a huge thank you because he taught me several useful management strategies that I used during both processes, but also have found useful in my day-to-day activities.

And that is a good reason to volunteer to chair a conference. One of the reasons you’ll hear most often for agreeing to do this kind of service is that it’s good for the community. And you do give your time, as you do as a reviewer, member of the program committee and so forth. Another is because it looks good on the vita. I was told, for example, that serving for CHI meant that the community trusted me with the products of their academic research. I’ll add another one into the mix. For anyone who has ever complained about the way a conference is run, or what happened to their paper, nothing beats seeing what the processes are by which the conference is put together. Actually, I think it should be mandatory that anyone who complains, especially more than once, have to get involved with the organization of the conference.

And today I want to offer another reason, what you learn in doing this. Papers Chairing throws up a myriad of management situations. Each one requires a thoughtful response, many require subtle negotiation to balance needs of the various parties. As a program chair, you are responsible for ensuring that everyone who is giving their time to review etc. gets a fair shake and feels that you support them in their service. I like doing that. I feel its a great way to say thank you. Sometimes it’s harder though, as you have to work something out as best you can. Sometimes there are difficult messages to write, and the practice in getting tone as well as content right is invaluable.

Knitting Needles dont Knit, People Do

In empirical, research on December 21, 2012 at 2:22 pm

I keep hearing this line about guns. Guns don’t kill people, people do. So I thought it would be interesting to explore the argument via knitting needles.

I knit, I create knitted artifacts. But, the knitting needles I use are pretty crucial to the experience. It’s not impossible to knit without knitting needles, I’ve tried with chopsticks, it’s possible but not as satisfying. You can also use the knitting needles for other, non-knitting things, I’ve used mine to tie my hair up. But they are better for knitting than as hair decorations.

Knitting needles shape the experience by being very intentionally designed for that experience (e.g., the different thicknesses suitable for different thicknesses of yarn, circular for working knitted objects in a round, double-pointed for socks, as well as the traditional straight needles). Knitting needles are designed to help people who knit knit. Without them people could knit, but the experience of knitting with knitting needles is the most common one and it’s not surprising, they were designed for it.

Beyond the design/function argument there is something else about knitting needles and knitting. When I have knitting needles in my hands, I am visibly a knitter. I’ve written before about the types of conversation that that starts up, about how to knit, what I am knitting, recollections of family members who knitted. It makes me a part of a world in which I am seen as a knitter, and in which others are a canvas of potential knitters or people who are curious. Just the other day I was knitting at my Godson’s school play, and so was the person sat next to me. Not only did we have conversations about our favourite local yarn stores, but we also received joking commentary from others about “keeping the knitters together.” I still don’t know her name, although I do know the name of her granddaughter who was also in the play (and about the same age as the children in the shooting that has triggered this reflections on knitting). Sometimes the associations are less amusing, I fly with knitting needles, its allowed, but it doesn’t mean that others on the plane don’t look at me, and the needles as if they are weapons and I am potentially a risk. Context matters, its uncomfortable for me to be seen as a terrorist risk when I knit on a plane, but it’s a space where contexts transform the meaning of the technology.

When I knit the technology that helps me do that is knitting needles. It changes what I can do, as well as supporting me in that, but it also changes my relationship to the world itself. I become associated with my needles. So, I don’t think you can separate guns from people, because you can’t separate the needles from the knitting.

Using those Six Years

In academia, academic management, computer science, discipline, research on December 11, 2012 at 10:49 am

One of the most common things I hear from new PhD students is that they do not want to be in graduate school for six years. There are a variety of reasons for that including explaining it to parents, wanting to earn a decent salary, and just not being able to imagine what one might spend six years doing.

It’s this latter point I want to take up today. What might one spend six years in graduate school doing. Recruiting for the next job is one way to think about it. Many students come into graduate school not really knowing what they want to do. I respect that, many people who embark on a PhD are fairly young, life is going to involve many changes (as an advisor one of the loveliest things I’ve experienced in graduate school is weddings, seeing my students and others find their partner). But that doesn’t mean that one shouldn’t give some thought to what you want to do post-graduate school.

Perhaps a different question is what are the options available that I would like to explore when I leave. And this begs a second and far more important question, what will I have to achieve while I am in school if I want to explore those options? Since you are probably only going to do one Ph.D. you might not want to leave these things to chance. And the later in graduate school the more your options have been made, either by what you chose, or by what you didn’t. This applies not just to how you are mentored while in graduate school but what you are being mentored for post-graduate school. Understanding that you are responsible as the student for finding someone who can do both is part of (as the book says) “getting what you came for.” I want to reiterate this point… in addition to finding someone who can mentor you effectively during graduate school, you want to find someone who can effectively mentor you for what you want to do post-graduate school. You can not leave this to chance. Or you can, but then you’ll have effectively made some choices that you may never have realized.

So, how do you know what you want to do? Look at what students who’ve graduated have gone on to do. Can’t decide among those choices, then plan for the one that looks the hardest for you to accomplish. You can course correct later on, but only if you’ve started down a path that allows for it. This is what you should use those six years for, and this is what you, as a graduate student, are responsible for doing. Irrespective, I would say, of what you might get told, you need to develop that internal sense of what you want and decide whether there’s appropriate alignment. You may have to ask to get what you want, or whether that’s even possible. Its crucial to understand that your goals may not align completely with another person’s, but it’s still your responsibility to get what you want (or risk becoming what someone else wants you to be).

Aside: to me this is just a variant of understanding evaluation. Every year in corporations, and regularly in academia, people are evaluated routinely as well as for promotion. Understanding how you are going to be evaluated is essential to understanding what you need to do to be successful. There is not always a clear alignment between what others may want and what you need to succeed. But your odds of succeeding are so much better if you pay attention to the means by which you will be evaluated and you plan to achieve it.

Theory

In computer science, discipline, HCI, research on October 15, 2012 at 2:53 pm

There was a panel about theory in HCI at the NordiCHI conference today. I wonder what they discussed. It made me think about questions I’ve been asked about the role of theory in human-centered computing, particularly in the context of teaching the Introduction to Human Centered Computing class (which is a survey of a variety of theories about technology). In that class, I recently said something about my own relationship to theory, and now I’m wondering whether it’s true for others, so here goes.

I find some theories more personally compelling than others. They speak a type of truth to me, one that I find very engaging. They bring out, for want of better words, the researcher in me. What I find very useful about this is that because theory is related to methods and questions, I can use these connections to focus in on particular problems. One of the first theories I ever used was Grounded Theory, it’s in my dissertation. I took the route of building more theory atop of articulation work (and to some degree social worlds) to develop my theory of software recomposition. The theory, modestly, explains some of the reasons that software is hard to produce, its because the process of modular decomposition creates a division of labor full of dependencies that are often underspecified and then change during the course of development. All of these dependencies are discovered at integration, unless they are well managed, and that makes production difficult.

Grounded Theory spoke to me. The works I read seemed to address problems that I found compelling, and using methods that I enjoyed to use. It didn’t help me find the domain, that was a different inspiration, one largely stemming from a sense that HCI had overly focused on the end-user taking the engineer to task and I wondered whether the work worlds of developers were as socially and technologically complex as those of the end-users and if so, whether we had to make inroads there in order to make the world of the end-user better. But it did help me formulate questions, operationalize them empirically, and do analysis.

Focus is the enemy of a new researcher. Being reflective on what your passions are might be one way to do this. I’m very lucky I also know the things I do not like to do. I am glad when others do them because I don’t want to.

Well that leaves all the big questions open like what is a theory in HCI, what should it do, what does it mean that we have multiple theoretical approaches, should we develop our own theories or use those developed by other disciplines. But, it is an answer to what problems should I work on? If you can find research that speaks to you very deeply, that supports the answering of questions you find interesting and using methods that you find enjoyable to practice, that seems useful.

Concerns about the Omission of HCI in Impacts of CS “Tire Tracks” Diagram

In academic management, computer science, discipline, HCI, research on August 15, 2012 at 9:18 am

Recently, the “tire tracks” diagram of how Federally funded research has led to impact on the Computing industry and American experience was updated. Appreciative of the work that it does to continue to make the case for basic research in Computer Science, I was keen to see it. Imagine my disappointment when I realized that HCI was not present.

Maybe HCI is omitted because the “tire tracks” diagram focuses on product and not business practice outcomes. One major impact of HCI on industry is User Experience design. Don Norman and his team at Apple first popularized that term, in a paper they wrote about Apple’s User Experience (UX) practice, the research that continued to inform those corporate practices, writing it up for an HCI conference, the ACM’s Conference on Human Factors in Computing Systems (CHI). Now consider, Google’s focus on keeping it simple. I’ve asked UX professionals and global estimates (based in part on membership in professional societies) are as high as 44,000 people who work in the field (another estimate was that about 10K of those people work in the United States). That’s a pretty significant type of employment impact!

Returning to the product-oriented outcomes, Brad Myers has done an eloquent job of explaining the role of HCI in the software and systems we take for granted today. He has even produced tire tracks. His diagrams suggest to me that HCI may have gotten folded into Computer Graphics in the CRA’s version of “tire tracks”. But, to omit explicit recognition of HCI is to preclude innovations that came from explicit consideration of humans, whether it be as information processors or soci0-technical beings. Yes, sure, the product innovations happened in the graphical user interface and how it is manipulated and interacted with, but reading the history of innovations at that interface it’s clear that the inspiration and the basic science that was drawn on included science about people, and developing a basic science of the interaction between people and machines. One particularly fruitful area for development that is of interest within HCI and appears to be missing from the tire tracks is input devices and techniques…

PARC features in some of the industrial work, and it is worth noting that the culture of PARC was to have a focus on the relationship between people and machines. J.C.R. Licklider’s Introductory Note to an anthology of papers about Computers and Systems in the book about the first decade of PARCs research: A Decade of Research: Xerox Palo Alto Research Center 1970-1980 captured this as follows:

In my opinion, the interaction between people and computers is the most exciting new frontier of our time; here, in these pages, are many superb contributions by colleagues who not only share the excitement of that frontier but are developing the new land and building a new way of life in it.

These papers are an eloquent testimoney to the fact that, in the ten years since it’s conception, the Palo Alto Research Center has flourished. In that short time, many computer scientists, I among them, have come to consider it as the leading center of research in interactions between (or, in 1960’s terms, the symbiosis of) human beings and digital computers. …

The science and technology of the human use of digital computers are being created right now. p3

I’m disappointed that as we continue to chart this frontier of novel people machine interactions that the role of HCI has been omitted by the people making the case for the future of our science. I think Computer Science will come up short without taking people seriously. After all, in the marketplace we can ask whether the truly successful technologies are just a matter of their hardware and software, or is it that they deliver something that people want, desire, find useful and useful, and connect them to others.

Metrics: Numbers and Processes

In academic management, discipline, research on August 6, 2012 at 9:45 am

Surprise, surprise, another post about metrics. Its not just the numbers themselves that can be problematic, but here’s a recent encounter with the processes used to compute the numbers.

Some time ago, Paul Krugman wrote this:

As I see it, the economics profession went astray because economists, as a group, mistook beauty, clad in impressive-looking mathematics, for truth. (Full article).

I love this phrase, and I tend to concur with him. Perhaps its because he has a Nobel Prize in Economics (2008).

But, I don’t think you even need to have very impressive-looking mathematics. Perhaps in Economics, perhaps to justify research outcomes the nature of the mathematics matters. Not, however, for a variety of metrics.

I was recently asked how I computed my h-index. While many people use tools, I compute mine manually. The mathematics of this is not complicated, although the truth is.

R. Grinter and R. Grinter are two different people. Some of my citations omit the E. that I use (Rebecca E. Grinter). This can, when not caught, lead to grade inflation. One way to catch it is by looking carefully for any focused on spectrosopy. But, even taking those rogues out you can see the consequences of interdisciplinarity on the h-index. Many of the tools let you refine by discipline, but while that removes most of the Chemistry, it doesn’t catch the interdisciplinary pieces that the other R. Grinter worked on. More over, if you get too restrictive, R. E. Grinter and of course my nemesis mis-cite, R. Grinter get lost categorised into other disciplines. Finally, there’s my nick-name. Beki. Happy to be Beki, but people will pick me up as B. Grinter. So, I usually need to pay attention to that. In retrospect I wish I’d just started with B actually, because as far as I can tell there are no other B Grinter’s currently research active.

So that’s why I compute my h-index manually, I do to provide a truth, one that reflects my awareness of the flaws associated with the tools. But, by computing manually I’ve separated my h-index from the one computed by tools, and thus to compare mine to others requires computing all the h-indexs manually. And comparing is frequently at the heart of metrics. And this to me seems like another fine example of the problem of numbers as truth. In the process of trying to supply truth (an accuracy through manual computation) I’ve simultaneously taken away another type of truth, one that is comparative based on using the same process to compute that “truth.”