Making New Mistakes: what bad things happen when we get things right

Dana Chisnell
Hi, everyone, first, thank you to the IA conference team for pulling together a whole new way of having my favorite conference. I am really honored to be a speaker and I’m excited that this community still gets to come together, at least virtually. Welcome to my office, where we’re going to talk about what bad things can happen when you get things right.

Dana Chisnell
This is a ballot from the 2000 presidential election. Notice that it has candidates listed on both sides of the spread. The local election administrator knew that she had a lot of older people in her just jurisdiction, and what to older people always want. Bigger type. Now, she wasn’t a trained designer, but she made a decision to increase the type size because older people told her that that would be helpful. Anyone might have made the same decision. This is how This ballot actually works.

Dana Chisnell
The ballot looks like a book, restin and apparatus in which a card slides behind the rectangular card is printed with numbers. around each number is a perforated square. These are Chad’s, the numbered boxes line up with the holes in the book to vote a voter pushes a pin through the hole to press the tab from the punch card. This is a great technology in the 1960s for computing boats and a whole bunch of other things too, and it actually worked really well for decades. Ideally, the ballot looks like this with all the candidates lined up on the left. Both sides flow the candidates onto both pages interlaced delivering the sort of butterfly effect. So if you wanted to vote for Al Gore and Joe Lieberman, which hole Do you punch, thousands of people thought it was the second but it’s Actually the third, the second whole cast the vote for Pat Buchanan, a conservative religious fundamentalist design decision seemed right. Took users into account. But there wasn’t any testing. So there wasn’t really any data on which to base this decision. It was good public administration, but not ideal design. Again, any one of us might have made the same decision. But the outcome was eventually that Supreme Court stopped recounts about and made

Dana Chisnell
George Bush, the 43rd President of the United States, nothing like this had ever happened in the United States history. It might be a coincidence then that after the 2000 election, the US found itself in two major wars and the world economy collapse because of us domestic monetary policy and lacks banking regulations. But could we have foreseen any of that? Is there any way to look at what the ripple effects called second order effects might be Want to make the case that design has a tool for this, and it is vastly underused? They’re called stakeholder maps, at least in my experience, we might make a stakeholder map to visualize who the main user is, yeah, that’s a good activity. But of course, you want to be focused on the user’s needs. Of course, I’m here to make the case that we can go beyond that to using stakeholder maps as a way to look at who might be directly affected if we solve a problem for our our core user. And a way to talk about who’s affected if we don’t get the design just right.

Dana Chisnell
So imagine this is your stakeholder map, put your user at the center of this, who are they what do they need? What problems are they trying to solve? We are trained to ask these questions and you probably are also thinking about risk. If you Get the design wrong, maybe it’ll be frustrating. Maybe people won’t make it to the conversion funnel. Maybe support costs will go up the stakeholder not the stakeholder map has given us a tool for looking at the risk by asking the question, if you solve their problems, what are the risks of getting it wrong? I am here to ask you to think much bigger than that. I want you to ask, what are the risks? getting it right? What happens in the larger system, the overall ecosystem if we do a good job of solving problems for our core user?

Dana Chisnell
Second, order effects are what ripples out from any given action. If you do this thing in one place, what effect does it have outside of that place? As designers we’re so focused on our user and their needs that we often don’t think about the other in the system. We are actually trained to work this way. Yes know the user’s needs. But let’s think about what happens next.

Dana Chisnell
Come back to the butterfly ballot in Florida. We want to serve a large proportion of our voters. They happen to be older, there in the center. Who else is affected? Play the story out. Well, other voters obviously might be easier or harder for them as well. Even if they’re not older. co workers may have to give more instruction or help people get a new ballot if they made a mistake. Election administrators end up with more work because they had to run recounts. Candidates obviously care about this because they want to win. The media is all over this. Eventually, the candidate campaigns put cases into court to challenge the outcomes of the election the results. So the political parties start to be involved because this is where the legal funds are. So now you have lots of lawyers eventually gets to the Supreme Court. There’s a question about the Electoral College. We come back to Congress, this might window away in our reputation with our international allies like NATO, the National Alliance of the North American Treaty Organization, the North American Treaty Organization. This affected that stock market because there was a lot of uncertainty. And eventually we went to war in Iraq, and another one in Afghanistan.

Dana Chisnell
So maybe that all sounds kind of extreme. From One design decision, maybe there are a lot of elements, right? It could be scary to think about all this stuff. There’s politics and policy, government processes, laws and courts and how that process works the economy, the Constitution.

Dana Chisnell
So let’s look at some other examples that have nothing to do with elections, or politics. This is a story from the New York Times in 2019, in which we learn that the president of family tree DNA, one of the country’s largest at home genetic testing companies, has apologized to his users for failing to disclose that it was sharing DNA data with federal investigators working to solve violent crimes. There are a bunch of issues here. Not least that, unbeknownst to its users, this company quietly and voluntarily agreed In 2018, to open a database of more than2 million records to the FBI. So the company didn’t disclose something that it should have to its users, they allowed an outside party to access the data, the company allowing the data to be used for something other than what the users provided it for. So did they think about how this might be used against innocent people? This was a this is a simple business decision, but there should have been a discussion as the database was designed and developed about how that design could safeguard identities.

Dana Chisnell
What happened in the meeting where the decision was made? Was there a meeting if there was a designer there, I hope they protested. So let’s do this story and stakeholder map. Maybe you’re someone who wants to learn about your ancestry you’re working on genealogy or you’re writing a family history. Cool, you do a swab and you send it in.

Dana Chisnell
You get to share what you learn with your known family, which they might get excited about because it’s interesting to learn about yourself, right? But soon This connects to you and your family to family, maybe you didn’t know about. Maybe step siblings then no one knew about until now, is that welcome? DNA sequencing companies start to compete for this processing and develop massive databases of very personal data. This could be a handy tool for law enforcement to use. After all, you can collect DNA from crime scenes and compare it to these databases, without permission

Dana Chisnell
from the people whose DNA it is. It turns out that it doesn’t matter whether you put in your DNA into the pool because it’s your first or second cousins are there It’s enough to find you.

Dana Chisnell
Now Crime Victims can know who perpetrated the crime. With science. This is satisfying and can close cases much more quickly. And having your DNA available to your doctors can be a first step in personalizing your medical treatment, treating your conditions much more precisely, and perhaps effectively.

Dana Chisnell
But this data could also be used for eugenics and other evil purposes and you don’t own it. lawyers and courts have started to be involved in cases related to who owns your DNA and what they can do with it. It is very likely that the Supreme Court will have cases brought before it to determine the answers to these questions.

Dana Chisnell
In the middle of the 19th century, we met a brand new technology for lighting homes and businesses up until this point It was basically candles and lanterns. But now we had gas light. It ran through the building through pipes, but the flow could be controlled from one central valve. So the picture that you’re seeing here is a still image from a movie called gas light where Charles Boyer and Ingrid Bergman play a married couple at the turn of the last century where Charles is working very hard to draw a grid kinda crazy by messing with the central valve on the gas lights, flickering, turning them off, turning them on, and now you know where the term gaslighting comes from. So all this has happened before all this will happen again. And now with smart devices and the internet of things you can make your domestic abuse interoperable. People are using virtual assistants and smart home devices to abuse the people who live there. People called into help hotlines and domestic violence shelters said that they felt as if they were going crazy. One woman turned on her air conditioner but said that switch off with our without her touching it. Another said that the code of numbers for the digital lock in front door it changed every day and she couldn’t figure out why. Still another told an abuse hotline that she kept hearing the doorbell ring but nobody was there. Their stories are part of a pattern of behavior of domestic abuse cases tied to the rise of smart home technology. internet connected locks and speakers and thermostats and lights etc. are being used as a meme for harassment, monitoring revenge and control. That’s a story from the New York Times. Now What were the teams that designed these things thinking about? And what experiences did they bring to their designs? Who the heck was on that team? And what was the user research and usability testing? Like?

Dana Chisnell
Basically, we’re turning a blind eye to how bad people can use these technologies when we make them. Well, let’s have a look at the map for the story. So let’s put partner one in the center. Partner two is certainly directly affected by the use of smart devices in the home. Maybe it’s a nice story of one of them making the temperature comfortable and turning on the lights in the kitchen just before the other one gets home. Kids are here and pets too. And let’s say that only one of the partners controls all the smart devices. Or maybe if that’s the case, one of the partners is making it impossible to unlock the door. Or turn the lights on. And the temperature is way too cold or way too hot. And now everyone in the house is afraid and the virtual assistant comes on to loud blaring music or sounds. And now suddenly you have a domestic abuse situation. Maybe the people in the house don’t suspect the controlling partner, they call the support number for a given device. But pretty soon we’re in the range of therapy and counseling. Maybe the abuse partner figures it out or maybe kids are involved and that means that law enforcement may also become involved and the social services system. Of course, there will be lawyers if one of the partners is accused of a crime and or custody becomes a question looking beyond the home to the neighborhoods.

Dana Chisnell
We have a story from Barcelona. How many Airbnb properties are in your city? Are there enough to change the character of the neighborhoods? In Barcelona as of May 2019, there were nearly 20,000 listings in Barcelona. 1.5 million people stay in them each year. Now, when Airbnb brings 1000 rooms to a neighborhood is like, like building 1000 room hotel, but if they were actually doing that there would be public hearings and community debate. That doesn’t happen when Airbnb enters the market. And this is changing the field the city as well as the housing inventory and they’ll population. I want to meet locals where I go. How does this happen if everyone moves out? In Barcelona, the resident population declines By 45% over just a dozen or so years. So it feels like Airbnb may have lost sight of the original reason for developing this business which was to take care take advantage of excess capacity, like spare rooms or couches. The growth of the company was happening at the expense of the character of the places people wanted to go visit and stay. And apparently not nobody engaged anybody in the city who might be planning or might be in charge of the city. Only the people who put listings on the site are involved. So let’s say you get this right for the people who need rooms, we’ll call them renters. There’s a landlord of course they make the money but they also have to manage the property. neighbors are affected by this local businesses. Now, if 45% of people in the neighborhood move out, and now you have mostly visitors, businesses may change to serve those visitors instead of people who are resident all the time, this affects local services as well and what happens in parks and why the tax authorities care about this because the tax base changes, who’s paying which taxes and how do those get paid? Are there would there normally be taxes on hotel rooms that are not being collected? Because these are private rental units. Law enforcement has different job to do when it is a large number of visitors versus a large number of residents because the dynamic is just different. people behave differently healthcare workers. preservation advocates. their jobs actually change healthcare workers serving visitors need to know different languages. Preservation advocates have a different fight on their hand if they want to preserve the architectural character or the history of a neighborhood. So now imagine if you took all of those stories together and looked at where the maps overlap.

Dana Chisnell
I’m gonna leave you with that, to contemplate on your own. While I give you a story from the US federal government. Let’s look at these questions for a minute. I did a bunch of different things while I was at the US Digital Service, but one of them was working with us citizen immigration services to help them move from all paper to all digital processing of applications for benefits. One of the more fascinating and frustrating parts of this process was looking at and updating forms, there were about 150 of them and you had to fill out some combination of them no matter what status or benefit you were applying for, from lawful permanent residents to get a green card to getting a student visa to answer about 50 questions about your moral character. These questions say everything about how Americans judge one another, and how we judge visitors and immigrants. Keep in mind that these are questions about your reported morality, while some are about the legality of a thing you might have done. Not all of them are. After all, we could just do a background check for that. This is my favorite one. But I don’t know why it’s the first one in the list like it seems so minor compared to the rest of the list, but whatever.

Dana Chisnell
I came to know these good moral character questions pretty well. Most of the questions had been around for a long time. Now, while I was there, the main form to apply for citizenship was expiring and then needed to be revised. The Department of Justice seize this opportunity to add some new good moral character questions. And the US Citizenship and Immigration Services was used to that sort of thing. The people we were working with directly didn’t seem to think twice about this idea that DOJ was there, adding questions to their forum. The new questions focused on terrorist activities, I guess it was a sign of the times. Now, making ethical concern with the burden of answering questions in the form and in the interview. But there were bigger issues to think about here. What was this form? trying to do? What was the purpose really of the form? And what was the intention of us?

Dana Chisnell
For the form? What did the Department of Justice intend by asking the good moral character questions? So there I am helping to make software for immigration officers. Those people go to the center. People directly affected by solving problems for immigration officers or applicants. The program manager of USC is immigration lawyers, organizations that advocate for immigrants and refugees. unions are also involved to some extent because represent immigration Officers but also obviously, friends and family of applicants. What are the risks of getting the system wrong? That’s screwing up how we saw the immigration officers problems for the immigration officers and for the others in the ecosystem. Things could take longer if we do a bad job of the design, right. And when they take longer, that means that backlogs form which is people have submitted applications that can’t be processed in a timely way. And when that happens, the agency gets bad press. They also get bad marks from oversight committees and inspectors general.

Dana Chisnell
They don’t like that. But worse for individuals, people who are applying for status might miss opportunities like getting a job because they don’t have the right paper. Work. This could mean that your people enter the US workforce, who can bring important skills to bear. And that means that there might not be enough jobs, or there might not be enough people to fill all the jobs that we have. And that affects our economy. These are all things that we sort of thought about or had even observed. We tried to take these into account in how we design the service.

Dana Chisnell
So when you start to think about these things, now you’re in the territory beyond just the design of the form, right. But we did ask a lot of questions about how the data is being collected, because that’s something how the data will be stored and how it will be used, who will have access to it and what will they have access to it for in that time, in 2015, all of this came on to the couch. Glory of cyber. I remember being with my USGS teammates talking about this and honestly laughing a little bit about the question of what could happen if someone evil got their hands on the data of applicants and immigrants. At the time, it seemed like a really crazy idea, we treated the threat scenario very lightly. Because we just didn’t take it seriously. We didn’t think it could happen here. So we usually look at risk as what bad things will happen if we get it wrong if we don’t meet our users needs, but what if we do get it right for our main users? Let’s imagine that it’s possible to build beautiful, usable software that moves us immigration into the 21st century. The users will love it. Now what happens? Let’s have a look. Same stakeholder map different point of view So one of the risks of getting it right?

Dana Chisnell
Well, immigration officers change in what their tasks are every day rather than sort of ticking off boxes that people have supplied. The right documentation, all that stuff can be done automatically. And now immigration officers are adjudicating the harder, more complex cases, they want more money for doing that, who can blame them, and that means that the union probably gets involved. And then the program manager also needs to come to the negotiating table and they have varying skills at managing those kinds of situations and the political outcomes from those kinds of conversations. Ultimately, advocates, third parties and nonprofits and community organizations to do if this works really well for immigration officers and applicants And they could lose their funding and their purpose for existing. Some of the stuff did play out. What do they have to do with what we designed intent? Really mistakes. If they were unintended, maybe where they knew mistakes. Let you judge. I think we made a massive mistake. We didn’t challenge our assumptions. We assumed that the trend of the country becoming better, more liberal more open and just becoming a more inviting and tolerant and forgiving place was going to continue. We assumed that the arc of the moral universe was going to continue bending toward what we can to centered justice. Sure, it would be bad if the systems were hacked, but we weren’t to prevent that. And yes, it would be bad if an immigration officer became an inside threat.

Dana Chisnell
But there were ways to deal with that, too. So what if? What if making it easier for Americans to live and work also made it easier for bad actors to act badly? Just as one example, what if we were invaded by a foreign enemy and the government was occupied? That’s not going to happen. But what if the democratically elected government turned out to be corrupt? We did not imagine the future that we are in now. We treated the whole thing pretty casually. We had this is not going to be a big deal. Fast forward though, from 2015 when we’re working on immigration systems to the beginning of 2017 by the dateline in this article from the New York Times, the new administration had instituted a ban on foreign nationals from seven countries. Executive Order had been challenged several times in federal court. The Chief Justice of the Supreme Court even made a statement about the form and the good moral character questions. Robert seemed particularly concerned that the government was asserting that it could revoke citizenship through criminal prosecution for trivial lies or even emissions. He noted that, in the past, he himself had exceeded the speed limit while driving. So, why do you do this work? It’s probably not for the money. I do it to make the world a better place. But there are a lot of competing interests, weighing indecision that’s expedient for now. For how it ripples out in the future. is a designer’s responsibility. We talked about the happy path. We’re kind of squeamish about dealing with the edge cases, but it is at the edges. In the fringes where people get hurt, that work should be designing. It is our responsibility to do more to understand the larger context of use of the designs we put into the world where they fit into the ecosystem, what the ecosystem is like.

Dana Chisnell
So let’s talk about a few tools that we already have to try to prevent bad things from happening. This goes way beyond use cases and scenarios for personas for a given product. Let’s think about the second and third order effects of what we design and the decisions we make. designers are responsible not only for the direct experience of the core user, but it What happens after that behavior takes place? What happens to the next person and the next? And the next person after that? We, lots of us have a seat at the table. That means being accountable to. So imagine this is your stakeholder map, put your user at the center, who are they? What do they need? What problems are the trying to solve ? If you solve their problems? What are the risks of getting it wrong? What are the risks of getting it right? So as you can see, these issues are not exclusive to any given sector. You get the outcomes that you’re processing organization are designed for.

Dana Chisnell
We should be asking ourselves at every decision point. If we have information from users, whatever it is, we’ll use it. We use it for UI prevent harm. We affect the social contract with every design decision we make. What I mean by this is, we’re all in this together for the world to work, we have to consider not only individual self interest now, or only the needs of our core user, but also how we will continue to live together in the future into the future.

Dana Chisnell
If we make a design decision that changes the outcome for one person, how does that ripple out? If design is the rendering of intent, if all we intend to do is to make the world a better place, why did these things keep happening? We intend to solve problems for people, but there’s some kind of regressions that we’re not checking. So I have some thoughts. This. As designers were focused on users needs, our code changes how people behave and what they believe. So how do we anticipate harmful effects on the humans that were centered on? How do we mitigate risks without becoming paralyzed? First, describe how you will know what it looks like if you fail your user.

Dana Chisnell
It isn’t always just the opposite of success criteria, and you need to take this seriously. At this point, you can go back to those Chinese journey maps and identify points at which there’s a risk that could prevent your solution and put the could prevent the solution you’re designed to put out into the world from reaching success criteria.

Dana Chisnell
Your question is What could possibly go wrong and for whom?

Dana Chisnell
Ask yourself, what is something Are we making

Dana Chisnell
as we describe what failure looks like? Then you can flip those assumptions upside down. Success criteria come out into the world through asking, Well, if we do this really well.

Dana Chisnell
How will the world be different? How will life be better for users? How will life be better for other people and other institutions in a larger ecosystem? And then measure the heck out of them? measure the ripple effects measure what what would be different in the world because you made this thing put it out there. Look for signs and signals that will tell you we’re on the right track. One of the ways you can do this is through stakeholder maps. Work your work with your team. iterate, try it with different people or roles at the core without thinking about what the solutions are. brainstorm about what’s different if the problem you’re solving is different for each of the stakeholders. brainstorm about what is different if the problem as you are defining it is solved for each of the stakeholders.

Dana Chisnell
Start with the core user and work your way out. Before you look at successes have the best possible experience you can design start with the worst possible when you can design the disaster scenarios, mix them and match them. Then flip the assumptions that you made in that one, to inform your stakeholder maps for the best possible scenarios and outcomes. Take your worst cycle map and turn it inside out. If we’re right, who we screw up This is our model. I think, you know, if you look at security, it has threat models. professional role players iterate over how to break or not break something to imagine how what we’ve made or done could be used against us. Agriculture and architecture have environmental impacts and analyses and reports. urban planners and government planners do fiscal impact models, medicine has morbidity and mortality conferences, mortality conferences, which are kind of retrospective. design has scenarios Could they be elevated to be something more than use cases? We need some kind of equivalent. What’s the impact on the ecosystem of implementing this design for this user to pull off some kind of modeling or scenario planning is useful, but you also need a cross functional team So if you’re a team, I’ll look like you have similar life experience to you are driven by shipping rather than outcomes for your users. Like as a philosophy, you’re probably not going to get the longer term positive outcomes, even for your core user that you want. second, and third order effects will just

Dana Chisnell
keep being not good and not intended. But now, you won’t be surprised at least. So besides having a team where people bring different kinds of life experiences the most successful teams I’ve been a part of expand the idea of design team way beyond the people you think of as designers. So look at everyone who’s influencing the design go big on intersectional ism. Your team needs not only engineers and designers, it needs people from legal and finance they need you need an economist, of course you have to have people who are thinking about security And privacy and they’re not the same thing and sometimes they can compete. Having people who are planners not have projects, necessarily, but have communities and cities and spaces. People were involved in labor and labor movements and labor unions, people who do labor. And most of all, you want people who are different from you. I hope you have a team room. And I hope you actually sit there with your team and collaborate. I hope you chat and laugh and produce amazing work. I bet you do. But most of all, I hope you ask yourselves if we can do this, should we? If we get this right, what happens and other people might have tried this already. What was mistakes they make on the way that we can learn from. We need that exquisite practice of modeling outcomes ahead of releases. To use it as a planning tool, yes, but also help us make the best possible design decisions to make intentional design choices that deliver the outcomes we want for people we’re trying to reach.

Dana Chisnell
Thank you for coming along with me through these stories and listening so intently today, may you only make new mistakes. Thank you.

Transcribed by https://otter.ai