2013 Main Conference Talk
Federal Government Website Citizen Engagement through Large-Scale Usability Studies
Minimum samplings are necessary to ensure the validity and statistical significance of usability studies. But too large a sampling creates more work without increasing significance or validity. Nevertheless, there is a case to be made for casting a wide net and engaging a larger audience than is necessary to actively engage users.
Today, IAs and usability professionals are employing large-scale online usability studies to promote President Obama’s Digital Government Strategy (DGS) and its goal of citizen-centric websites. This presentation will consider how users drive change on private vs. public websites, provide an overview of the DGS, and dive into real-world examples of usability-driven citizen engagement on federal websites. It will discuss appropriate tools and techniques (including recruiting) and important considerations for successful large-scale usability studies (methodology, scope, the UX lifecycle, etc.). And importantly, the presentation will weigh in on the logistical, technical, quantitative, and qualitative challenges of this approach.
About the speaker(s)
Jeff Pass: …before I get off this slide into my proper introduction slide, let me just say that I hope, God willing, to get through what I have to say in about 30 minutes. So we have 10 to 15 minutes for Q&A. But, more than Q&A, what I would really like to hear is if any of you are doing anything similar to what I’m talking about here. Because some of the concepts I’m talking about here are things I want to take a little bit farther. I’d love to hear what other people are working on as well.
As is the case with sessions where this is QA, we have books to give away. At the end of this one, we have an O’Reilly book. I won’t tell you the title of it yet. Keep suspense high. I will go ahead and get started. It starts with a guy introducing himself. I am that guy. My name is Jeff Pass. I’m an information architect. Hi, thank you for waving, very obliging. My business card says, “Lead User Experience Consultant.” That communicates a lot, I’m sure. I’m here to talk about citizen engagement and large scale IA-focused usability activities.
So, I started by talking about myself. Next, I want to talk about this guy. Apologies to Shepard Fairey, the creator of this original image, for basically changing the word “change” to “engage.” But he’s made plenty of money lately. So I’m not too worried about hurting his feelings.
President Obama supports a lot of things. A few of them relate very specifically to what I want to talk about today and this whole notion of the new web, for the US people and the US government. He’s really keen on innovation, transparency, information and feedback. And this whole idea of “We the people.” I’ll tell you that was a contentious bullet point.
Originally, I had citizens. But not everyone’s a citizen. What does that mean? The American people? Native American? Whatever I was going to put there was wrong, so forgive me if I’ve offended anyone. But it means the people that are contributing to this country. Those of you who are here by whatever means you’re here. Don’t want to go into that too much further. So instead, we’ll go to the next slide.
Obama commissioned this strategy and then signed a memo around the digital government strategy. It happened on May 23rd of last year. There was this memo, basically, that was launched and then there was the larger strategy. Collectively, they’re known as the “digital government strategy”. The memo itself was a memo, and as such, not very excited. But basically what it did was it put the government on notice. It said, “This strategy is happening. There’s a timeline. There are things that you have to achieve and accomplish. You’ve got 12 months to do it.”
What was much more interesting than the memo itself was the strategy, which was introduced by the CIO and CTO. I just want to read this quote directly, since it’s a quote and I don’t want to get it wrong, “The federal government must be able to deliver information and services to the American people, anytime, anywhere, and on any platform or device.”
Has anyone heard this phrase bandied about at the conference so far? [laughs]
This is not a particularly cutting edge theme, but to have it being codified by the government is pretty cool. There’s four over-arching principles. Got a slide for those later. And then also, 12-month agency milestones, another slide for those.
The over-arching principles. I’m only going to read two of them, because only two of them really apply. If you guys want to read, you can. Basically, information-centric and customer-centric. Information-centric basically meaning, give them what they need. Customer-centric meaning, involve them in the process.
The 12-month agency milestones. Just want to go through these really quickly, because they really don’t apply at all. But if I’m giving high level background of the digital government strategy, I ought to address it. There were three-month, six-month and 12-month goals.
The three-month and six-month goals are behind us. The three month goals are, sort of, to identify your strategy, to enunciate your strategy, to determine some APIs and mobile apps that you’re going to make available by the six-month milestone. Then, of course, you have to deliver on those two things. Those are both green checkmarks.
The third column, 12-months. You have to have the two APIs implemented. You have to have the mobile services implemented. Everything new, of an IT sort that’s going from that point forward, needs to adhere to the digital government strategy. And then, there’s all the GSA stuff. That’s a lot of work and they’ve got about 23 days, eight hours and 55 minutes left to do it, if I am on track for time. I’m probably running a little slow. Now this slide is completely inaccurate.
So digital content of, for, and by the citizens. Forgive the not quite correct reference to some of our founding documents. It isn’t really of and by the people, but what it is is involving the people directly. Going back to those two over-arching principles. Information-centric, you have to give people something that is useful to them. We have to be thinking about the consumer. This is nothing new to anyone in this room. But I’m can tell you, it’s pretty revolutionary for some government agencies who are thinking, “But we’re the experts. We know what we need to give to people.”
You’ve been hearing throughout the conference and, if any of you were at the pre-conference workshops, you know this whole notion of…we know that the IA is benevolent, or that the author, the content strategist or the content author is benevolent… We have the very best of intentions, but we don’t know how, when or why people are going to use the information we create. We have to take that into account.
Customer-centric, allowing customers to shape this whole thing, allowing new technologies, new tools, to enable to government to reach out to the populace and get them involved in actually shaping the websites that their tax dollars are paying for, to be quite frank.
These are both pretty important, I think, for someone like me, who works at a company that exclusively does federal consulting. Just a quick show of hands, how many of you have either federal or state clients, either domestic or… I know there’s a number of people from…OK. So you’re all right there with me on this.
This sounds like a job for me, and you, and everyone in this room, IAs and UX folks, interaction designers, all of the people that fall under this larger, we don’t quite have the right name, but I’m going to say, user experience field. Any of you who were here on Wednesday know that was quite a little discussion. What do we call ourselves?
It’s something that we can do, at least in significant part. If you think back to a couple of those over-arching policies, some of them were very security-related. Some of them were very big data-related. Obviously, we will have less impact there or perhaps none. But on those two, about being customer-centric and about putting the content out there, we can really make a big difference.
There’s a lot that we can do. I want to focus, specifically, on large scale online card sorts. In this case, I’m going to talk, primarily, about closed sorts. Although we have done a number of open sorts and reverse card sorts as well. Let’s talk about what we have in our toolbox, that we can use to start putting forward this digital government strategy, to help the people guide the websites that they’re paying for.
If we go rummaging through the IA toolbox, there’s a whole bunch of stuff that I’m sure a lot of us know. But, for the benefit of those who are, maybe, coming at this from other than a usability testing point of view, I want to just go over some of them. We’ve got open card sorts. That’s where you give people a bunch of cards, with the names of topics or content, and you ask them to create categories and to place them. Pretty straightforward.
Closed card sorts, you give them the categories, you tell them to put the cards where they belong. Reverse card sorts, or tree sorts, you have come up with some sort of a hierarchical folder organization…think about Microsoft folder view…then the users have to figure out where stuff should be in that folder structure.
Unmoderated usability tests, things like Usabilla, where you’re basically setting some tasks and having people do it. It’s all being captured remotely. It’s all being captured online. It’s all unmoderated and you’re just collecting pure data.
One-click tests, basically, giving someone a question or a task and seeing where they click first, and collecting some data there. When we get feedback tests, these five minute tests where you show someone a screen and ask them some questions afterwards or, give them a free-text field in which they can comment on whatever. That’s typically more design-oriented, but it can certainly help us.
I promised myself I wouldn’t read the jokes but, “I have this ultimate set of tools and I can fix it.” Anyone who was alive in the seventies will appreciate my bad Spicoli impersonation and I’m going to stop there and move forward.
Still rummaging. There’s a lot of tools that many of us in this room have used, or ought to be using, if we’re not, that aren’t specifically-directed, large-scale usability tools, but that allow us to gather data that help us to do our jobs or help our clients to achieve success.
Surveys, obviously Survey Monkey’s a good example. Page-based feedback mechanisms, this is that thing you’ll find on a lot of federal websites. It should be all, at this point, quite frankly, that says “how am I doing”, “rate this page”, “feedback”, whatever it may be. Giving people the ability to, on any given page in your site, or on any given piece of content that you’re serving out, to say, “Hey, was this useful? Was this helpful?”
An interesting little thing there, it’s a side conversation that we could have, about how do you do that correctly? Because you really want to be able to say, “Tell me about my site. Does my site work for you?” You also want to say, “Tell me about my page. Does this content work for you?” Is that two surveys? Is there a page-based survey and a site-based survey? There’s a lot of different approaches that are ongoing. I’ve been involved in executing on a few of them. I’d love to hear if any of you are doing anything there.
Customer-satisfaction tools, like 4C, using the ACSI, any of those sorts of tools. Click-analysis tools. Heat-mapping tools, which are typically mouse based, but there are some that are eye-tracking. User-research tools, like Ethnio, that’s collecting data you might not be able to collect otherwise, certainly not through standard visitor data. And, crowd-sourcing feedback tools.
These are all things that, if they are used strategically and judiciously, can really help us to do our jobs. If they’re not done strategically and judiciously, they can create tremendous amounts of additional work, analysis, reporting, and you’ll be lucky if it gets read. So use caution, but they’re definitely tools that are in our tool box.
I’ve got a case study I want to talk about. It’s going to be completely anonymous. It’s actually a series of large-scale closed-card sorts that we did for a federal client. Here’s the background. Basically, this is an IA redesign of a public facing “website”… and I’m putting “website” in quotes…for a government agency. The reason I’m putting “website” in quotes is because it consists of well over 100 content collections. That’s kind of a funny word, content collections, but it’s funny on purpose because we didn’t really know what to call these things because they’re not all websites.
Here, we’re talking about the main domains themselves, sub-domains, sub-folders that are separate and discrete, virtual sites that are within the normal folder structure, sub-sections that have a unique look and feel, the whole gambit of identifiable, cloistered, chunks of content.
The goal was to end up with no more than five domains, with a single IA, and put it all in one CMS, and have it all work perfectly. We’re in the middle of it. It’s happening. Can’t tell you it has succeeded yet, but I’m very, very hopeful that it will. Keep your eyes open. At some point, once it does succeed, I’ll reveal the name of the agency. We’ve done a lot of testing, many, many rounds of card sorting, both closed, open, and reverse or tree-sorting.
We’ve done usability tests with wireframes. We’ve done a whole bunch of interviews. We have created personas. Pretty much everything that you would expect in the user-centered life cycle. In fact, I’ll do a shameless plug right here. Myself and one of my colleagues, Weiman Ho, from User Works, have two posters that are going to be in the poster session tonight, both focused on different aspects of this case study. So there’s a lot more detail there. I’d be very happy to discuss that detail with you at the poster session.
Next, I want to talk about card-sorting basics for any of you who don’t know them. Is there anyone who’s never done a card sort, doesn’t know the basics? Because I will blast through this super fast, if not. Open, closed, tree, you get the idea. This goes back like 100 years. Literally, real card sorts were being done 100 years ago. If you trace back the idea, it goes back to the Greeks. This is nothing new, obviously, but the ability to do it online, large capacity, using tools like OptimalSort, that is new and pretty novel.
So, a card sort looks like this. This shouldn’t be a surprise to anyone. This is a standard OptimalSort closed cart sort. There’s a bunch of conventional wisdom about card sorts. You can even get it going to sites like OptimalSort.
They’ll say, “How many people do you need?” I’ll say, “Up to 50. 50 is a good number. Everyone agrees, there are certain minimum numbers you need to have a valid sampling, to have a valid card sort. There’s not nearly so much agreement on how many people you should actually survey. But everyone seems to be in this 25 to 35 participants range, before the statistical significance drops off and, basically, you’ve already got your answer. You’re not going to improve it by adding more people.
And, in fact, the conventional wisdom is that the more people you add, all you’re really doing is adding a lot of additional analysis and reporting work for yourself. It makes perfect sense. So how many participants should you have? I’m going to go through the next series of slides pretty quickly. But just in the interest of having something that I could cite and there are source cites at the end of our slide show, which I’ll put up on SideShow afterwards, if you want to scrutinize them. But let’s go through a few of the luminaries.
Freed, in 2012, said 15 minimum, 20 optimum. Gaffney, in 2004, four to six, pretty low number. Nielsen, 16, although, I’m sure he’d like to say five, because that is a magic number. It’s not three anymore, no matter what you think. Hall, six to twelve and that was from 2008. Robertson, four to eight, another low number.
We’re going to start getting into some higher numbers now. Spencer and Warfull, seven to ten. Here we start going up precipitously. Tullus and Wood, in the first of three either Woods’ or Tullus’ you’re going to see, in 2004, they figured 20 to 30. A year later, that number goes up to 30 to 40. I believe, it’s actually a year earlier, Wood and Wood decided well, actually, 25 to 30 should do it. But you’ll notice that all those top out around that 30 number. Above that number, you’re not getting any additional, real decent quantitative analysis.
The numbers aren’t improving. You’ve pretty much already discovered the information you’re going to discover. How many participants did we use in our case study? I should probably say “case studies” because it was a number of different, discrete, card sorts. Well, we did over 1000. It worked really well. It worked really well. We didn’t have any problems to speak of. I actually have a slide on this so I shouldn’t really get ahead of myself too much.
How did we make it work? How did we recruit for it? How did we do the thing? First of all, we leveraged social media. We leveraged blogs. We leveraged online tools like OptimalSort and Treejack. Basically, we did it for free, or at least on the cheap. Some human hours when into writing the blog post, to tweeting, to posting on Facebook. But basically, with a captured audience, we were able to do an awful lot without creating extra work for ourselves.
When I’m talking about a captured audience, what I’m really talking about here is if your client has a mail list, if they have Facebook followers, if they have people that are following them on Twitter, you have got a built in audience that you can draw from. You don’t need a recruiter. What you do need is a way to, sort of, sort people out, a way to do your own screening. Screening can be pretty expensive as you well know. Anyone who really knows how to do a good job, it’s not an easy task.
The way we addressed that was through a blog post. We put out a blog post that said, “Hey, here’s what we’re doing. Here’s what we need from you. We need your help making this website better.” We didn’t want to collect unnecessary PII about these people so what we did was this.
We said, “Hey, look, here are a bunch of links. If you are a member of the general public, click on this link and take the same card sort as everybody else. If you are a member of the scientific community, click on this link and take the same card sort as everybody else. If you actually are associated with this federal agency, if you’re a contractor, an employee, click on this link and take the same card sort as everybody else.” So everybody’s taking the same card sort, but we’re collecting some data about who’s taking it without really collecting data about who’s taking it.
The result, we gained really valuable insight and were able to make some fantastic IA improvements. There’re ongoing. There are as many as, I think, nine more usability tests that may happen during this particular project phase within this calendar year. We were able to do an awful lot with that quantitative data, but there was also a lot of qualitative data that we got. There was a lot of things that we were able to do.
Some of the other benefits of this approach were it served as an outreach and a feedback mechanism. At the end of some of the sorts, on the final page where we allowed people to actually just add comments and free text, we also allowed them to say, hey, share this on Facebook. Tweet to your friends that you did this. We were able to recruit from the actual exercise itself, raising awareness about what was going on and kind of getting people excited about the fact that they’re actually being asked to help use this website that, at least for some of them, they’d been finding difficult to use over the years.
It allows us to put quantitative and qualitative research side by side. The way you do that is through some very, very carefully crafted survey questions or through making the commitment…I did say no extra analysis and no extra reporting, but that would not necessarily be true if you’re going to actually ask some focus survey questions, because then you have go through it, parse it all out. But, from a quantitative standpoint, you can avoid that. You’re able to collect all of this qualitative data that you can analyze on its own later while you’re getting all this quantitative data that you can act on immediately.
It raises awareness of the contribution of usability studies, ups the presentation of online content. I really honestly believe that, to the world at large, how websites get made is sort of a mystery. To people that come to especially federal websites maybe get very, very confounded, because they are organized, say, around an organizational hierarchy or they’re organized around, let’s say, phrases of art within a scientific community or something. [sarcastically] That’s incredibly frustrating.
For people to see that these usability tests are ongoing…It’s great if you use a recruiter and get 12 people, and they come in, and they give you some great data, and you can improve your website. But the rest of the world doesn’t know it’s happening. If you’re out there and public about it, saying, “You know what? Help us make this better,” you can get tremendously large amounts, voluminous amounts of feedback. You can make people feel a little bit of ownership in the websites that they’re using.
Of course, this all supports the digital government strategy. It is allowing people to determine the content that’s most important to them, where it gets placed, how it gets placed. Depending on the qualitative feedback that you get back, it might help you make decisions about design. It might help you make decisions about responsive versus mobile. It just really depends on how you craft the test and how you craft those questions at the end.
It really doesn’t result in unnecessary analysis and reporting, at least not in terms of the quantitative data. But there are a lot of challenges and a lot of lessons that we learned, and I want to share some of those.
First of all, you have to have a really clear, well-established methodology. It’s really easy to throw together a card sort and just hope that you get something good. If you’re not very carefully selecting cards, making sure that they represent the depth and breadth of what you have available, if you don’t really have some expectation about what you’re going to find and therefore are not able to sort of be surprised and enjoy serendipity of people saying, “You know what? I think we’re going to go in this direction instead,” if you haven’t thought all that out in advance, you actually are going to create a lot more work for yourself. Because you’re going to have a mass of data and you’re gong to have to figure out what to do with it.
You need to have a clearly defined goal and a clearly defined scope. These things can really get out of hand, especially if you’re adding a bunch of survey questions at the end or if you have a large, large…in our case, these 100 content collections, we’re talking three-quarters of a million discrete pieces of content. For me to do card sorts that represent the depth and breadth of that content, you could do 200 card sorts and not get to a tenth of the content that’s in the system.
Fortunately, in keeping with the digital government strategy, we’re trying to consolidate as much as possible, so hopefully it’ll be a leaner site when we’re done. You have to make sure you’re not giving them too many cards. All the basics that go into crafting a usability study, in this case, a card sort.
You have to have a tool that can handle what you’re doing. There’s a lot of great card sorting tools out there. Many of them are free, many of them you have to pay for, and they do different things better. Some do very, very beautiful, rich reporting, gorgeous stuff that you can just drop into a report and really tell a story with it, but maybe they choke over 250 respondents when it comes time to actually churning out that data and churning out that analysis. You have to make sure you select your tool carefully and that you know how to use it. These are some pretty basic and obvious lessons learned.
So what do you think? This is the part where I’ve actually gone much faster than I thought I would, hopefully not too, too fast, where I want to know if any of you are doing anything similar. I don’t know if anyone from Canada or the UK is in the room, but I understand that there’s some people that are involved in the Canadian and the UK equivalent of the digital government strategy. I’d love to hear what you guys are working on there. Then, of course, to get any questions you might have, because I’m listening.
I’ve got a slide here that just says who I am and what I am, but what I really want to do is just end on this slide which is…I originally had preview but you can’t really see the thing. It’s scarcely a preview, but the two posters that we have out. They’re literally four foot by eight foot. They’re frickin’ massive. Everything that I went through here and a whole lot more is there in detail. So, specific questions certainly we could talk on the poster session tonight, but I’d really love to hear what you guys are doing and any general questions you might have.
To the person who asks the first question, and I don’t know how that would be determined, there’s a book to be had! It was probably one of you two but I don’t know who. You can pick one of the lovely ladies here. [laughs] Since I don’t know you, I’ll start with you so I can avoid any appearance of bias.
Audience Member: Hi there. I am working on a similar much, much, much smaller project. It is a federal website. In keeping with the open government initiative, we’re trying to leverage as many metadata artifacts as we can and finding very few available taxonomies, ontologies, thesauri, controlled vocabularies. So when you are done with your massive project, will that data be available for other federal websites? It would belong to the government at that point. Is that correct?
Jeff: It already actually belongs, if you are a US citizen, it belongs to you. It’s already out on data.gov, and it is out on the agency’s website, and it is made public. However, this agency does a lot of publication. And, the taxonomy…in fact, we are leveraging the taxonomy for the publication portion of the agency and are appending it with a lot of stuff we need in order to drive content on the website.
So, in fact, it might not be the most useful taxonomy because it is so very focused. Certainly, in terms of looking at how it is created, what assets are being used, it could be very helpful. Afterwards, I’d be happy to tell you where you actually download the thing.
Audience Member: Sure. Can I do a really quick follow-up?
Audience Member: What metadata schema are you using to be compliant? Is it Dublin Core?
Jeff: We’re using modified Dublin Core and I didn’t do the taxonomy, so I can’t… Do you happen to know, Shawn? No. I can get you an answer to that if you want to exchange information afterwards, but I know it’s modified Dublin Core or Dublin Core-based. I don’t know if there’s a lean version or just what.
To your larger question, and I’m not a taxonomist, so I will just answer to the highest level possible, metadata and taxonomy are similar, but they are not 100 percent overlap. It’s very important that, if you don’t have a good taxonomist, you do your legwork and figure out everything that you need to figure out so that you can set realistic expectations with your client. There’s a large movement across the government to sort of come up with agency taxonomies.
An excellent example, if you haven’t seen it, you should go to it, NASA has their taxonomy published. Their taxonomy is just brilliant. It’s flat, well, it’s not flat. It’s two levels. They had, I think, close to a thousand different taxonomies across the administration…I think they’re an administration technically. They had a multiple-year project. They consolidated down into one. It is absolutely brilliant. They have an entire website dedicated to their taxonomy. It describes how it plays into metadata on the NASA site and all the things that are metadata that are not captured by the taxonomy. It’s a fantastic resource.
Audience Member: I’ll check that out. Thank you.
Jeff: There’s a whole bunch of taxonomies available on data.gov and possibly also on info.gov.
Audience Member: I am from Canada, and we’ve been running fairly large studies as well. But I have a question about understandability of the instructions. Often, for example, with the OptimalSort tools, we’ve been using them, but we’ve had to refine the instructions over and over, because a lot of people don’t understand them. Did you refine them? To get a thousand people, you must have refined them and, if so, are those available?
Jeff: I could make them available. The sorts are closed. I don’t think they’re out there other than in documentation about the actual project. My colleague, Wei Min Ho, who’s here in the audience did most of the work there. We did a lot of iterative revision on the instructions.
Correct me if I’m wrong, they don’t look anything like the original instructions that OptimalSort provides you out of the box. They’re much leaner and yet they’re also much richer.
I will say that the recruiting for the test tried to address some of that. The blog posts where we were saying, “Hey, help us out. We’ve got a card sort. Here’s what a card sort is,” in very clean, simple, plain language narrative, we described the process before we even put them into using the tool. I can’t say how much they read it.
Certainly people commented on the blog post, certainly people clicked through from the blog post, although I’m not sure how detailed our metrics are. We tweeted and posted on Facebook links and certainly other people shared them. Obviously, there were other sources that people used to come to the discrete tests for the members of the public versus the members of the scientific community versus the govies, that sort of thing.
Shawn: You want to call on someone?
Jeff: I guess since I skipped you at the very beginning because I wasn’t sure who was first… [laughs]
Audience Member: I have that book. I have all of the books. I already bought it.
Jeff: Oh, well then, I’m glad I skipped you. Did you already have the book? Cause I’ll take it back. [laughs] Give it to somebody else.
Audience Member: Hi. It’s not really a question. But since you asked who in the audience is dealing with some of these large-scale, citizen-facing sites, I wanted to stand up because I’m dealing with a huge citizen-facing site called eBenefits. It’s a VA project. We get 250 million visitors a month. There’s lots and lots of traffic. It’s very high profile. What I’m interested in saying is, beyond what you opened up with, I’d be so interested to hear more about what you’re doing with your project, which you shared with the table a little bit at lunch that day.
But, also, I want to get a sense of who in the group are working on these large scale government sites, because I feel like, from a IA perspective, there’s not a coherent group of people that you can go to to ask questions. There are some government-based online bulletin boards and stuff about all things Web but not really that focus on the problems that we have and how we solve them. You talked about one aspect of research. We’ve got so many versions of that, what I deal with including cross channel, visual design, hierarchy, just millions of topics.
This is an indication of people who are actually doing that kind of work. I don’t know what to propose. I just feel like it’s time to have a group of people and a place to go.
Jeff: There is one thing that’s happening. My company’s involved with it. I’m not specifically. You company in a way will be. In fact, anyone here who’s with a company may be involved because there are at least 40 contractors involved in the new healthcare.gov where you’re going to be able and go and do all the insurance enrollment.
They’re doing something that’s just crazy innovative. It’s an iterative IA. What launches originally is going to be telling you about the thing while you wait for the ability to actually sign up and get into these healthcare exchanges. The IA is going to move through different phases. There’s some really cool innovative stuff going on, so I suspect that because of the over arc sharing. That’s like $250 million citizens potentially that are users for that site. I would hope that that group will have some lessons learned at the end, but they are very much heads down in the middle of the work.
Audience Member: Yeah, Jeff. What I want to let you guys know is, it’s a really good point, we’re revamping usability.gov right now, so that’ll be sort of the new [inaudible 28:39] . If you haven’t been able to, just so you’re on the mic.
Shawn If you haven’t been able to be participating in any of the user research and professional user research around usabilty.gov, drop Jeff a note or myself. Catch me later and that should be, hopefully, one of the forums that we can do this kind of work on.
Jeff: Actually, yeah. We actually just had a UX community group meeting at our company. The people that are working on the healthcare exchange are the people that are working on usability.gov. I mean, we’re all hanging out. Yeah, there may be some opportunities there and I’ll take that back with me.
Audience Member: I’m wondering if you’re doing any user testing with people with disabilities because that would be a minority group that does work with the federal government. I was wondering if you’re doing any testing with people with disabilities.
Jeff: It is certainly possible that some of the people that have been involved in our testing so far have had those sorts of considerations. I’m not aware, specifically, that any have.
I do know that we have…Shawn might be able to add here.
Shawn: Jeff and I work together.
Jeff: While he’s getting the microphone, we’ll say that we did have some specific involvement from individuals who were consumers, I’ll say, of the work of the agency that we’re working with. Those would be, I would classify them as at-risk communities. We definitely did have some people with special considerations in usability testing, but not that I’m aware of from an accessibility or other standpoint.
Shawn: On some of our other projects across the Department of Health and Human Services, we are, actually, including folks with disabilities in our usability studies. Down at HHS, we have the usability labs run labs down there, and we’ll always include them in our studies.
In addition, we have a visually paired member of our UX team down on HHS that does reviews of a lot of our work products.
Jeff: Also, my company has a fair amount of work with VA. I don’t know if this actually relates to any of the work that we’re doing, but I was a VA with a different company before Veteran’s Administration, Veteran’s Health Administration, VA and VHA.
They’re doing a very, very good job of trying to involve people, especially who, for instance, don’t have limbs. Wounded Warriors, I think, is the preferred phrase. Individuals who have traumatic brain injuries, for instance, who have very specific needs with regards to websites. You have a flashing GIF for some sort of a crazy interface, and it could literally set people off if they’ve suffered a traumatic brain injury.
There is some stuff in the community, and I could maybe try to put you in touch with some people doing that afterwards.
Dana Chisnell: Two things. The first is a shameless plug. I’m Dana Chisnell, and if you liked this session, you might like my flag session about crowdsourcing IA research for county government sites.
My question for you is, at what point do you actually observe people interacting with the information architecture to see if what you’re seeing, that they’re self-reporting or that their card sorting actually works?
Jeff: Great question. I’ll just skip back to this slide only because we’re still rummaging through the IA UX toolkit. I had crowdsourcing there. I mentioned it maybe very quickly or skipped it because it’s not something I’ve done much work with, but it’s something I’m really keen to hear more about because that seems like a real Pandora’s box if you don’t do it right, but it seems like it could be a tremendous tool if you use it with surgical precision.
To your question, the large-scale stuff that I’ve been talking about in this particular presentation, online card sorts, reverse card sorts, treejack, some survey-based stuff, it’s obviously very difficult, arguably impossible to observe people actually interacting with the system. We have done multiple iterations of usability testing—moderated usability testing by my colleague Wei Min Ho behind you, and some other colleagues as well—of iterative wireframes representing the current sites state, future site states.
We have, I wouldn’t quite call these observations, but we did an awful lot of focus groups and end-user interviews. We asked people to talk about how they used the site. In some cases, people will actually take us through “This is my typical experience on your website,” which was very revealing and also very frustrating at times to see someone who’s maybe using your website on a weekly or monthly basis struggling and not seeing a navigation path that’s maybe obfuscated by poor IA in the current legacy site. We have done that.
From the standpoint of the large-scale, heavily automated stuff, I’m not aware of a good way to do direct user observation. Maybe some of you have come across tools that allow that capability. I’d love to hear from you if you have, but it’s kind of a pseudo answer.
Michelle Cronister: Hi, I’m Michelle Cronister. I work for USA.gov, which is part of the general services administration, so obviously, I work for a very large federal website that tries to be everything to everybody. We have a lot of these issues. We do lots of iterative design and testing of those designs, but not on this type of scale just because of time and resources and also the requirements that we have to go through to get those things approved in the government.
I wanted to mention another tool that’s available to federal government agencies, which is the First Friday’s Usability Testing program, which is run out of another part of my office. If you go to howto.gov, you can find information about it.
They usually do testing of three and four individuals, where if you want to confirm your results from something like this, you can do formal moderated testing. It’s through us, so we help you with the tasks and things like that. I know a lot of agencies don’t always feel like they have the expertise or resources to do this alone. I wanted to mention that that’s a resource to help with some of these.
Jeff: That’s a fantastic resource. I know people have used it. I’ll do a shameless pitch for some of the work my company is doing. HHS has a usability testing lab. I don’t know the specifics, but I know certainly anyone within HHS, within Health and Human Services, any of the different operational divisions are able to use that lab.
I know that they’ve extended the lab to VA. The lab is fully staffed. There are a lot of resources that people don’t necessarily know about. Note to self, I’ll take back the usabilty.gov folks, maybe the idea of putting together a page or something that lists what’s available, especially to federal agencies that already exist.
There are a lot of agencies that set up a usability lab at some point. Some of them are collecting dust and some of them are being sued vigorously. It’d be great to see more of those things taken advantage of as well as the tools you’re talking about.
Audience Member: I actually just had a couple of comments and whatnot. I’m from Canada as well. I was working with the province of British Columbia. They had a citizen engagement strategy called Citizens at the Center and whatnot.
One of their first projects was collecting qualitative feedback on changing some legislation around the modernization of the Water Act. I found your presentation really interesting because we kind of totally overlooked the idea of collecting quantitative feedback through social media and whatnot.
We collected qualitative feedback, but overlooked the idea of collecting quantitative feedback. So I found that really great and useful.
Jeff: Yeah. It’s really cool. My company isn’t doing it, but there are some companies that we’ve been affiliated with that are doing really innovate metrics analysis of new media, of social media.
In fact, HHS says there are several agencies I know that have set up a person who’s sort of like the new media metrics person. Once we figure out how to really use the tremendous amounts of data we get there, I think that’ll be really, very, very telling.
Audience Member: I agree. The other thing I just wanted to comment on with the person who got my book. You were talking about having trouble finding an ontology to match government processes and things like that.
Don’t forget that you don’t have to just use one ontology. You can use bits and pieces from different vocabularies and whatnot, too. I did some work creating identify standards and whatnot, which wasn’t really for websites. It’s more for exchanging information within and organization.
I think that was one thing that we forgot, at that point. We looked just for what is the one solution that will apply to all of our problems. The other way to look at it is, what bits and pieces from different things can we use to match our own internal model, too.
Audience Member: We ended up with some more solutions to what you’re describing. It’s just always my hope that the government, being as large as it is, and with as many people doing the same kinds of work, that we could leverage what’s there, maybe come to some sort of standard.
Host: People in the corner couldn’t hear what you were saying. Can you just summarize?
Audience Member: Yeah…
Jeff: With the microphone.
Audience Member: I was saying that yes, I agreed with his suggestion and that’s what we ended up doing. My looking for artifacts that we could leverage was just because I would like to bring us into some sort of a standard method of describing our data that the government is already using just because there can be so many differences with the way you describe data, even if it means the same things semantically.
Jeff: Again, as a non-taxonomist, I would say I had the slide lessons learned about governance and process. So important for taxonomy and so important for meta data. It’s not a silver bullet. It has to be reviewed iteratively. You have to have a process for reviewing it. Those considerations can’t be emphasized enough.
Host: I got 3:29, if you want to take one more rapid fire.
Jeff: Anybody else have one last question?
Audience Member: What’s your next step?
Jeff: Next step. We have just finished our gazillionth usability test. We are doing IA refinements now. We’re launching a beta site very, very soon, start getting feedback on that, tweaking the beta site, and then going live, hopefully with a lean mean agency site within the year. The next time you see me, hopefully I’ll have something more to say.
Audience Member: It kind of strikes me that this is an interesting case study on card sort best practices. I’m wondering if you guys did an assessment of whether the findings and the data were actually that much different after using 1,000 participants than what they would’ve been say from the first 30 or 50.
Jeff: I don’t believe we did formally. What I will say is that on some of our open card sorts and also in a tree sort, where there was a little bit more data produced, we did look into sampling the total number and did find that we were able to successfully sample without losing any validity or fidelity in the data that we were capturing. We haven’t done anything formal. Did you want to add to that, Wei Min?
Wei Min Ho: What we did for card sorting basically, we are using that to inform later as stated. What we have learned, we were able to use that in the wire frame that was developed, which was tested using moderated usability testing.
Jeff mentioned the poster. We talked about the serious user-centered design process, what we have done and what influenced later stages and how…
Jeff: Yeah, the whole process is illustrated there. I can’t actually point. The column here, on the…Do I have a cursor? This column here, all these are center activities we went through. When you see it on…you will not be able to read it here. When you look at it on the poster, it’ll be pretty clear, the steps we went through and what followed what and why.
I guess that’s it because everyone wants to have tea. For those people who have posters, you have 15 minutes to put them up during tea.
Thank you guys. Really appreciate all of your feedback.