Jeff Eaton  
Hi there, welcome to personalization foundations at the 2020 information architecture conference, all online, all digital all streaming. Today we're going to be talking about how to build the basically a same foundation for content and digital personalization for your online publishing projects or website or anything involves lots of content and the need to tailor it for individuals who are visiting. I'm Jeff Eaton. I'm digital strategist at Lola bot, a content strategy design and implementation shop. We love content stuff. We tend to do a lot of big CMS builds for clients for whom content is really central to their business, intranet sites, large publishing organizations entertainment, education, they're basically orgs for whom content is central to what They do, or the core of their communications. And, yeah, so we've come across a lot of odd and interesting situations along the way that relate to personalization. And hopefully we'll be able to learn some fun stuff today. Before we go on, I'm going to give a brief tour of the presentation you're about to see, if we were to look at it as a simple flowchart. It'd be pretty straightforward, fairly linear. There'd be the opening greeting in which I introduce myself, maybe try to make a lame joke. There would be the moment where I sort of tell you what I'm going to tell you. That's where we are right now. Then later on in the presentation, I'm going to cook Glossary of sort of the core fundamental concepts and terminology that we're going to be using go through a case study or to some warnings about caveats and you know, things you should look out for another example or two and then some key takeaways, and hopefully, you'll be able to walk away with something useful. That's, you know, a pretty straightforward system, but it's also very linear. And the problem is, it's like all the people who are here listening today, there's a lot of you and you come from different industries, different practices, different, you know, career disciplines. And it'd be really wonderful. If I could give you know, each of you something different to, you know, go home with something that's tailored to you. So if I were to, like, go out and find out about all of you who were going to be attending, and if I were to make a bunch of extra slides, and, you know, skip over someone talking to you and rearrange a couple or spend more time on another slide, when I'm talking to you over there, you know, all of that stuff would allow me to make like a tailored version for every individual here and make sure that everybody gets the best version or that eccentric My company's services in just the right light or, you know, whatever. So like, you know, one person might get this path and another person might get a completely different one. It's fantastic. Now, that's the heart of personalization. And you know, it's a little odd to try to apply this to a keynote, because the keynote is traditionally very linear, very real time, even a little interactive here in person. But for a lot of web content and a lot of other digital delivery mechanisms, you know, there's nothing really preventing us from delivering those kinds of tailored experiences, other than, you know, the time and the energy and the infrastructure to make it happen. You know, we could just manually send out a custom, you know, email to every individual, it's written just for them.

Jeff Eaton  
That's the problem. Now, the the sort of Scrooge McDuck skiing through a pile of gold vision of what personalization could deliver that highly tailored, highly personalized experience. And that's just right for every single person who you know, touches a web property or uses an application or whatever, that's really cool. And everyone sort of feels like that could be the possibility. But in the example of my presentation, you know, best case scenario, if I were to tailor to everybody, I'd be on the hook for maybe, you know, 789 different permutations of my talk, just just to make sure that, you know, some people in the room got a tweaked and tailored version. And for a large enough, you know, web property, you could be scaling them up the amount of work by orders of magnitude. And that's sort of what brings us to the gist of this presentation. To quote Karen McGrane. You know, I in content strategy, sort of, I won't call her a gray beard that feels a little bit weird as as a category but she's she's one of the people who's been around for many, many years. And one of the one of the things that she often says is that most conversations about content personalization are basically fanfiction. For content marketing officers they're an awesome an amazing story about what what they wish were possible if only all of the pieces were in place you know can date Spock and fanfiction and you know downloading you know a new Drupal plug in or signing up with a service that can you know flip a switch and triple your sales is all possible in the world of content, personalization fanfiction. But the reality is a lot harder and you know, product pitches and technology set pitches aside. There's a lot of really deep foundational work that needs to happen before any of those systems can actually deliver what they promise. And that's actually what this talk is going to be about not individual products or how to do something with Optimizely or how to customize, you know, Sitecore or something like that. We're going to be stepping back and looking at the basic concepts of that your organization needs to figure out in order to have a game plan to either evaluate products, or start scoping out a personalization initiative or something like that. It's the groundwork of understanding. So that will help you evaluate stuff more intelligently, and hopefully actually accomplish your goals more effectively. So at a very high level, this is the model that we tend to use when thinking about personalization and starting conversations with any of the organizations we work with the the sort of block at the top goals and metrics, hopefully, that's fairly obvious. We're going to talk about it later. But it's not working, what we're going to dwell on initially. And the piece at the bottom appropriately structured content is sort of a term of art that we're going to delve into a little later as well. It's foundational, but it's not what we're going to dwell on right away. The first thing that we're going to talk through is the The center stripe here signals, scenarios and reactions. When a lot of organizations and a lot of product companies talk about personalization, they're talking about those three things. That may not be what it's called in a particular product or you know, in a particular, you know, service. But that's a sort of common framework. We're going to go through those one by one, and see how they interact with each other. So the first one signals, that's basically what you know about the current situation of someone interacting with your online presence or your app or your content, all of the different inputs you can use to determine what's going on who you're talking to stuff like that. Some of the really common ones are basic context, like at this moment, this interaction that I am having with a user What is going on in that context? Like, it could be stuff like, what web browser are they using? what time of day? Is it? What IP address are they visiting from? And what can we determine from that? An example is the best buy website. It uses the local time of the computer that the user is visiting from and on the web to determine what the local timezone is, and what the probable location of the user is using geolocation basically, what IP address are they visiting from, where are they? And it takes that local time and address it says, What's the closest Best Buy store? And what time will it open? Or what time will it close. So when you go to bestbuy.com, up in the top, you know, navigation bar, if you give it permission to access that information from your web browser, it will just automatically tell you when the closest Best Buy to you opens. That's an example of very, very simple But useful personalization based on sort of ambient context clues that are available just from the fact that someone is visiting the website. Another common signal that's used is behavior. Basically something that someone is doing while they are on the website or while they're using your product or service, if you can look at what they're doing, and make some best guesses about what additional information they will need or what else they might be interested in, because they're doing x or doing y. That's basically behavioral customization. The easiest and like most common example people are familiar with is like a company like Amazon, recommending books to you, based on what you've already viewed when visiting that website, or what you've already purchased or something like that. That it's it's a fairly low risk, low impact type of personalization because If someone's doing x, the odds that they will also want to do something very closely related to x are fairly high. And if you recommend something that's obviously, you know, intuitively related to what they're currently doing, even if it's not exactly what they want, it's not gonna necessarily feel out of place or weird and that that sort of mismatch between a user's expectations and what they're currently thinking about and what you serve up to them, using personalization is a big factor later. So this behavior based customization is one of those things that is fairly common. Another example is user profiles, you know, accumulating a sort of body of information that persists across multiple visits. So that you can not only you know, customize what what people are seeing based on what they do right now, but also what they've done in the past or what you expect they might be interested in doing in the future based on that past history. This can be explicit like in the case of Netflix, when you log on, you can create a profile name yourself and say, I'm Jeff, my wife Katherine, she has a different Netflix profile because she watches a lot more k dramas and I watch a lot more science fiction. And this allows us allows Netflix to provide useful recommendations that are tailored to our different, you know, or different profiles. Now, theoretically, most personalization plans involve building up or accumulating some sort of history of what a given user has been doing. Whether or not they explicitly create a profile or create you know, say I'm Jeff keep track of me. You know, even if they don't know it, there's some sort of body of information that's being accumulated about what they do either in like a web cookie or you know, in an organization's you know, database of, you know, people who've interacted with us. But generally speaking, we tend to group explicit profile management by a user and the accumulation of a pool of what they've done in the past, under this sort of profiles umbrella. Another big factor, you can get signals of what kind of you know what bits of data are out there that you can act upon from business partners, like for example, Facebook and Instagram are related to each other. So Facebook may not have lots of information about what somebody is doing. But Instagram may have tons of information about who they're connected to who they interact with, how frequently they log on stuff like that. And they exchange information. If you go and look at something on Instagram, or interact with someone on Instagram, it will go and recommend it to you on Facebook that perhaps you could friend them or perhaps you might also be interested in seeing ads about x or y or z because you tend to favorite photos that were tagged with those terms on Instagram. Now, this is not without its downsides. You know, there's a lot of fear advantages to taking in this kind of information signals from other external properties. In order to customize your content, I like to sum it or summarize it with you know, the worst case scenario here is you could get GDPR lawsuits if you are not careful about how information is shared how information is used. And if you aren't explicit with users about that stuff. We're gonna put a pin in this and talk about it a little bit later. But the idea is in the world of personalization, you know, taking in information from multiple sources, not necessarily just your own site and your own properties, is, I'll say commonish, but also tricky sometimes because of the legal and privacy implications. So it happens and it's talked about a lot, but be careful. And then the final sort of terminal point of this is The idea of large third party databases, individual signals that you can accumulate explicitly via your interactions with someone things like, what is someone's email address, what company do they work at where they often login from, you can accumulate some of that. But there are also giant third party databases out there that have accumulated downright terrifying amounts of information about about people from how long they've been at a current job, what their income is, what health conditions they have stuff like that. And many of these organizations are perfectly happy to provide as a service, the ability to say, hey, you've got some signals about this person, if you give us that information, we'll match it up with other stuff we have about them and give you a giant, big dusty air on the individual that you're trying to target content to. All this stuff is technically possible. Um, we're, I keep saying this, but we're going to put a pin in this and come back to some of the moral and ethical questions that ripple out from this. The reason I bring it up explicitly rather than just in dwelling on some of the simpler signals like ambient context or user behavior is because if you start talking about large scale personalization projects, you will start dealing with and interacting with product and services companies that will be selling this kind of stuff or will be leveraging this kind of stuff. And it's good to be aware and think through what the implications are, before just signing up and saying, Yeah, we can get all kinds of information. So think it through be ready to have the organizational institutional conversation about where the red lines are, before you sort of get pulled down that path. So that was signals basically, just the pile of facts, the raw material you have about what's going on when you interact with an individual. That's the starting point, the raw materials that you have for personalization.

Jeff Eaton  
The next step in that path is scenarios. They're basically like ways of turning all those raw signals about a particular interaction into like a meaningful narrative about what's going on right now. It's the story that you tell yourself about all of those signals. They reduce the world of data points into manageable buckets, so you can start making decisions. An example of that. So let's say you've got just a giant pile of information about what's going on right now. You want to boil that down and say, let's cluster that into a couple of useful, you know, piles of information. Instead of looking at all of those vegetables, we're going to say that all of those signals, we're going to boil it down to red things, yellow things or green things and all of our future personalization data you In this project, we're just going to base it on red, yellow, or green, rather than just the full giant pile of stuff. So this is one way of teasing out meaningful, like, you know, meaningful decision points from the giant pile of signals, taking all of those and saying, Here's three specific ones or three aggregates that we're going to use as decision points. It may be, you know, like, say, anybody who's on the west coast, we're going to treat as this bucket and anybody who's on the east coast will treat as this bucket or maybe browser related or what, you know what the source of their visit was, for example, someone who comes in from a marketing campaign will treat one way, and somebody who comes in from Google via organic search, we're going to treat another way. Another way, is that a very common approach is to start grouping people into personas. Now, what signals you use to group people into different personas is a whole nother Like, that's a whole nother art form. But one of the common approaches is to say, you know, we want to customize things based on whether somebody is a fanboy of our products, student or a shopper who's just browsing around or something like that. You know, these kinds of personas, how different organizations develop these personas can differ a lot. But basically, it's you know, like, what kind of person are they in? What kind of relationship do they have with us, is the common factor. And generally, it boils down to looking at those raw signals and determining of these valuable different personas that we've identified. What signals are we going to use to determine which bucket the person who's currently interacting with us falls into? So that's, that's another way of slicing and dicing things. One of the ones that I really tend to tend to lean on a lot is anticipated tasks. Like we may not know exactly who This person is, but based on what they've done in the past and what they're doing right now, we think that they're about to evaluate our product or we think they are in the process of trying to obtain support for a product or something like that. And personalizing what content you present to somebody based on the tasks you you think that they are trying to accomplish, is a great way to you know, sort of reduce some of the cruft from an overloaded, you know, communications experience to just what someone's looking for. Obviously, as with all like personalization, if you end up making the wrong judgment call and somebody who's trying to obtain support is getting nothing but in but you know, hard sell information or calls to action about purchasing a product. There'll be annoyed, but we'll get to you know, some of the whoops scenarios around around personalization later. Another one another common like way of approaching this scenario question is behavioral contexts, things like what situation does the user find themselves in, we don't know exactly what they're doing, but they might be traveling right now, or we think they're studying or we think they're hungry, or something like that, you know, an example of like a traveler might be, we know that this user always logs in from, you know, one of two different locations, and suddenly, they're logging in from a different laptop on the other side of the country. That could mean they're traveling, and we're gonna tailor that towards what we think they will need as a traveler. Now, again, that can go wrong. traveler could also mean someone stole their identity and is logging in from a completely different laptop. So it can go either way.

Jeff Eaton  
Demographic categories are another way that some people that some organizations end up doing personalization. That's one of the sort of scenarios that they tailor things around like this person is a Gen Xer. So we're going to show them content that we think is well done. tailored for that or on our marketing pages, we are going to use, you know, a giant full bleed image of Kurt Cobain as the backdrop for the webpage instead of something else. And that's probably a terrible way of selling products unless you're unless you're literally selling Nirvana CDs. But, you know, is this kind of demographic breakdown is at least one of the common examples that people use when talking about personalization. Now, you know, obviously, different organizations have success with different things we've actually found in a lot of our, like projects that this kind of demographic targeting is a lot less successful than some of the task based or scenario based stuff. Figuring out from context, what someone's trying to accomplish, tends to be a lot more effective than figuring out whether they're a grandparent or not, or something like that. exist example of how this plays out. Is Angie's List. It's a A company that we worked with a couple of years back. If you're not familiar with it, it's basically like a hyperlocal services company. They build giant databases of all the, say plumbers or gardeners or AI doctors in every city in the country. And they basically collect reviews and feedback from Angie's List members on how well those different service providers perform. So the idea is you move to a new city, you're looking for a plumber and you can hit Angie's List. And whereas Google would you know, just give you a general listing of everybody who you know he's in the area or has a webpage that claims they're in a particular city Angie's List would bring up a you know, vetted list of things that other Angie's List customers have written up reviews of and have worked with them and stuff like that. So it's basically a service directory for hyper local services with a with a high trust factor because other members have rated and evaluated them. And one of the things that they do Part of what, as a part of their, you know, web, you know, their web marketing efforts is they provide, like highly tailored hyperlocal search pages for like every zip code in the country for almost every service. So if you search for, like a plumber, Chattanooga, Tennessee, you are probably going to find an angie's list page that lists plumbers in Chattanooga, Tennessee, very close to the top results on Google. That's fantastic for them. And it's actually a fairly good way of finding plumbers in Chattanooga. But the interesting thing is, is that what they really want is for people to see this page, start reading some of the information and find that it's so valuable. They want to sign up for an Angie's List account. That's that's the purpose of these pages. And they do a good job. But what they found was they really wanted to experiment with tailoring each of those different pages more effectively to the kinds of users they were getting, and they would you know bump certain, you know calls to action up in the hierarchy, they would change language around things, they would change the design. But with a one size fits all approach this landing page, they kind of hit a wall, they'd hit the point where anything that boosted the effectiveness of the page for one group of users take it for another and it ended up being a wash. And the unfortunate thing is that we didn't have a lot of data signals coming in other than someone went to Google and search for plumbers in Chattanooga. So without a lot of additional information, how could they start doing something more effective to personalize that stuff to really reach different kinds of users with different messages. They had tried a lot of demographic and you know, psychographic grouping and you know, using what limited signals they had to assume, you know, to make some assumptions about scenarios. But we ended up working with them and talking through what they did know and what they felt really distinguished the different variations of content that they could effectively produce and present to people and What we ended up deciding was that it might be more effective to break down their scenarios into like essentially the the level of emergency that someone is usually in when looking for a particular kind of service. like are they in a, I'm buying something or I'm browsing for something or I am currently panicking mode. I think we called it like the the task urgency level.

Jeff Eaton  
And an example would be, if I'm looking for LASIK eye surgery, I'm probably looking to buy or I'm evaluating, you know, doctors specifically because I'm interested in having LASIK eye surgery. But if I'm looking for kitchen cabinets, let's say I might be in more of a browse mode. I'm probably you know, it's not that much of an emergency and I may be looking around and just you know, trying to evaluate my options and I want to be inspired but what different kinds of cabinets I could see Whereas if I'm searching for gas leak Chattanooga, Tennessee, we'll call that a panic scenario. Very few people aspirationally search for information about fixing gas leaks. So this The idea was breaking this down into the level of urgency someone is usually in we're looking at a particular service allowed us to tailor what kind of messaging got prioritized on a given on a given landing page, based on information we knew we always had that is what service are people looking for. And then we could build the technical infrastructure in the CMS to do things like reprioritize different parts of the page, accentuate a CTA, or give access to like a, you know, one time free view all of our, you know, reviews of local providers this once and you know, we'll give you a code to sign up in panic scenarios that basically takes the paywall away and lets people get a one time you know, access Access to, you know, high quality reviews. Whereas with kitchen cabinet browsing, you know, that's not as much of a thing and people aren't necessarily going to be to have a negative impression of the company, because of that being there. So we were able to make a lot of interesting decisions that prioritize different kinds of messaging, prioritize different communications in ways that wouldn't annoy or offend or frustrate people if we made the wrong call. Because we were able to be sort of a little more ambitious in what we, you know, pushed at them with brows, and, you know, dial back the hard sell in panic scenarios to be more helpful. And that basically gave them groundwork for personalization, they were able to build out sort of a, you know, a basic, you know, ability to deliver personalized pages. But the signals even though they were sparse, they were able to base it on signals that were always going to be there and we're always reliable, and later as they started developing the ability to do more effectively Signal gathering and they started getting better sources of information about the users who were visiting their site, they might be able to make more and different kinds of customizations to those pages. As their signal information got richer, but they rolled it out in specific locales, they started testing it. And they saw it. They saw a lot of success there. It wasn't ambitious as what they had initially intended, but ended and it ended up being very, very solid and performed well. So that's scenarios again, just to revisit basically, it's, you know, you take the raw signal, it signals what you know about a user's interaction with you. And then you make scenarios of the stories about what's currently going on, based on those signals. And the next step of that we already hinted at that a little bit with the the Angie's List. Story is basically reactions, what you are going to do, based on the scenario that you believe you're in And so the signal tells you, you know, what's the raw bits of data we have about an interaction? The scenario is the story you tell yourself based on that data. The reaction is what am I going to do? Based on my belief in you know about what scenario we're currently in. Now there's a lot of different things that can be done especially in like a you know, digital comes, you know, scenario or you know, a website. But some of the common patterns are recommendation. This is ubiquitous on the internet, whether you know, blogs or you know, shopping websites, or sales and support websites, you're reading a thing. And there's a list of additional things that you might also be interested in seeing. It's basically recommending other stuff.

Jeff Eaton  
Now, on most sites, even if there's no real personalization, occuring these kinds of you know, content rotators or related stories, elements of a design are usually a reflection of like your core IAA, maybe you've got an internal tagging system where things are tagged with topics, and what you can recommend his other pieces of news with a given topic or other products in a similar product category or something like that. So in in one sense, you know, other articles like this one is a reaction to the signal of what page is someone looking at right now, in a scenario that, you know, we would imagine is, oh, well, they're looking for information about a topic and they're interested in other things that are about that topic. But you know, generally providing customized recommendations to a particular person and they're based on their behavior or what their interests are. ground zero for a lot of content personalization.

Jeff Eaton  
Another common pattern We see is notification.

Jeff Eaton  
And this can either occur on a given website, where something new that, you know, something, some new information that's available, or a new feature that's been rolled out. If a given user hasn't, you know, come to the site in a while, you can pop up notifications, or if they have, there's something that you want to promote to them individually, swapping information in, you know, to a given slot on, you know, pages layout, with a call to action or a promo like card or something like that, for the thing you're trying to promote to this person is a common form of notification. Now, this can could also take the form of like scheduled emails that will go out to them later because they did something in particular on the site, or, you know, other other forms of contact, if you gather their, you know, SMS, you know, their SMS phone number or you get permission to send push notifications to And those are also common vectors for notifying giving people personalized notifications. Now, the downside of this is it can become incredibly annoying if it's abused. Or if you use those kinds of communications channels to basically spam people with, you know, what they with more information than they actually care about. Or there's a mismatch in what's promised and what's delivered, like you say, oh, would you like to, you know, get notified when news about things you care about, you know, is available, and what you'd always tell them is, hey, we've got a new product, we've got a new product, we've got a new product, that's obviously not necessarily what they were, you know what they wanted, even if the even if your organization wants to tell them about it, they may not be interested in hearing about it. So again, notification is a common pattern, but it can also it can backfire.

Jeff Eaton  
Variation is another one of those umbrella terms we use for basically taking a page and altering some part of it or how its organized or how its structured based on what we know about the current scenario. So the Angie's List example that we were talking about, here's a screenshot from the Angie's List homepage. In some scenarios, we had reviews of a service, like in this case, it's, you know, plumbers in Denver, Colorado, there were reviews that were right there, and it was presented using a fairly vanilla layout like this, depending on the user scenario, like in browse situations. We used a different design, and which one of these two modules like you know, does it which which performs better users, you know, just a list of reviews or like a ABC rating of each one of the service providers. Variation was The technique used here in some situations, we swapped in one version of this particular module and others, we swapped in another version. And we were able to test you know, which one worked better in different user scenarios. So that idea of taking a given page, and either swapping out different modules of the page or reprioritizing them or presenting them using a different design treatment or something like that. That's another way that different, you know, different content can be personalized for a particular audience. It's sort of the next step beyond that. And this is where it starts getting into stuff that is often tied deeply into a CMS platform rather than grafted on using a third party tool is dynamically composing content for a particular audience, like just, you know, dynamically generating the contents of a given page. For example, it's you can think of it as the same idea as variation, but dialogue Up to 100. And it's not always flashy, you know, high, you know, high gloss stuff either. This is an example of a tech company HR site that we worked on. Their basic goal was they had about 20,000 pages in their HR intranet, talking people through everything from benefits, you know, travel arrangements, you know, insurance, stuff like that. And they wanted it flattened down to 50 pages. That was their goal was ambitious and a little terrifying. But it led us in interesting directions. What it meant was pursuing a hyper personalization strategy, they wanted 50 pages that covered 50 key topics. And each page would be dynamically generated based on everything they knew about someone who was logged in on their intranet. language they spoke what department they worked for, what location they work in, what's their insurance they signed up for, what's their age? What's their inseam the whole deal. And data accuracy was an interesting challenge. Although they only realize that during testing, they thought, Oh, these are our employees, we know everything about them. But it turned out, you know, something being wrong in the employee database is one thing, but dynamically generating their HR instructions for say, applying for insurance based on incorrect information is a whole nother thing. So there was a follow on project to go through and actually that a lot of there, a lot of their HR database, because suddenly the stakes were higher. They weren't just looking up a particular page variation themselves, but the page they were being shown was pre customized to them. The end result was actually really, really cool, and it worked well. But it was a huge lift, they had to train editors to rewrite everything on these pages. It's like tiny snippets that can be swapped in and out dynamically. Translation and localization complexities. Were, let's say non trivial, because small snippets, like, if you're traveling outside the United States on business, you're covered by medical benefits abroad. That, you know, could be translated easily. But other short phrases that need to be reused might not have sufficient context to be effectively translated, you know, in isolation. Anyways, it's big, hairy project. But at the end, it worked fairly well. Interesting thing is that the big tangle we ran into was had nothing to do with actually assembling these pages, but rather, it had to do with what happened when they called their HR support team to ask a question about the page that they were reading, because suddenly every single page was dynamically generated for every single person who is logged in and it was just for them. So when you call Support Center, how exactly do they know what you're even looking at?

Jeff Eaton  
The problem was, there was no real way to actually let the Support Center read the same page without exposing every single bit of information that that the company knew about every single employee. And that got complicated because it also meant that some of their employees were working in countries where homosexual relationships were punishable by death. And things like what's the gender of your spouse was actually necessary for it to generate spousal benefit pages? And that caused some real questions that they they had to grapple with how important was it to have a perfectly tailored web page, if it meant that they were also generating a giant tool for call center people to be able to immediately access all of someone's private information As I've said many times, well, we'll put a pin in this well, we'll come back to this later, they did end up resolving it in a way that they felt sufficiently protected everyone internally, but it revealed some really big questions that they hadn't yet grappled with. So the third piece of this pipeline reactions is basically what you do. Based on the scenario you're you believe you're in could be promoting something that a user might know about. It could be recommending content to them that you think will work with the next thing that they want to read. It could be providing variations on the page, either visually or structurally, or, you know, maybe minor messaging variations like language or photography, or it could be completely dynamically generating content that's tailored for the particular scenario you believe you're in either a very individual view of that user, or the broad category that you think that user falls into. So that's it signals, scenarios and reactions. Easy peasy. Now we're going to jump back to that goals and metrics piece where we were talking about, you know, everybody loves goals and metrics. But in this context, what we mean by that is a couple of very simple questions. First, what is it that you want to change? Based on what you're doing with this personalization effort? When you have a particular reaction on your site or in your messaging platform or something like that? What do you want to occur that would not have occurred if you did not do that customization or have that automatic reaction? If there's nothing you want to change, if it's just this sense of wanting to personalize things, you may not want to dive in and you know, tackle the expensive have this kind of stuff, it's good to actually know what what, like what outcomes you're trying to achieve. The second question is, are you actually measuring it? are you measuring the outcome you are trying to change? It's shocking how often an organization will say, Oh, well, we're measuring x. But what we really want to happen is why. And there may not necessarily be any connection between those two things. So the third question is no, really, are you actually measuring what you're trying to change? I, it seems ridiculous. But we've come back to it a lot. And we're going to talk about why that's tricky in a second. But the idea is that without this idea of what you're trying to change, a clear idea of what you're measuring in order to determine whether it's changing and high confidence that what you're measuring, does actually correlate to the real thing that matters.

Jeff Eaton  
You're flying blind.

Jeff Eaton  
One of the biggest ways that this, despite us is something called the availability heuristic in psychology. It's basically the fact it's this idea that human beings, if information is available to them, tend to assume that it's important and that it's relevant to what they're trying to figure out. Or at least that it's more important than information they don't have easy access to. The good example is, let's say dashboards. You know, stakeholder dashboards are ridiculously popular feature in almost every kind of enterprise software. And the running joke is often that the dashboards don't necessarily need to correspond to anything useful. It's just people like seeing little needles on dials and crafts, and they feel good about it. And sometimes and you know, classic, you know, SEO and analytics. There's very common indicators that people are used to looking at things like what's the traffic to that page? Or what's the bounce rate for that page that everyone's used to ascribing value to we assume that, oh, if everybody comes to this page, but leaves the page, well, that must mean the page is bad. But that doesn't necessarily correlate to what you're trying to accomplish. In some scenarios, it might, if your goal is getting people to sit on that page and read something for five minutes or go to that page and then make a purchase high bounce rate is bad. But if all you want to make sure is that they know the answer to a particular question in the middle of a complicated task, then them hitting that page and getting their question answered and immediately leaving is your success scenario. So that's the availability heuristic is a cognitive bias that causes us to pay attention to the wrong things because the information is easy to find. And that's a real danger when looking at these at the metrics that that you're using to determine the success of a project. Jared Spool has summarized this in terms of like SEO and, and page analysis. I say basically, if a change to your website boosted sales by 10%, but doubled your bounce rate, like twice as many people just leave the second thing, they arrive, because they decide Nope, this isn't for me, but your sales went up by 10%. Would you make that change? Now, obviously, you should make the change if your sales go up, and it's not a fundamentally immoral or unethical or, you know, disingenuous, you know, thing that you're doing in order to make the change. But a lot of people have an instinctive reaction to metrics. They're used to paying attention to going in a direction that they think they shouldn't bounce rate going up is bad, even if sales go up, but counter intuitively, that that's the website doing its job. An interesting example of this is the state of Georgia. We worked with them for probably about a year building out a suite of websites for their different state agencies. And a lot of what they did was trying to build up useful and effective metrics for the kinds of information they were presenting to individual residents of the state and visitors to the site. They did all kinds of stuff. They were measuring traffic time on site, you know, what kinds of what kinds of incoming questions people were, you know, using in search engines to arrive at the site, stuff like that. But one of the ironic things that they discovered was that improving the information that was exposed to search engines simplifying and streamlining the content, making the topic Content better by all qualitative measures actually lowered the time that people spent on the site. It increased the bounce rate. And it decreased traffic to the site by all of the metrics they were used to using. They were failing. But they also knew from interviews with users, and what they what the changes they were making, were they they knew they were doing better, because people were getting the answers faster. So they spent less time on the site. They increased the bounce rate, technically, because people were coming to the site, getting information and leaving almost immediately. That was a win, but they had no way of measuring the fact that that win was an improvement. So we ended up stepping back and looking at other kinds of metrics, what stuff could we measure that we thought would correlate more clearly to the quality of the web content improving and the information getting into the hands of the right people more accurately. And what they ended up settling on was call center value. Because what they did track across all the different agencies was the kinds of topics people were calling in to the state of Georgia's 800 number about what they were asking how confused they were. So what they decided was they they may not be able to look at like time on page, but they could say, when we start revamping, the Department of Health website, is our call volumes going down about those related questions to the call center. And that can be our indication that regardless of what's happening with web traffic, the website is doing its job, you know, all other changes aside, is the call volume going down when the web content gets reworked. And the new information architecture goes up and are you know content tailored. mechanisms start rolling out. It wasn't perfect, but it was a much better metric that allowed them to more accurately measure what their real goal was. So goals and metrics, without those measurements, you may be accurately gathering signals coming up with useful scenarios. And, you know, wiring up cool products and you know, CMS, you know, mechanisms to generate interesting and novel reactions, but it may not be accomplishing what you care about. And without those goals and metrics, you won't have any way of really knowing that. The final piece of the puzzle is appropriately structured content. This is pretty critical. And if you've ever you know, talked to say a content strategist, or someone who's been responsible for managing a large pool of web content, this will come as no surprise. It's very easy to sketch out a fabulous vision of content personalization, but Without the appropriately structured content and without the support systems there to generate it and maintain it, the whole thing is either a, you know, Ghost Town, or it collapses under its own weight. Now, what do we mean when we say appropriately structured content? Things like you know, is it tagged so that you can actually map different pieces of content to the scenarios you care about? Is it broken down into small enough pieces so that you can actually swap them in or out? Based on the scenarios you're you're in? Do you actually have enough content, the different reactions that you've planned to roll out on different scenarios have content that matches them? Those are all really important ones. It's common for conversations about structured content in these you know, big in big personalization projects to basically treat content as either a big bad on pile, you know, unstructured. pile of stuff, bad content. to contrast that with good content, like somehow it's in some in some broad general form, it's structured well, and it's tagged. And that means we can use it with the personalization engine or something like that. But really, the the good and bad versions of that, you know, are boiled down to a lot of different kinds of scenarios like bad could just mean, we write nothing but blog posts all the time. So we want to start showing, you know, events in a sidebar rather than just additional articles. But we don't actually create that stuff. And we're not, no, we're not we're not creating it in a way that allows us to call it out differently or treated differently. It could mean that we're already doing personalization work, but it's all in one off a B tests. You know, we got a product like Optimizely and we're identifying it Individual URLs, where we're swapping in different copy for headlines, or CTAs, or something like that, which works on, you know, on a page by page basis, but it's effectively impossible to scale. content that isn't sufficiently tagged or isn't, isn't capable of being automatically pulled in in different scenarios or isn't only manually assembled into aggregate pages. And you can't set up automated mechanisms to swap those things out based on different signals. Those are all different kinds of unstructured content for the purposes of personalization. It may not look unstructured, it may not seem like bad content, but it may not have what you need to actually accomplish the stuff you're trying to plan, exam, you know, and there's also different kinds of good content, depending on what your plan is.

Jeff Eaton  
You may need you know, like a decoupled Content API, so that you You know, your, your marketing, you know, your messaging platform can pull out bits of content and assemble them into an email, in the same way that your website can swap out a different CTA. Or you might just need a better taxonomy system, a better mechanism for tagging and categorizing the content that's already there, just so that it can be recommended or cross, you know, cross referenced to each other in different scenarios more effectively. So, what appropriate structure is, varies based on the kinds of scenarios you're looking at and what kind of reactions you're planning. An example would be that HR website we talked about a while ago, the you know, the benefits and you know, travel information and stuff like that. They had those 50 topics they wanted to break down the, you know, site to so instead of 20,000 pieces, you know, 20,000 pages, they were only And it would deliver up the customized version of each one of those topics. Now what that meant was, we actually had a hierarchy of different content components that made up the site. At the top, there may be a travel benefits page for as far as the user is concerned. But under the hood, we're actually different pieces of content tailored to different audiences like North America, Europe, or China. And then underneath those large like chunks of content that were regionally specific region specific. There were also individual snippets of text inside each one of those large documents that had custom XML markup or actually custom html5 markup that allowed them to flag individual paragraphs or even individual sentences or statements as being targeted towards very specific groups. So there might be a particular paragraph on the travel information page that talks about insurance benefits, but it only Actually rendered out to the visitor who's reading the webpage, if they had insurance in the USA. And they don't, they also had things like little, you know, tokens that can be placed inside of the text, that would be a link, and you know the name of their actual insurance plan and a link to that insurance plan. Rather than writing different versions of the paragraph for each permutation, they dropped in those tokens. Now, this is a very, very fine grain like, you know, high, high complexity example of what I'm talking about. But it's what we mean when we say appropriately structured content. Swapping in different content recommendations based on a user's favorite topics does not require this level of complexity, but dynamically generating a highly tailored highly focused, you know, visitor specific, you know, either sales pitch or page of support information. It does require fine grained information and sometimes a different approach, then, you know, different tools are capable of delivering, and being able to figure out what underlying content mechanisms are necessary, what kinds of structures are needed to support the granularity and relationships that your personalization plan needs. That's an important part of the process without content to match. A lot of this stuff just comes back to what Karen McGrane said, it's fanfiction for CMOS. So what does appropriately structured content mean? It's structure that fits the kinds of variations that you're trying to generate. It's granular at the level that you need. You need metadata that captures the the meaning and you know the semantics of a given piece of content so that it can be pulled in and out in an automated fashion, not just manually. You need stuff that's appropriate in multiple contexts and multi use multiple use cases, if you have, if you're dynamically generating aggregate content especially, you need to make sure that you don't have to write infinite numbers of variations for that stuff. Very few organizations have just the raw manpower for that. And you also need process that supports your scale, often, tools will be excellent at one off a B testing or small scale content variations. But the kinds of things that are very impressive in demos can't be done at scale, or at least are very labor intensive. Other kinds of stuff, you know, you may be able to, you know, do certain kinds of content swapping at scale, but it may be fully automated and you can't manually override it for the

Jeff Eaton  
for specific signals or scenarios that you want to so understanding what you're trying to do, whether you have the content to support it, whether the processes of the tool you're evaluating actually supported at the scale. You're going to To be using it, those are all important considerations. So if appropriately structured content, the final piece of this puzzle, and ideally, together, all of these things are working, working with each other and informing each other. You start with your goals and metrics, what are we trying to accomplish with content personalization that feeds into the evaluation of what signals you're going to try to gather, what scenarios you're going to plan for what reactions you want to, you know, actually use to deliver custom content. And that feeds back into the goals of saying, you know, what, what are we going to measure in order to see whether or not we're actually making progress towards those goals are the reactions that we're triggering actually accomplishing what we want. And then the goals and metrics and the signals scenarios, reactions also inform planning for what content that you're either going to need to produce, or what changes are going to need the need to be made to the existing content, and sometimes you'll find it there. Just a hard stop, you are only going to be able to get so far in like an MVP rollout for a personalization project. Because there's limits in the content and there's only so much you can produce in a period of time, you may only be able to do it in a particular section or for a particular project. And if that's successful, more resources, and you know, more tooling may be able to be rolled out in order to convert more or larger swaths of your content to a personalization effort. So, all that said, what happens when it goes wrong? We're gonna wrap up with a with a sort of digression into that set of questions. The things that we see most commonly when when things have not been planned effectively or when bad choices have been made in you know, either the implementation or the execution. The most common one is mismatched content structures are no metadata. You know, an organization may have a giant pool of content, but There's no metadata to actually pull it in in an automated fashion. Or they have content that was built out for visual design, you know, restructuring of pages, but not necessarily for swapping in messages that are tailored for different audiences. So even if they have structured content, it may not be structured for what they need to do in a personalization, no plan, or they have metrics that don't really correlate to what they're trying to accomplish. They tend not to these projects tend to sort of drift around and never really actually changed much, even if on a month to month or quarter to quarter basis. They're demoing all the cool stuff that they're doing with personalization. unreliable signals are also a big issue. Basically, you know, some of those pieces of data that we assume we're going to use to construct a particular scenario or to swap out different reactions on we have to be fairly confident that data is accurate, or we need to plan for what happens when it goes wrong. Or you know, what happens when it's missing entirely. Simple example is a website that assumes that we'll always be able to get a user's location in order to recommend a, you know, closest, you know, closest store or something like that is lost if someone views it from a desktop browser and doesn't grant permission to their location data, or they have location data turned off on a cell phone or something like that. You have to plan for that kind of stuff. And, you know, assume that sometimes it will not be available. Just so assumptions are another one. This is sort of like the, the the stories we tell ourselves about the scenarios are not necessarily what is true. We can spin all sorts of interesting tales about what a given signal that we can detect means about what a user is doing. But it can actually just be you know, it was True once and we saw it happen once. But it's not always true when that signal is being received in the example is, you know, assuming that someone, someone who's using their phone to look at a website is on the go in some fashion or they're traveling or something like that, in the early days of, you know, like mobile sites, the assumption was like, if you're on your phone, you must be like, you know, jogging through an airport looking for quick information. But these days, you know, just as many people are using their phones, to browse the web or to interact with, you know, organizations content from their couch, as or bed as they are from an airport. So, the, the scenarios that we construct, have to be tested against the real world to make sure that we're not just telling ourselves certifying well, that we're not just writing more fanfiction for our CMOS, reproducibility is another one. Sometimes we can construct such a laborat scenarios in such elaborate, like multi dimensional combinations of data that we're trying to use, especially in the dynamic construction scenarios, that it's almost impossible to reproduce what a given user is seeing without just being that person and logging in from their computer. And this is problematic because it means that it's really hard to measure the effectiveness and preview what something is going, you know, what some what someone is actually going to encounter. Is it possible to effectively reproduce what in a given scenario will be shown to a given user? editorial overload, that's another big one, it's very easy to construct a magical wonderland of content personalization that requires 10 times as many content creators and content editors on a site and that is rarely what most organizations are willing to sign up for. So sometimes that means figuring out how to effectively In creatively reuse the content that's there, sometimes it means breaking it up into smaller pieces so that it can be reshuffled, rather than being rewritten from scratch. But at some point, it has to be planned for. And you have to start thinking early about what the impact on workload for the content creators and managers will be. creeping creepiness is another one, depending on what signals you're using, especially if it's something that a user doesn't think of as being a signal that you're receiving from their interaction, or if it's information that they didn't explicitly tell you or give you permission to access. It's very, very easy to weird people out, creeps them out and give them the sense that you're you're really violating their privacy by tailoring something to something that they didn't really intend to tell you. But all of these, you know, their their let's say their execution problems. They're well within the realm of correctable errors.

Jeff Eaton  
There's, there's a big issue in the world of content personalization. That I feel like it needs specific attention. I think of these as failure states that go beyond harm to your organization, and become harm to the actual users of your service or to the people that are in you know, that are in the vicinity of your service. A good example is bias amplification. Things like creating customization, or sorry, creating user profiling systems that deliver really perfectly tailored information to a user's tastes, but that just end up amplifying what they already think in ways that cause them to make money. poor decisions. This can also be a feedback loop. For those of us who are building these systems, we can build, we can look at signals, build scenarios in our minds and build reactions to them that all they really do is amplify our assumptions about who we're talking to, you know, tailoring something to people in a particular socio economic group, and then saying, hey, look, most of our users are in that socio economic group. Because we tailored the messaging to them. Those sorts of feedback loops are very easy to construct. It's also shockingly easy using, you know, vanilla content personalization, to engage in things like illegal discrimination depending on what business you're in. Facebook recently got into trouble because people were using their art their their ad targeting system to create housing ads, that elite discriminated against racial minorities, they were creating basically housing ads that could never be seen by races that they didn't want to rent out their houses to. And that's actually illegal in the United States. And it's very easy to build systems that facilitate that kind of stuff and and harm both groups, and your organization because of the consequences of those illegal actions. subversion by bad actors is another really significant issue. Any system that uses ambient signals or explicit signals to customize what people see and especially what other people see, for example, popular topics today on our website include x and y and z. Anytime you give people out there, the ability to change in some fashion What you're generating and presenting to everyone, the potential for bad actors to reverse engineer the mechanisms of your of your personalization system and use it to deliver messages they want other people to see. That's fairly significant. The easiest example of this is people who use, you know, dummy dummy accounts to purchase and rate products on Amazon cancel the orders, the rating stays up. But suddenly they've gamed the system with a bunch of positive ratings and now everyone is seeing you know, is seeing, you know, a product that may not otherwise merit as much attention. This particular scenario can get very bad indeed. And I like the absolute worst case scenario that I've seen is literal genocide, based on user on content personalization. If you've read about if you've read about the Myanmarr genocide, basically there's there's a long term conflict with the government of Myanmar and Facebook was one of the key sources of information in that region. For news, social connections, stuff like that was the primary interaction that most people there had with the internet. And at one point, a couple years ago, Facebook De prioritized official news sources to refocus on personal interactions. Now, there were a lot of reasons for that. They felt like you know, there were so many different, you know, websites out there that were using Facebook as a messaging platform to reach users with this core idea of keeping in touch with people you know and care about was being drowned out. So this effort to reprioritize what people would see in their news feeds seemed like it made sense and was a new decent and good thing to do. However, what it meant was that in Myanmar, the military was able to create hundreds, even thousands of propaganda accounts as fake individuals, or even, you know, sending out their messages through other through real individuals that were working for the Myanmar military, and they broadcast calls to violence against minority Muslims. More than 25,000 people ended up being killed in Myanmar, more than half a million people became refugees. And a lot of it the UN actually had released a report on it about two years ago. A lot of this was traced back to the propaganda efforts that were powered by Facebook's newsfeed reprioritization algorithms. Facebook didn't intend to build that system. They did not intend to build a genocide engine, but by creating some basic rules for what kinds of things To be prioritized, what kinds of things they assumed their users would want to see what scenarios they were imagining based on the signals that they had and what reactions they wanted. Based on that. They built a system that allowed the bad actors to manipulate and and cause immense harm. If we take the work of building content personalization system seriously, we are building systems that identify, monitor, and even train and reward real people. For you know, what they consume, what they read what they do. And increasingly, as we automate that stuff, we're delegating the decisions to tools that learn and automated and amplify our assumptions. It's very easy for us to create systems that feedback loops for our own blind spots and our own biases, and hurt and harm other people based on. So a question that I want to dwell on as we as we wrap this up is basically how do we, how do we build more responsibly? How do we build personalization systems that are more robust against this kind of failure, state failure states that don't just embarrass us on twitter but actively harm vulnerable people? A good first step is integrating marginalized groups. If, whenever possible, seek out and seek the input of groups that could be hard. And you know, this often gets lumped under like diversity and inclusion initiatives in teams and stuff like that, but From a systems building perspective, it means that we, we need people on our teams who understand and can recognize the harms that could possibly come from a system. Because they're part of the groups and the marginalized individuals that that will be harmed. And if we don't have those people represented our teams, we need to seek them out. And we need to get their input and we need to take it seriously. Another another key point is to do Red Team exercises, to to go through testing processes that allow and empower members of our teams to approach our systems and our personalization tools and our data gathering systems in an antagonistic fashion and try to figure out how they could be exploited to do you know to leverage the data and the rules and the metrics that we're creating, in a way that we wouldn't want them to be to be leveraged, we often tend to approach testing as a way to ensure that the right thing happens in a given scenario. But Red Team testing is about making is about figuring out how you could get the wrong thing to happen. And that allows us to figure out what we need to change in systems to make it less possible for those negative outcomes to occur.

Jeff Eaton  
And also study the systems in which our work exists, not just the specific tools that we're working on or the specific projects that we're that we're building. What are the societal and economic and political systems in which these tools exist, what what's the surroundings of the stuff that we're making? All of our work exists in the real world. And ignoring that means that misuse or, or, you know, harm is much more likely. And finally, the other aspect of this is to own the choices we make, and go on record as much as possible. If we think that harm may come might come from a particular path, telling other people on the team, making that explicit calling it out and saying, we need to make a choice about this, even if it's not something that you have the power to prevent, making sure that that that people have to go on record and make an explicit choice with a warning is very powerful. Some good reading on these, in this realm is technically wrong by Sarah Walker botcher. Algorithms of oppression and living in information are both great books as well. They explore both how data automation can Reinforce, you know, oppression and harm and living information. fantastic book about the broader concept of examining the the context in which our work lives and approaching it responsibly. So deep breath that was that was a heavy hit. But we'll, we'll, we'll move on.

Jeff Eaton  
What are the what are the key takeaways? What are the key things that we can do to help a person or content personalization projects succeed? before we ever like pick a project or sorry, pick a product or a platform to use? First one is knowing the data that we have to work with and its limits. That's both the signals that we can draw on to do our customization and the underlying content that we have to work with, in building out customized communications and experiences Know what your goals are, and measure carefully, like choose what it is that you're measuring, that's as close as possible to the goal itself. And if you can't measure the thing you actually are trying to change directly, then try to choose your corollaries as carefully as possible. Like in the case of Georgia. You know, we couldn't measure whether or not somebody came to Google found the answer their question without ever visiting our website and, you know, went on with their day, but we could measure decreased call center volume, and that was something that was well not exactly what we what we were trying to do, it was close enough that it was a better corollary than just you know, traditional page metrics, planning invest in structure and metadata. And, you know, also the the editorial resources and the content management resources in terms of people that are necessary to be build and maintain the structure to support your plan and crawl before you can walk. It's better to build out like a, an MVP that looks at a couple of particular data, you know, pieces of data matches one scenario, if it looks like it's true, and rolls out one or two test reactions that you can measure whether or not they're effective, it's better to do something like that and see how well it goes and plan to scale it up than to try to go hot, go go all out and build a giant system that crumbles under its own weight after a year or two of development and you know, testing and finally,

Jeff Eaton  
identify and prioritize safety and ethics as key considerations before any like business KPIs. And this is easy to say, but if we're seeing About not doing harm to other people with our work. This has to be an integrated part of the planning and assessment for any kind of user data gathering and targeting effort. Otherwise, we're acting irresponsibly, and we're essentially certain to harm.

Jeff Eaton  
if you want to heckle me on Twitter, I'm eating. If you want to read more of my writing or my work, you can visit lb cm slash eaton. It's a quick link to my page on Lola bot com and you can join in visit the information architecture conferences 2020 Slack, and there's a bunch of us there chatting and and dialoguing about these issues.

Jeff Eaton  
That's it, drive carefully. Come back soon. Thank you for listening and I hope you find the information useful.

Transcribed by https://otter.ai