Amanda Hi, I'm Amanda. Alex And I'm Alex. Leora And I'm Leora. We're all located here in Austin, Texas. Amanda And we're sorry, we can't give our talk in person, but we're still excited to be presenting. Alex Welcome to our presentation on adventures in user centric taxonomy, or how to get job seeker feedback into the back end. Our talk combines three different perspectives: user research, taxonomy and product will tell you about how we took an idea and developed it into a product that you can now see live on Indeed, indeed is a job search engine available all across the world, on both mobile and desktop. As the world's number one job site, indeed has over 250 million unique visitors every month. Our mission is to help people get jobs. Job seekers use our site for free, and we constantly strive to provide the best search experience. Let us now introduce ourselves. I'm Alex Ali was working on the taxonomy team at Indeed, our team's goal is to met the data behind the search engine. We create and curate metadata structures that allow us to index content in a meaningful way. We work behind the scenes to enable document matching, semantic search and a richer UI experience. Amanda And I'm Amanda, a UX researcher. UX researchers talk directly to customers to understand problems that products might solve. And we also test new features and products with users. And although I'm now with duo I worked with my co presenters at indeed for two years. My work in indeed focused on engaging job seekers in particular to deliver the best search experience possible on the site. Leora And I'm Leora the segmentation manager on the product team who works cross functionally across the business to configure our site for unique audiences. We'll be speaking more shortly on what a second Manager is, but suffice to say it's a similar role to a product manager. This is a story about how cross team collaboration created better UX. We're going to share with you a case study, our journey started with the realization that as Margaret Kelsey from envision noted, design for everyone is a design for no one. Our categories, filters, and our search results were designed for the average job seeker. But the more we talk to our customers, the more we realize that there really isn't an average job seeker. And we already knew that our site did not serve all job seekers very well. Amanda So let's set the scene. It's early 2018. And at this time, many people on both the UX and the taxonomy teams were trying to facilitate cross teamwork. On the UX side. This was challenging because UX was fairly new, and indeed, it had only really gotten going in late 2016. And UX was often thought of in terms of designing the user interface or front end while other teams Such as taxonomy were thought of as being more on the back end. Note that what we're calling the back end for the purposes of this talk is meant to be a generic catch all for the data and algorithms that power the site. While front end refers to the display that you see on the screen. Across taxonomy, and UX, we all knew that the back end was crucial to being able to design innovative products. We also knew that taxonomy work was already in place for job occupations, meaning nurses, truckers, etc, and that this data could be used to classify search results. But there was still no established process within the product organization for collaborating across teams. So despite a general desire to design with taxonomy, data, and many one on one meetings, we let the momentum for further work. Now, Around this time, I've been talking to a product director who had an idea for a new cross functional team, which would deliver tailored experiences to users who will call job seekers At indeed, any starting around quarter two of 2018, we began the process of forming and hiring for this team now called segmentation. I was brought on to help design the UX research strategy that would be an integral part of the team, which was very exciting for me. But before we talk about that, we should talk more about what segmentation is. Leora What is a segment? This team worked on segments, which we define as categories, verticals, occupations, or Behaviors of a select group. For instance, when we speak about categories, An example might be people who work remotely or veterans. industry verticals might be workers, such as those who work in the government sector, and occupations would be cooks, lawyers, truck drivers, architects. And lastly, an example of a behavior might be stay at home parents. Why was this team created, segmentation was created to identify options communities to customize our site for unique users. Rather than treating everyone the same, we acknowledge and celebrate the differences. Amanda One decision we made early on was that the segment managers should be able to do their own UX research and that they should understand taxonomy processes and tools at a high level. For this reason, applicants interested in the segmentation team completed a UX research exercise and a taxonomy exercise as part of the interview process. The taxonomy director at the time and myself both served on the hiring committee and conducted individual interviews with candidates for the taxonomy exercise. The director familiarize the candidates with the concept of taxonomy by reviewing the shoe section of the Nordstrom website and talking about categories for shoes, boots, slippers, types of heel color, etc. She then asked the candidates how they would classify jobs in a similar fashion. And this wasn't so much about technical detail About the ability to categorize the facets of a job. The UX exercise consisted of a customer interview, we gave candidates time to write a basic script, then had them complete a short customer call. The intent here was to see where candidates were in terms of their ability to talk directly to customers, especially in a non sales context. Once the segment managers were hired, I trained them in essential recruitment and interviewing skills so that they would always be able to conduct research with the customers they were representing. taxonomists train them in tools and existing data. After this training, we were ready to start interviewing job seekers. Leora We decided to start our research with the nursing segment. The country has experienced a nursing shortage for decades, but an aging population means the problem is about to get much worse. That's one statistic that led us to make this decision. another statistic was that employment and registered nurses is projected to grow 12% From 2018 to 2028, much faster than the average of all occupations. So in other words, the Bureau of Labor Statistics is telling us that the marketplace has demand from both employers and job seekers. Alex In order for your EIN Amanda to collect user feedback from nurses, they needed to find the right participants for their interviews. So they turned to the taxonomy T. We had done extensive analysis and research of the US job market and had identified a comprehensive set of occupations including nurses, or occupations taxonomy had definitions in place for who was or wasn't a nurse, and how their duties were different from those of other health care professionals. We also knew what employers were looking for when hiring nurses, things like license certifications or skills. your EIN Amanda use this information to set up the screening process. They filter out potential breaches The sequence that didn't fit the study criteria, such as positioning systems, certified education, AIDS, surgical technicians, or phlebotomist, and only kept those that were a good fit. Leora Once the participants were selected, we performed one on one moderated discovery interviews in q3 of 2018. We spoke to a range of nurses, certified nursing assistants, licensed vocational nurses, registered nurses and nurse practitioners. Here are some of the memorable quotes from the participants. I think that there are certain things that shouldn't even be in this category. If I'm searching for an RN, a registered nurse, medication aide doesn't need to be there, or nursing assistant or home health aide, a nursing assistant aide is not going to look for an RN job. To be honest, that's a little offensive. And here we have this quote by Terry who's an RN and we wanted to make sure to let you know that we've changed our names to ensure That our participants remain anonymous. For our next quote, Andrea, the RN says the emergency department can be kind of blunt, very concise, quick, not super in depth type B nurses, ICU or Intensive Care Unit nurses are the flip side. They are very intense with their care. They're very type a medical surgical is the, quote, dark and stormy place of nursing. Most people don't necessarily want to work med search, but it's known as a good place to start. Amanda We reviewed the themes from the interviews and shared our findings with stakeholders in q4. And this brings us to 2019. Both the designers and taxonomists had been working through the end of 2018 and by January of 2019, we were ready to test some prototypes. As we started putting prototypes in front of users taxonomy began its work on data for medical specialties. Alex I should note here that on the taxonomy team, we considered using medical specialties as facets as early as 2017. They show up across healthcare and education occupations, and we knew they had great potential for increasing search relevance. But there were trade offs involved. And we had to prioritize implementing what we already had in our occupations but sooner rather than continue refining it. Without implementation or data will not be disseminated within the indie ecosystem. We needed to maximize impact and knowledge good be the enemy of great. You're on Amanda's UX readout created an ideal opportunity. It showed that medical specialties are vital to nurses are job seekers, and that our website needed to extract this information in order to make it easier for nurses to find a job that meets their needs and preferences. We finally had the momentum to start building a taxonomy of medical specialties. This involves using both internal and external sources to do research, we found that there were already multiple taxonomies of medical specialties out there, including the woman team by the American Board of Medical Specialties. We're going to just import them as we needed. them either was designed with our users in mind, and would reflect the language that employers using jobs, as well as the queries that job seekers enter when searching for those open positions. We also had to make our internal decisions on whether we wanted separate taxonomies for medical specialties, and medical departments are just one, or whether medical specialties should be specific to nurses, or cover all healthcare professions. The medical specialties you see on the right, are all part of our taxonomy. And were the first to make their way into user testing. Amanda And on the left, you see a section of our very first prototype You'll note that it shows an attempt at skills categorization as well as a proposed UI to filter search results. We were at the point where we needed user feedback on both of those elements. Our overall question was, were nurses happy about seeing this information as part of their search? And could they use it to get better search results? And as you can see, we didn't get it quite right. I'll read the quote on the right from one of our nurses, I guess I'm a little confused about the basic IV because that's required by all nurses. So I think that's kind of redundant. We as non experts had made the wrong call about what data it was important to display. And since taxonomy was working on medical specialties in parallel, this kind of feedback helped them decide what their categories were going to look like. multiple rounds of feedback like this led us to ultimately choose a filter interface, which you'll see in a few moments Alex By closely collaborating with segmentation and UX research, we on the taxonomy team were able to quickly update and we find our categories to ensure they fit our users mental models. Unlike the skill filters in the prototype medical specialty filters like medical imaging, cardiac catheterization, or performs well in these initial tests. This was a sign that we could move from the research phase to the operations phase in which we implement or the song and indeed, this means extracting from unstructured text, all the custom concepts we define, so they become available to all our infrastructure. The abstracted text we analyze comes from multiple sources, with jobs and resumes being the bulk of it. We use different tools and models to process natural language automatically. Many of these are built in house by our engineers, as well as help from a large team of experts. Kuwait the modal models output. This is a complex process that requires us to identify and map all the various ways in which employers and jobseekers refer to concepts such as medical specialties. For example, terms like intensive care, critical care CCU, I see you need to pq are all linked to one meta data item, namely the medical specialty, critical and intensive care, provided they appear in the right context. If I turn my critical care appeared in the company description of a job hiring for hospital accountants, we wouldn't link it to the medical specialty. This is the magic of mapping and indexing, meaning, not just strings. That were just messy though. And as you can imagine, we had to adapt our strategies. We were also getting new quantitative information from our users. We had to incorporate into how we had set up the concepts in Arctic song. Well, we initially envisioned as a wonderful process with operations following research. You gave an agile process in which the two phases continually important each other. By the end of Q2 2019, we had research created and implemented 58 extractable medical specialties. In the meantime, the UX team continued to be finding the design. Leora haven't already done a small scale test with the metadata, we were ready to expand our reach within product testing. In q3 2019, we launched an A B test on our site. For those who are unfamiliar with a B testing, it allows users to be automatically broken into two buckets to see a different user interface. We monitor user behavior to identify which group A or B performs best. And then decide if we want to modify the product or maintain its current status. So in this case, we looked at analytics to identify the top nursing related queries and targeted these users. 50% of the users experience the normal interface on indeed, and 50%. In the test group, where they saw the medical specialty filters. Here's what it looked like. You'll notice that we added three filters, specialty license and patient type. We started with about three in the patient type category, geriatrics, pediatrics and primary care. And we noticed that this filter did not perform well. After further investigation and discussion with taxonomy. We learned that treating different types of patients like geriatrics, pediatrics and primary care is actually considered a medical specialty. We modified the filters and decided on two different ones specialty and license for specialty we decided that rather than add in 15 attributes we would include the ones that were most common. For example, we added in travel nurse labor and delivery and home health, and then we kept license the same. Alex On the taxonomy side, we followed up on the results of the A B testing with current cert studies. To further improve filter disclaiming a card study is one where participants are asked to sort items according to criteria that make sense to them. Or goal in doing this was to identify names of specialties that were familiar, but also distinct enough, was are better than surgery, or vice versa. We also want you to avoid conceptual overlap. How similar or dissimilar, were the terms medical surgical and surgery for our nurse jobseekers. Leora And the good news is that by showing these medical specialty filters after having completed the card, sort which confirmed our titles. We went ahead and notice that bump them, the filter usage increased significantly. This means that the nurses found these filters valuable. Having seen success with nursing in q4 2019, we rolled out the feature and replicated the formula for other segments. Amanda For instance, we repeated a similar process with trucking. We performed interviews worked with taxonomy to build attributes for trucking and tested filter prototypes. As you can see here, we added driver type filters and I want to draw particular attention to the first item on the list OTDR driver, O TR stands for over the road and among truck drivers. OTDR is an acronym that is as familiar as ATM might be to us. OTDR work generally means a trucker will be doing long hauls across many states and will be away from home for long periods of time. For this reason, many truckers told us that otrs As a way of putting in your time before having the experience to choose other less grueling types of driving. So those filters and data categories at our local designated regional driver are based on what truckers told us in our interviews. Again, by showing driver type, we saw a filter usage increase significantly. Also, the impact of these and other related AV tests was larger than any of the other tests that we ran that quarter. And we have gone on to try this approach with other segments with similar results. The segment team has been working with retail employees, caregivers returning to work and other groups of job seekers that will benefit from targeted search. Alex After two years of work on the project, here are our takeaways. taxonomy creation is not traditionally thought of as an agile process. A big reason is that it's hard to define what counts as an MVP. A good taxonomy should be comprehensive. Our project taught us how to import qualitative and quantitative analysis in an iterative fashion and create an agile taxonomy. That's data informed and user centered. We made trade offs. We structured our internal workflows, and collaborated cross functionally to ensure our users voice is hurt. And but the search experience best addresses their needs. Amanda The second takeaway is that managing UX research, product decisions and back end implementation require deliberate collaborative processes and roles. And being deliberate means a few things because we all know that collaboration is a good thing. But we have to acknowledge that it isn't something that's going to just happen. As you saw this very successful collaboration took two years and that's okay. deliberate means being really explicit about building collaboration functions into roles. This is often something that UX researchers are expected to do for example, but for me, it was eye opening, and wonderful To work with other allies who are also assigned this role. For this reason, it's worth thinking about how you define roles. Again, it's easy to write cross functional collaboration in a job description. But can you be more specific about what that looks like? And if not, maybe that's part of the job building out the playbook with other team members. Finally, when hiring and promoting for such roles, collaboration is a skill you need to identify and prioritize with interview questions and reward and reviews. Leora Our learnings with the nursing segment have allowed us to scale to other segments. Currently on the segmentation team. we've adopted many more segments so that we can go ahead and repeat our success. Alex To wrap up, we learned that collaboration is the cornerstone of successful product. Before we got together, there was no handbook for how product UX and taxonomy could collaborate. Our project paves the way for to collaboration. And by this we don't mean collaboration as a buzzword Work. We're working together on solutions from the very beginning. Leora We enjoyed sharing this case study and hope our insights about product collaboration were helpful. We appreciate that you joined us today. I want to encourage you to stay in touch with questions. Our emails are above on this slide. Many thanks to our colleagues that indeed for their contributions, and now we're ready to take questions. Transcribed by https://otter.ai