vanderwal.net

Understanding Taxonomy and Folksonmy Together

I deeply appreciate Joshua Porter's link to from his Taxonomies and Tags blog post. This is a discussion I have quite regularly as to the relation and it is in my presentations and workshops and much of my tagging (and social web) training, consulting, and advising focusses on getting smart on understanding the value and downfalls of folksonomy tagging (as well as traditional tagging - remember tagging has been around in commercial products since at least the 1980s). The following is my response in the comments to Josh' post...

Response to Taxonomy and Tags

Josh, thanks for the link. If the world of language were only this simple that this worked consistently. The folksonomy is a killer resource, but it lacks structure, which it crucial to disambiguating terms. There are algorithmic ways of getting close to this end, but they are insanely processor intensive (think days or weeks to churn out this structure). Working from a simple flat taxonomy or faceted system structure can be enabled for a folksonomy to adhere to.

This approach can help augment tags to objects, but it is not great at finding objects by tags as Apple would surface thousands of results and they would need to be narrowed greatly to find what one is seeking.

There was an insanely brilliant tool, RawSugar [(now gone thanks to venture capitalists pulling the plug on a one of a kind product that would be killer in the enterprise market)], that married taxonomy and folksonomy to help derive disambiguation (take appleseed as a tag, to you mean Johnny Appleseed, appleseed as it relates to gardening/farming, cooking, or the anime movie. The folksonomy can help decipher this through co-occurrence of terms, but a smart interface and system is needed to do this. Fortunately the type of system that is needed to do this is something we have, it is a taxonomy. Using a taxonomy will save processor time, and human time through creating an efficient structure.

Recently I have been approached by a small number of companies who implemented social bookmarking tools to develop a folksonomy and found the folksonomy was [initially] far more helpful than they had ever imagined and out paced their taxonomy-based tools by leaps and bounds (mostly because they did not have time or resources to implement an exhaustive taxonomy (I have yet to find an organization that has an exhaustive and emergent taxonomy)). The organizations either let their taxonomist go or did not replace them when they left as they seemed to think they did not need them with the folksonomy running. All was well and good for a while, but as the folksonomy grew the ability to find specific items decreased (it still worked fantastically for people refinding information they had personally tagged). These companies asked, "what tools they would need to start clearing this up?" The answer a person who understands information structure for ease of finding, which is often a taxonomist, and a tool that can aid in information structure, which is often a taxonomy tool.

The folksonomy does many things that are difficult and very costly to do in taxonomies. But taxonomies do things that folksonomies are rather poor at doing. Both need each other.

Complexity Increases as Folksonomies Grow

I am continually finding organizations are thinking the social bookmarking tools and folksonomy are going to be simple and a cure all, but it is much more complicated than that. The social bookmarking tools will really sing for a while, but then things need help and most of the tools out there are not to the point of providing that assistance yet. There are whole toolsets missing for monitoring and analyzing the collective folksonomy. There is also a need for a really good disambiguation tool and approach (particularly now that RawSugar is gone as a viable approach).

The Social Enterprise

I am just back from Enterprise 2.0 Conference held in Boston, where I presented Bottom-up All The Way Down: How Tags Help Businesses Organize (thanks to Stowe Boyd for the tantalizing session title), which was liveblog captured by Sandy Kemsley as "Enterprise 2.0: Thomas Vander Wal". I did not catch all of the conference due to some Boston business meetings and connecting with friends and meeting digi-friends whose work I really enjoy face-to-face. The sessions I made it to were good and enlightening and as always the hallway conversations were worth their weight in gold.

Ms. Perceptions and Fear Inside the Corporate Walls

Having not been at true business focussed conference in years (until the past few weeks) I was amazed with how much has changed and how much has stayed the same. I was impressed with the interest and adoption around the social enterprise tools (blogs, wikis, social bookmarking/folksonomy, etc.). But, the misperceptions (Miss Perceptions) are still around and have grown-up (Ms. Perception) and are now being documented by Forrester and others as being fact, but the questions are seemingly not being asked properly. Around the current social web tools (blogs, wikis, social bookmarking, favoriting, shared rating, open (and partially open collaboration) I have been finding little digital divide across the ages. Initially there is a gap when tools get introduced in the corporate environment. But this age gap very quickly disappears if the incredible value of the tools is made clear for peoples worklife, information workflow, and collaboration, as well as simple instructions (30 second to 3 minute videos) and simply written clear guidelines that outline acceptable use of these tools.

I have been working with technology and its adoption in corporations since the late 80s. The misperception that older people do not get technology, are foreign to the tools, and they will not ever get the technical tools has not changed. It is true that nearly all newer technologies come into the corporation by those just out of school and have relied on these tools in university to work intelligently to get their degree. But, those whom are older do see the value in the tools once they have exposure and see the value to their worklife (getting their job done), particularly if the tools are relatively simple to use and can be adopted with simple instruction (if it needs a 10 to 200 page manual and more than 15 minutes of training to start using the product effectively adoption will be low). Toby Redshaw of Motorola stated on a panel that he found in Motorola (4600 blogs and wikis and 2600 people using social bookmarking) "people of all ages adopt these tools if they understand the value connected to their work". Personally, I have seen this has always been the case in the last 20 years as this is how we got e-mail, messaging, Blackberries, web pages, word processing, digital collaboration tools (the last few rounds and the current ones), etc. in the doors of small to large organizations. I have worked in and with technically forward organizations and ones that are traditionally thought of as slow adopters and found adoption is based on value to work and ease of use and rarely based on age.

This lack of understanding around value added and (as Toby Redshaw reinforced) "competitive advantage" derived from the social tools available today for use in the enterprise is driven by fear. It is a fear of control that is lost from the top-down. But, the advantage to the company from having this information shared and easy found and used for collaboration to improve knowledge, understanding, and efficiency can not be dismissed and needs to be embraced. The competitive advantage is what is gained today, but next month or next quarter it could mean just staying even.

Getting Beyond Fear

But, what really is important is the communication and social enterprise tools are okay and add value, but the fear is overplayed, as a percentage rarely occurs, and handling the scary stuff it relatively easy to handle.

Tagging and Social Bookmarking in Enterprise

In the halls I had many conversations around tagging ranging from old school tagging being painful because the experts needed to tag things (meaning they were not doing the job as expert they were hired to do and their terms were not widely understood) all the way to the social bookmarking tools are not scaling and able to keep up with the complexity, nor need to disambiguate the terms used. But, I was really impressed with the number of organizations that have deployed some social bookmarking effort (officially or under somebody's desk) and found value (often great value).

Toby Redshaw: I though folksonomy was going to be some Bob Dillon touchy-feely hippy taxonomy thing, but it has off the chart value far and above any thing we had expected.

My presentation had 80 to 90 percent of the people there using social bookmarking tools in some manner in their organization or worklife. The non-verbal feed back as I was presenting showed interest in how to make better sense of what was being tagged, how to use it better in their business, how to integrate with their taxonomy, and how to work with the information as the tools scale. The answers to these are longer than the hour I have, they are more complex because it all depends on the tools, how they are set-up and designed, how they are used, and the structures of information inside and outside their organization.

New Profession Unfolding In Beauty and Geekery

A week or more ago I ran across the incredible video of Blaise Aguera y Arcas presentation of Photosynth and Seadragon at TEDTalks 2007. The video is stunning work of Seadragon and Photosynth bringing a collection of images to life from one or more resources.

While the video and ideas behind the tools are incredible displays of where we are today with technology and where we are heading, this caused some ideas I have been trying to get to gel to finally come together. In this video Blaise states (my own transcription):

So what the point here really is, is we can do things with the social environment taking data from everybody, from the entire collective memory of what the earth looks like, and link all of that together and make something emergent that is greater than the sum of the parts. You have a model that emerges of the entire earth, think of it as the long tail to Stephen Lawlers Virtual Earth work. This is something that grows in complexity as people use it and whose benefits become greater to the users as they use it. Their own photos are getting tagged with metadata that somebody else entered. If somebody bothered to tag all of these saints and say who they all are, then my photo of the Notre Dame Cathedral suddenly gets enriched with all of that data. I can use it as an entry point to dive into that space in that metaverse, using everybody else's photos, and do a cross-modal and cross-user social experience that way. Of course a by product of all of that is an immensely rich virtual models of every interesting part of the earth, collected not just from overhead flight and satellite images, but from the collective memory.

Torrent of Human Contributed Digital Content

What this brought together was the incredible amount of human contributed digital content we are sitting on top of at this moment in time. This is not a new concept, but what is different is the skills, tools, and understanding to make use and sense of all this content are having to change incredibly. The Photosynth team is making use of Flickr content that has been annotated by humans (tags, titles, and descriptions), as well as by devices (date, time, location, etc.). This meta information provides hooks put pull disparate information back from its sole beauty and make an even greater beauty and deeper understanding. The collective is better than the pieces, but pulling to collective together in a manner that is coherent, adds value, and brings deeper appreciation is where get hard.

Much of information understanding and sense making to date has relied on human understanding and we have used tools to augment our understanding. But, we now need to rely on deeper analytics in quantitative methods and advanced algorithms to make sense and beauty out of the bits and bytes. The models of understanding are changing to requiring visualizations methods (much like those of Stamen Design) to begin to grasp and see what is happening in our torrent of information at our finger tips and well as make sense of the social interactions of our digitally networked and digitally augmented lives.

Amalgamation of Designer and Quant Geek

What gelled in my mind watching the Blaise demonstration is there is a skill set missing in the next generation comprised of amalgamated design, information use, analytical foundation, and strong quantitative skills. I have clients in start-up businesses and in enterprise that are confronting these floods of information they need to make sense of from folksonomies and customer generated content (including annotations and regular feedback). The skills needed for building taxonomies are not translating well when the volume of information the information managers are dealing with is orders of magnitude higher than what they dealt with previously. The designer, information architect, and taxonomist who have traditionally have dealt with building the systems of information order, access, and use are missing the quantitative skills to analyze and make sense out of a torrent of loosely structured information and digital objects. Those with the quantitative and strong analytical skills have lacked the design and art skills to bring the understanding into frame for regular people grasp and understand.

This class of designer and quant geek is much like the renaissance men, but today the field of those forging new ground is open to men and women. The need to understand not only broad but deep sets of data and information so to contextualize it into understanding is the realm of few, unfortunately as there is a need for many.

I know of limited pockets of people with the skills to do the hard work of querying the vast array of information, objects, and raw data then make something of value of it. But, there needs to be more of these people getting trained as designers with solid quantitative and analytical skills (or the converse). Design shops are missing the quant geeks and engineering shops are missing the visualization geeks that bring this digital world rich in opportunity into something that makes sense and beauty.

If you know people like this that are bored, please let me know as I am finding opportunities flowing.

Full-Day Social Web & Folksonomy Workshop at d.construct

Tickets for the d.construct Workshops go on sale June 14, 2007. Buying a ticket for a full-day workshop provides free access to the full d.construct conference on September 7th. I am presenting the following workshop on September 6th...

Building the Social Web with Tagging / Folksonomies

Folksonomy Provides 70 Percent More Terms Than Taxonomy

While at the WWW Conference in Banff for the Tagging and Metadata for Social Information Organization Workshop and was chatting with Jennifer Trant about folksonomies validating and identifying gaps in taxonomy. She pointed out that at least 70% of the tags terms people submitted in Steve Museum were not in the taxonomy after cleaning-up the contributions for misspellings and errant terms. The formal paper indicates (linked to in her blog post on the research more steve ... tagger prototype preliminary analysis) the percentage may even be higher, but 70% is a comfortable and conservative number.

Is 70% New Terms from Folksonomy Tagging Normal?

In my discussion with enterprise organizations and other clients that are looking to evaluate their existing tagging services, have been finding 30 percent to nearly 70 percent of the terms used in tagging are not in their taxonomy. One chat with a firm who had just completed updating their taxonomy (second round) for their intranet found the social bookmarking tool on their intranet turned up nearly 45 percent new or unaccounted for terms. This firm knew they were not capturing all possibilities with their taxonomy update, but did not realize their was that large of a gap. In building their taxonomy they had harvested the search terms and had used tools that analyzed all the content on their intranet and offered the terms up. What they found in the folksonomy were common synonyms that were not used in search nor were in their content. They found vernacular, terms that were not official for their organization (sometimes competitors trademarked brand names), emergent terms, and some misunderstandings of what documents were.

In other informal talks these stories are not uncommon. It is not that the taxonomies are poorly done, but vast resources are needed to capture all the variants in traditional ways. A line needs to be drawn somewhere.

Comfort in Not Finding Information

The difference in the taxonomy or other formal categorization structure and what people actually call things (as expressed in bookmarking the item to make it easy to refind the item) is normally above 30 percent. But, what organization is comfortable with that level of inefficiency at the low end? What about 70 percent of an organizations information, documents, and media not being easily found by how people think of it?

I have yet to find any organization, be it enterprise or non-profit that is comfortable with that type of inefficiency on their intranet or internet. The good part is the cost is relatively low for capturing what people actually call things by using a social bookmarking tool or other folksonomy related tool. The analysis and making use of what is found in a folksonomy is the same cost of as building a taxonomy, but a large part of the resource intensive work is done in the folksonomy through data capture. The skills needed to build understanding from a folksonomy will lean a little more on the analytical and quantitative skills side than the traditional taxonomy development. This is due to the volume of information supplied can be orders of magnitude higher than the volume of research using traditional methods.

Folksonomy Book In Progress

Let me start with an announcement. I have not had any answer for continual question after I present on tagging and folksonomy (I also get the question after the Come to Me Web and Personal InfoCloud presentations), which is "where is your book?" Well I finally have an answer, I have signed with O'Reilly to write a book, initially titled Understanding Folksonomy (this may change) and it may also be a wee bit different from your normal O'Reilly book (we will see how it goes).

I am insanely excited to be writing for O'Reilly as I have a large collection of their books from over the years - from the programming PHP, Perl, Python, and Ruby to developer guides on JavaScript, XHTML, XML, & CSS to the Polar Bear book on information architecture, Information Dashboard Design, and Designing Interfaces: Patterns for Effective Interaction Design along with so many more.

Narrowing the Subject

One of the things that took a little more time than I realized it would take is narrowing the book down. I have been keeping a running outline of tagging and folksonomy related subjects that have been in my presentations and workshop as well as questions and answers from these sessions. The outline includes the deep knowledge that has some from client work on the subject (every single client has a different twist and set of constraints. Many of the questions have answers and for some the answers are still being worked out, but the parameters and guidelines are known to get to viable solutions.

Well, when I submitted the outline I was faced with the knowledge that I had submitted a framework for a 800 to 1,000 page book. Huh? I started doing the math based on page size, word counts, bullets in the outline, and projects words per bullet and those with knowledge were right. So, I have narrowed it down to an outline that should be about 300 pages (maybe 250 and maybe a few more than 300).

What Is In This Smaller Book?

I am using my tagging and folksonomy presentations as my base, as those have been iterated and well tested (now presented some version of it well over 50 times). While I have over 300 design patterns captured from tagging service sites (including related elements) only a few will likely be used and walked through. I am including the understanding from a cognitive perspective and the lessening of technology pain that tagging services can provide to people who use them. There will also be a focus on business uses for intranet and web.

When Will This Be Done

Given that I have a rather busy Fall with client work, workshops, and presentations I set a goal to finish the writing by the end of Summer. It sounds nuts and it really feels like grad school all over again, but that is my reality. I have most of the words in my head and have been speaking them. Now I need to write them (in a less dense form than I blog).

Your Questions and Feedback

If you have questions and things you would like covered please e-mail me (contact in the header nav above). I will likely be setting up a blog to share and post questions (this will happen in a couple weeks). I am also looking for sites, organizations, and people that would like to be included in the case studies and interviews (not all will end up in the book, but those that are done will end up tied to the book in some manner). Please submit suggestions for this section if you have any.

Hoping About

Things have been a wee bit busy the past few weeks. I am off to Banff tomorrow (Monday) and boomerang right back after the WWW Conference Tagging Workshop as I have two days of private workshops in DC just following days.

I am off to the 2007 Identity Workshop for a few days, in part to talk to the some TagCommons people face to face, as well as have some client work to follow-up on. I am then off to south of the boarder, for some work before heading back to the office.

Tagging Workshop at d.construct

I should let you know early that I will be doing a full-day tagging workshop prior to d.construct 2007 in Brighton, England in September. I will post more about this as it is announced. The space is rather limited so keep an eye out.

Workshops Near You & Packages

Those interested in the workshop and I will add you to a list for announcing them. If you would like one in your organization or city I can help get this moving. Toward the end of May I will be providing more information about these, the getting smart packages (workshop or presentation combined with days of advising over a set of months). As I have mentioned these to people the interest is quite strong, so if you want more info on these before they are public on the InfoCloud Solutions site drop me an e-mail.

E-mail

If you have an e-mail that may have an answer that would be longer than 2 minutes to write I likely still have it lined up, but if you want to jump that line, send the e-mail again (or a series of short e-mails with single questions). I am deeply sorry for my delay in responding. I have no excuse, but the need for more time.

Tagging That Works Presentation Links

Today#&039;s presentation at the O'Reilly Web 2.0 Expo seemed to go very well. My session was Tagging That Works had really good feedback, which I thought was good as I really did not know the audience coming in to the presentation.

The presentation can be downloaded as PDF from Tagging That Works or can be viewed on Tagging That Works at SlideShare or view below...

From the HQ Office

It has been a good stretch of travel, mostly for work/professional life, but also took a trip to Florida for family holiday. At the moment I am back in the office working on proposals for upcoming projects of various lengths for clients and working through the process of writing, which involves dead trees at the final stages.

Web 2.0 Expo and SF Bay Area

I am soon off again to the San Francisco Bay Area to speak at the O'Reilly Web 2.0 Expo and have business meetings around the Bay Area (please ping via e-mail if you would like to meet-up during this time).

WWW2007 Workshop on Tagging and Metadata

Early next month I am off to Banff to keynote the WWW2007 Workshop on Tagging and Metadata for Social Information Organization. I am not sure how long this trip will be as I will have some pressing work around this time.

Social Software Summit

Lastly, I should note there will be a Social Software Summit that will run at the same time as the ASIS&T IA Summit next Spring (Spring 2008) in Miami. The Social Software Summit is still in its early stages of planning (the idea, dates, location, and interest have been launched). The dates are April 10-11 2008. I have a role in the planning and preparation for this event, along with some other incredible people.

On SXSW Tag You're It Panel

I am a panelist on the Tag You're It Panel at South by SouthWest in Austin, Texas. Others on the panel are the ever fantastic: Heath Row (moderator), George Oates, and Ben Brown. The panel information:

Tag You're It on Saturday 10 March 2007 at 2:00 pm to 3:00 pm. The panel will be looking at what people are actually doing inside social tagging systems and where things have gone in the past couple years with tagging. We will get beyond the notion that tagging is cool by providing examples of how people are really using the tools in innovative and useful ways.

Stop by and say hello.

Analysis of LibraryThing vs. Amazon Tagging Analysis

I have posted my Breaking Down LibraryThing vs. Amazon Tagging Analysis over at Personal InfoCloud. I have spent a lot of time analyzing what Amazon is doing with tagging over the past 16 months. Much of it was missed, but that was not the disappointing bit in the analysis. Most of the comparisons being made were really apples to oranges.

At some point I will write-up an analysis of what Amazon is doing well (some things they are doing brilliantly) and where it has yet to really take advantage of what they have. This will likely have to be broken in to a series as my the two impromptu presentations on the subject have been about an hour long (yes, there is that much there), not including the background set up bits.

I have been including a highlevel view of Amazon tagging in my normally full tagging/folksonomy presentation for the last year or so as there are valuable examples in what is there and their iterations.

Ma.Del Tagging Bookmarklet

Out of frustration with Yahoo MyWeb no longer surfaces results in Yahoo Search, which made Yahoo Search much better for me than any other web search. Now that this is no long functioning and there is no response as to if or when it will return I am back to Google and Microsoft Live Search (the relevancy is better for me on many things). But, this change also removes the value of MyWeb and this has me looking back to Ma.gnolia (also am a huge fan of Raw Sugar and their facets, but that is another longer post).

New Tagging Combo Bookmarklet

When I became a fan of MyWeb I used some glue to make Del.icio.us and MyWeb Combo Bookmarklet.

So now I have done the same for Ma.gnolia and del.icio.us with the Ma.Del Marklet (drag to bookmark bar in FireFox and Safari only). This was built using the Ma.gnolia bookmarklet Ma.rker Mini as its base.

Static Folksonomy Definition Posted

After many requests, mostly from academics wanting a place to point their cite, I have written and posted a static Folksonomy page. This page may get some minor edits in the next day or so, but it is set. This page will not change, nor be impacted by any change to the blogging software (still my own hand-rolled tool from 2001).

You are now free to quote and point.

Yes, this took to long to put in place. It took a few requests in the past three weeks to put it higher up on the priority list.

Pew Research on Tagging

Lee Ranie of the Pew Internet and American Life Project announced the release of Pew Internet Project Report on Tagging in America. The report also includes an extensive interview with David Weinberger on the subject of tagging. The most interesting parts of this report at the percentages of people in America who tag (includes those who add categories). Based on their survey, which randomly selected and spoke to 2,373 adults, 28% of Americans online have added tags or categories. The survey found 7% of the respondents tag/categorize daily.

I am really happy with the report as it looks at the numbers from a use perspective. Up to this point I have been using tagging service provider numbers (few are made public) along with Alexa hit reports across many services and took that total and divided by the Neilsen report number of total people on the web (approximately 750 million people). This approach provided about .85% of all the people on the web are tagging (does not include tagging on blogs as that is more ad hoc categories, but that is a long post to explain or done over a beer or two).

The difference between the percentages in the Pew report and the numbers I backed into is the Pew is just an American view and mine was looking at things globally. Pew looks at tags and categories and many systems have categories. I am really comfortable with the daily number of 7% on the web are tagging/categorizing and I will likely use that number in future presentations. The 28% number is really surprising, but for one time use it is accurate. This represents a much larger user base than I thought, but is also includes categories with tagging.

Separating Tagging and Categories

The Pew Report on Tagging combines categories and tagging. While optimally it would be great to separate the two out, explaining the difference between the two to a regular person (non-geek) in America will be difficult. Asking if somebody has used certain functionality on a service or one of the 130 or so social bookmarking tools or the many hundreds of products that include tagging will negatively impact the results. The terms tagging and categories combined for a research question make for a question that is more easily answered yes or no.

The Pew Report provides a starting place for future research, hopefully delving into the subject with a little more clarity, where tagging and categories are separated.

Tagging and Race

If one looks at tagging as a means to refind information and looks at tagging as adding context by adding a person's own vocabulary and social terminology as one of the tools of tagging then looking at various social groups is a simple way to start to validate this (a much better approach is to ascertain why somebody is adding a tag). One simple way to look at different social structures is race. The inclusion of the break down of who tags by race can provide a good argument that people who tag are adding missing language terms, if the assumption is made that the content is missing metadata or is provided by somebody not of that race.

The Pew Report indicates the following tagging breakdown by race:

• 26% of White, non-Hispanics
• 36% of Black, non-Hispanic
• 33% of English-speaking Hispanic

These higher numbers of people tagging who are not white seems to support the idea that those whose vocabulary and terminology is not represented will tag to ease their refinding the information. When things are in familiar terms they are easier to find and having the ability to tag from one's own context eases refindability. The Report does not dive into this and it is a really good subject for future research.

[I initially posted this at Personal InfoCloud :: Pew Research on Tagging, which has comments open

Beneath the Metadata - Pointer to Replies

I have read the "Beneath the Metadata: Some Philosophical Problems with Folksonomy in the November 2006 DLib. Boy, did I have problems with it, but so did David Weinberger, who responded with Beneath the Metadata - a reply.

My reading of the DLib article brought back the pain of taxonomies and the "expert" arrogance that occasionally accompanies taxonomies. I have worked in many enterprise organizations who have gone down the taxonomy route and had decent results. Taxonomies are essential parts of building information management tools. But taxonomies are never done, they are not easily emergent, they are incredibly resource intensive (they need a lot of money and people to build them and maintain them), and they don't work for everybody (sometimes large portions of the people who use the information system can not find or refind information they want or need using the taxonomy). When we would survey the people using our intranet or the internet application, one of the top responses was things were difficult to find as they were not called what they expected them to be called. This can be before or after a few hundred thousand dollars were spent building a taxonomy.

The remainder of this reply is posted at: Beneath the Metadata - Replies, where comments are turned on.