Jive SBS Syndication Feed

How Popularity Works in Clearspace

aaron@jivesoftware.com — Thu, 04 Sep 2008 05:19:13 GMT

I've heard this question a number of times here on Jivespace and I think even a couple times on our intranet (which is called Brewspace) but I've yet to see it documented anywhere, mostly because no one has ever taken the time to explain it to our awesome docs guy Steve (by the way, have you seen the new docs? They're amazing!). I doubt any of you are losing sleep about it, but just in case, here's my simplified, albeit technical, take on things. If you've got a source build handy, I'd encourage you to follow along in your favorite IDE.

PopularityDeterminationTask is the class that handles the majority of the work. It runs every 15 minutes (as configured in spring-taskContext.xml, which means you can change it via a plugin in 2.5 if you want) and works closely with ActivityManager. Every time it runs it does the following things:

iterates over the set of containers that exist in Clearspace (communities, projects, social groups and blogs)
calculates the popularity of each item in the in-memory copy of activities that activity manager maintains for the current day. The score (or popularity) of each item is calculated by adding together the score of each activity on an item, the scoring is as follows:

each comment (blogs and documents) or reply (thread) is given 5 points
each modification (documents only) is given 5 points
each view is given 1 point

calculates the popularity of items in each container stored the database (a popularity score is computed for each item based on the activity that happened that day every night at midnight, this data is stored in the jivePopularity table) for the last seven days. The individual day score is adjusted for each day old that the score is recorded. We call this decay: the computation is done in SQL but if it were done in Java it would look something like this:

Calendar c = Calendar.getInstance();
c.add(Calendar.DATE, -daysAgo);
Date now = new Date();
Date nDaysAgo = c.getTime();
long dateDiff = now.getTime() - nDaysAgo.getTime();
long dateAdjustment = dateDiff / 86400000; 
long decayedScore = score / (1 + dateAdjustment);

add the results of step 2 and step 3 together. So for example, let's say I published a blog post 2 months ago and through some sort of magic, it got 3 comments every day for the last 2 months (and it's already received 2 comments today that haven't been archived to the jivePopularity table). We only take into account the popularity score for the last seven days so the total score is going to be the sum of the following scores:

Date
Number of Comments
Recorded Score
Score With Decay
6/5/2008 (today)
3
15
15
6/4/2008
3
15
7
6/3/2008
3
15
5
6/2/2008
3
15
3
6/1/2008
3
15
3
5/31/2008
3
15
2
5/30/2008
3
15
2
Total Score
37
Finally, each item is added to a SortedSet and then copied back to ActivityManager, where it lives until the next time popularity is calculated fifteen minutes later.

Date	Number of Comments	Recorded Score	Score With Decay
6/5/2008 (today)	3	15	15
6/4/2008	3	15	7
6/3/2008	3	15	5
6/2/2008	3	15	3
6/1/2008	3	15	3
5/31/2008	3	15	2
5/30/2008	3	15	2
Total Score			37

So when you see a widget that shows the most popular items for a given community / space, project, social group or blog, you're getting the n items who have the highest combined score, which is a combination of comments, replies, edits and views, decayed over the last seven days. ¿Comprende?

Search Improvements in 2.0.x and 2.1

aaron@jivesoftware.com — Mon, 16 Jun 2008 18:38:23 GMT

During the last couple weeks of the 2.0 development cycle we pushed some really helpful search improvements (some of them bug fixes) into Clearspace. There are a numberof posts scattered around our intranet (which is called Brewspace) where the actual improvements / bug fixes are discussed but I don't believe those improvements made it into the documentation (there are a number of improvements listed in the changelog, but no description of what the improvements are). Hence this blog post.

First, in 2.0, the default search operator was changed to 'AND' from 'OR', the end result being that if you did a search like this:

clearspace openid

Clearspace would look for all the blog posts, discussions and documents that contained the term "clearspace" AND contained the term "openid". Way back in Clearspace 1.0 the thought was that we should deviate from what Google does (they AND the terms you input) because we're not searching the entire web; our thinking was that most of our installations would only have a couple thousand documents, blog posts and threads and so we didn't ever want a search for 'clearspace openid ldap' to return nothing if there was a document that discussed two of the three. The reality is that when the search operator was 'OR' the number of results from a search query in Clearspace was almost always greater than 500 results (the maximum number of results we would return in a search): in fact, the more words you used in your query, the more likely that you'd end up with a large number of results, which in theory is great (we found a bunch of stuff that for you!) but in practice doesn't make for a great user experience (thirty four pages of search results? come on!). One of the articles (discussed below) had this to say about lots of search results:

Users sometimes measure the success of a query primarily by the number of results it returns. If they feel the number is too large, they add more terms in an effort to bring back a more manageable set.

So not only did "OR" result in more results per query, if you decided that you wanted to refine your query by adding a term, the number of results would actually grow, not shrink, which is the opposite of what you'd expect.

The funny thing about changing this default is that when we turned it on for brewspace (our intranet, no other changes had been made at that point), a number of people noticed right away and were amazed at the 'improvement'. It's really crazy how something as simple as "AND" versus "OR" could make such a big difference in user experience.

Before I move on, here are a couple interesting articles I found that talk about search as it relates to user experience:

Greg Linden, an ex-Amazon guy, wrote a great blog post a couple months agothat summarized a talk that Marissa Mayer gave about Google and their results page and the number of results per page and also added some notes about his own experience while working at Amazon. The bottom line from the presentation? Speed kills. The faster that we can return search results, the happier Clearspace users will be (although the comments on that post tell a potentially different story, don't miss'em).
A BBC News article from 2006 had this to say about search results:

At most, people will go through three pages of results before giving up, found the survey by Jupiter Research and marketing firm iProspect. It also found that a third of users linked companies in the first page of results with top brands. It also found 62% of those surveyed clicked on a result on the first page, up from 48% in 2002. Some 90% of consumers clicked on a link in these pages, up from 81% in 2002. And 41% of consumers changed engines or their search term if they did not find what they were searching for on the first page.

Takeaway? Relevant results are more important than many results.

The guys and gals at Boxes and Arrows have some really great articles on search patterns:

The essential problem of search — too many irrelevant results — has not gone away.

More and more, our ongoing research is telling us that Search has to be perfect. Users expect it to "just work" the first time, every time.

One thing that was easy to add which has also come up a couple times was search by author. I'm happy to report that in 2.1 we added the ability to search for content authored by a specific user. So just like you can click 'more options' on the search results page today and choose what types of content you want to search for, you'll be able to select a user whose content you want to find and filter the results using that selection. Side note: that functionality has always been in our API and if you're a URL hacker like I am, you can actually perform Clearspace searches using a pretty URL like this:

http://example.com/clearspace/search/openID

or if you want to search for any content written by the user 'aaron' that contains the word 'openID', you'd use this URL:

http://example.com/clearspace/search/~aaron/openID

Another thing that I believe has worked in the past but that we haven't talked about is the idea of a simple syntax for search. Much like the operators you can use in Google (ie: 'site:jivesoftware.com lucene' will find all the references to 'lucene' on the domain 'jivesoftware.com'), we now support the following operators: 'subject:', 'body:', 'tags:' and 'attachmentstext:'. While I admit that they're not the most user-friendly things to type in, it does give advanced users a little bit more flexibility. For example: you can now ignore tags if you want by doing a search like this: 'subject:lucene OR body:lucene'. The search syntax operators are scoped to be in the search tips documentation that sits right alongside the search box. Again, this is for 2.1.

Those were the improvements. Now the bug fixes (which just so happen to also really improve your searching experience).

Search stemming doesn't seem to be working (CS-3645): Not sure how long this wasn't working, but the existence of this bug meant that if you put the word "error" in a document and then did a search for "errors", our search engine wouldn't find your document. Read more about stemming if you're curious about that sort of thing. If you're seeing this bug, make sure a) you upgrade to at least Clearspace 2.x and b) make sure that you're using a stemming indexer. The default analyzer does not stem. You can change the indexer by going to the admin console --> system --> settings --> search --> search settings tab --> indexer type.
Group search results by thread setting in admin console doesn't change search behavior (CS-3656): I'm not sure how long this feature has been around, but there is a search setting in the admin console that gives you the ability to group all messages in a thread into one result in a search results page so that the messages in a single thread don't overwhelm the search results (since messages share the subject and tags from the thread, that actually happened quite a bit). Fixed in 2.0, I highly recommend turning it on in your instance if you haven't already.
Search updates to better balance queries across content types (CS-3638): Some improvements were made in 2.0.0 toward this issue, but it's 100% fixed in 2.0.3 and 2.1. There were two really big but really really hard to see problems with the way that we were executing our search queries. First, a quick background on how a search query is performed in Clearspace against all content types. We have a single Lucene index for all the content in Clearspace (there is a separate index for user data, but that's a different story) so when a search for 'bananas' is executed, we did something like this (don't read too much into the language I'm using, I'm just trying to illustrate how it works at a 30,000 foot level):

get blog posts that match query

find all the blog posts where the subject matches 'bananas' OR the body matches 'bananas' OR the tags matches 'bananas' OR the attachments matches 'bananas' or the blogID matches 'bananas'

get discussions that match query

find all the messages where the subject matches 'bananas' OR the body matches 'bananas' OR the tags matches 'bananas' OR the attachments matches 'bananas' or the threadID matches 'bananas'

get documents that match query

find all the documents where the subject matches 'bananas' OR the body matches 'bananas' OR the summary matches 'bananas' OR the fieldsText matches 'bananas' OR the tags matches 'bananas' OR the attachments matches 'bananas' or the documentID matches 'bananas'

merge results from steps 1-3 using the relevance score from each item in the result set as the comparator
display results

The assumption we made when writing this code was that the scores that Lucene returns for each item of all content type will be relatively similar. More concisely, if I had a document and a blog post which for some reason had identical content, I'd expect they would both have the exact same Lucene relevance score if they came up in the results of a search. But that assumption turned out to be wrong, not once but twice.

First, as you can see from glancing at the sample queries I pasted above, we searched a different number of fields per content type. Who cares right? Why would the number fields that you search on influence anything? Turns out that Lucene cares: the way scoring works in Lucene is that if you search on ten fields and only get a hit on two of them in document 'X' that the resulting relevance score should be less than a hit on blog post 'Y' where you search on five fields and get a hit on two of them. It makes perfect sense when you think about it: it's just like the tests you had in school. Getting a 4 out of 5 on a test works out to be 80%, about a B. If you got 4 out of 10 on a test, that's 40%, you failed. You probably called them grades... maybe sometimes even a score, which just so happens to be exactly how Lucene refers to the relevance that a particular document has to a given query (if you're curious about how Lucene does scoring / relevance you should check out the JavaDoc for the Similarity class and also read this document on scoring). Anyway, this behavior was actually fixed in 2.0: so now when we execute a search on mixed content we search the exact same number of fields for each content type: subject, body, tags, attachmentsText.

The second assumption that turned out to be wrong was just as nebulous. I illustrated above how we search in Clearspace: we do searches for each content type and then we merge the results of those searches into a single result set. In order to do a query for just blog posts that match the word 'token', we do a query that in Lucene looks like this:

+objectType:38 +(subject:token body:token attachmentsText:token tags:token)

It kinds of looks like a SQL subselect: get me all the things where one of subject, body tags or attachmentsText match and then, from those results, only return the results where the objectType is 38 (which is the int that JiveConstants.BLOGPOST refers too). The thing that killed us here was outer statement

+objectType:38

because:

a) when Lucene executes a query, it computes a query weight and field weight for each statement in your query and multiplies those two values together to get the total weight for that statement and

b) the query weight is basically a measure of how many times the key (in this case 'objectType') appears divided by the number of times the value appears (in this case '38') which means that

c) content objects that you have less of (in our case: blog posts) will tend to have a score much higher than content objects that you have a lot of (in our case: documents). Again, this makes sense: items that appears in the index a relatively small number of times are in some sense rare and so they should get a relatively higher weight. Regardless, it turns out there's a really easy fix for this problem as well: you can boost specific fields in your query like this:

subject:token^3

and you can effectively neuter a field by boosting it to zero:

subject:token^0

which means that Lucene will look for all the items in the index whose subject is 'token' but the weight, which usually influences the score that it assigns to the field 'subject' will not influence the score that the resulting item receives.

We're continually looking to improve the search tools in Clearspace. If you're seeing something you don't expect or if there's something cool you'd like us to add, please pipe up in our Support and Feature Discussion spaces here on Jivespace.

New Widgets

aaron@jivesoftware.com — Fri, 11 Jan 2008 00:48:02 GMT

In a future release of Clearspace we're talking about adding a personalized homepage feature, much like My.Yahoo.com or Google.com/ig, where users can add Clearspace widgets like 'Recent Content' or 'Top Members' to their homepage. A lot of the widgets that we debuted in the 'customize a community' feature (have you seen the video?) will be available for users on the homepage, but we'd like to hear from you: assuming you had a customizable homepage and you could use Clearspace like you use My.Yahoo.com or Google.com/ig, what widgets (Clearspace related or unrelated) would make your day?

Linking in rich text editors

aaron@jivesoftware.com — Thu, 23 Aug 2007 06:18:16 GMT

Jon Udell <a href="http://blog.jonudell.net/2007/08/17/unexamined-software-idioms-1-linking-in-rich-text-editors/">recently blogged</a> about how the user interface for creating links in rich text editor hasn't changed in years:

The idiom goes like this:

1. Select the text to which you want to attach your link.

2. Click the Link button.

3. Type (or paste) the URL.

I’ve watched novices struggle with this for years, and it’s no wonder that they do.

We've spent a number of cycles making the rich text editor linking process easier in Clearspace. In case you've never explored the little link button in the editor, clicking on it pops up a window that looks like this:

The pop up window defaults to showing you a search interface: do a quick search and you can link to anything in Clearspace. Conveniently there are two other options: the history option:

which shows a list of your most recently viewed blog posts, forum threads and wiki documents and second: the web address option:

where you can type in or paste a URL and then a link title. We've gone through a number of rounds of design on this pop up window and I'm sure there's still room for improvement (there is no perfect design). We'd love to hear your feedback though: what do you like about linking in the editor? Which of the three tabs do you find you use the most? Did you even know it was there? And for any developers out there, has anyone tried to or wanted to customize the editor? If so, what did you did do to it?

Clearspace and Meaningful URLs

aaron@jivesoftware.com — Thu, 21 Dec 2006 17:53:32 GMT

Early on, during one of our Clearspace product development meetings, we (the Clearspace development team) discussed the importance of 'hackable' or 'meaningful' URL's. In other words, if someone sent you a link to a Clearspace blog via IM or email[1|#urls-1], you should be able to figure out a lot about the blog post without viewing it in your browser. So, for example, if my boss Bill sent me an IM that looked like this:

Hey Aaron, check out this blog entry:
http://mysite.com/do/entry?publicid=4B4C0B46DC743154ECB68300531D6A04&token=

I'd probably click on it, but I sure as heck wouldn't know what Bill was sending me too. Is he sending me another link to a blog that discusses the virtues of living in Iowa[2|#urls-2] or is it something work related? I can't tell. If however, he blogged about the link in Clearspace, his IM would look like this:

Hey Aaron, check out this blog entry:
http://mysite.com/blogs/bill/2006/12/19/iowa-jokes/

With only a cursory glance, it should be immediately obvious what this blog post is about (another Iowa joke), when it was posted (December 19th, 2006) and who posted it (Bill). We can all agree this is 'a good thing', right? Hold on though to your corn husks though, it gets better.

Most blog software products in the wild give up at this point: they answer the question of who, what and why by looking at the URL of a single blog post and that's it. Clearspace, on the other hand, goes the extra mile for you. Wouldn't it be nice to see a list of all of Bill's posts on the 19th of December?

http://mysite.com/blogs/bill/2006/12/19/

or how about all the posts in December?

http://mysite.com/blogs/bill/2006/12/

or 2006[3|#urls-3]?

http://mysite.com/blogs/bill/2006/

What about all his posts that are tagged with 'iowa'?

http://mysite.com/blogs/bill/tags/iowa

Bill's a smart guy, I should subscribe to his blog, I wonder what feeds he has available?

http://mysite.com/blogs/bill/feeds

Great, I can subscribe to all of his posts:

http://mysite.com/blogs/bill/feeds/posts

and a feed of all of his posts that are tagged with 'iowa':

http://mysite.com/blogs/bill/feeds/tags/iowa

but I'd really like those in Atom format rather than the default[4|#urls-4] RSS 2.0 format:

http://mysite.com/blogs/bill/feeds/posts/atom

and it would be nice to easily be able to include a list of his posts on the homepage of our intranet:

http://mysite.com/blogs/bill/feeds/posts/json

So there you have it: hackable and meaningful URL's galore. If you're interested in learning about how we did it, you can read about the gory details over on my blog.

Now if I could just get Bill to stop sending me links about Iowa.

¹You're not using still using email are you?

²If you consider it a sport to gather your food by drilling through 18 inches of ice and sitting there all day hoping that the food will swim by, you might live in Iowa. (source)

³Hat tip to Tim Bray for nice way he implemented URL's on his blog.

⁴Yes, you can switch the default to be Atom.

Blogs: Not just for breakfast anymore

aaron@jivesoftware.com — Thu, 14 Dec 2006 16:32:02 GMT

Over the last couple months, we've had a number of people[#1] come into the office to give Clearspace a test drive. We watched while they explored the application and asked questions. When asked what they thought of just the blogging[#2] part of the application, almost every person said that they liked the individual and team blogs but then said something like, "but blogs are just opinions," insinuating (and sometimes explicitly saying) that white papers, knowledge base articles, wiki documents and forum posts are more valuable and authoritative in a corporate environment than blogs are.

I'll be honest: I took it personally every time someone equated blogging with being nothing more than opinions and navel gazing because I think blogs are more valuable and more authoritative than all the white papers, knowledge base articles, wiki documents and forum posts in the world combined[#3]. Am I exaggerating to get your attention? Yeah. But if you're one of those people who falls into the "blogs = just opinions" camp, consider these broad brush strokes:

Blogs are authoritative +because +they're transparent. Blogs are generally written by a single person or a small group of people, all of whom you can read about by going to the "about me" or "about us" page on the blog. If the "about us" page doesn't cut it, you can go and read the other blog posts to get a sense of the blogger's background and interests. Finally, you can use a tool like technorati[4|#3] to see what other people think about the blog. So when you read a blog post, you know who's standing behind it. Conversely, white papers and knowledge base article are faceless and opaque: no one person is standing behind the document saying "this is true."

Blogs are valuable because they are written by people for people. White papers, powerpoint presentations, knowledge base articles and wiki documents are written by companies for companies.
Blogs are valuable because they're about subjects people care about and take seriously. It's an environment that allows for self-expression~~not like the rest of the applications you'd find in Microsoft Office or~~ack!--things that they have to write about to complete their monthly knowledge base quota.
Blogs are widely read because they're not white-washed, corporate-speak. People prefer the truth, it's why a lot of them have stopped watching mainstream news. There's more truth in The Daily Show.

What does all this have to do with Clearspace? Tune in next week for some hot blogging screenshots.

By the way, my name is Aaron. I'm an engineer on the Clearspace team and I have a blog.