I still think we collectively learned the wrong lesson from the Netflix Prize contest

The collective conclusion from 2006’s Netflix Prize competition was this: You can’t beat massive model ensembles. Ten years on, I’m still convinced the larger lesson was completely overlooked: that sometimes, when modeling human behavior, there are better KPIs than RMSE.

In 2011, I began work on my own SaaS recommendation service (called Selloscope). As part of that work, I evaluated the performance of a Slope-One model (similar at the time to Netflix’s Cinematch algorithm) against a spiffied-up object-to-object collaborative filter. The latter doesn’t produce an error score, so I devised a different goalpost: If I remove random recommendations from each user’s profile, how well does the model fill in those gaps?

Continue reading “I still think we collectively learned the wrong lesson from the Netflix Prize contest”

Centralized vs Decentralized Analytics: All You Need To Know

Having spent the past dozen years in an Analytics role, both as an individual contributor and as a manager, I’ve had the opportunity to see the analytics function work within centralized, decentralized, and matrixed organizational structures. The relative merits of building a centralized versus decentralized Analytics organization depend largely on exactly what you expect to get out of the Analytics function in your organization, so it’s important to consider the pros and cons — and to structure the Analytics function correctly to meet your organizational goals.  Continue reading “Centralized vs Decentralized Analytics: All You Need To Know”

Enter the ( SEO ) Matrix

There are two very commonly known things about search engine optimization (SEO).

1) Great, targeted content drives PageRank.
2) Google’s page rank algorithm is recursive.

Given how common this knowledge is, it’s surprising that so many Web sites haven’t put 1 + 1 together: Your pages earn a PageRank based on content and links, and then they lend that PageRank power to your other pages based on how you link to them.

In effect, your internal link structure may be causing your most powerful, highest-ranked pages to spread their PageRank weight across your entire site, instead of to the specific campaigns or initiatives you mean to support with those individual pages.

The takeaway: Instead of thinking of your SEO efforts as a collection of channels (FaceBook, Twitter, YouTube, and the obligatory blog) meant to drive content — think of SEO as a structured set of Content Tiers, each of which are built and interlinked to support your marketing and SEO objectives.

Enter: the SEO Matrix.

Continue reading “Enter the ( SEO ) Matrix”

SEM for Longer Time-to-Convert Businesses: How to Stop the Bleeding

I’m always surprised that you don’t see more direct marketers hired into Web analytics roles — they’re expert in a number of skills such as predictive modeling, experimental design, and statistical analysis that really turn the Web analytics role into a strategic source of business revenue. Along those lines, I wanted to give an example of how a tried-and-true direct marketing technique — using regression modeling to predict how likely each user is to convert — can allow media buyers and SEM managers to significantly reduce the amount of time it takes to determine which campaigns are winners and campaigns are losers.

Companies rely on pay-per-click and online ad buys to drive growth.  While buying clicks is easy, ensuring that individual campaigns are profitable – before significant losses have been incurred – presents a host of challenges to media buyers.  Continue reading “SEM for Longer Time-to-Convert Businesses: How to Stop the Bleeding”

A Rationalist Responds to Godin’s Blog Post “Are You Rational?”

In his recent post Are You Rational?, Seth Godin makes a broad argument that nobody is rational all the time (true enough): that some decisions fall squarely within the domain of  rational methods (e.g., analyzing your Adwords click-thru rate) while other things are best approached irrationally: falling in love, appreciating music or wine, or generating ideas for new businesses and startups.  He goes on to say that “irrational passion is the key change agent of our economy.”  Simply put, he proposes that there are entire domains of human endeavor  that are better managed  with the  “gut” — a common belief and arguably a staple of American culture itself.

A great professor once told me, “If a good essay is one that’s fun to argue with, yours is a great essay.”  It’s in this spirit that I can’t resist offering a counterpoint to Godin.  My argument: that passion and irrationality are two different things.  Passion has place, but the time when it was OK to “go with your gut” is well behind us. Continue reading “A Rationalist Responds to Godin’s Blog Post “Are You Rational?””

How to identify real trends in user behavior

There is a perennial question in Web analytics:  “Are the numbers up?”

Certain web metrics can be highly variable on a day-to-day or week-to-week basis.  Daily Unique Visitors (UVs) and Daily Visits are just two examples of metrics that can change dramatically from one day to the next.  These big swings in day-to-day numbers can make it difficult for managers to tell whether their KPIs are really trending up or down.

Continue reading “How to identify real trends in user behavior”

Using search engines to measure social behavior

Let’s start with a simple premise: The more often something happens, the more often people write about it.

Sounds reasonable, right?

Albert Saiz and Uri Simonsohn, in their article “Downloading Wisdom from Online Crowds,” demonstrate that the relative frequency of documents returned by a search engine can be a good measure of how frequently a phenomenon occurs.  For example, if you want to know the relative cost of living in all U.S. cities (or the relative amount of corruption, or perhaps even how good the golfing is) then simply searching for “Dallas cost-of-living” and “San Francisco cost-of-living” may give you a great index.  If it works as advertised, this is a fantastic general research tool for analysts and marketing researchers. Let’s take a look.
Continue reading “Using search engines to measure social behavior”

What it takes to compete (and win!) on analytics

Things have come a long way since the days of analyzing Web server logs (raise your hand if you were at THAT party.) Web analytics tools like Google Analytics, SiteCatalyst, and WebtrendsAnalytics have brought basic site tracking and reporting to the masses, often with little more than dropping some javascript on the page.  Closer to home, Business Intelligence (BI) and data warehousing tools such as MicroStrategy and Microsoft’s suite of tools provide a structured view into in-house data.  Google Site Optimizer, SiteSpect, and tools like Omniture Recommendations go so far as to provide intelligent testing, targeting, and recommendation abilities to companies that even a few years ago would have met serious challenges developing these competencies in-house.

But these tools are not how you compete on analytics. Continue reading “What it takes to compete (and win!) on analytics”

Why “The One Number You Need To Grow” is one number you should probably avoid

In 2003, Frederick Reichheld published a Harvard Business Review article entitled The One Number You Need to Grow.  Reichheld’s article described a method for computing a simple, easy-to-understand customer satisfaction metric called the Net Promoter Score — and ushered in a flavor-of-the-month management practice that has left a bad taste in the mouth of academics and serious marketing researchers Continue reading “Why “The One Number You Need To Grow” is one number you should probably avoid”

Is the online dating industry propping up Google’s ad revenue numbers?

A few days ago, Frédéric Peters (Cupidon at Cupidon.be) posted an interesting question to the LinkedIn’s Internet Dating Executive Alliance group: Given the oft-cited success of the online dating industry, what percentage of Google’s ad revenue is driven by the online dating industry?

Based on Frédéric’s question, I felt compelled to run a quick back-of-the-envelope estimate of the share of Google’s ad revenue contributed by the U.S. online dating industry based on Continue reading “Is the online dating industry propping up Google’s ad revenue numbers?”