October 2006


30 Oct 2006 01:38 am

DSC06327 smallIt seems that every good web analyst / practitioner / director / vp’s wish list of a perfect web analytics tool starts with a desire to get “real-time” data.

The thought being that with the fast pace of the web and everything changing all the time getting real-time data is mandatory to being able to take advantage of all that the web has to offer from its ability to cough up so much data.

This customer desire seems to be so pervasive that every little and big web analytics vendor prominently advertises how real-time their data is. Someone says I can do every five hours, the next guy says I see that and I raise you three hours and the next gal says you guys are sissies because I can give you real real-time and give you live traffic streams.

But is getting real-time data really relevant? Do you are really need it? What’s the cost of getting real-time data?

This spoils the surprise but the answer to the first and second question is mostly no. Since most of us think it is yes it makes the answer to the third question a little bit scary (and rather sub optimal for most businesses in terms of impact).

So what is the typical impact of getting data near real-time (roughly defined as faster than every couple hours)? Here are the five that we have generally observed:

1. Much more reporting, much less analysis: We already live and swim in a world microscope smallof too much data, so much that we really have a hard time finding any actionable insights from what we have even after we hack at it for hours and days. Real-time data usually worsens that by giving you even more data faster and you are left to find the proverbial needle in the haystack (a non trivial task as you can imagine).

2. Detrimental impact on resource allocation: It is a common theme in the industry that we don’t apply the 10/90 rule. One reason is that it is hard to find the right people with the right skills for the job. But a secondary reason is that in the world of Web Analytics we have a lot of very complex data that can never reconcile to anything else. Now imagine what happens with real-time data.

Our finite resources now have to make sense of all this, pardon me, mess but with less time on hand and provide insights. Almost always because of real-time data there is a negative impact on the resources and bandwidth allocation because there is organization and management pressure to justify return on investment (remember real-time data is not really free, you pay to have access to data that fast).

3. Choice of sub optimal web analytics solutions: This one is really common. Web Analytics tools are chosen based on complicated all encompassing RFP (request for proposal) processes (here’s a alternative suggestion for selecting a tool). Everyone wants everything so these things are usually a joy to behold. : ) Usually top of that list is “need data in real-time” (remember the mindset, who would not want data real-time; its like asking someone “when did you stop hitting your spouse” its a lose lose).

choice smallThe impact is that the committee that is narrowing from 200 tools to 2 will reject any vendor that is not “real-time” because that is a deal breaker. Most often this means that lots of tools that might have met other important criteria (say advanced segmentation or integration with other sources) get kicked out. In the end you might end up choosing a tool that is real-time (and expensive) yet in a few months when we are smart enough to dig deeper we’ll find limitations.

Let me relate a personal story. Everyone wanted real-time data and that is what the big three vendors were selling as well (including data almost real time streamed over to pagers and smart phones). Yet we choose the tool we have because of all the features it brings to us, and it can’t do real-time and we don’t care. It is much cheaper to boot (software, hardware).  Another story, this time from a friend’s company, is that their team wanted real time PPC / SEM (pay per click / search engine marketing) data and they simply decided to take it all outside the company and created a data / decision making silo that did not have a end to end view and optimized for that silo (usually this is a sub optimal scenario).

4. Increased complexity in systems and processes: Most practitioners don’t realize that real-time is not just buying a powerful web analytics tool. There are other collateral requirements.

1) If you have a in-house solution then real-time means having to buy increasingly powerful machines (usually multi-cpu and loads of memory) that can capture data and process it fast enough to make it available in real-time for you to use it. [Remember that you can’t actually use raw data logs (web logs or javascript tag based).]

complexity2) In order to pull real-time off we will also have to implement increasingly complex processes inside and outside the company.

In your company for example you’ll have to have to have faster processing schedules implemented and allocate some resource (maybe 0.25 person) to watch and make sure everything happens as expected and finally implement reports to run to process all the data to humans.

From a outside perspective you’ll have to put processes in place that will pull data from outside sources (say adwords or affiliates). This adds more steps and complexity into your systems / processes, complexity that is often ignored and not considered by marketing folks but it is complexity that inserts a non-trivial cost into the ecosystem.

5. False sense of confidence: There is not much to say here except that sometimes you’ll observe a false sense of confidence that all is well with the world because we have real-time data streaming into our blackberries. Of course this is not every organization. But it exists more than we might prefer. This false sense of confidence means that we are less likely to look at what the real cost is of getting the data and what is the downside.

In summary the impact of real-time data is that you will pay more for your web analytics tools than might be optimal, you’ll fuel a culture that will do more reporting than analysis and you will end up adding complexity to your systems and supporting processes which in turn will add lots of hidden costs.

Did you realize this? What is the true “cost” of your real time data? Do you disagree with the five impacts outlined above?

This obviously does not mean that you should never want real-time data. Here is a simple check-list to use to judge if your organization is ready for real-time data and increase the odds that you will get enough bang for the, end-to-end, increased bucks you’ll spend:

1. statistical tables “Statistical significance”: You get enough visitors to your website that you can make statistically significant decisions using real-time data.  You not only have to get enough overall traffic to the site but you also have to get enough data in segments you want to make real-time decisions.

For example if you want to make real-time decisions about marketing promotions or adwords campaigns then do you get enough traffic and outcomes (orders / leads) to make a statistically significant decision? If you get 13 visitors and 2 outcomes from two different campaigns every four hours then you probably can’t make a confident decision comparing that to anything else.

Statistical significance is not just about raw numbers, you don’t need a million visitors a day to get significance. But you do need enough visitors exhibiting the right behavior you are looking for and for them to do it often enough every hour for you to separate signal from all the noise.

2. Good analytical capabilities: You can not only capture data real time but you have dedicated analysts who can analyze the data very quickly to find nuggets of valuable insights by looking not just at one piece of data but end to end. For example they would not only notice that we got lots of clicks on this new creative from Google / Yahoo PPC campaigns but this traffic is also placing more orders for the right products than other sources of traffic.

Along with analytical capabilities you also need people who have optimal business acumen (maybe super optimal). Numbers no matter how fast they come at you and in how much quantity by themselves won’t help you make good decision. For that you need people will good business acumen (as defined by people who understand your business really well, have a great grasp of your web ecosystem, have lots of common sense). 

As a wise person :) said reporting is not analysis!

3. Diversified & Empowered decision making structure: Does your company have a decision making structure where a “front line” analyst can make decisions and authorize / thumbs up smallexecute changes based on data? Do you require VP approval before web pages go on or off? Do you need a HiPPO to sign off on promotions / campaigns changes?

For action to be taken from real time data decisions need to be made fast. Usually it will be your Analyst or Marketing Manager observing these statistically significant differences. Often these kind folks don’t actually have the authority to stop or green signal anything based on data. That happens via a company labyrinth that needs to be navigated.

If the answer to all of the questions above is No then you are all set, you are empowered and ready.

4. Awesome website / structural operational execution capabilities: Your company has a web operations team that can execute on a dime. They are able to push out the right creative, remove non performing promotions, change the adwords strategy, update landing pages, change email blasts that are already in the queue, send different instructions to your ad / search / affiliate marketing agencies who can also make changes very quickly.

Essentially if it takes you two days to execute changes to your website / campaigns / agencies then value of real time might be really questionable.

Four extremely simple rules / requirements. If your organization capabilities meet all of the above requirements then you are well set to to gain a advantage from getting your data real-time. But if even one of the above requirements are not met then it is perhaps more the case that you want to know (real-time) because you want to know and not to take action. That knowing can be extremely expensive (people, process, $$$) and distracting.

Real-time is perfect in one scenario, if there are micro decisions that a automated system can make based on rules that humans can input. In this scenario some, but not all, of the above issues become less critical. Data helps technology to react real-time to create unique customer experiences. More on this in a future post.

What do you think? Are there other requirements for a organization ready to leverage data real time? Is this post off base? Something missing from the analysis outlined above? I welcome your feedback and critique.

[Like this post? For more posts like this please click here.]

26 Oct 2006 12:47 am

DSC03659 smallAt conferences and summits there are always a lot of vendors and I imagine these events are really hard for them. Two reasons:

    1) They have to stand out in a large crowd.
    2) They have to do #1 when the most get is two to five mins with each person who stops at their booth.

But from a customers perspective this is a very interesting situation because it might be easy to see how good each vendor is by how they react to this high pressure environment.

I took some time at the recent emetrics summit in Washington DC to get to know all the vendors who had booths in the exhibition area. I stopped by each table and asked the first person that was available to talk at the booth two simple questions:

    Question # 1: Unique: What is the most unique thing about your company / solution compared to all other competitor solutions out there?

    Question # 2: Unique Feature: Is there a feature in your product that differentiates you from your competitors? Perhaps a report, a metric or a feature?

The first question was geared towards getting a 60,000 feet view about differentiation (because there is a rumor that they all do the same thing!) and the hope with the second question was that the vendor would highlight somethings specific about the product, the 6 feet view if you will.

Before we go any further the dictionary definition of the word unique is:

    Unique:


    1) Being the only one of its kind
    2) Without an equal or equivalent; unparalleled.

Here are some very insightful answers from each vendor, if you are looking for a solution this might be helpful to you or if you are looking for some fun read then you will also find this interesting.

[The "ORbAK Golden Star Award" for the best answer goes to Karen Hugins from Unica Affinium NetInsight NetTracker! She win's brownie points. :) Honorable mention for Steve Phillips from Numeric, he also gave a great answer.]

Web Side Story (Dave Datars)
Unique: "The way we capture data. Our solution uses page tagging and not web logs as others do."
Feature: "We are very good at reporting and metrics for streaming media. We also have Visual Sciences that can track you across channels by placing sensors"

ExactTarget (Pat Donnely)
Unique: "We have developed a open API and that leverages industry standards."
Feature: "Data is trapped in different systems in a company and we can pull all that data and use it to send timely emails."

Omniture (Trent Dava)
Unique:
“I have not used other web analytics products but Omniture's ability to create hundreds of custom variables is unique. We also have built in bid management in the latest version.”
Feature: “All the new optimizations that we have built into the tool. As a example you can measure RIA's (rich internet applications) and track across RIA's. You can also do Blog Value optimization and if you are getting the conversion rates that you like. With Discover you can get much deeper segmentation.”

X+1 (Heath Podvesker)
Unique:
"What we do is real time and we both product and service focussed. Our customers get a 20 – 60% by using our product."
Feature: "Kefta will optimize based on creative, we can go much deeper when we do optimization."

Atlas (Erin) [Represented just the Web experience optimization group. They compete, if that is the right word, with lab usability, focus groups etc. ]
Unique: "We use much larger sample of website users, initially we will go in and collect two weeks of data which is a lot more than ."
Feature: "Our ability to tie to quantitative data and ability to do deep segmentation."

Numeric (Steve Phillips)
Unique:
“Our background is in Direct Marketing Analytics, highly quantitative positions and we are not dedicated to web analytics and brining our background with us makes us unique.”
Feature: “We don't have recruiters who are focussed on generic positions, they are specialized in analytics and they provided very personalized high level service. We also offer a 90 day money back guarantee.”

iPerceptions (Jonathan Levitt)
Unique: “Our methodology is what makes us unique, the new iPSI we have introduced is the only index that is measuring the pure online experience. We believe that there are different customer expectations from a online experience and creating a pure index is better.”
Feature: “The ability to get a 360 degree view of the customer experience, looking not just at customer satisfaction but also how specific segments of customers behave and what each segment is looking for.”

ForeSee Results (Eric Head)
Unique: “The ACSI (American Customer Satisfaction Index) model is very unique and exclusive in our tool. The model is time tested, it is credible and the model links Satisfaction to Business Performance.”
Feature: “A unique feature in our product is the ability to assign a score to the impact of a chance, it is all about the numbers.”

Optimost (Scott Simonelli)
Unique: “Out ability to use true multi variable testing methodology and uses "optimal design", our system / platform was built from the ground up just to do MVT. Other vendors are just can test "islands" and not truly multi variable tests.”
Feature: “Reporting is very customizable, you can customize to your KPI's to your success metrics.”

Site Spect (Eric Hansen)
Unique: “Our solution is non intrusive, there is no code to integrate and you can have a managed or hosted solution.”
(On a follow up question I requested a clarification of what non-intrusive” means: "We have a solution, hardware or software, that sits between the customer and your web servers and in real time can detect the content going out and send the content that you want to go out for the test to be executed." Think of how packet sniffers work by sitting on your servers (software) or in your data center (hardware)).
Feature: “Our solution will dynamically build orthogonal arrays based on all the variables that you want to test.”

Unica Affinium NetInsight (Karen Hudgins)
Unique: “We provide the most robust reporting for marketers doing cross channel marketing. We have built integration robustness into the tool, for example easy integration with your CRM data.”
Feature: “Our "drag and drop" functionality. For example you could be looking at a group of referrers and you want to see which referrer traffic stayed on your site for ten minutes or more. This is really simple, just drag and drop the time from the menu and drop it on the referrers and your report is ready. The ease of use of our tool is unique, for both new users or long term users.”

Google Analytics (Jeff Gills)
Unique: “Scalability. We have thousands of users and we can tap into their voices and feedback to select features that we implement, features that will be most relevant to them.”
Feature: “Cross segmentation, for example you have the ability to look at a webpage and which keyword brought people to that page and analyze that. You can also compute life time value.”

indexTools (Dennis Mortensen)
Unique: “Web analytics is increasingly becoming a commodity with similar features showing up in all the tools. If you have $100k to spend on web analytics then our tool will give you all the features that you are looking for a fraction of the cost of other vendors and you'll have money left over for investing in analysts.”
Feature: “Ability to segment and the ease of doing segmentation. We make it easy to take the idea that you have in your head and help you easily do that segmentation in our tool.”

Instadia (Anders Jorgensen)
Unique: “Instadia is a integrated service solution that will give you clickstream analysis but it will also give you the ability to understand why people who visit your website do what they do (via a built in surveying capability). We have a architecture that allows us to store unaggregated data what in turn allows our customers to have the ability to do infinite segmentation and aggregation so that they can look at the data any way they want.”
Feature: “The ability to create your prefect dashboards / reports / views using excel and then uploading it into Instadia and going forward Instadia can produce your excel created dashboard and populate that with the latest data (even from other non web analytics data sources) and produce new versions and email them to your decision makers.”

Maximine (Michael Fung)
Unique: “Our customer retention rate is greater than 90%. It is because 80% of our features from our customer's feedback and our ability to scale to meet our customers needs.”
Feature: “Our "web map" to represent the web architecture is very unique. Out ability to store all the information in a spatial database which makes things very relevant but allowing us to answer questions very quickly.”

[I have to apologize to two vendors who were at the summit but I could not interview. Ironically I have a great deal of affection for both of them, and I forgot. So I feel worse. The vendors are ClickTracks and Visual Sciences. John, Eric, I am sorry, you can answer the questions via comments with benefit of hindsight!]

From my quick interviews here were some overall, surprising, observations:

* It is amazing how many times the vendor representatives don't / can't / won't answer the question who their competitors are. Some seemed to be genuinely perplexed that they have competitors.

* A lot of them answer the question "what is unique about you" but sharing what is not great about their competitors. That might not be optimal framing when speaking to your customers, even if the competitors suck.

* I have the privilege of having used many of solutions these vendors were selling. Hence it was especially interesting to hear answers of these vendors. For many I would have answered the two questions completely differently from how they answered. So a difference in what a company thought was unique, and what I think it is from my perspective as a customer. Which could be ok, but in some cases I could have answered the questions better for them! : )

So what can you do with this blog post:

Vendors: Is what is below truly reflective of your unique and differentiated value proposition? Does everyone in your company know why you are so unique and can they articulate it to all your customers? [Perhaps this post might be of value: Make a Great Vendor / Agency / Consulting Pitch - Win Big Contracts.]

Customers (practitioners): Even though the two questions are seemingly simple they are very tough questions (very few people are good at answering to ambiguity). As you consider these vendors hopefully this post has given you some food for thought.

What do you all think? Are there other answers above that you would have voted as the best answer? Do you agree with answers to the two unique questions from your current vendor? Do you have a better answer to these questions for your vendor? Please share your feedback via comments.

[Like this post? For more posts like this please click here.]

Next Page »