January 2007

30 Jan 2007 01:03 am

DSC00115 small1The post on questions to ask WA Vendors started with a observation that it is perhaps more important that you ask yourself a few questions before you get into the Web Analytics tools selection process. These questions would help you understand your company needs in a very unique way which would help you pick the right tool.

The core point of that argument is simple: Most tools selection processes have very little self reflection built in, they might even be cookie cutter processes that might be from some book (or worse from some random blogger!! :)). This is not optimal.

Each company is unique and your approach to picking the right tool should be preceded with a lot of self reflection to ensure you understand who you really are and what you really need. (The “radical” selection process recommended in a earlier post forces this self reflection.)

This post covers three questions that should prompt the kind of critical self reflection that should increase the possibility that you’ll pick the right web analytics tool for yourself.


    # 1: Do you want reporting or analysis?

    # 2: Do you have IT strength or Business strength? Or Both?

    # 3: Are you solving for the Trinity or Clickstream reporting / analysis?

Now the questions, context and an attempt at providing some guidance…..

Do you want reporting or analysis?

    This is a very difficult question to answer because most organizations have a very hard time being honest about what they need, humans are also poor at being self critical.

    The other reason this is hard is that its like asking if you like a $0.99 Hershey chocolate bar or a piece of L'Artisan du Chocolat. It would be a sin not to say L'Artisan du Chocolat.

    Everyone wants analysis, yet few organizations (especially ones that are greater than 100 people in total size) actually want analysis. They want reporting.

    There are numerous reasons for this, including:

    • Decentralized decision making.
    • Company cultures (consensus, “cover your back”, layers of management, matrixed, paperwork driven and so on and so forth).
    • Availability of tools / features. History.
    • Propensity of risk (as in no propensity to take risk and risk taking actually harmful for career).
    • Distribution of knowledge in people / teams.
    • Availability of raw analytical brain power.
    • And more….. (suggest your own in comments below).

    measurementThe bottom line is that while everyone actually says that they want analysis. Admitting that you simply want reporting would be a sacrilege. Yet most companies simply want reporting. They want a web analytics tool and a web analytics team that simply facilitates reporting that is requested from Marketers, Business Leaders, VP’s, CMO’s and others.

    The web analytics team might be told that they should provide analysis (remember given a free choice no one will opt for the Hershey’s bar), but they are not set up to have time or knowledge to provide analysis and if by their free will they do provide analysis then it rarely translates into acceptance and action. Success of the web analytics tool implementation is measured by the number of reports it provides, the number of KPI's it is pushing out, the number of Marketers who say they are getting reports.

    If you are deciding on what web analytics tool to choose you should take a really hard self critical look at your company, its decision making structure, its needs and then be honest and decide if you want reporting or analysis. Then decide on what tools in your selection criteria are good at reporting and what tools are good at analysis. Ignore the category you are not interested in (usually analysis).

    This is a really hard choice to make, even harder to justify (because everyone thinks they want analysis). But having the tough conversation will ensure that your company will be happy with the choice that it makes, it will in all probability save money (reporting tools are much cheaper) and it is highly likely that on the long run you will be successful.

    You choose the wrong tool (say a true analysis tool), it will only turn choice smallpeople off the tool and using data (because it will suck at reporting) and on the long run hurt you a lot more. You choose the right tool for your company (say that is really powerful at reporting) and you will glean a lot more traction since atleast they will look at something and your web analytics people will know what they are getting into. Over time your company, if the culture and org structure and risk taking are all lining up, will get smarter and maybe you’ll be ready to move to a analytical tool.

    I am going out on a limb here but currently there are potentially only two true analysis tools: ClickTracks and VisualSciences. They come at massively different price points and have a very differentiated set of features and performance (speed, complexity, depth and breadth, sexiness). They are both right for the right kind of organizations, in case you decide you are it, and both need a deep self reflection on your part before you send a chq.

    Other tools are also making great progress getting there, empowering true analysis out of the box to complement their far superior strength in being able to do awesome reporting.

    IMPORTANT: Hopefully you’ll see that there is no judgment being made above if reporting is better or analysis is better. Each serves its purpose. The emphasis is on figuring out what you really need and then buying what you really need.

Do you have IT strength or Business strength? Or Both?

    strength smallSome companies are good at IT (technology), others are good at the Business side of things (marketing, analysis, strategic decisions etc etc). A rare few are good at both, or have environments where the two are the same when it comes to Web Analytics.

    Pulling off a successful web analytics implementation is complicated and it is often easy to get it wrong. A professional services company recently shared with me that seven out of ten times they find the tool implementation is wrong (and the client has not known about it for a while, and have been using wrong data).

    If you have solid IT (technology) and Business strength in your company then go at it all by yourself and you’ll be fine (here are Web Analytics Technical Implementation Best Practices).

    If you don’t have strong IT strength (by that I mean IT who knows and gets Web Analytics and not just standard IT) then go with a partner, say my buddy Justin at EpikOne  (Justin let’s talk about my cut of this deal! – all joking aside I have not financial or other ties with EpikOne).

    If you don’t have IT strength and you don’t want (or can’t afford a consultant) please consider following this process: How to Choose a Web Analytics Tool: A Radical Alternative.

    Here is one last reason to assess IT strength: if you want to do in-house and not asp. If you want to consider hosting the data collection and analysis in-house (say with WebTrends, ClickTracks or Unica NetInsight) then you need some serious IT/Technology strengths (in atleast one person) to pull it all off. Ensure that you are covering this important consideration.

    If you don’t have strong Business strength (realize that just by figuring this out honestly and you are already ahead of your peers) consider following this process: Web Analysis: In-house or Out-sourced or Something Else. It provides a great framework for the evolution of a effective and efficient web analytics program.

Are you solving for the Trinity or Clickstream reporting / analysis?

    This is a mind-set question. It is a question that tries to judge what you are solving for. It tries to get to help you understand the level at which you are approaching the solution set. It is all about trying to know if you need to buy a tool that will help you “understand clicks” (which is ok) or do a lot more than that. It tries to help you crystallize what your short term goals are and what your long term goals are.

    trinity web analytics strategy1

    The Trinity mind-set & strategy calls for having a robust Qualitative and Quantitative analysis in your web analytics approach with the goal of: Understanding the customer experience explicitly (via Research), to then influence customer behavior on your site (via Clickstream analysis), leading to win-win Outcomes (via outcomes analysis) for your company.

    Your consideration criteria will be vastly different depending on where you are and what your own point of view and approach is in this context. In one case a simple log parser is fine, in another you need a tool that integrates with other data sources and in yet another case you need a tool will pay ball with your data warehouse.

    See this post: Web Analytics Tool Selection: 10 Questions to ask Vendors. The questions you’ll ask and what you’ll stress with change with the answer to the question above.

That’s it. Three questions.

A bit of my own self reflection & guidance:

I  am on record saying “don’t spend too much time on worrying about web analytics implementation, turn on Google Analytics on your website and follow a simple, but “radical”, process”. Given that I feel a bit embarrassed at publishing two consecutive posts on Web Analytics Tool selection.  Both posts combined I suspect highlight how complex the process of selecting a tool can be. To some of you it might even look scary as to how much thought you need to put into selecting the right tool for your company.

four steps small1To put a nice bow around all this (and to make it easy for all of you), here is how to think about web analytics tool selection & implementation:

It’s that simple.

My apologies if there is any inadvertent confusion. On the bright side regardless of your size and what tools you own now you have a end to end overview of how to go from nothing to something to something awesome. Good Luck!!

What would be your own self-reflection questions? Will the above questions work for you? Am I missing anything obvious? Is this a good use of time? Vendors what else can we ask ourselves before we ask you to bare your soul? Please share your feedback via comments.

[Like this post? For more posts like this please click here.]

23 Jan 2007 05:58 pm

DSCF2190 smallJerome N, a reader from the UK, asked me to write a post on the questions that should be presented to a Web Analytics Vendor during tool selection process. My instinctive reply to Jerome was “I think it is more important to know what questions to ask of yourself (your company) before you go into tool selection because the answers to those questions are a lot more critical and will tell you what you are ready for”.

My reaction was that because I feel companies don’t do a good enough job of understanding themselves (in a deeply self critical way) and hence pick a sub optimal solution. But it was also sourced from the fact that in vendor pitches all the answers are “yes we can do it” and it is very hard to discern reality. (If you are a Vendor read this post on tips on doing optimal presentation, if you are a Client read this post on web analytics tool selection Process).

But since Jerome was so kind to ask I wanted to respect his request. It is important to note that most of my suggestions below you would ask the for-fee (paid) Vendors, but you should also evaluate the free vendors in this context. I’ll also fess up that in a vendor pitch or conversation it is hard to get tease out the nuances but hopefully there is enough meat in rest of the post to empower you to tease out some reality.

10 Questions to ask Web Analytics Vendors: Summary:

    1) Main differences with free tools.
    2) Types of versions and flexibility.
    3) Types of data collection options and entrenchment.
    4) TCO (Total Cost of Ownership).
    5) Type of Support (options, pricing, technical or business).
    6) Segmentation awesomeness (post data capture).
    7) Exporting data (options, history, data ownership).
    8) Integration with other sources of data.
    9) What’s up next, the competitive edge.
    10) Types of business lost, why.
    11) Bonus Question.

Jerome here are ten questions you should ask a Web Analytics Vendor before you get too deep into selection……..

1) What is the difference between your tool / solution and Google Analytics? Please share your top five, just five, reports that are different from GA and why?
(In a few months you can replace GA with our friend Ian’s Gatineau analytics package from Microsoft, or if you want log file based solutions then the free ClickTracks Appetizer.)

    Might as well acknowledge the white elephant in the room. If there are good analytics tools out there for free why should you pay for a good analytics tool? Why not focus on following the 10/90 rule for web analytics success?

    hot 20air 20balloon smallThe answer you are looking for is not that Google and Microsoft are big and bad and your privacy is under threat. That’s a cop out if there ever was one. You are also not looking for a answer that you won’t get support for a free tool or that they will die and wither away.

    You are looking for specific and tangible examples of reports and metrics that your for-fee vendor will provide that Google (and later Microsoft) don’t provide. Any analytics Vendor worth paying for will have a crisp answer that focuses on differential features, reports, metrics, integration points etc and not scare tactics.

    I want to stress that GA and Gatineau are not right for everyone and don’t have every feature you want. You should carefully consider fit of those tools for your company. But it should be based on facts and not some of the FUD (fear, uncertainty and doubt) that is put forth.

2) Are you 100% ASP or do you offer a software version we can buy and install in-house? Are you planning to do a software version?

    I am on the record saying one of the challenges Vendors will face is that Clients will want to have solutions that are software based and in-house rather than ASP based. Please see Five “Ecosystem” Challenges for Web Analytics Vendors.  Currently most vendors are ASP based with no software based offerings (except for WebTrends and ClickTracks, I believe). That is ok for now, might not be so in a year or two.

    wwwWith this question you are trying to probe for how ready the Vendor is for the future in terms of having differentiating offerings for you should you want them (and I think you will, not tomorrow but in the near future). You are also looking for the intangible: how they react to this question, just as much as the content of their answer.

    You can also ask them about first part and third party cookies, which one they use and how much pain, and cost, would it be to use first party. You should almost always use first party cookies, most Vendors enable this. What you are looking for when you ask this question is their reaction. Did they proactively advice you to have first party cookies? Did they insist on it? It shows the mindset of the Vendor.

3) What data capture mechanisms do you use? Javascript only? Javascript and web logs? Web logs only? Others?

    There are many different ways to capture data, javascript tags and web logs and packet sniffers and “sensors” and so on and so forth. Javascript tags and web logs are the most common, each have their merits (if you are stuck between making a choice between them, after having considered all options, here’s a post to help you make a choice).

    data capture small1In the answer to this question you are looking for a the kind of flexibility that a Vendor has natively in dealing with different data capture formats, you are looking for a Vendor who might evolve beyond javascript tags (or logs or sniffers) as the web evolves and becomes much harder to track (as in flash, flex, RIA’s, RSS, mash-ups and things we don’t even know). No current methodology will survive for a long time, is the Vendor you are considering ready for the near future data capture challenges?

    You are not looking for them to brainwash you that javascript tags are God’s answer to all your prayers (or web logs or packet sniffers or “sensors”). If they try that chalk it up in the sub optimal sales choice column.

4) Please help me calculate the TCO (Total Cost of Ownership) for your tool.

    Most Vendors will tell you that that the cost of their tool will be $30 per month for any site of any size and it is a all you can eat buffet. Ok so I am stretching that a bit, but by not that much. :)

    TCO direction small1The Total Cost of Ownership of a web analytics tool can be massively different depending on who you are as a company, what you already have it in place and, mostly, who your Vendor is and what their pricing strategies are. I encourage you to poke and dig for data to get a clear understanding of what the TCO is for each Vendor you are selecting.

    Here are elements or TCO that you would have to consider:

       = Cost Per Page View (since most ASP based vendors charge per page view)

      + Incremental costs beyond initial lump-some (charge if you go over your allocated page views, are there any “advanced” features – say RIA tracking or RSS or whatever in terms of extra modules that would cost more, what else could we buy from the vendor that we might need later – for example Pay Per Click integration with Google / YSM or keyword bidding feature or whatever, better to know now)

      + Annual support costs after year one

      + Cost of professional services (initial install and then post launch trouble shooting or customizations)

      + Any additional hardware you need at your end (pc’s, laptops, web servers, data storage drives etc, could vary by Vendor)

      + Cost of “administration” (manage vendor relationship – could be partial head count, someone to create all the reports and publish them, someone to coordinate between vendor and IT and marketers – all these could be one person, better to know now)

      + Cost of analysts needed to draw insights (you can lump this with the one above if you want but it is important to be aware of the 10/90 rule and realize that you can’t just buy the tool, you also have to hire a relatively intelligent brain to interpret the data – there are vendors who have stated that their tools are so smart that you don’t need Analysts, this would work in your favor but make sure you buy that)

      + Additional head count (partial or full) to maintain the tags, liaise with IT, update pages on the site etc

    Total that up across Vendors and make a informed choice. The only thing above that it not totally in “unique to each Vendor” camp are the Analysts you’ll need, stress test that it is ok to break the 10/90 rule because the tool inherently is so smart.

    Also notice that while some of the above are eliminated with a free tool other things above will still apply even in the case of a free vendor.

5) What kind of Support do you offer? What is included for free and what would cost more? Is free 24×7? What are the limits? How far will you go to help me solve my technical problems and answer “silly” questions from my business users?

    During Vendor pitches you’ll hear that everything is free (and some web analytics Vendors do indeed offer loads and loads of absolutely free support as long as you stay with them). But often there are limits and caveats that are not explicit and you’ll have to dig them out.

    support small1Signing and contract and implementing the solution signifies the start of your problems and not the end of your data problems. It is critical that you understand exactly what you’ll get and exactly how much it will cost to get what you need (say if a Vendor will only provide business hour support during week days then what would 24×7 cost, or if they will only answer questions about the tool and not why the tool is not working with your site then what would that cost – just to stress these are just suggestions to get your juices flowing, you’ll have to make up your own unique questions).

    You will need support and professional services, understand what the Vendor will provide or what the Vendor’s “Authorized Consultants” will provide etc.

6) What features do you have in your tool that allow me to segment the data?

    Ian posted recently asking if we need Power or Simplicity (please cast your vote and help him). I agree with his grouping of sub features in each, except for one. Segmentation is not in Power. Segmentation should be in Simplicity. Segmentation should be everywhere. There should be no web analytics tool that does not allow for massive segmentation, in fact every tool should allow for massive segmentation  in a very simple way!  

    segments smallWithout segmentation all data is not very insightful, I know that sounds extreme but segmentation is the fastest path to insights. You should deeply stress your Vendor on exactly what options you have to segment data and how easy it is to do it. Try it yourself and see if you can segment data in the tool.

    Also ask for one important point: Do you have to pre-code everything in custom javascript tags on each page on your site to be able to segment the data post capture? Or you can capture data with a standard tag and do segmentation later (even though you did not implement all the segmentation possibilities before launch in custom javascript tags)?

    Most vendors are in the former camp, custom javascript tags on pages to enable any segmentation, and that makes segmentation much harder (how can you think of all the questions you’ll ask of the data up-front before you install the tool?). Again understand where the Vendor is and make a informed choice.

7) What options do I have to export data from your system into our company system? Can I get all the raw data? Can I export processed data? How easy it is for me to export 100,000 rows of Processed (not Raw) data out of your tool into my other company systems? What happens if I terminate my contract with you?

    Ok I admit that’s a lot of questions, but it really is one important question: Who owns the data and if the Vendor stores it and you want to export it do you get the raw logs (huge data files with no intelligence in terms of computed metrics so you have to figure out how to do that) or Processed data (computed data that is much easier to integrate into where ever you are taking the data).

    This question is as much a process of you discovering what you need and realizing that you are not going to get it and then being ok with that (or not).

    export small1Typically most vendors will say you can export everything. As them the specific questions above, understand exactly what you can actually export (remember that you can get a excel dump is not the answer which is why 100,000 rows are mentioned above) and then form a opinion if that is sufficient for your company or not.

    Let me stress that I am not recommending that you insist on getting it all (data) or in a particular way, I am recommending that you ask the hard questions above so you don’t have any disappointment later about what you’ll get should you need it. I have gone into this rather naively and won’t do that again.

8) What kinds of features do you provide for me to integrate data from other sources into your tool?

    There is so much on this blog about the Trinity model and multiple sources of data and how to get a complete picture and all that nice stuff. Bottom-line your clickstream data, no matter what Vendor you use, will feel limiting after a while when you want real insights and you’ll be forced to integrate it with other sources of data. Since the export route (#7 above) is so hard you’ll have to bring that data in. This question goes to the heart of figuring out how easy it is to do that for each Vendor you are considering.

    integrate small1Examples of data you might want to bring into your tool are: meta-data from other sources in your company, or you CRM data, or data from you ad / search agency, or data from  surveys that has the primary key in there (like cookie values), or a/b / multivariate testing integration so you can measure conversion from the MVT tool but also analyze Site Overlay (Click Density) for each recipe, etc. You have to be able to import data efficiently (without needing humans if possible) and then use it for segmentation or reporting.

    Some Vendors can automatically pick up data from Google AdWords and do direct integration without you having to do anything. Others will require you to do a daily download yourself and upload a text or csv file into your tool. Others still can’t do anything. Figure out what the line in the sand is for the Vendor you are considering for the kinds of data you want to integrate.

9) What are two new features / tools / analytical advancements / acquisitions is your company cooking up that would keep you ahead of your competition for the next three years?

    At the core of this question you are trying to see if your Vendor is good at the today or if they are worried about the tomorrow and have plans to deal with tomorrow.

    leap hurdle 2D1 small1It also gives you a sense at how much they know about their own position and that of their competitors (hence we are not asking what are two things you are doing that are good, the framing is in context of competition). As our E-metrics 2006 DC summit experiment proved some Vendors are much much better at having a good reality check about themselves (read: Hello, My Name is Avinash. What is Unique About You?).

    You want to be impressed by atleast one of the two things you hear being a complete surprise to you (which means they are ahead of you, always a good thing), and you want to get the feeling that your Vendor has a good sense for themselves and their competitors.

    If you ask this exact same question across a few Vendors they will talk about each other and the differing perspectives are a source of valuable insights for you.

10) Why did the last two clients you lost cancel their contracts with you? Who are they using now? Would you permit us to call one of them?

    I have to admit this is not one of my questions, a Vendor taught me this question and it is truly fantastic. patriots smallYou want to be confident that you are making the right choice and there is no better way than to learn why each Vendor recently lost someone’s business.

    Now the reality is that you will probably hear marketing / sales speak and not reality but even hearing the marketing / sales speak can be of value. Of a whole bunch of vendors only two have ever answered this question honestly (both were VP of Sales answering the question) and the interesting thing is that we are doing business with both today, even though in both cases they are not the most awesome Vendor technologically.

    Remember with any Vendor you are actually buying a relationship and not just the tool and on the long run (even near short term) people you can do business with will far outstrip value with than working with the most advanced tool on earth whose people you can’t have a relationships with. 

Bonus question to ask:

11) Do you read Occam’s Razor by Avinash Kaushik and what do you think of his “radical” postings that tend to have a anti-vendor slant?

    This one is only partly fun.  A friend who is a respected industry authority and works at a vendor did say to me that he and the other Vendors got together and the sentiment that was expressed was “what does Avinash have against us”, he actually used even stronger language.

    It was a huge surprise to hear this because this is absolutely not true and some my industry best friends are Vendors. But since you might ask them some of the above questions it is fair for you to ask them question 11 so they can give you their point of view on my stated and unstated biases.

    kissing booth smallJust for the record, I don’t have anything against any Vendor (explicit or implicit). We are lucky to have a good crop of tools in our industry, some good, some better, some awesome. I am glad that we can pick and choose and I am glad that my friends at Omniture, ClickTracks, Google Analytics, WebTrends, CoreMetrics, Instadia, indexTools, HBX, Visual Sciences, Unica NetTracker, etc read this blog, subscribe to it and often make public comments and send me private emails. I thank you all and I promise you there is absolutely no anti-vendor slant on any vendor.

What do you all think? Of #11 and all other questions? What am I missing in the list? Do you have your favorite question to ask any Vendor? Please share your feedback via comments.

[Like this post? For more posts like this please click here.]

Next Page »