DSCF2190 smallJerome N, a reader from the UK, asked me to write a post on the questions that should be presented to a Web Analytics Vendor during tool selection process. My instinctive reply to Jerome was “I think it is more important to know what questions to ask of yourself (your company) before you go into tool selection because the answers to those questions are a lot more critical and will tell you what you are ready for”.

My reaction was that because I feel companies don’t do a good enough job of understanding themselves (in a deeply self critical way) and hence pick a sub optimal solution. But it was also sourced from the fact that in vendor pitches all the answers are “yes we can do it” and it is very hard to discern reality. (If you are a Vendor read this post on tips on doing optimal presentation, if you are a Client read this post on web analytics tool selection Process).

But since Jerome was so kind to ask I wanted to respect his request. It is important to note that most of my suggestions below you would ask the for-fee (paid) Vendors, but you should also evaluate the free vendors in this context. I’ll also fess up that in a vendor pitch or conversation it is hard to get tease out the nuances but hopefully there is enough meat in rest of the post to empower you to tease out some reality.

10 Questions to ask Web Analytics Vendors: Summary:

    1) Main differences with free tools.
    2) Types of versions and flexibility.
    3) Types of data collection options and entrenchment.
    4) TCO (Total Cost of Ownership).
    5) Type of Support (options, pricing, technical or business).
    6) Segmentation awesomeness (post data capture).
    7) Exporting data (options, history, data ownership).
    8) Integration with other sources of data.
    9) What’s up next, the competitive edge.
    10) Types of business lost, why.
    11) Bonus Question.

Jerome here are ten questions you should ask a Web Analytics Vendor before you get too deep into selection……..

1) What is the difference between your tool / solution and Google Analytics? Please share your top five, just five, reports that are different from GA and why?
(In a few months you can replace GA with our friend Ian’s Gatineau analytics package from Microsoft, or if you want log file based solutions then the free ClickTracks Appetizer.)

    Might as well acknowledge the white elephant in the room. If there are good analytics tools out there for free why should you pay for a good analytics tool? Why not focus on following the 10/90 rule for web analytics success?

    hot 20air 20balloon smallThe answer you are looking for is not that Google and Microsoft are big and bad and your privacy is under threat. That’s a cop out if there ever was one. You are also not looking for a answer that you won’t get support for a free tool or that they will die and wither away.

    You are looking for specific and tangible examples of reports and metrics that your for-fee vendor will provide that Google (and later Microsoft) don’t provide. Any analytics Vendor worth paying for will have a crisp answer that focuses on differential features, reports, metrics, integration points etc and not scare tactics.

    I want to stress that GA and Gatineau are not right for everyone and don’t have every feature you want. You should carefully consider fit of those tools for your company. But it should be based on facts and not some of the FUD (fear, uncertainty and doubt) that is put forth.

2) Are you 100% ASP or do you offer a software version we can buy and install in-house? Are you planning to do a software version?

    I am on the record saying one of the challenges Vendors will face is that Clients will want to have solutions that are software based and in-house rather than ASP based. Please see Five “Ecosystem” Challenges for Web Analytics Vendors.  Currently most vendors are ASP based with no software based offerings (except for WebTrends and ClickTracks, I believe). That is ok for now, might not be so in a year or two.

    wwwWith this question you are trying to probe for how ready the Vendor is for the future in terms of having differentiating offerings for you should you want them (and I think you will, not tomorrow but in the near future). You are also looking for the intangible: how they react to this question, just as much as the content of their answer.

    You can also ask them about first part and third party cookies, which one they use and how much pain, and cost, would it be to use first party. You should almost always use first party cookies, most Vendors enable this. What you are looking for when you ask this question is their reaction. Did they proactively advice you to have first party cookies? Did they insist on it? It shows the mindset of the Vendor.

3) What data capture mechanisms do you use? Javascript only? Javascript and web logs? Web logs only? Others?

    There are many different ways to capture data, javascript tags and web logs and packet sniffers and “sensors” and so on and so forth. Javascript tags and web logs are the most common, each have their merits (if you are stuck between making a choice between them, after having considered all options, here’s a post to help you make a choice).

    data capture small1In the answer to this question you are looking for a the kind of flexibility that a Vendor has natively in dealing with different data capture formats, you are looking for a Vendor who might evolve beyond javascript tags (or logs or sniffers) as the web evolves and becomes much harder to track (as in flash, flex, RIA’s, RSS, mash-ups and things we don’t even know). No current methodology will survive for a long time, is the Vendor you are considering ready for the near future data capture challenges?

    You are not looking for them to brainwash you that javascript tags are God’s answer to all your prayers (or web logs or packet sniffers or “sensors”). If they try that chalk it up in the sub optimal sales choice column.

4) Please help me calculate the TCO (Total Cost of Ownership) for your tool.

    Most Vendors will tell you that that the cost of their tool will be $30 per month for any site of any size and it is a all you can eat buffet. Ok so I am stretching that a bit, but by not that much. :)

    TCO direction small1The Total Cost of Ownership of a web analytics tool can be massively different depending on who you are as a company, what you already have it in place and, mostly, who your Vendor is and what their pricing strategies are. I encourage you to poke and dig for data to get a clear understanding of what the TCO is for each Vendor you are selecting.

    Here are elements or TCO that you would have to consider:

       = Cost Per Page View (since most ASP based vendors charge per page view)

      + Incremental costs beyond initial lump-some (charge if you go over your allocated page views, are there any “advanced” features – say RIA tracking or RSS or whatever in terms of extra modules that would cost more, what else could we buy from the vendor that we might need later – for example Pay Per Click integration with Google / YSM or keyword bidding feature or whatever, better to know now)

      + Annual support costs after year one

      + Cost of professional services (initial install and then post launch trouble shooting or customizations)

      + Any additional hardware you need at your end (pc’s, laptops, web servers, data storage drives etc, could vary by Vendor)

      + Cost of “administration” (manage vendor relationship – could be partial head count, someone to create all the reports and publish them, someone to coordinate between vendor and IT and marketers – all these could be one person, better to know now)

      + Cost of analysts needed to draw insights (you can lump this with the one above if you want but it is important to be aware of the 10/90 rule and realize that you can’t just buy the tool, you also have to hire a relatively intelligent brain to interpret the data – there are vendors who have stated that their tools are so smart that you don’t need Analysts, this would work in your favor but make sure you buy that)

      + Additional head count (partial or full) to maintain the tags, liaise with IT, update pages on the site etc

    Total that up across Vendors and make a informed choice. The only thing above that it not totally in “unique to each Vendor” camp are the Analysts you’ll need, stress test that it is ok to break the 10/90 rule because the tool inherently is so smart.

    Also notice that while some of the above are eliminated with a free tool other things above will still apply even in the case of a free vendor.

5) What kind of Support do you offer? What is included for free and what would cost more? Is free 24×7? What are the limits? How far will you go to help me solve my technical problems and answer “silly” questions from my business users?

    During Vendor pitches you’ll hear that everything is free (and some web analytics Vendors do indeed offer loads and loads of absolutely free support as long as you stay with them). But often there are limits and caveats that are not explicit and you’ll have to dig them out.

    support small1Signing and contract and implementing the solution signifies the start of your problems and not the end of your data problems. It is critical that you understand exactly what you’ll get and exactly how much it will cost to get what you need (say if a Vendor will only provide business hour support during week days then what would 24×7 cost, or if they will only answer questions about the tool and not why the tool is not working with your site then what would that cost – just to stress these are just suggestions to get your juices flowing, you’ll have to make up your own unique questions).

    You will need support and professional services, understand what the Vendor will provide or what the Vendor’s “Authorized Consultants” will provide etc.

6) What features do you have in your tool that allow me to segment the data?

    Ian posted recently asking if we need Power or Simplicity (please cast your vote and help him). I agree with his grouping of sub features in each, except for one. Segmentation is not in Power. Segmentation should be in Simplicity. Segmentation should be everywhere. There should be no web analytics tool that does not allow for massive segmentation, in fact every tool should allow for massive segmentation  in a very simple way!  

    segments smallWithout segmentation all data is not very insightful, I know that sounds extreme but segmentation is the fastest path to insights. You should deeply stress your Vendor on exactly what options you have to segment data and how easy it is to do it. Try it yourself and see if you can segment data in the tool.

    Also ask for one important point: Do you have to pre-code everything in custom javascript tags on each page on your site to be able to segment the data post capture? Or you can capture data with a standard tag and do segmentation later (even though you did not implement all the segmentation possibilities before launch in custom javascript tags)?

    Most vendors are in the former camp, custom javascript tags on pages to enable any segmentation, and that makes segmentation much harder (how can you think of all the questions you’ll ask of the data up-front before you install the tool?). Again understand where the Vendor is and make a informed choice.

7) What options do I have to export data from your system into our company system? Can I get all the raw data? Can I export processed data? How easy it is for me to export 100,000 rows of Processed (not Raw) data out of your tool into my other company systems? What happens if I terminate my contract with you?

    Ok I admit that’s a lot of questions, but it really is one important question: Who owns the data and if the Vendor stores it and you want to export it do you get the raw logs (huge data files with no intelligence in terms of computed metrics so you have to figure out how to do that) or Processed data (computed data that is much easier to integrate into where ever you are taking the data).

    This question is as much a process of you discovering what you need and realizing that you are not going to get it and then being ok with that (or not).

    export small1Typically most vendors will say you can export everything. As them the specific questions above, understand exactly what you can actually export (remember that you can get a excel dump is not the answer which is why 100,000 rows are mentioned above) and then form a opinion if that is sufficient for your company or not.

    Let me stress that I am not recommending that you insist on getting it all (data) or in a particular way, I am recommending that you ask the hard questions above so you don’t have any disappointment later about what you’ll get should you need it. I have gone into this rather naively and won’t do that again.

8) What kinds of features do you provide for me to integrate data from other sources into your tool?

    There is so much on this blog about the Trinity model and multiple sources of data and how to get a complete picture and all that nice stuff. Bottom-line your clickstream data, no matter what Vendor you use, will feel limiting after a while when you want real insights and you’ll be forced to integrate it with other sources of data. Since the export route (#7 above) is so hard you’ll have to bring that data in. This question goes to the heart of figuring out how easy it is to do that for each Vendor you are considering.

    integrate small1Examples of data you might want to bring into your tool are: meta-data from other sources in your company, or you CRM data, or data from you ad / search agency, or data from  surveys that has the primary key in there (like cookie values), or a/b / multivariate testing integration so you can measure conversion from the MVT tool but also analyze Site Overlay (Click Density) for each recipe, etc. You have to be able to import data efficiently (without needing humans if possible) and then use it for segmentation or reporting.

    Some Vendors can automatically pick up data from Google AdWords and do direct integration without you having to do anything. Others will require you to do a daily download yourself and upload a text or csv file into your tool. Others still can’t do anything. Figure out what the line in the sand is for the Vendor you are considering for the kinds of data you want to integrate.

9) What are two new features / tools / analytical advancements / acquisitions is your company cooking up that would keep you ahead of your competition for the next three years?

    At the core of this question you are trying to see if your Vendor is good at the today or if they are worried about the tomorrow and have plans to deal with tomorrow.

    leap hurdle 2D1 small1It also gives you a sense at how much they know about their own position and that of their competitors (hence we are not asking what are two things you are doing that are good, the framing is in context of competition). As our E-metrics 2006 DC summit experiment proved some Vendors are much much better at having a good reality check about themselves (read: Hello, My Name is Avinash. What is Unique About You?).

    You want to be impressed by atleast one of the two things you hear being a complete surprise to you (which means they are ahead of you, always a good thing), and you want to get the feeling that your Vendor has a good sense for themselves and their competitors.

    If you ask this exact same question across a few Vendors they will talk about each other and the differing perspectives are a source of valuable insights for you.

10) Why did the last two clients you lost cancel their contracts with you? Who are they using now? Would you permit us to call one of them?

    I have to admit this is not one of my questions, a Vendor taught me this question and it is truly fantastic. patriots smallYou want to be confident that you are making the right choice and there is no better way than to learn why each Vendor recently lost someone’s business.

    Now the reality is that you will probably hear marketing / sales speak and not reality but even hearing the marketing / sales speak can be of value. Of a whole bunch of vendors only two have ever answered this question honestly (both were VP of Sales answering the question) and the interesting thing is that we are doing business with both today, even though in both cases they are not the most awesome Vendor technologically.

    Remember with any Vendor you are actually buying a relationship and not just the tool and on the long run (even near short term) people you can do business with will far outstrip value with than working with the most advanced tool on earth whose people you can’t have a relationships with. 

Bonus question to ask:

11) Do you read Occam’s Razor by Avinash Kaushik and what do you think of his “radical” postings that tend to have a anti-vendor slant?

    This one is only partly fun.  A friend who is a respected industry authority and works at a vendor did say to me that he and the other Vendors got together and the sentiment that was expressed was “what does Avinash have against us”, he actually used even stronger language.

    It was a huge surprise to hear this because this is absolutely not true and some my industry best friends are Vendors. But since you might ask them some of the above questions it is fair for you to ask them question 11 so they can give you their point of view on my stated and unstated biases.

    kissing booth smallJust for the record, I don’t have anything against any Vendor (explicit or implicit). We are lucky to have a good crop of tools in our industry, some good, some better, some awesome. I am glad that we can pick and choose and I am glad that my friends at Omniture, ClickTracks, Google Analytics, WebTrends, CoreMetrics, Instadia, indexTools, HBX, Visual Sciences, Unica NetTracker, etc read this blog, subscribe to it and often make public comments and send me private emails. I thank you all and I promise you there is absolutely no anti-vendor slant on any vendor.

What do you all think? Of #11 and all other questions? What am I missing in the list? Do you have your favorite question to ask any Vendor? Please share your feedback via comments.

[Like this post? For more posts like this please click here.]

Social Bookmarks:

  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite
  • services sprite