USASearch Blog (FeedBurner)

Highlighted Feature

You can now access your raw HTTP logs. These logs contain valuable data on searches, clicks and USASearch tag page loads. 

Read How to Access Your Raw Logs for more details about how to request secure FTP access to your logs.

Other Features

• We open sourced all of the code that powers our Recalls Server API in Github.
• Searchers are now able to search and see results from collections that are not visible in the left-hand navigation on your “main” web results. You can use this option to provide a search box on a small subset of content (such as a policy manual or meeting minutes) while not cluttering up the left-hand navigation on your main results page. (It also helps reduce the likelihood that searchers experience a “dead end” on your main results page.)

Chores

• We upgraded to CentOS 6.3.
• We optimized the routing of search queries so that they’re served by the geographically closest datacenter. 

You can directly access your raw HTTP logs. The logs include valuable data on searches, clicks, and USASearch tag page loads.

With the raw logs, you can conduct in-depth, ad hoc analyses to answer any questions that you can’t answer using the standard analytics reports in the Admin Center. You can also automatically integrate your search logs with your other reports and business processes. 

To access your raw logs, you’ll first need to upload your public key on the Raw Logs Access page in the Admin Center. 

We’ll set up your public key on our secure FTP servers and send you an email when your logs are available for download. 

Once your key is set up, you’ll be able access your log files via secure FTP. We upload your logs each day, usually within several hours after midnight GMT. After a week, we purge the older files.

We recommend that you set up a cron script to automatically download the latest files each day.

Troubleshooting tip: On Unix systems, you can generate a RSA public key with no passphrase by typing in the following command. Once you’ve created a key, it’ll be located at /home/user/.ssh/id_rsa.pub.

% cd ~
% ssh-keygen -q -b 4096 -t rsa
Enter file in which to save the key (/home/user/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:

USASearch > Admin Center > YourSite > Raw Logs Access

One of the most common search tasks conducted by public is looking for current job openings. To help searchers quickly find relevant jobs we have created the federal Jobs GovBox.

When you turn on the federal Jobs GovBox, searchers looking for jobs on your site will see up to three relevant USAJobs listings on your results page.

• Federal Department Websites—If your website is affiliated with a specific federal department (e.g., Department of Commerce), the federal Jobs GovBox will return relevant, location- and keyword-matched job openings across all agencies within your department.
• Federal Agency or Office Websites—If your website is affiliated with a specific federal agency (e.g., NOAA) or office (e.g, National Ocean Service), the federal Jobs GovBox will return relevant, location-and keyword-matched job openings at your agency.
• State and Local Websites and Cross-Agency Portals—If your website is a state or local government site or a federal cross-agency portal (e.g., USA.gov), the federal Jobs GovBox will return relevant, location- and keyword-matched job openings across all federal agencies.

Searchers will also see a direct link to USAJobs.gov for all current job openings (specific to your department or agency, if applicable), as shown in the sample below for a search on jobs on the USDA.gov.

Highlighted Feature

The display of search results pages is constantly evolving. Some changes are subtle. Others are more noticeable.

In line with industry best practices and commercial search engines, we’ve moved the tools for filtering and sorting results to the top of results list (rather than on the left side). We’ve also added an option to clear all filters.

 

Chores

• We upgraded to Rails 3.2.11.

Fixes

• Searchers no longer see Did You Mean suggestions for your excluded domains.
• We fixed a cross browser compatibility issue on the Query Logs page in the Admin Center so that the ‘number of clicks’ column is now visible.
• We labeled the Safe Search radio options on our advanced search page as a fieldset to improve accessibility for screen readers.

Highlighted Feature

This week we buckled down and analyzed data on jobs-related searches to ensure that the new federal Jobs GovBox lists the most relevant job openings. 

Searchers can now effectively search for jobs by keyword, agency, state, or work schedule (part-or full-time).

We also focused on not showing the Jobs GovBox when it isn’t relevant (for example, it is no longer shown when searchers look for job safety posters or employment verification forms).

Chores

• We added our posting channel ID to all USAJobs links.
• We set up a cron job to fetch and process job files from USAJobs.
• We added impression and click tracking for the federal Jobs GovBox module.
• We resolved performance discrepancy between our two datacenters.
• We no longer use XML sitemaps to add documents to the USASearch web index.

Fixes

• You can now delete keywords from text best bets.
• Newly added misspellings are now being applied to type-ahead suggestions.
• Searchers can now find jobs in non-standard state locations (for example someone searching for jobs in Washington, DC, will find postings the the location of Washington DC Metro Area, District of Columbia).

Highlighted Feature: The federal Jobs GovBox

We’re excited to announce that searchers looking for jobs on your site can now see all of your agency’s current job openings.

Our new federal Jobs Govbox displays up to three location- and keyword-matched job openings on your results page. For more information on how to turn on this GovBox, see How To Select The Modules That Appear On Your Results Page.

We’ve also open sourced all of the code that powers our Federal Jobs Server API in Github.

Fixes

And, we’ve already pushed two bug fixes for the federal Jobs GovBox to production. Searchers no longer see yet-to-be-opened or already closed jobs. 

Highlighted Feature

Are you curious about the most searched for terms on your site in the past year? They’ll arrive in your inbox in our new, annual Year-in-Review email.

We’ll send you an email every December 15 (along with cheer and goodwill) with a list of the top 1,000 terms searched for on your site over the past year and how many times each term was searched.

Other Features

• Developers using our jobs API can find agency-specific jobs by searching for an agency’s name, abbreviation, or other variant.

Chores

• We upgraded to Rails 3.2.10.
• We sped up our interaction with the Bing Search API.
• We set up an automatic download of current jobs opening from USAJobs’ jobs export.

Fixes

• Searchers and API users are now able to maintain a HTTP connection for multiple requests resulting in shorter response times.

Highlighted Feature

Did you know that mobile search traffic increases during the holidays? 

You can now customize the header on your mobile search results page to create a more seamless experience for visitors to your mobile website. Our new feature allows you to upload your mobile logo and specify a mobile homepage URL.

USASearch > Admin Center > YourSite > Header and Footer

Other Features

• Searchers see Bing image results followed by results from your Flickr photostream. (Example: search for images about hurricane sandy on DOL.gov.)
• Searchers can now sort RSS-based results (such as press releases, blogs, etc.) by date or relevance.
• Searchers now only see tweets from up to three months ago.
• Searchers now only see articles from up to one year ago in the News GovBox.
• You can now exclude specific subdomains or folders from your search results on the Excluded Domains page in the Admin Center.
• You can now segment the keywords for your text Best Bets.
• You now see your domains listed in A-Z order on the Domains page in the Admin Center.
• You now see your requested 3rd Party Tracking scripts in the Admin Center.
• You can now access collection results via our API.

Chores

• We refined the rules of our fetcher to index only HTML pages with the USASearch Javascript tag and any PDF or Microsoft Office documents linked from the tagged HTML page.
• We automatically remove the repetitive @ if you include it as part of your Twitter handle on the Social Media page in the Admin Center.
• We automatically resize your mobile header image if it is too big.
• We updated our Linux kernels, Jenkins, New Relic, and lots of RubyGems.

Fixes

• Searchers can now set a custom date range even if type-ahead search is disabled on your results page.
• Searchers no longer see inactive best bets as recommended pages in the type-ahead search.
• Images no longer overlap on image search results pages.
• We now show both queries and clicks in the analytics reports for customers with really long site handles.
• We fixed a bug with how we recorded clicks for searches with apostrophes.
• We show you an error message when you try to upload an improperly encoded URL  via the URLs & Sitemaps page in the Admin Center.

We offer our search results as a hosted results page or as an application programming interface (API). 

When to Use the Hosted Results

Most agencies use our hosted search results page. We recommend that you to use our hosted page for the typical use case of returning search results on your website. Using the hosted page has many benefits such as allowing you to:

• Devote minimal development resources for the initial set up.
• Benefit from new features automatically as they’re rolled out.
• Stay current with search best practices, such as type-ahead search and formatting of search results.

When to Use the API

You may need to access our search results to present them outside of a typical search results page. We offer our results as an API for these use cases.

You can access the API and its documentation at Admin Center > YourSite > Search API. 

The API returns results from each of our indexes—web, image, document (ODIE), RSS (news), video, best bets (boosted content), and related searches—in JSON, JSON-P, or XML.

We recommend you use our API for limited use cases such as:

• Displaying search results within a native mobile application (i.e., an Android or iPhone app).
• Integrating USASearch results within existing applications (such as your intranet or publications catalog).
• Creating data mashups.

Examples of how agencies are currently using the API follow.

Use Case 1. Display Results in Your Mobile App

USA.gov and the White House use the Search API to render USASearch results within their mobile apps. 

Screenshot of a USASearch results page in the USA.gov iPhone app

 
Screenshot of a USASearch results page in the White House iPhone app

Related Links

• USA.gov mobile app
• WhiteHouse mobile app
• White House Github account (information on and source code for the iOS and Android mobile apps)

Use Case 2. Integrate USASearch Results within existing applications

RI.gov manages hundreds of distinct websites. The state provides a consistent search experience across all of its websites. Some sites are centrally controlled by RI.gov. Others are not. Using the Search API, RI.gov programmatically tells USASearch the site’s domain from which the user searched (e.g., tmc.dot.ri.gov). RI.gov then highlights results from this specific domain within RI.gov’s common look and feel.

See the sample results page below that shows a set of results displayed on Rhode Island’s Transportation Management Center for a search on traffic.

  Step 1. Searcher on www.tmc.dot.ri.gov types traffic in the search box

Step 2. Searcher sees results on RI.gov with a common look and feel that highlights results from the original site

Did you know? The Search API is available for government websites only. Other, publicly available APIs (i.e., for use by anyone) are at http://www.usa.gov/About/developer-resources/developers.shtml.

USASearch > Admin Center > YourSite > Search API

Your pre-launch checklist will be unique to your agency’s workflow, requirements, and deadlines. A typical punch list to go live with USASearch follows.

Connect Your Site to USASearch

Most agencies add these two snippets of code to the template(s) in their content management systems (rather than adding them to individual pages).

 1. Does your search box point to USASearch? Update your search box form code point to USASearch so that, when users execute a search, they’ll see your USASearch-hosted results page.

<form accept-charset="UTF-8" action="http://search.usa.gov/search" id="search_form" method="get"><div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="&#x2713;" /></div>
<input id="affiliate" name="affiliate" type="hidden" value="YourSiteHandle" />
<label for="query">Enter Search Term(s):</label>
<input autocomplete="off" class="usagov-search-autocomplete" id="query" name="query" type="text" />
<input name="commit" type="submit" value="Search" />
</form>

 2. Do you have the USASearch Javascript tag on your webpages? This Javascript helps us find your new and updated pages. It also powers the type-ahead search.

<script type="text/javascript"> //<![CDATA[ var usasearch_config = { siteHandle:"YourSiteHandle" }; var script = document.createElement("script"); script.type = "text/javascript"; script.src = "http://search.usa.gov/javascripts/remote.loader.js"; document.getElementsByTagName("head")[0].appendChild(script); //]]>
</script> Double Check Your Content Sources

 3. Have you told us which domain(s) you want to search? List one or more domains that you want to search on the first page of web results.  Most agencies opt to leave out the ‘www’ to include all subdomains (e.g., list usa.gov to include answers.usa.gov, blog.usa.gov, publications.usa.gov, etc). Some agencies—especially department-level portals—opt to also include their bureaus’ websites (e.g., Commerce.gov includes commerce.gov, noaa.gov, trade.gov, etc.)

 4. Have you told us about your RSS feeds? We love RSS feeds. Tell us about all of your feeds, even if you opt not to show them  (see #7 below). They’re the quickest way for you to tell us about your new and updated content. Feeds also have well-structured titles, URLs, descriptions, and dates that we use to improve your search results page. (Tip: Preview your feeds from the RSS page in the Admin Center to confirm that we’ve indexed your content.)

 5. Have you told us about your social media accounts? Flickr photostreams, tweets, and YouTube videos are a great way to highlight your recent content and to let searchers know about your social media channels. Tell us about all of your social media, even if you opt not to show them (see #7 below). 

Customize Your User Interface

 6. Have you made the search results page look like your website? Customize the look and feel (colors, fonts, favicon) and header and footer of your search results page to create a seamless experience for users as they search and browse your website.

 7. Have you told us what to show on your results page? Tell us which results modules (i.e., the stuff in the center of the page) and sidebar options (i.e., the navigation in the left-hand column) you want to appear on your search results.

Set Up Nice-to-Have Features (Optional)

 8. Have you masked your domain?  We offer DNS masking, which allows you to show searchers your search.YOURSITE.gov (instead of search.USA.gov). If you leverage DNS masking you will have to change your search box form action from search.usa.gov to search.YOURSITE.gov. 

 9. Have you added the Javascripts for your other third party web services to your results page? If you have a third party service running on your website, you likely want it to run on your results page. Agencies often include their 4Q, AddThis, Foresee, Google Analytics, Omniture, or WebTrends code, among others.

Don’t Forget SEO (Optional)

 10. Have you optimized your site so we can find your content? We follow search engine optimization (SEO) best practices. Verify that you have a robots.txt and an XML sitemap. These two files are typically located in the root directory of your website (i.e., agency.gov/robots.txt and agency.gov/sitemap.xml, respectively). Also verify that your robots.txt file allows our user agent (usasearch) to access your public content.

 11. Have you registered your site with the major commercial search engines? Register for both Bing Webmaster Tools and Google Webmaster Tools to maximize the coverage of your content in their search results.

Highlighted Feature

Searchers often know the exact page they’re looking for. We’ve added a feature to help them find that page even more quickly. 

As searchers type in their query, relevant text Best Bets are shown as Recommended Pages, as shown in the sample below for a search on fa… on DOL.gov.

Other Features

• Searchers can narrow RSS news results by custom date ranges.
• Searchers now see more images above the fold on image results pages.
• Searchers see integrated image results across both the images on your website and in your Flickr photostream.
• You can now add the Digtital Analytics Program’s government-wide analytics tag to your search results pages. Email the Digital Analytics Program at DAP@gsa.gov for more information on setting up the government-wide analytics tag.
• When you add your Twitter handle in the Admin Center, we now show the Twitter GovBox by default.

Chores

• We upgraded the tweetstream and twitter gems.
• We upgraded Jenkins.
• We upgraded to Cloudera 4.1 and installed Impala.

Fixes

• We fixed the URLs within your tweets so that searchers can now click on the links.
• We fixed a bug so that infinite scroll now works on news and image results pages.
• We now index your ISO-8859-1 encoded pages.
• We now always show your no results section on the Query Logs page in the Admin Center.
• We fixed some bugs with how the ends of the day and month were calculated so that the data displayed in our reports is consistent.
• We fixed a bug so that the query timeline chart now displays in all browsers.
• We synced the Query Logs and Click Stats reports so that you now see today’s data for both.

The Low Click-Thru Rate (CTR) Queries page in the Admin Center reports on the top 10 searches with low click-thru rates. It shows the most popular search terms that people generally don’t act (click) on. 

You’ll see a list of up to 10 terms that returned search results, but resulted in a click less than 20 percent of the time over the past day or so. In other words, fewer than one in five people who searched for that topic clicked on a search result. (Terms must have been searched by at least 10 unique searchers per day to be included in this list.)

Use this report to identify issues with coverage. Create a Best Bet, add pages that may be missing from the results, or do both to improve recall. Incorporate language from these popular search terms into your titles and descriptions to improve the relevance of results. 

Click on any linked word or the  symbol in the list, and you’ll be taken to a Query Timeline chart. There you’ll be able to select a period of time for the term, and you can also add a second query for comparison.

Click on the  symbol, and you’ll be taken to a list of the top clicked URLs for that term.

Did you know? This low click-thru report lists only those terms that returned search results. Visit the Query Logs page to see the top queries that returned no results.

USASearch > Admin Center > YourSite > Low CTR Queries

The Trending Queries page in the Admin Center reports on the top 10 changing search terms. 

You’ll see a list of up to 10 terms with the greatest gain between yesterday and today. (Terms must have been searched by at least 10 unique searchers per day to be included in this list.)

Use this report to identify newly popular terms. Create new content or update existing content to ensure it’s current, accurate, and complete. 

See the sample report below that shows the 10 trending queries on USA.gov on Monday, November 5, 2012—the day before Election Day.

Click on any linked word or the symbol in the above list, and you’ll be taken to a Query Timeline chart. There you’ll be able to select a period of time for the term, and you can also add a second query for comparison.

Click on the symbol, and you’ll be taken to a list of the top clicked URLs for that term.

To investigate why the term may be trending, click on the Twitter, news, or trends icons to see what’s happening on other websites.

USASearch > Admin Center > YourSite > Trending Queries

Highlighted Feature: Common Search Metrics for the Digital Strategy

Do you ever feel like you’re drowning in data? Do you feel like finding meaningful insights from the data is like searching for a needle in a haystack? 

To standardize and streamline how agencies collect and report this data, the Digital Government Strategy requires federal executive branch agencies to implement common performance and customer satisfaction metrics. HowTo.gov has published guidance on common metrics, including describing what you should measure, what to do with the data, and common tools to help you collect and analyze your metrics.

To support you in collecting and reporting your site search metrics for the Digital Government Strategy, we’ve added four new reports to the Admin Center.

1. Top “no results” queries

The most popular, valid searches that don’t return any results on your site’s search results page (e.g., because you don’t have the content on your site, or the content isn’t findable by the person’s search term). Review your content to see if you need to add new content, or update existing content to include the words your readers search for. Read our post, How to Analyze Your Query Logs, for more details.

2. Top searches with low click through rates

The most popular search terms that people generally don’t act (click) on. Consider incorporating language from these popular search terms into page titles or descriptions, to encourage searchers to click on your relevant page. Read our post, How to Analyze Your Queries with Low Click-Thru Rates, for more details.

3. Top changing search terms (movement up/down)

Shows trending topics, what’s hot or not. Use this report to investigate why people might be losing interest in once-popular content and consider archiving if it’s no longer needed. For newly popular terms, create new content or update existing content to ensure it’s current, accurate, and complete. Read our post, How to Analyze Your Trending Queries, for more details.

4. Top clicked URLs for specific queries

The most common results that searchers found most relevant or appealing, for specific search terms. Review these popular pages and other customer top tasks to make sure they are current, accurate, complete and compelling. Read our post, How to Analyze Your Click Stats, for more details.