Downloadable Tools
We are developing tools to support combinatorial testing. No license is required and there are no restrictions on distribution or use. If you would like a copy, just send us an email request. Please include the name of your company or university - this information helps us with maintaining management support for the project. All software is provided free of charge and will remain free in the future. NIST is an agency of the US Government.
Software:- Advanced Combinatorial Testing System (ACTS) - generates test sets that ensure t-way coverage of input parameter values; includes support for constraints and variable-strength tests.
- Combinatorial coverage measurement - computes a number of coverage measures of an existing test set.
- Access Control Policy Test (ACPT) - uses combinatorial testing with model checking to produce tests for access control policies
- .NET Configuration test file generator - provide combinatorial coverage for .NET systems that have a large number of configuration options.
- Combinatorial sequence test generator - generates sequence covering arrays, useful for event driven systems including GUIs, protocols, hardware testing.
- Practical Combinatorial Testing - tutorial-style introduction to combinatorial methods for software testing (pdf)
- ACTS User Guide - how to use the ACTS test generation tool (pdf)
- Combinatorial Coverage Measurement - explains various coverage measurements and how to use the tool for computing these (pdf)
Advanced Combinatorial Testing System (ACTS) can compute tests for 2-way through 6-way interactions. An easy-to-use GUI is included, in addition to a command line version suitable for use in scripts or system calls from another tool. A comparison of ACTS with similar tools shows that ACTS produces smaller test sets (with the same degree of coverage) and is faster than others. ACTS is Java-based, so it will run on any Java platform.
For a detailed comparison of ACTS with other combinatorial testing tools, see spreadsheet and graphs here.
This software is provided free of charge and will remain free in the future. NIST is an agency of the US Government. Please Note: Prior to Feb 09, the name "FireEye" was used for the ACTS tool. Some papers and articles on the site refer to FireEye; this is the same tool now called ACTS.
User guide for combinatorial testing tool here.
Who uses our combinatorial testing tool? We have
approximately
200 users as of July 2009, in IT, defense, finance, telecom, and many
other industries. Here
is a breakdown of our user base.
User
Feedback
Users
have been very positive, and are applying ACTS to a wide variety of
software.
-
"The tool is pretty easy to use, and has already reduced our planned number of tests by 20% or so." -
"At present I am working with a development team to enable automated testing of a complex
rating algorithm under development within a property/casualty insurance system. The system
component under test accepts several hundred independent input variables which are used
to produce rating factors as output. I have had some opportunity to use [NIST-ACTS] as a
means to calculate test coverage and identify missing test cases in our current population of
test data. I'm pleased to report that the combinatorial coverage data has proven useful to us
in identifying gaps. Eventually, I would like to use this data to extend our test coverage as
well -- at coverage strength 3 we're producing a volume of test cases that should be
manageable when paired with automated test data generation. I can also foresee other uses
which would pair automated testing tools with [NIST-ACTS] output to improve our test
coverage for other systems." -
"Our feedback on [NIST-ACTS] is extremely positive. The tool implement in a very efficient way
the algorithm of N-Wise reduction we were looking at. In fact, we would be interested in
including your technology in our tool [ ]." -
"[NIST-ACTS] is a good application. It was easy to figure out and use. ... I would use this
application again." -
""I've fund it pretty intuitive and greatly helped to reduce the test size. I'll continue to use this tool
at next opportunity." -
" Not only does [NIST-ACTS] handle phenomenal combinatorial complexity without breaking a sweat,
it is also very easy and straightforward to use. The user interface is very well thought out,
intuitive, and delightfully uncluttered. "
Combinatorial coverage measurement: a tool to measure combinatorial coverage of an existing test suite. This is currently an alpha release, but will eventually be open source.
To request a copy, send email to Rick Kuhn - kuhn@nist.gov.
User guide for coverage measurement tool here.
Access Control Policy Test (ACPT) tool allows policy authors to conveniently specify access control models (such as RBAC and Multi-Level models) and rules as well as access control properties. From the specified models and rules, the ACPT tool automatically synthesizes deployable policies in XACML and generates combinatorial tests to verify security policy implementations. Complete test cases are generated, consisting of test inputs and expected output for each set of inputs. ACPT uses ACTS to provide 2-way to 4-way combinatorial testing of policies. To request a copy, send email to Vincent Hu - vhu@nist.gov.
.NET Configuration test file generator produces a collection of .NET config files from ACTS output. This tool can be used to provide combinatorial coverage for systems that have a large number of configuration options.
To request a copy, send email to Rick Kuhn - kuhn@nist.gov.
Sequence covering array generator can produce arrays for more efficient testing of event-driven systems. See chapter 5 of the Practical Combinatorial Testing tutorial, here, for an explanation of sequence covering arrays. To request a copy, send email to Rick Kuhn - kuhn@nist.gov.
