public class PairedTTester extends Object implements OptionHandler, Tester, RevisionHandler
-D <index,index2-index4,...> Specify list of columns that specify a unique dataset. First and last are valid indexes. (default none)
-R <index> Set the index of the column containing the run number
-F <index> Set the index of the column containing the fold number
-G <index1,index2-index4,...> Specify list of columns that specify a unique 'result generator' (eg: classifier name and options). First and last are valid indexes. (default none)
-S <significance level> Set the significance level for comparisons (default 0.05)
-V Show standard deviations
-L Produce table comparisons in Latex table format
-csv Produce table comparisons in CSV table format
-html Produce table comparisons in HTML table format
-significance Produce table comparisons with only the significance values
-gnuplot Produce table comparisons output suitable for GNUPlot
Modifier and Type | Class and Description |
---|---|
protected class |
PairedTTester.Dataset
Utility class to store the instances pertaining to a dataset
|
protected class |
PairedTTester.DatasetSpecifiers
A list of unique "dataset" specifiers that have been observed
|
protected class |
PairedTTester.Resultset
Utility class to store the instances in a resultset
|
Modifier and Type | Field and Description |
---|---|
protected int[] |
m_ColOrder
The sorting of the columns (test base is always first)
|
protected int[] |
m_DatasetKeyColumns
An array containing the indexes of just the selected columns
|
protected Range |
m_DatasetKeyColumnsRange
The range of columns that specify a unique "dataset"
(eg: scheme plus configuration)
|
protected PairedTTester.DatasetSpecifiers |
m_DatasetSpecifiers
The list of dataset specifiers
|
protected int[] |
m_DisplayedResultsets
An array containing the indexes of the datasets to display
|
protected int |
m_FoldColumn
The option setting for the fold number column (-1 means none)
|
protected Instances |
m_Instances
The set of instances we will analyse
|
protected ResultMatrix |
m_ResultMatrix
the instance of the class to produce the output.
|
protected int[] |
m_ResultsetKeyColumns
An array containing the indexes of just the selected columns
|
protected Range |
m_ResultsetKeyColumnsRange
The range of columns that specify a unique result set
(eg: scheme plus configuration)
|
protected FastVector |
m_Resultsets
Stores a vector for each resultset holding all instances in each set
|
protected boolean |
m_ResultsetsValid
Indicates whether the instances have been partitioned
|
protected int |
m_RunColumn
The index of the column containing the run number
|
protected int |
m_RunColumnSet
The option setting for the run number column (-1 means last)
|
protected boolean |
m_ShowStdDevs
Indicates whether standard deviations should be displayed
|
protected double |
m_SignificanceLevel
The significance level for comparisons
|
protected int |
m_SortColumn
The column to sort on (-1 means default sorting)
|
protected int[] |
m_SortOrder
The sorting of the datasets (according to the sort column)
|
Constructor and Description |
---|
PairedTTester() |
Modifier and Type | Method and Description |
---|---|
void |
assign(Tester tester)
retrieves all the settings from the given Tester
|
PairedStats |
calculateStatistics(Instance datasetSpecifier,
int resultset1Index,
int resultset2Index,
int comparisonColumn)
Computes a paired t-test comparison for a specified dataset between
two resultsets.
|
boolean |
displayResultset(int index)
Checks whether the resultset with the given index shall be displayed.
|
Range |
getDatasetKeyColumns()
Get the value of DatasetKeyColumns.
|
int[] |
getDisplayedResultsets()
Gets the indices of the the datasets that are displayed (if
null
then all are displayed). |
String |
getDisplayName()
returns the name of the tester
|
int |
getFoldColumn()
Get the value of FoldColumn.
|
Instances |
getInstances()
Get the value of Instances.
|
int |
getNumDatasets()
Gets the number of datasets in the resultsets
|
int |
getNumResultsets()
Gets the number of resultsets in the data.
|
String[] |
getOptions()
Gets current settings of the PairedTTester.
|
ResultMatrix |
getResultMatrix()
Gets the instance that produces the output.
|
Range |
getResultsetKeyColumns()
Get the value of ResultsetKeyColumns.
|
String |
getResultsetName(int index)
Gets a string descriptive of the specified resultset.
|
String |
getRevision()
Returns the revision string.
|
int |
getRunColumn()
Get the value of RunColumn.
|
boolean |
getShowStdDevs()
Returns true if standard deviations have been requested.
|
double |
getSignificanceLevel()
Get the value of SignificanceLevel.
|
int |
getSortColumn()
Returns the column to sort on, -1 means the default sorting.
|
String |
getSortColumnName()
Returns the name of the column to sort on.
|
String |
getToolTipText()
returns a string that is displayed as tooltip on the "perform test"
button in the experimenter
|
String |
header(int comparisonColumn)
Creates a "header" string describing the current resultsets.
|
protected void |
initResultMatrix()
clears the content and fills the column and row names according to the
given sorting
|
Enumeration |
listOptions()
Lists options understood by this object.
|
static void |
main(String[] args)
Test the class from the command line.
|
String |
multiResultsetFull(int baseResultset,
int comparisonColumn)
Creates a comparison table where a base resultset is compared to the
other resultsets.
|
String |
multiResultsetRanking(int comparisonColumn)
returns a ranking of the resultsets
|
String |
multiResultsetSummary(int comparisonColumn)
Carries out a comparison between all resultsets, counting the number
of datsets where one resultset outperforms the other.
|
int[][] |
multiResultsetWins(int comparisonColumn,
int[][] nonSigWin)
Carries out a comparison between all resultsets, counting the number
of datsets where one resultset outperforms the other.
|
protected void |
prepareData()
Separates the instances into resultsets and by dataset/run.
|
String |
resultsetKey()
Creates a key that maps resultset numbers to their descriptions.
|
void |
setDatasetKeyColumns(Range newDatasetKeyColumns)
Set the value of DatasetKeyColumns.
|
void |
setDisplayedResultsets(int[] cols)
Sets the indicies of the datasets to display (
null means all). |
void |
setFoldColumn(int newFoldColumn)
Set the value of FoldColumn.
|
void |
setInstances(Instances newInstances)
Set the value of Instances.
|
void |
setOptions(String[] options)
Parses a given list of options.
|
void |
setResultMatrix(ResultMatrix matrix)
Sets the matrix to use to produce the output.
|
void |
setResultsetKeyColumns(Range newResultsetKeyColumns)
Set the value of ResultsetKeyColumns.
|
void |
setRunColumn(int newRunColumn)
Set the value of RunColumn.
|
void |
setShowStdDevs(boolean s)
Set whether standard deviations are displayed or not.
|
void |
setSignificanceLevel(double newSignificanceLevel)
Set the value of SignificanceLevel.
|
void |
setSortColumn(int newSortColumn)
Set the column to sort on, -1 means the default sorting.
|
protected String |
templateString(Instance template)
Returns a string descriptive of the key column values for
the "datasets
|
protected Instances m_Instances
protected int m_RunColumn
protected int m_RunColumnSet
protected int m_FoldColumn
protected int m_SortColumn
protected int[] m_SortOrder
protected int[] m_ColOrder
protected double m_SignificanceLevel
protected Range m_DatasetKeyColumnsRange
protected int[] m_DatasetKeyColumns
protected PairedTTester.DatasetSpecifiers m_DatasetSpecifiers
protected Range m_ResultsetKeyColumnsRange
protected int[] m_ResultsetKeyColumns
protected int[] m_DisplayedResultsets
protected FastVector m_Resultsets
protected boolean m_ResultsetsValid
protected boolean m_ShowStdDevs
protected ResultMatrix m_ResultMatrix
protected String templateString(Instance template)
template
- the templatepublic void setResultMatrix(ResultMatrix matrix)
setResultMatrix
in interface Tester
matrix
- the instance to use to produce the outputResultMatrix
public ResultMatrix getResultMatrix()
getResultMatrix
in interface Tester
public void setShowStdDevs(boolean s)
setShowStdDevs
in interface Tester
s
- true if standard deviations are to be displayedpublic boolean getShowStdDevs()
getShowStdDevs
in interface Tester
protected void prepareData() throws Exception
Exception
- if the TTest parameters have not been set.public int getNumDatasets()
getNumDatasets
in interface Tester
public int getNumResultsets()
getNumResultsets
in interface Tester
public String getResultsetName(int index)
getResultsetName
in interface Tester
index
- the index of the resultsetpublic boolean displayResultset(int index)
displayResultset
in interface Tester
index
- the index of the resultset to check whether it shall be displayedpublic PairedStats calculateStatistics(Instance datasetSpecifier, int resultset1Index, int resultset2Index, int comparisonColumn) throws Exception
calculateStatistics
in interface Tester
datasetSpecifier
- the dataset specifierresultset1Index
- the index of the first resultsetresultset2Index
- the index of the second resultsetcomparisonColumn
- the column containing values to compareException
- if an error occurspublic String resultsetKey()
resultsetKey
in interface Tester
public String header(int comparisonColumn)
public int[][] multiResultsetWins(int comparisonColumn, int[][] nonSigWin) throws Exception
multiResultsetWins
in interface Tester
comparisonColumn
- the index of the comparison columnnonSigWin
- for storing the non-significant winsException
- if an error occursprotected void initResultMatrix()
public String multiResultsetSummary(int comparisonColumn) throws Exception
multiResultsetSummary
in interface Tester
comparisonColumn
- the index of the comparison columnException
- if an error occurspublic String multiResultsetRanking(int comparisonColumn) throws Exception
multiResultsetRanking
in interface Tester
comparisonColumn
- the column to compare withException
- if something goes wrongpublic String multiResultsetFull(int baseResultset, int comparisonColumn) throws Exception
multiResultsetFull
in interface Tester
baseResultset
- the index of the base resultsetcomparisonColumn
- the index of the column to compare overException
- if an error occurspublic Enumeration listOptions()
listOptions
in interface OptionHandler
public void setOptions(String[] options) throws Exception
-D <index,index2-index4,...> Specify list of columns that specify a unique dataset. First and last are valid indexes. (default none)
-R <index> Set the index of the column containing the run number
-F <index> Set the index of the column containing the fold number
-G <index1,index2-index4,...> Specify list of columns that specify a unique 'result generator' (eg: classifier name and options). First and last are valid indexes. (default none)
-S <significance level> Set the significance level for comparisons (default 0.05)
-V Show standard deviations
-L Produce table comparisons in Latex table format
-csv Produce table comparisons in CSV table format
-html Produce table comparisons in HTML table format
-significance Produce table comparisons with only the significance values
-gnuplot Produce table comparisons output suitable for GNUPlot
setOptions
in interface OptionHandler
options
- an array containing options to set.Exception
- if invalid options are givenpublic String[] getOptions()
getOptions
in interface OptionHandler
public Range getResultsetKeyColumns()
getResultsetKeyColumns
in interface Tester
public void setResultsetKeyColumns(Range newResultsetKeyColumns)
setResultsetKeyColumns
in interface Tester
newResultsetKeyColumns
- Value to assign to ResultsetKeyColumns.public int[] getDisplayedResultsets()
null
then all are displayed). The base is always displayed.getDisplayedResultsets
in interface Tester
public void setDisplayedResultsets(int[] cols)
null
means all).
The base is always displayed.setDisplayedResultsets
in interface Tester
cols
- the indices of the datasets to displaypublic double getSignificanceLevel()
getSignificanceLevel
in interface Tester
public void setSignificanceLevel(double newSignificanceLevel)
setSignificanceLevel
in interface Tester
newSignificanceLevel
- Value to assign to SignificanceLevel.public Range getDatasetKeyColumns()
getDatasetKeyColumns
in interface Tester
public void setDatasetKeyColumns(Range newDatasetKeyColumns)
setDatasetKeyColumns
in interface Tester
newDatasetKeyColumns
- Value to assign to DatasetKeyColumns.public int getRunColumn()
getRunColumn
in interface Tester
public void setRunColumn(int newRunColumn)
setRunColumn
in interface Tester
newRunColumn
- Value to assign to RunColumn.public int getFoldColumn()
getFoldColumn
in interface Tester
public void setFoldColumn(int newFoldColumn)
setFoldColumn
in interface Tester
newFoldColumn
- Value to assign to FoldColumn.public String getSortColumnName()
getSortColumnName
in interface Tester
public int getSortColumn()
getSortColumn
in interface Tester
public void setSortColumn(int newSortColumn)
setSortColumn
in interface Tester
newSortColumn
- the new sort column.public Instances getInstances()
getInstances
in interface Tester
public void setInstances(Instances newInstances)
setInstances
in interface Tester
newInstances
- Value to assign to Instances.public void assign(Tester tester)
public String getToolTipText()
getToolTipText
in interface Tester
public String getDisplayName()
getDisplayName
in interface Tester
public String getRevision()
getRevision
in interface RevisionHandler
public static void main(String[] args)
args
- contains options for the instance ttestsCopyright © 2015 University of Waikato, Hamilton, NZ. All rights reserved.