|
ActiveTcl User Guide
|
|
|
[ Main table Of Contents | Tcllib Table Of Contents | Tcllib Index ]
math::statistics(n) 0.1.1 "Math"
math::statistics - Basic statistical functions and
procedures
TABLE OF
CONTENTS
SYNOPSIS
DESCRIPTION
GENERAL
PROCEDURES
STATISTICAL DISTRIBUTIONS
DATA
MANIPULATION
PLOT
PROCEDURES
THINGS TO DO
EXAMPLES
KEYWORDS
package require Tcl 8
package require math::statistics 0.1.1
The math::statistics package contains functions
and procedures for basic statistical data analysis, such as:
- Descriptive statistical parameters (mean, minimum, maximum,
standard deviation)
- Estimates of the distribution in the form of histograms and
quantiles
- Basic testing of hypotheses
- Probability and cumulative density functions
It is meant to help in developing data analysis applications or
doing ad hoc data analysis, it is not in itself a full application,
nor is it intended to rival with full (non-)commercial statistical
packages.
The purpose of this document is to describe the implemented
procedures and provide some examples of their usage. As there is
ample literature on the algorithms involved, we refer to relevant
text books for more explanations. The package contains a fairly
large number of public procedures. They can be distinguished in
three sets: general procedures, procedures that deal with specific
statistical distributions, list procedures to select or transform
data and simple plotting procedures (these require Tk).
Note: The data that need to be analyzed are always
contained in a simple list. Missing values are represented as empty
list elements.
The general statistical procedures are:
- ::math::statistics::mean data
- Determine the mean value of the given list of
data.
data - List of data
- ::math::statistics::min data
- Determine the minimum value of the given list of
data.
data - List of data
- ::math::statistics::max data
- Determine the maximum value of the given list of
data.
data - List of data
- ::math::statistics::number data
- Determine the number of non-missing data in the given
list
data - List of data
- ::math::statistics::stdev data
- Determine the standard deviation of the data in the
given list
data - List of data
- ::math::statistics::var data
- Determine the variance of the data in the given
list
data - List of data
- ::math::statistics::median data
- Determine the median of the data in the given list
(Note that this requires sorting the data, which may be a costly
operation)
data - List of data
- ::math::statistics::basic-stats
data
- Determine a list of all the descriptive parameters: mean,
minimum, maximum, number of data, standard deviation and
variance.
(This routine is called whenever either or all of the basic
statistical parameters are required. Hence all calculations are
done and the relevant values are returned.)
data - List of data
- ::math::statistics::histogram limits values
- Determine histogram information for the given list of data.
Returns a list consisting of the number of values that fall into
each interval. (The first interval consists of all values lower
than the first limit, the last interval consists of all values
greater than the last limit. There is one more interval than there
are limits.)
limits - List of upper limits (in ascending
order) for the intervals of the histogram.
values - List of data
- ::math::statistics::corr data1 data2
- Determine the correlation coefficient between two sets of
data.
data1 - First list of data
data2 - Second list of data
- ::math::statistics::interval-mean-stdev data confidence
- Return the interval containing the mean value and one
containing the standard deviation with a certain level of
confidence (assuming a normal distribution)
data - List of raw data values (small
sample)
confidence - Confidence level (0.95 or 0.99 for
instance)
- ::math::statistics::t-test-mean
data est_mean est_stdev confidence
- Test whether the mean value of a sample is in accordance with
the estimated normal distribution with a certain level of
confidence. Returns 1 if the test succeeds or 0 if the mean is
unlikely to fit the given distribution.
data - List of raw data values (small
sample)
est_mean - Estimated mean of the
distribution
est_stdev - Estimated stdev of the
distribution
confidence - Confidence level (0.95 or 0.99 for
instance)
- ::math::statistics::quantiles
data confidence
- Return the quantiles for a given set of data
data - List of raw data values
confidence - Confidence level (0.95 or 0.99 for
instance)
- ::math::statistics::quantiles
limits counts confidence
- Return the quantiles based on histogram information
(alternative to the call with two arguments)
limits - List of upper limits from histogram
counts - List of counts for for each interval in
histogram
confidence - Confidence level (0.95 or 0.99 for
instance)
- ::math::statistics::autocorr data
- Return the autocorrelation function as a list of values
(assuming equidistance between samples, about 1/2 of the number of
raw data)
The correlation is determined in such a way that the first value is
always 1 and all others are equal to or smaller than 1. The number
of values involved will diminish as the "time" (the index in the
list of returned values) increases
data - Raw data for which the autocorrelation
must be determined
- ::math::statistics::crosscorr
data1 data2
- Return the cross-correlation function as a list of values
(assuming equidistance between samples, about 1/2 of the number of
raw data)
The correlation is determined in such a way that the values can
never exceed 1 in magnitude. The number of values involved will
diminish as the "time" (the index in the list of returned values)
increases.
data1 - First list of data
data2 - Second list of data
- ::math::statistics::mean-histogram-limits mean stdev number
- Determine reasonable limits based on mean and standard
deviation for a histogram
Convenience function - the result is suitable for the histogram
function.
mean - Mean of the data
stdev - Standard deviation
number - Number of limits to generate (defaults
to 8)
- ::math::statistics::minmax-histogram-limits min max number
- Determine reasonable limits based on a minimum and maximum for
a histogram
Convenience function - the result is suitable for the histogram
function.
min - Expected minimum
max - Expected maximum
number - Number of limits to generate (defaults
to 8)
- ::math::statistics::linear-model xdata
ydata intercept
- Determine the coefficients for a linear regression between two
series of data (the model: Y = A + B*X). Returns a list of
parameters describing the fit
xdata - List of independent data
ydata - List of dependent data to be fitted
intercept - (Optional) compute the intercept (1,
default) or fit to a line through the origin (0)
The result consists of the following list:
- (Estimate of) Intercept A
- (Estimate of) Slope B
- Standard deviation of Y relative to fit
- Correlation coefficient R2
- Number of degrees of freedom df
- Standard error of the intercept A
- Significance level of A
- Standard error of the slope B
- Significance level of B
- ::math::statistics::linear-residuals xdata ydata intercept
- Determine the difference between actual data and predicted from
the linear model.
Returns a list of the differences between the actual data and the
predicted values.
xdata - List of independent data
ydata - List of dependent data to be fitted
intercept - (Optional) compute the intercept (1,
default) or fit to a line through the origin (0)
In the literature a large number of probability distributions
can be found. The statistics package supports:
- The normal or Gaussian distribution
- The uniform distribution - equal probability for all data
within a given interval
- The exponential distribution - useful as a model for certain
extreme-value distributions.
- PM - binomial, Poisson, chi-squared, student's T, F.
In principle for each distribution one has procedures for:
- The probability density (pdf-*)
- The cumulative density (cdf-*)
- Quantiles for the given distribution (quantiles-*)
- Histograms for the given distribution (histogram-*)
- List of random values with the given distribution
(random-*)
The following procedures have been implemented:
- ::math::statistics::pdf-normal
mean stdev value
- Return the probability of a given value for a normal
distribution with given mean and standard deviation.
mean - Mean value of the distribution
stdev - Standard deviation of the
distribution
value - Value for which the probability is
required
- ::math::statistics::pdf-exponential mean value
- Return the probability of a given value for an exponential
distribution with given mean.
mean - Mean value of the distribution
value - Value for which the probability is
required
- ::math::statistics::pdf-uniform
xmin xmax value
- Return the probability of a given value for a uniform
distribution with given extremes.
xmin - Minimum value of the distribution
xmin - Maximum value of the distribution
value - Value for which the probability is
required
- ::math::statistics::cdf-normal
mean stdev value
- Return the cumulative probability of a given value for a normal
distribution with given mean and standard deviation, that is the
probability for values up to the given one.
mean - Mean value of the distribution
stdev - Standard deviation of the
distribution
value - Value for which the probability is
required
- ::math::statistics::cdf-exponential mean value
- Return the cumulative probability of a given value for an
exponential distribution with given mean.
mean - Mean value of the distribution
value - Value for which the probability is
required
- ::math::statistics::cdf-uniform
xmin xmax value
- Return the cumulative probability of a given value for a
uniform distribution with given extremes.
xmin - Minimum value of the distribution
xmin - Maximum value of the distribution
value - Value for which the probability is
required
- ::math::statistics::cdf-students-t degrees value
- Return the cumulative probability of a given value for a
Student's t distribution with given number of degrees.
degrees - Number of degrees of freedom
value - Value for which the probability is
required
- ::math::statistics::random-normal mean
stdev number
- Return a list of "number" random values satisfying a normal
distribution with given mean and standard deviation.
mean - Mean value of the distribution
stdev - Standard deviation of the
distribution
number - Number of values to be returned
- ::math::statistics::random-exponential mean number
- Return a list of "number" random values satisfying an
exponential distribution with given mean.
mean - Mean value of the distribution
number - Number of values to be returned
- ::math::statistics::random-uniform xmin xmax value
- Return a list of "number" random values satisfying a uniform
distribution with given extremes.
xmin - Minimum value of the distribution
xmin - Maximum value of the distribution
number - Number of values to be returned
- ::math::statistics::histogram-uniform xmin xmax limits number
- Return the expected histogram for a uniform distribution.
xmin - Minimum value of the distribution
xmax - Maximum value of the distribution
limits - Upper limits for the buckets in the
histogram
number - Total number of "observations" in the
histogram
TO DO: more function descriptions to be added
The data manipulation procedures act on lists or lists of
lists:
- ::math::statistics::filter varname data expression
- Return a list consisting of the data for which the logical
expression is true (this command works analogously to the command
foreach).
varname - Name of the variable used in the
expression
data - List of data
expression - Logical expression using the
variable name
- ::math::statistics::map varname data expression
- Return a list consisting of the data that are transformed via
the expression.
varname - Name of the variable used in the
expression
data - List of data
expression - Expression to be used to transform
(map) the data
- ::math::statistics::samplescount varname list expression
- Return a list consisting of the counts of all data in
the sublists of the "list" argument for which the expression is
true.
varname - Name of the variable used in the
expression
data - List of sublists, each containing the
data
expression - Logical expression to test the data
(defaults to "true").
- ::math::statistics::subdivide
- Routine PM - not implemented yet
The following simple plotting procedures are available:
- ::math::statistics::plot-scale
canvas xmin xmax ymin ymax
- Set the scale for a plot in the given canvas. All plot routines
expect this function to be called first. There is no automatic
scaling provided.
canvas - Canvas widget to use
xmin - Minimum x value
xmax - Maximum x value
ymin - Minimum y value
ymax - Maximum y value
- ::math::statistics::plot-xydata
canvas xdata ydata tag
- Create a simple XY plot in the given canvas - the data are
shown as a collection of dots. The tag can be used to manipulate
the appearance.
canvas - Canvas widget to use
xdata - Series of independent data
ydata - Series of dependent data
tag - Tag to give to the plotted data (defaults
to xyplot)
- ::math::statistics::plot-xyline
canvas xdata ydata tag
- Create a simple XY plot in the given canvas - the data are
shown as a line through the data points. The tag can be used to
manipulate the appearance.
canvas - Canvas widget to use
xdata - Series of independent data
ydata - Series of dependent data
tag - Tag to give to the plotted data (defaults
to xyplot)
- ::math::statistics::plot-tdata
canvas tdata tag
- Create a simple XY plot in the given canvas - the data are
shown as a collection of dots. The horizontal coordinate is equal
to the index. The tag can be used to manipulate the appearance.
This type of presentation is suitable for autocorrelation functions
for instance or for inspecting the time-dependent behaviour.
canvas - Canvas widget to use
tdata - Series of dependent data
tag - Tag to give to the plotted data (defaults
to xyplot)
- ::math::statistics::plot-tline
canvas tdata tag
- Create a simple XY plot in the given canvas - the data are
shown as a line. See plot-tdata for an explanation.
canvas - Canvas widget to use
tdata - Series of dependent data
tag - Tag to give to the plotted data (defaults
to xyplot)
- ::math::statistics::plot-histogram canvas counts limits tag
- Create a simple histogram in the given canvas
canvas - Canvas widget to use
counts - Series of bucket counts
limits - Series of upper limits for the
buckets
tag - Tag to give to the plotted data (defaults
to xyplot)
The following procedures are yet to be implemented:
- F-test-stdev
- interval-mean-stdev
- histogram-normal
- histogram-exponential
- test-histogram
- test-corr
- quantiles-*
- fourier-coeffs
- fourier-residuals
- onepar-function-fit
- onepar-function-residuals
- plot-linear-model
- subdivide
The code below is a small example of how you can examine a set
of data:
|
# Simple example:
# - Generate data (as a cheap way of getting some)
# - Perform statistical analysis to describe the data
#
package require math::statistics
#
# Two auxiliary procs
#
proc pause {time} {
set wait 0
after [expr {$time*1000}] {set ::wait 1}
vwait wait
}
proc print-histogram {counts limits} {
foreach count $counts limit $limits {
if { $limit != {} } {
puts [format "<%12.4g\t%d" $limit $count]
set prev_limit $limit
} else {
puts [format ">%12.4g\t%d" $prev_limit $count]
}
}
}
#
# Our source of arbitrary data
#
proc generateData { data1 data2 } {
upvar 1 $data1 _data1
upvar 1 $data2 _data2
set d1 0.0
set d2 0.0
for { set i 0 } { $i < 100 } { incr i } {
set d1 [expr {10.0-2.0*cos(2.0*3.1415926*$i/24.0)+3.5*rand()}]
set d2 [expr {0.7*$d2+0.3*$d1+0.7*rand()}]
lappend _data1 $d1
lappend _data2 $d2
}
return {}
}
#
# The analysis session
#
package require Tk
console show
canvas .plot1
canvas .plot2
pack .plot1 .plot2 -fill both -side top
generateData data1 data2
puts "Basic statistics:"
set b1 [::math::statistics::basic-stats $data1]
set b2 [::math::statistics::basic-stats $data2]
foreach label {mean min max number stdev var} v1 $b1 v2 $b2 {
puts "$label\t$v1\t$v2"
}
puts "Plot the data as function of \"time\" and against each other"
::math::statistics::plot-scale .plot1 0 100 0 20
::math::statistics::plot-scale .plot2 0 20 0 20
::math::statistics::plot-tline .plot1 $data1
::math::statistics::plot-tline .plot1 $data2
::math::statistics::plot-xydata .plot2 $data1 $data2
puts "Correlation coefficient:"
puts [::math::statistics::corr $data1 $data2]
pause 2
puts "Plot histograms"
.plot2 delete all
::math::statistics::plot-scale .plot2 0 20 0 100
set limits [::math::statistics::minmax-histogram-limits 7 16]
set histogram_data [::math::statistics::histogram $limits $data1]
::math::statistics::plot-histogram .plot2 $histogram_data $limits
puts "First series:"
print-histogram $histogram_data $limits
pause 2
set limits [::math::statistics::minmax-histogram-limits 0 15 10]
set histogram_data [::math::statistics::histogram $limits $data2]
::math::statistics::plot-histogram .plot2 $histogram_data $limits d2
.plot2 itemconfigure d2 -fill red
puts "Second series:"
print-histogram $histogram_data $limits
puts "Autocorrelation function:"
set autoc [::math::statistics::autocorr $data1]
puts [::math::statistics::map $autoc {[format "%.2f" $x]}]
puts "Cross-correlation function:"
set crossc [::math::statistics::crosscorr $data1 $data2]
puts [::math::statistics::map $crossc {[format "%.2f" $x]}]
::math::statistics::plot-scale .plot1 0 100 -1 4
::math::statistics::plot-tline .plot1 $autoc "autoc"
::math::statistics::plot-tline .plot1 $crossc "crossc"
.plot1 itemconfigure autoc -fill green
.plot1 itemconfigure crossc -fill yellow
puts "Quantiles: 0.1, 0.2, 0.5, 0.8, 0.9"
puts "First: [::math::statistics::quantiles $data1 {0.1 0.2 0.5 0.8 0.9}]"
puts "Second: [::math::statistics::quantiles $data2 {0.1 0.2 0.5 0.8 0.9}]"
|
If you run this example, then the following should be clear:
- There is a strong correlation between two time series, as
displayed by the raw data and especially by the correlation
functions.
- Both time series show a significant periodic component
- The histograms are not very useful in identifying the nature of
the time series - they do not show the periodic nature.
data analysis , mathematics , statistics