> William Stein wrote:

>> On Thu, Dec 31, 2009 at 9:13 PM, Tim Daly <daly@axiom-developer.org> wrote:

>> The output is used both for human use and for regression testing. Its

>> primary use is human -- it's an example in the Sage reference manual:

>>

>> sage: float(e)

>> 2.7182818284590451

>>

>> This is something a user will look at when reading the documentation

>> for some function. It illustrates what happens when they convert the

>> symbolic constant e to float.

>>

>> William

>>

>>

> And therein lies the problem. We use a regression that does a comparison

> of the

> printed representation of the output of the run with a stored copy of

> the output.

>

> All of our regression tests were passing until I installed another,

> unrelated program.

> Suddenly about 30 regression tests started failing. It turns out that

> the unrelated

> program upgraded one of the system libraries. The net effect of that

> change was

> to cause the last digit in the output to "wobble" so that some of the

> table values

> differ in the nth place (20th, 30th, or thereabouts digit). This caused

> the regression

> comparisons to fail.

>

> Common lisp will give you the exact bit pattern of the float and this value

> does not wobble so the text comparison succeeds with both the old and

> the new libraries against the bit pattern.

>

> So I can tell you from experience that what you would like to do is not

> going to succeed.

What I would like to do does succeed. The goal of the doctest suite

in Sage is:

(1) to provide examples for users that illustrate every function in Sage,

(2) assure that these examples run as claimed, in the sense that a

given input pasted into the Sage command line, results in the claimed

output (with certain well-defined assumptions about state).

You can view (1) as documentation for users, and view (2) as one form

of "regression testing". There are many types of regression tests.

Regarding your example, I would consider it a bug if the specific

examples of Sage's floating point output -- as claimed in the examples

in the official documentation -- are wrong in the presence of certain

standard system-wide shared libraries. That is to say, if the Sage

documentation says:

sage: foo(bar)

1.2345

but in fact in Sage one has

sage: foo(bar)

1.2351

then I would consider this misleading documentation, i.e., a bug.

Your statement above suggests that you *don't* consider this a bug at

all. In Sage, one possible solution (and it depends on context

whether this it the right solution or not) would be to change the

documentation to

sage: foo(bar) # last few digits numerically unstable

1.23...

The "..." in Python's doctest framework is a wildcard. The Sage

code hasn't been changed, but the documentation has been. The point

is that a read of the documentation for the function foo will see

explicitly -- right there in the documentation -- that there are

numerical noise issues on certain platforms. We have followed

exactly this approach in all of the Sage example docstests, which now

total over 110,000 lines of input, testing over 19,000 functions (or

80.9% of the functions in Sage).

-- William

>

> Our solution to the human vs regression problem is to include the stable

> bit values in the actual compare and keep the human values in a latex

> table. This is easy to do with literate input.

>

> Tim

>

>

>

>

>

>

>

> --

> To post to this group, send an email to sage-devel@googlegroups.com

> To unsubscribe from this group, send an email to sage-devel+unsubscribe@googlegroups.com

> For more options, visit this group at http://groups.google.com/group/sage-devel

> URL: http://www.sagemath.org

>

--

William Stein

Associate Professor of Mathematics

University of Washington

http://wstein.org

--

To post to this group, send an email to sage-devel@googlegroups.com

To unsubscribe from this group, send an email to sage-devel+unsubscribe@googlegroups.com

For more options, visit this group at http://groups.google.com/group/sage-devel

URL: http://www.sagemath.org