Home / Learn / 14 Testing

Release v0.4.0Edition 0.2

14.4 What the verifier proves, and what examples show

Page 5 of 10

14.4.1 Two kinds of evidence

A contract (chapter 5) and an example both state what a function must do, and Una checks both. A contract or an example, in section 5.2, compared them in a table. This section shows the difference on real bugs:

Neither is weaker. A contract covers inputs no one thought to try. An example states the exact answer, which a contract that gives a range does not.

14.4.2 Why one kind is not enough

Take the store's shipping fee: 4.90, and free from 50.00 on. The rule comes from the store's terms, so the step records it as a requirement and ties the contract to it with derived from, as section 5.5 teaches:

steps/23-contracts/src/shipping.unacheckruntestuna
module shipping

requirement SHIPPING_1
  text "Orders of 50.00 or more ship for free"
  source policy "Store terms, section 4"
  owner checkout
  status active

--- The shipping fee of an order, in cents: 4.90, and nothing from 50.00 on.
pub shipping_fee(subtotal: Int) -> Int
  ensures result >= 0 and result <= 490
  ensures subtotal >= 5000 implies result == 0
  derived from SHIPPING_1
=
  if subtotal >= 5000 then 0 else 490

pub main() -> Unit
  uses @console: Console[print]
=
  @console.print("49.99 pays {shipping_fee(4999)}, 50.00 pays {shipping_fee(5000)}")

example "an order of 49.99 pays 4.90"
  do shipping_fee(4999)
  expect result == 490

example "an order of exactly 50.00 ships for free"
  do shipping_fee(5000)
  expect result == 0

Now compare two bugs.

14.4.2.1 A bug the verifier catches: off by one

In the first, the threshold is off by one, > instead of >=:

src/shipping.unadiff
--- a/src/shipping.una
+++ b/src/shipping.una
@@ -15 +15 @@
-  if subtotal >= 5000 then 0 else 490
+  if subtotal > 5000 then 0 else 490
Diagnostic
error U5001 at src/shipping.una:15:34: postcondition of `shipping_fee` is refuted
  counterexample: result = 490, subtotal = 5000

The verifier rejects it before anything runs, and its counterexample, subtotal = 5000, is exactly the order the bug charges.

14.4.2.2 A bug only an example catches: a dropped zero

In the second bug, a zero is missing from the fee:

src/shipping.unadiff
--- a/src/shipping.una
+++ b/src/shipping.una
@@ -15 +15 @@
-  if subtotal >= 5000 then 0 else 490
+  if subtotal >= 5000 then 0 else 49

The package checks clean: 49 cents is between 0 and 490, so every promise still holds. Only the example sees it:

Terminal
$ una test .
example shipping "an order of 49.99 pays 4.90" ... FAILED
  src/shipping.una:24: expect result == 490
  it is false, with `result` = `49`
example shipping "an order of exactly 50.00 ships for free" ... ok

2 examples: 1 passed, 1 failed

The contract states a range, and the example states one exact value. Neither replaces the other.

14.4.3 The parts of the shipping rule

  1. requirement SHIPPING_1. The rule in words, where it came from, and who answers for it (section 5.5).
  2. ensures. Two promises, proven for every subtotal: the fee is between 0 and 4.90, and an order of 50.00 or more pays nothing.
  3. derived from SHIPPING_1. Ties the contract to the requirement.
  4. The examples. Two chosen cases, one on each side of the threshold: 49.99 pays 4.90, and exactly 50.00 pays nothing.

14.4.4 How proofs and examples work together

14.4.4.1 una test proves before it runs

una test . checks the package first, proofs included. A bug that breaks a contract therefore stops every example of the package: the verifier reports it, with a counterexample, and nothing runs. A bug that keeps every contract reaches the examples, which are the only check left.

14.4.4.2 A counterexample becomes an example

The counterexample of the off-by-one bug, subtotal = 5000, is the order of the second example, "an order of exactly 50.00 ships for free". From a counterexample to an example, in section 5.2, keeps such a case as an example. The proof already covers it; the example shows a reader the edge the rule turns on, in the words of a real order.

14.4.4.3 What each one covers

A contract covers every input, but only the facts it states. An example covers one input, but states the whole answer, and with fakes, the effects. So write the rule as a contract, and add an example at each edge of the rule, such as each side of a threshold, and for each exact value a reader should see.

14.4.5 Choosing a contract or an example

Which to use:

You want to… Use
A rule that must hold for every input, such as a bound or a threshold ensures in the header, proven by una check
A precondition that callers must meet requires
An exact value for a case a reader should see an example with expect result == …
A failure path an example with expect fails
What a call did, or did not do, to a service an example with fakes and expect performed
Where a rule came from a requirement and derived from (section 5.5)

14.4.6 Check yourself: proofs and examples

A third bug: the fee becomes 5.00 instead of 4.90.

src/shipping.unadiff
--- a/src/shipping.una
+++ b/src/shipping.una
@@ -15 +15 @@
-  if subtotal >= 5000 then 0 else 490
+  if subtotal >= 5000 then 0 else 500

Which catches it first: una check, or the example "an order of 49.99 pays 4.90"?

Show the answer

una check, before any example runs:

Diagnostic
error U5001 at src/shipping.una:15:35: postcondition of `shipping_fee` is refuted
  counterexample: result = 500, subtotal = 4999

500 is above the bound of the first ensures, result <= 490, so the verifier refutes it, with an order of 49.99 as the counterexample. The example would catch it too, but una test checks the package first and runs nothing when it does not check. A bug that breaks a contract is caught for every input; a bug that stays inside the contract, like the dropped zero, needs an example.

Next:14.5 Day-to-day: checkout