Type inference for datalog with complex type hierarchies

Abstract

What is disclosed are a novel system and method for inferring types of database queries. In one embodiment a program and associated database schema that includes a type hierarchy is accessed. The program includes query operations to a database that contains relations described by a database schema. types are inferred from definitions in the program by replacing each database relationship in the program by the types in the database schema. A new program is generated with the types that have been inferred with the new program only accessing unary relations in the database. In another embodiment, testing of each of the types that have been inferred is performed for type emptiness. In response to type emptiness being found for a type that have been inferred, a variety of different operations are performing including removing the type, providing a notification regarding the emptiness found for the type, and more.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

states that the first column of the extensional relation salary will contain values of type employee, and the second one will contain floating point numbers. A database schema provides this column-wise typing information for every extensional relation.

Given the schema and the hierarchy, type inference should infer the following implications from the definitions in the program:
bonus(x, y)→employee(x) float(y)
factor(x, y)→employee(x) float(y)
query(x, y)→senior(x) manager(x) float(y)

Note how these depend on facts stated in the type hierarchy: for example, in inferring the type of factor, we need to know that employee is the union of junior and senior.

Type inference will catch errors: when a term is inferred to have the type ⊥, we know that it denotes the empty relation, independent of the contents of the database. Also, the programmer may wish to declare types for intensional relations in her program, and then it needs to be checked that the inferred type implies the declared type.

The benefits of having such type inference for queries is not confined to merely checking for errors. Precise types are also useful in query optimisation. In particular, the above query can be optimised to:
query(x, y)←manager(x) senior(x) salary(x, z) y=0.15xz

This rewriting relies on the fact that we are asking for a manager, and therefore the disjunct in the definition of bonus that talks about students does not apply: a student is a parttime employee, and managers cannot be parttime. Again notice how the complex type hierarchy is crucial in making that deduction. The elimination of disjuncts based on the types in the calling context is named type specialisation; type specialisation is very similar to virtual method resolution in optimising compilers for object-oriented languages. Similarly, the junior alternative in the definition of factor can be eliminated. Once type specialisation has been applied, we can eliminate a couple of superfluous type tests: for instance, student(x) is implied by manager(x). That is called type erasure. The key to both type specialisation and type erasure is an efficient and complete test for type containment, which takes all the complex facts about types into account.

Overview The general problem we address is as follows. Given the class of all Datalog programs , we identify a sublanguage of type programs. Type inference assigns an upper bound ┌p┐ in the language of type programs to each program p. To say that ┌p┐ is an upper bound is just to say that p (plus the schema and the type hierarchy) implies ┌p┐. The idea of such ‘upper envelopes’ is due to Chaudhuri [8, 9].

In Section 2 we propose a definition of the class of type programs: any program where all extensionals called are monadic. Type programs can, on the other hand, contain (possibly recursive) definitions of intensional predicates of arbitrary arity. We then define a type inference procedure ┌p┐, and prove that its definition is sound in the sense that ┌p┐ is truly an upper bound of p. Furthermore, it is shown that for negation-free programs p, the inferred type ┌p┐ is also optimal: ┌p┐ is the smallest type program that is an upper bound of p.

For negation-free programs, the definition of ┌p┐ is in terms of the semantics of p, so that the syntactic presentation of a query does not affect the inferred type. It follows that the application of logical equivalences by the programmer or by the query optimiser does not affect the result of type inference.

The restriction that programs be negation-free can be relaxed to allow negation to occur in front of sub-programs that already have the shape of a type program. Not much more can be hoped for, since sound and optimal type inference for programs with arbitrary negations is not decidable.

To also do type checking and apply type optimisations, we need an effective way of representing and manipulating type programs. To this end, we identify a normal form for a large and natural class of type programs in Section 3. The class consists of those type programs where the only negations are on monadic extensionals (i.e., entity types). There is a simple syntactic containment check on this representation, inspired by our earlier work reported in [13].

That simple containment check is sound but not complete. A geometric analysis of the incompleteness in Section 4 suggests a generalisation of the celebrated procedure of Quine [22] for computing prime implicants. We present that generalised procedure, and show that the combination of the simple containment check plus the computation of prime implicants yields a complete algorithm for testing containment of type programs.

The algorithm we present is very different from the well-known approach of propositionalisation, which could also be used to decide containment. The merit of our novel algorithm is that it allows an implementation that is efficient in common cases. We discuss that implementation in Section 5. We furthermore present experiments with an industrial database system that confirm our claims of efficiency on many useful queries.

Finally, we discuss related work in Section 6, and conclude in Section 8.

In summary, the original contributions of this paper include:

• • The identification of a language of type programs for Datalog, suitable both for precise error checking and for type-based optimisations.
  • The definition of a very simple type inference procedure that is proved sound and optimal.
  • A containment check for type programs that relies on a novel generalisation of Quine's method of computing prime implicants.
  • Experiments in an industrial database system that show that this containment algorithm is efficient and useful.

2. Type Programs

It is customary to write Datalog programs as a sequence of rules as seen in FIG. 1, but this syntax is too imprecise for our purposes. In the remainder of the paper, all Datalog programs will be written as formulae of Stratified Least Fixpoint Logic (SLFP) over signatures without function symbols.

This logic, presented for example in [10], extends first order logic (including equality) with fixpoint definitions of the form [r(x₁, . . . , x_n)≡φ] ψ. Here, r is an n-ary relation symbol, the x_i are pairwise different element variables, and φ and ψ are formulae. Intuitively, the formula defines r by the formula φ, which may contain recursive occurrences of r, and the defined relation can then be used in ψ.

The x_i are considered bound in φ and free in ψ, whereas the relation r is free in φ and bound in ψ. To ensure stratification, free relation symbols are not allowed to occur under a negation, which avoids the problematic case of recursion through negation. For example, the formula
[s(x, y)≡x≐y ∃z.e(x, z) s(z, y)]s(x, y)  (1)
defines the relation s to be the reflexive transitive closure of e. By way of illustration (and only in this example), free occurrences of element variables and relation variables have been marked up in bold face. The formula is trivially stratified since no negations occur anywhere.

In this program, the relation symbol s is an intensional predicate since it is given a defining equation. On the other hand, relation e is an extensional, which we assume to be defined as part of a database on which the program is run. We will denote the set of intensional relation symbols as and that of extensional relation symbols as , and require them to be disjoint.

Model-theoretically, the role of the database is played by an interpretation which assigns, to every n-ary relation symbol e ∈ , a set of n-tuples of some non-empty domain. To handle free variables we further need an assignment σ that maps free element variables to domain elements and free relation variables to sets of tuples.

Just as in first order logic, we can then define a modelling judgement _{σ φ} stating that formula φ is satisfied under interpretation and assignment σ. In particular, stratification ensures that formulae can be interpreted as continuous maps of their free relation variables, so that fixpoint definitions can be semantically interpreted by the least fixpoints of their defining equations.

For a formula φ with a single free element variable x, we set [[φ]] :={c| _x:=c φ}, omitting the subscript where unambiguous.

Just as the semantics of the intensional predicates is determined with respect to the semantics of the extensional predicates, the types of the intensional predicates are derived with respect to the types of the extensional predicates. These are provided by a schema which assigns, to every n-ary extensional predicate, an n-tuple of types. As is customary, types are themselves monadic predicates from a set ⊂ of type symbols. For consistency, we require that the schema assigns a type symbol to itself as its type.

For the program in (1), for example, we might have two type symbols a and b, with the schema specifying that (e)=(a, b) and of course (a)=(a), (b)=(b).

Semantically, we understand this to mean that in every interpretation of the extensional symbols conforming to the schema, the extension of a given column of an extensional symbol should be exactly the same as the extension of the type symbol assigned to it by the schema.

We can define a first order formula, likewise denoted as , which expresses this constraint: For every n-ary relation symbol e ∈ \ and every 1≦i≦n where u ∈ is the type assigned to the i-th column of e, the formula contains a conjunct
∀x_i.(∃x₁, . . . , x_i−1, x_i+1, . . . , x_n.e(x₁, . . . , x_n)) u(x_i)  (2)

Our example schema above, for instance, would be expressed by the formula
(∀x.(∃y.e(x,y)) a(x)) (∀y.(∃x.e(x, y)) b(y))

In the literature, the types assigned by the schema are often not required to fit exactly, but merely to provide an overapproximation of the semantics. We could achieve this by relaxing (2) to use an implication instead of a bi-implication.

Instead, we take a more general approach by allowing arbitrary additional information about the types to be expressed through a hierarchy formula . Our only requirements about this formula are that it be a closed first order formula containing only type symbols (and no other relation symbols).

In particular, the hierarchy may stipulate subtyping relations; in our example, the hierarchy could be ∀x.a(x)→b(x), expressing that a is a subtype of b. Perhaps more interestingly, it can also express disjointness of types, e.g. by stating that ∀x. a(x) b(x); we could then deduce that in the definition of the reflexive transitive closure e can, in fact, be iterated at most once. Many other kinds of constraints are just as easily expressed, and provide opportunities for advanced optimisations.

We now approximate programs by type programs, which are programs that only make use of type symbols (and no other extensional relation symbols), and do not contain negation. (Compare this with the hierarchy which can contain negations, but no fixpoint definitions.) To every SLFP formula φ we assign a type program ┌φ┐ by replacing negated subformulae with

which is semantically equivalent to
x≅y a(x) b(y)

We will see in the next section that, in fact, fixpoint definitions can always be eliminated from such type programs, yielding formulae of monadic first order logic.

As the example suggests, the types assigned to programs semantically over-approximate them in the following sense:

Theorem 1 (Soundness). For every program φ we have
, , φ ┌φ┐

That is, every interpretation and assignment satisfying the schema, the hierarchy and φ also satisfies ┌φ┐.

Proof Sketch. We can show by induction on φ the following stronger version of the the—orem: For any interpretation with and any two assignments σ and σ′, which assign the same values to element variables, and where σ(r) ⊂ σ′(r) for every r ∈ , we have _σ′ ┌φ┐ whenever _σφ.

From this, of course, it follows that , φ ┌φ┐, and again , , φ ┌φ┐ by monotonicity of entailment.

Perhaps more surprisingly, our type assignment also yields the tightest such over-approximation, in spite of our very lax restrictions on the hierarchy:

Theorem 2 (Optimality). For a negation-free program φ and a type program , if we have
, , φ
then also
, , ┌φ┐

To prove this theorem, we need a monotonicity result for the type assignment.

Lemma 3. For two negation-free programs φ and ψ, if it is the case that , , φ ψ then also , , ┌φ┐ ┌ψ┐.

An easy way to prove this result is to make use of cartesianisation, which is the semantic equivalent of the typing operator ┌•┐. For a relation R ⊂ Dⁿ, we define its cartesianisation cart (R)=π₁(R) x . . . . xπ_n(R) to be the cartesian product of the columns of R. Likewise, the cartesianisation cart() of an interpretation is the interpretation which assigns cart((e)) to every relation symbol e.

It is then not hard to see that
cart() _{σ φ σ}┌φ┐  (3)
for any negation-free formula φ, interpretation and assignment σ whenever . Also, cart() if , and cart() iff .

From this observation, we easily get the

Proof of Lemma 3. Assume , , φ | ψ, and let , σ be given with , and _σ ┌φ┐.

By (3), we have cart() | , cart() | and also cart() ψ _σφ, so by assumption cart() | _σ ψ, but then by applying (3) again we get | _σ┌ψ┐.

We briefly pause to remark that Lemma 3 also shows that type inference for negation-free programs respects semantic equivalence: If φ and ψ are semantically equivalent (under and ), then so are their types. This does, of course, not hold in the presence of negation, for given a type symbol u, we have ┌u(x)┐=u(x) yet ┌ u(x)┐=τ.

Continuing with our development, we can now give the

Proof of Theorem 2. Assume , , φ ; by Lemma 3 this means , , ┌φ┐ ┌┐. But since is a type program we have ┌┐=, which gives the result.

So far we have handled negation rather crudely, trivially approximating it by τ. In fact, we can allow negations in type programs and amend the definition of ┌•┐ for negations to read

$\lceil ⫬\phi \rceil =\{\begin{array}{cc}⫬\phi & \mathrm{if}\phi \mathrm{is}a\mathrm{type}\mathrm{program}\\ \top & \mathrm{otherwise}\end{array}$

All the above results, including soundness and optimality, go through with this new definition, and the optimality and monotonicity results can be generalised to hold on all programs which contain only harmless negations, i.e. negations where the negated formula is a type program.

There is little hope for an optimality result on an even larger class of programs: Using negation, equivalence checking of two programs can be reduced to emptiness, and a program is empty iff its optimal typing is ⊥. As equivalence is undecidable [23], any type system that is both sound and optimal for all programs would likewise have to be undecidable.

Our proposed definition of type programs is significantly more liberal than previous proposals in the literature. Frühwirth et al. in a seminal paper on types for Datalog [16] propose the use of conjunctive queries, with no negation, no disjunction, no equalities and no existentials.

As an example where the additional power is useful, consider a database that stores the marriage register of a somewhat traditionally minded community. This database might have types male and female, specified to be disjoint by the type hierarchy, and an extensional relation married with schema (female, male). (Note that this means that all male and female entities in the database are married, i.e. occur somewhere in the married relation.)

Using our above terminology, we have ={male, female} and = ∪ {married}; is ∀x.male(x) female (x).

Let us now define an intensional predicate spouse:
[spouse(x, y)≡married(x, y) married(y, x)]spouse(x, y)

What is the type of spouse(x, y)? In the proposal of [16], we could only assign it (female(x) male(x)) (female(y) male(y)), so both arguments could be either male or female. By contrast, when employing our definition of type programs, the inferred type is
(female(x) male(y)) (male(x) female(y))

This accurately reflects that x and y must be of opposite sex. By properly accounting for equality, we can also infer that the query spouse(x, y) x≐y has type ⊥ under the hierarchy: nobody can be married to themselves.

3. Representing Type Programs

For query optimisation and error checking, the single most important operation on types is containment checking, i.e. we want to decide, for two type programs and ′, whether | ′, which we will write | ′ for short.

As mentioned in the introduction, we are often interested in checking whether a formula is unsatisfiable, that is whether it always gives an empty set of results regardless of the interpretation of the extensional relations. If such a formula occurs in a user program, it is regarded as a type error and the user is alerted to this fact; if it occurs during optimisation, it provides an opportunity for logical simplification. By the results of the previous section, a formula φ containing only harmless negations is unsatisfiable iff its type is equivalent to ⊥, that is iff ┌φ┐ ⊥.

Our type programs only make use of monadic extensionals, and their containment can be decided by a straightforward extension of the well-known decision procedure for monadic first order logic [6]. This construction, however, incurs an exponential blowup in formula size and does not directly yield a practical algorithm.

Under some mild restrictions on the class of type programs and the hierarchy, however, we can represent type programs in a compact manner that allows efficient containment checking in practice.

The class of type programs we deal with falls in between the two classes discussed in the previous section by allowing negation, but only in front of type symbols (and never before equalities). The definition of ┌•┐ is easily changed to accommodate this by setting ┌φ┐=φ if φ is of the form u(t) for u ∈ , and τ otherwise. We call a possibly negated application of a type symbol to a variable a type literal.

Furthermore, we require the hierarchy to be of the form ∀x.h(x), where h(x) does not contain quantifiers or equations, and only a single free variable x.

The basic unit of our representation are type propositions, which are Boolean combinations of type symbols, all of them applied to the same variable. In other words, a type proposition is a quantifier-free type program without fixpoint definitions or free relation variables, having precisely one free element variable. For example, the subformula h(x) of is a type proposition.

A type proposition φ behaves essentially the same as the propositional formula |φ| that results from it by replacing every atom u(x) with the propositional letter u; for example, it is easily seen that ∀x.y is (first order) valid iff |φ| is (propositionally) valid [20]. Hence we will sometimes identify the two notationally.

For example, if a and b are type symbols, then a(x) b(x) is a type proposition with |a(x) b(x)|=a b.

For the restricted class of type programs under consideration, fixpoint definitions can be eliminated. To see this, recall that fixpoint definitions can be expanded into infinite disjunctions [10]. It thus suffices to show that the type programs without free relation variables and without fixpoint definitions (which we shall henceforth call flat) form a complete lattice, so that even a prima facie infinite disjunction can in fact be represented by a single formula.

Lemma 4. The flat type programs form a complete lattice under the order .

Proof. The lattice structure is obviously given by the logical operations of conjunction and disjunction. To show completeness, we show that the set of flat type programs is finite (modulo equivalence under ).

Notice that every flat type program with m free element variables from X={x₁, . . . , x_m} can be written in prenex form as

$\exists y_1,\dots ,y_n·\underset{i}{⩔}\left[\underset{1\le j\le m}{⩓}{\psi }_{i,j}⩓\underset{1\le k\le n}{⩓}{\chi }_{i,k}\left(y_k\right)⩓\zeta \right]$
for a set Y={y₁, . . . , y_n} of element variables, where all the ψ_i,j and X_i,k are type propositions, and ζ is a conjunction of equations between element variables from X ∪ Y.

Existentials distribute over disjunctions, and also over conjunctions if the variable they bind is only free in one of the conjuncts. Also, note that ∃x.(x≐y φ) is semantically equivalent to φ with y substituted for x. Hence we can further bring the type program into the form

$\underset{i}{⩔}\left[\underset{1\le j\le m}{⩓}{\psi }_{i,j}^{\prime }\left(x_j\right)⩓\underset{1\le k\le n}{⩓}\left(\exists y_k·{\chi }_{i,k}^{\prime }\left(y_k\right)\right)⩓{\zeta }^{\prime }\right]$
where ζ′ now contains only variables from X. We will say that a formula of this shape is in solved form.

Clearly, there are only finitely many different formulae in solved form (modulo semantic equivalence), since there are only finitely many type propositions.

Notice that the set of flat type programs is no longer finite if we allow negated equalities. Indeed, for every natural number n, let ε_n be a formula asserting that there are at least n different individuals; for example, ε₂ is ∃x₁, x₂. x₁≐x₂. Then ε_n ε_n+1 for every n, yet all the ε_n are semantically distinct; so the lattice is not even of finite height anymore.

Corollary 5. Every type program without free relation variables is semantically equivalent to a flat type program, and hence to a formula of monadic first order logic.

Here is an example of how a type program (with only a single disjunct) can be brought into solved form.
∃z·a(x)z≐yb(y)c(x)x≐y
c(z)≡{sorting by variable}∃z·a(x)c(x)b(y)x≐y
z≐yc(z)≡{pushing in ∃}a(x)c(x)b(y)x≐y
∃z·z≐yc(z)≡{eliminating≐under ∃}a(x)c(x)b(y)
x≐yc(y)≡{sorting again}a(x)c(x)b(y)c(y)x≐y

We will now introduce data structures that represent formulae in solved form, and develop a containment checking algorithm that works directly on this representation.

We define an order on type propositions by taking φ<: ψ to mean | |h| |φ|→|ψ|, where |h| is the propositional formula corresponding to as defined above. It is not hard to see that this holds precisely when φ ψ. In the remainder, we will identify type propositions that are equivalent under <: so that it becomes a partial order. Where needed, we make equivalence under <: explicit by writing it as ≡.

If t is a single type proposition and C a set of type propositions, we write t<:C (C<:t) to mean that t<:c (c<:t) for some c∈C.

The order <: can be extended to sets of type propositions by defining C<:D to mean that for any d∈D there is some c∈C with c<:d. This order is clearly reflexive and transitive, and is anti-symmetric (and hence a partial order) on lean sets, i.e. sets C where c<:c′ for c, c′∈C implies c=c′. One way of making a set of type propositions lean is to discard its non-minimal elements, which we write as min(C).

We can represent every disjunct of a formula in solved form using the following data structure:

Definition 1. An n-ary type tuple constraint, or TTC, τ is a structure (t₁, . . . , t_n|p|C) where

• • each component t_i is a type proposition,
  • p is an equivalence relation over {1, . . . , n},
  • C is a set of type propositions called inhabitation constraints, such that
  • for all 1≦i, j≦n with i˜_p j (i.e., i and j belong to the same partition of p) we have t_i=t_j,
  • C is lean,
  • for all 1≦k≦n we have C<:t_k.

If p is a relation, we write p to mean the smallest equivalence relation containing it. The equivalence class of an element x under p is written [x]_p, where we omit the subscript p if it is clear from context. We call an equivalence relation q finer (or, more precisely, no coarser) than p if whenever x˜_qy then also x˜_py.

Whenever we have a pre-TTC τ=(t₁, . . . , t_n|p|C) that does not satisfy one of the requirements, we can build a TTC that does by setting
∥τ∥:=(∥t₁∥_p . . . , ∥t_n∥_p|p|,om(C∪{t₁, . . . , t_n}))
where ∥t_i∥_q={t_j|i˜_qj}.

For a TTC τ=(t₁, . . . , t_n|p|C), we define τ_i:=t_i, and τ_i:=r is the TTC resulting from τ by replacing the component t_i (and every component t_j where i˜_p j) by r and adding r to the inhabitation constraints.

The formula represented by a TTC is easily recovered:

Definition 2. Given a list of n different element variables x₁, . . . , x_n, we define the type program [τ](x₁, . . . , x_n) corresponding to a TTC τ=(t₁, . . . , t_n|p|C) as

$\left(\underset{1\le k\le n}{⩓}{t}_i\left(x_i\right)\right)⩓\left(\underset{i\sim {\hspace{0.17em}}_{p}j}{⩓}{x}_i\doteq x_j\right)⩓\left(\underset{c\in C}{⩓}\exists y·c\left(y\right)\right)$

A TTC is called degenerate if ⊥ is among its inhabitation constraints; it represents an unsatisfiable formula. The equivalence relation is called trivial if it is in fact the identity relation id, and the set of inhabitation constraints is called trivial if it only constrains the component types to be inhabited. When writing a TTC, we will often leave out trivial partitionings or inhabitation constraints.

We extend the order <: to TTCs component-wise:

Definition 3. For two n-ary TTCs τ=(t₁, . . . , t_n|p|C) and τ′=(t₁′, . . . , t_n′|p′|C′) we define τ<:τ′ to hold iff

• • for every 1≦i≦n we have t_i<:t_i′,
  • p′ is finer than p,
  • C<:C′.

It is routine to check that this defines a partial order with maximal element τ^TTC=(τ, . . . , τ), with binary meets defined by

$\begin{array}{cc}\left(t_1,\dots ,t_npC\right)\stackrel{\_}{⩓}\left(t_1^{\prime },\dots ,t_n^{\prime }p^{\prime }C^{\prime }\right)=\left(t_1⩓t_1^{\prime },\dots ,t_n⩓t_n^{\prime }p\bigcup p^{\prime }C\bigcup C^{\prime }\right)& \left(4\right)\end{array}$

Since all type programs can be brought into solved form, we can represent them as sets of TTCs.

Definition 4. A lean set of non-degenerate TTCs is called a typing.

The formula represented by a typing is
[T]:=

The order <: is extended to typings by defining T<:T′ to hold if, for every τ∈T there is a τ′∈T′ with τ<:τ′.

To convert any set of TTCs into a typing we can eliminate all non-maximal and degenerate TTCs from it; this operation we will denote by max_⊥.

The order on typings is again a partial order with least element ∅ and greatest element {τ^TTC}. Binary meets can be defined point-wise as
TT′=max_⊥({ττ′|τ∈T, τ′∈T′}  (5)
and joins are given by
TT′=max_⊥(T∪T′)  (6)

We will now show that every type program can be translated to a typing. We already have enough structure to model conjunction and disjunction; existential quantification is also not hard to support.

Definition 5. The i-th existential of a TTC τ=(t₁, . . . , t_n|p|C), where 1≦i≦n, is defined as
∃_i(τ):=(t₁, . . . , t_i−1, t_i+1, . . . , t_n|p′|min(C∪{t_i}))
where p′ is the restriction of p to {1, . . . , i−1, i+1, . . . , n}.

The existential of a typing is defined pointwise:
∃_i(T):=max_⊥{∃_i(τ)|τ∈T}

To see that this models the existential quantifier on type programs, we can easily convince ourselves of the following:

Lemma 6. Let τ be an n-ary TTC, an interpretation and σ a statement. Then | _σ [∃_i(τ)] iff there is a domain element d such that _{σ, xi:=d}[τ]. An analogous statement holds for typings.

Although typings cannot directly represent type programs with fixpoint definitions or free relation variables, we can translate every closed type program to a typing by eliminating definitions. Let us start with a translation relative to an assignment to free relation symbols.

Definition 6. Let φ be a type program without fixpoint definitions containing the intensional relation variables r₁, . . . , r_m and the element variables x₁, . . . , x_n. We translate it to a mapping φ: Typing^m→Typing by recursion on the structure of φ:
⊥({right arrow over (T)})=∅
x_i≐x_j({right arrow over (T)})={(τ, . . . , τ|{{ij}})}
u(x_j)({right arrow over (T)})={τ^TTC_j:=u}
u(x_j)({right arrow over (T)})={τ^TTC_j:=u}
r_i({right arrow over (x)})({right arrow over (T)})=T_i
ψX({right arrow over (T)})=(ψ)({right arrow over (T)})X({right arrow over (T)})
ψX({right arrow over (T)})=(ψ)({right arrow over (T)})X({right arrow over (T)})
∃x_i.ψ({right arrow over (T)})=∃_i(ψ({right arrow over (T)}))

It is easy to check that φ is a monotonic mapping for every φ with respect to the order <:, and since the set of typings is finite its least fixpoint can be computed by iteration. Thus we can translate every type program φ which does not have free relation variables into a typing φ which represents it:

Lemma 7. For every type program φ without free relation variables, we have φ[φ].

Proof Sketch. In fact, we can show that, for any type program φ with m free relation variables and every m-tuple {right arrow over (T)} of typings, we have the equivalence φ([{right arrow over (T)}]) [φ ({right arrow over (T)})], where [{right arrow over (T)}] is the m-tuple of type programs we get from applying [•] to every component of {right arrow over (T)}.

This can be proved by structural induction on φ by showing that [•] commutes with the logical operators (using Lemma 6 for the case of existentials) and with fixpoint iteration. The lemma then follows for the case m=0.

4. Containment Checking

We are now going to investigate the relation between the orders <: (component-wise or syntactic subtyping) and (semantic subtyping). The former is convenient in that it can be checked component-wise and only involves propositional formulae, so it would be nice if we could establish that T₁<:T₂ iff [T₁]| [T₂].

Unfortunately, this is not true. The ordering <: is easily seen to be sound in the following sense:

Theorem 8. For any two typings T and T′, if T<:T′ then [T] [T′].

However, it is not complete. Consider, for example, the typings T={(b c, a)} and T′={(a b, a), (c d, a)}. Clearly we have [T] [T′], yet TT′.

If we interpret type symbols as intervals of real numbers, this example has an intuitive geometric interpretation, shown in FIG. 4.

The individual TTCs then become rectangles in the two-dimensional plain, and it is easy to see that the single TTC in T (depicted by the single rectangle in heavy outline filled with vertical lines) is contained in the union of the TTCs in T′ (the two rectangles filled with slanted lines), but not in either one of them in isolation.

A similar problem can arise from the presence of equality constraints: Consider the typings T={(a b, a b|{{1, 2}})} and T′={(a, a), (b, b)}. The former appears in FIG. 5 as a heavy slanted line, which is not contained in either of the two squares representing the TTCs in T′, but only in their union.

Intuitively, the problem is that <: tests containment of TTCs in isolation, whereas the semantic inclusion check considers containments between sets of TTCs.

The following lemma paves the way for a solution:

Lemma 9. The mapping [•] is the lower adjoint of a Galois connection, i.e. there is a mapping (|•|) from type programs to typings such that
[T] iff T<:(||)

Proof. As we have established, typings and type programs form complete lattices under their respective orders. Since [•] distributes over joins it must be the lower adjoint of a Galois connection between these lattices.

An immediate consequence of this is that [T₁] [T₂] implies T₁<:(|[T₂]|). To check, then, whether T₁ is semantically contained in T₂, we compute (|[T₂]|) and check for component-wise inclusion.

One possible implementation of (|•|) comes directly from the definition of the Galois connection: For any type program , (||) is the greatest typing T such that [T] , and since there are only finitely many typings and containment is decidable we could perform an exhaustive search to find this T.

Clearly, this algorithm is not ideal, since one of our goals is to avoid having to explicitly decide the relation . Inspiration for a better solution comes from a perhaps somewhat unexpected direction, namely the theory of Boolean function minimisation:

Quine [22] introduces an algorithm whose first part determines the set of prime implicants of a Boolean formula, which are conjunctive formulae such that every conjunctive formula that implies the original formula implies one of the prime implicants.

Generalising from conjunctions to TTCs and from propositional formulae to typings, we will introduce a saturation mapping sat that generates the set of all prime implicant TTCs for a typing, which are TTCs such that every TTC that (semantically) entails the typing entails one of them componentwise. Hence whenever [T₁] [T₂] we have T₁<:sat(T₂), so sat is an implementation of (|[•]|). The saturation mapping also preserves semantics, so the other direction of the implication holds as well.

We can follow Quine's definitions and proofs almost paragraph by paragraph, starting with the generalisation of the consensus operation:

Definition 7. The consensus τ⊕_J τ′ over J of two n-ary TTCs τ and τ′, where J∈{1, . . . , n} is an index set, is defined by

$\left(t_1,\dots ,t_npC\right){\oplus }_J\left(t_1^{\prime },\dots ,t_n^{\prime }p^{\prime }C^{\prime }\right)=\left(u_1,\dots ,u_np\bigcup p^{\prime }\bigcup J\times JC\bigcup C^{\prime }\right)$

where

$u_i=\{\begin{array}{cc}{⩓}_{j\in J}{t}_j⩔{⩓}_{j\in J}{t}_j^{\prime }& \mathrm{if}i\in J\\ t_i⩓t_i^{\prime }& \mathrm{otherwise}\end{array}$

We will sometimes write _J τ for _j∈J^t_j.

Roughly, the consensus operation equates all columns in J and preserves all other equalities and inhabitation constraints of its operands, takes disjunctions over the columns in J, and conjunctions over all the others.

For example, in the example shown in FIG. 5 we can take the consensus of (a, a) and (b, b) over J={1, 2} to obtain the TTC (a b, a b|{{1, 2}}), which is precisely the missing piece to show that T is a subtype of T′.

In the particular case where J={j} is a singleton set, and both partitions and inhabitation constraints are trivial, the consensus takes conjunctions on all columns except j, where it takes a disjunction. This is a generalisation of the behaviour of the consensus operator used in the minimisation of Boolean functions.

For example, in FIG. 4 we can take the consensus of (a b, a) and (c d, a) on J={1} to obtain the TTC (a b c d, a) which covers (b c, a).

We need another kind of consensus to deal with inhabitation constraints:

Definition 8. The existential consensus of two n-ary TTCs τ and τ is defined by

$\left(t_1,\dots ,t_npC\right){\oplus }_{\exists }\left(t_1^{\prime },\dots ,t_n^{\prime }p^{\prime }C^{\prime }\right)=\left(t_1⩓t_1^{\prime },\dots ,t_n⩓t_n^{\prime }p\bigcup p^{\prime }C⩔C^{\prime }\right)$
where C C′={c c′|c∈C, c′∈C′}.

To see why this is necessary, assume we have type symbols a, b, c with ∀x.a(x) b(x)→c(x), and consider typings T={(c|id|a b)} and T′={(c|id|a), (c|id|b)}. Clearly, [T]=c(x₁) ∃z.(a(z) b(z))=[T′], yet TT′ since {a b}:{a} and {a b}:{b}. However, if we add the existential consensus (c|id|a b) to T′, we will be able to prove that T<:T′.

It is not hard to verify that adding the consensus of two TTCs to a typing does not change its semantics.

Lemma 10. For two TTCs τ, τ and an index set J we have [τ⊕_Jτ′] [{τ, τ′}] and [τ⊕_∃τ′] [{τ, τ′}].

Proof Sketch. Clearly, if J=∅ the consensus is just the meet, so nothing needs to be proved.

Otherwise, let and σ be given such that _σ[τ⊕_Jτ′]. Then, for some i∈J, either | _σj∈Jt_j(x_i) or | _σj∈Jt_j′; in the former case _σ[τ], and in the latter case _σ[τ′].

For the existential consensus, assume |_σ[τ⊕_∃τ′] and further assume there is some inhabitation constraint c of τ such that ┌y.c(y) (for otherwise _σ[τ] is immediate). We must have ∃y.c(y) c′(y) for all inhabitation constraints c′ of τ′, so certainly | ∃y.c′(y). We then easily see that _σ[τ′].

Remember that typings do not contain non-maximal elements. The following result shows that we do not lose anything by eliminating them.

Lemma 11. Consensus formation is monotonic in the sense that for TTCs σ, σ′, τ, τ′ with σ<:σ′ and τ<:τ′ and for an index set J we have σ⊕_Jτ<: σ′⊕_Jτ′ and σ⊕_∃τ<:σ′⊕_∃τ′

Proof Sketch. The components of the two TTCs are combined using conjunction and disjunction, which are monotonic with respect to subtyping, and the partitionings and inhabitation constraints are combined using set union which is also monotonic.

Definition 9. An n-ary typing T is said to be saturated if the consensus of any two TTCs contained in it is covered by the typing; i.e. for any τ, τ′∈T and any index set J⊂{1, . . . , n} we have τ⊕_Jτ′<:T and τ⊕_∃τ′<:T.

Lemma 12. Every typing T can be converted into a semantically equivalent, saturated typing sat(T) by exhaustive consensus formation.

Proof. We use Algorithm 2 to collect all consensus TTCs of two given TTCs. As shown in Algorithm 1, this is performed for every pair of TTCs in the typing, and this procedure is repeated until a fixpoint is reached.

Notice that in each iteration (except the last) the set of TTCs covered by the typing becomes larger, and since there are only finitely many TTCs, the termination condition must become true eventually.

We now generalise the concepts of implicants and prime implicants.

Definition 10. A TTC τ is an implicant for a typing T′ if we have [τ] [T′]; it is a prime implicant if it is a <:-maximal implicant.

More explicitly, a TTC π is a prime implicant for a typing T′ if

1. π is an implicant of T′

2. For any τ with π<:τ and [τ] [T′], we have τ<:π.

The second condition is equivalent to saying that for any τ with τ<:τ and τ:π we have [τ] [T].

Lemma 13. Every implicant implies a prime implicant.

Proof. The set of all implicants of a typing is finite, so for any implicant τ there is a maximal implicant π with τ<:π, which is then prime by definition.

Remark 14. If π is a prime implicant of T, then π<:T iff π∈T.

Proof. The direction from right to left is trivial. For the other direction, suppose π<:T, i.e. π<:π′ for some π′∈T. Certainly [π′] [T], so π′<:π since π is prime. But this means that π=π′∈T.

We want to show that sat(T) contains all prime implicants of T, so we show that saturation can continue as long as there is some prime implicant not included in the saturated set yet.

Lemma 15. If there is a prime implicant π of T with π∉T, then T is not saturated.

Proof. Let π be a prime implicant of T with π∉T. Consider the set
M:={τ|[τ] [π], τ:T}
This set is not empty (it contains π by the preceding remark) and it is finite, so we can choose a <:-minimal element ψ∈M.

The proof proceeds by considering three cases. In the first, we shall show that a consensus step is possible, thus proving that T is not saturated. In the second, we show that an existential consensus step can be made, again proving non-saturation. Finally, we show that either the first or second case must apply, as their combined negation leads to a contradiction.

• • Assume there is an index i for which ψ_i≡r′ r″ for two r′, r″ neither of which equals ψ_i.
  • Then we form ψ′:=ψ_i:=r′ and ψ″:=ψ_i:=r″. Observe that ψ and ψ are strictly smaller than ψ, so they cannot be in M. They both, however, entail π, so there must be φ′, φ″ ∈ T with ψ′<:φ′ and ψ<:φ″.
  • Note that ψ=ψ′ ⊕_[i] φ″, where [i] is the equivalence class of i in ψ. By Lemma 11, this means that ψ<:φ′⊕_[i]φ″, and since ψ:T we also have φ′⊕_[i]φ″:T, which shows that T is not saturated.
  • Assume for some c∈C, where C is the set of inhabitation constraints of ψ, we have c≡ r′ r″ for two r′, r″ neither of which equal c.
  • We form ψ from ψ by setting the inhabitation constraints to C\{c} ∪ {r′} and minimising, and similarly for ψ″. As above, those two are strictly smaller than ψ and we can find φ′ and φ″. The proof goes through as in the previous case, since it is easy to see that ψ=ψ′⊕_∃ ψ″.
  • Otherwise, for any index i and r′, r″ with ψ_i≡ r′ r″ we have either ψ_i≡ r′ or ψ_i≡ r″ (i.e. every ψ_i is join-irreducible) and likewise for all c∈C.

We will show that in this case ψ<:T, contrary to our assumptions. We know that [ψ] | [T], so any interpretation and assignment σ with and _σ [ψ] will satisfy some TTC τ∈E T. Put differently, if _σ(ψ) (x_i) for all i and ∃y.(c) (y) for all inhabitation constraints c and moreover a satisfies the partition of ψ, then we will find τ∈T with _τ[τ].

Observe that for every i the (propositional) formula |h||ψ_i| can be written as a conjunction l_{i, l} . . . l_{i, mi} of type literals. We can moreover assume that every type symbol occurs in precisely one of these literals, for if, say, type t does not occur, then l_{i, 1} . . . l_{i, mi} t l_{i, 1} . . . l_{i, mi} is a join reduction of |h||ψ_i|.

Let such a decomposition be fixed and let L_i:={1_{i, 1}, , . . . , l_{i, mi}}. Since ψ_i=ψ_j whenever i and j are in the same partition, we can clearly choose the decomposition such that L_i=L_j in this case. Likewise, for every inhabitation constraint c we can write |h||c| as l_{c, 1} . . . l_{c, mc}, and define L_c by analogy.

We define an interpretation over the domain {x_[1], . . . , x_[n]}∪C of variables modulo the partition of ψ plus the inhabitation constraints by
(b)={x_[i]|b∈L_i}∪{c|b∈L_c}
for every type symbol b. The assignment is simply defined by σ(x_i)=x_[i].

It is easy to see that x_[i]∈[[l]] iff l∈L_i for all 1≦i≦n, and c∈[[l]] iff l∈L_c for all inhabitation constraints c, from which we deduce _σ ( ψ_i) (x_i) for all i, and ∃y.() (y) for all inhabitation constraints c. By definition, σ satisfies the partition of ψ. So we have some τ∈T with | _σ[τ].

We claim that ψ<:T.

Indeed, let an index 1≦i≦n be given. Then we can write τ_i as a disjunction of conjunctions, such that one of its disjuncts, of the form l_{i, 1}′ . . . l_{i, mi′}′, is satisfied under and σ, meaning that x_[i]∈[[l_i,j′]] for all j, so all the l_i,j′ are in fact in L_i, so ψ_i<:τ_i.

Since σ satisfies the partition of τ, this partition must be finer than the partition of ψ.

Finally, let an inhabitation constraint d of τ be given. Since ∃y.() (y), the interpretation of d cannot be empty. As above, we can write d in a disjunctive form such that all the literals in one of its disjuncts are non-empty in . So there must be some domain element that occurs in the denotation of all these literals. If it is one of the x_[i], then we have ψ_i<:d, which implies c<:d for some inhabitation constraint c of ψ; if it is an inhabitation constraint c, then we have c<:d directly.

Taking these facts together, we get ψ<:τ, whence ψ<:T. This contradicts our assumption, and we conclude that this subcase cannot occur.

Now we can declare victory:

Theorem 16. If [T₁] [T₂], then T₁<:sat(T₂).

Proof. Assume [T₁] [T₂] and let τ∈T₁ be given. Then [τ] [T₂], so certainly [τ] [sat(T₂)], since saturation does not change the semantics. By Lemma 13 this means that there is a prime implicant π of sat(T₂) with τ<:π. By Lemma 15 we must have π∈sat(T₂), so τ<: sat(T₂). Since this holds for any τ∈T₁ we get T₁<:sat(T₂).

5. Implementation

It is not immediately clear that the representation for type programs proposed in the preceding two sections can be implemented efficiently. While the mapping ┌•┐ from programs to type programs is, of course, easy to implement and linear in the size of the program, the translation of type programs into their corresponding typings involves fixpoint iterations to eliminate definitions. Saturation is also potentially very expensive, since we have to compute the consensus on every combination of columns for every pair of TTCs in a typing, and repeat this operation until a fixpoint is reached.

We report in this section on our experience with a prototype implementation based on Semmle's existing type checking technology as described in [13]. The type checker computes typings for programs and immediately proceeds to saturate them to prepare for containment checking. It uses a number of simple heuristics to determine whether a consensus operation needs to be performed. This drastically reduces the over-head of saturation in practice and makes our type inference algorithm practically usable.

TTCs can be compactly represented by encoding their component type propositions as binary decision diagrams. The same can be done for the hierarchy, so checking containment of type propositions can be done with simple BDD operations.

As with any use of BDDs, it is crucial to find a suitable variable order. We choose a very simple order that tries to assign neighbouring indices to type symbols that appear in the same conjunct of the hierarchy formula . For example, if the hierarchy contains a subtyping statement u₁(x)→u₂(x) or a disjointness constraint (u₁(x) u₂(x)), then u₁ and u₂ will occupy adjacent BDD variables. This heuristic is simple to implement and yields reasonable BDD sizes as shown below.

To mitigate the combinatorial explosion that saturation might bring with it, we avoid useless consensus formation, i.e. we will not form a consensus τ⊕τ′ if τ⊕τ′<:{τ, τ′} or τ⊕τ′ is degenerate. The following lemma provides a number of sufficient conditions for a consensus to be useless:

Lemma 17. Let τ=(t₁, . . . , t_n|p|C) and τ′=(t₁′, . . . , t_n′|p′|C′) be two TTCs and J ∈ {1, . . . , n} a set of indices. Then the following holds:

• • 1. If N^⊥:={i|t_i t_i′=⊥} ⊂ J then τ⊕_Jτ′ is degenerate.
  • 2. If _∈Jt_j=⊥ or _∈Jt_j′=⊥, then τ_J′τ′<:{τ, τ′} for any J′⊃J.
  • 3. If we have l∉J with _j∈Jt_j<:t_l and _j∈Jt_j′<:t_j′, then τ⊕_J∪{l}τ′<:τ⊕_Jτ′.
  • 4. If _j∈Jt_j<: _j∈Jt_j′ or vice versa, then τ⊕_Jτ′<:{τ, τ′}.
  • 5. For j∈J, j′∉J with j˜_p∪p′j′ we have τ⊕_Jτ′<:τ⊕_J∪{j′}τ′.
  • 6. If N^⊥≠∅, then τ⊕_∃τ′ is degenerate.

Proof. Recall that τ⊕_Jτ′=∥(u₁, . . . , u_n|p∪p′∪J×J|C∪C′)∥ where

$u_i=\{\begin{array}{cc}{⩓}_{j\in J}{t}_j⩔{⩓}_{j\in J}{t}_j^{\prime }& \mathrm{if}i\in J\\ t_i⩓t_i^{\prime }& \mathrm{otherwise}\end{array}$
We prove the individual claims:

• • 1. Indeed, assume k∈N^⊥\J, then u_k=t_k t_k′=⊥.
  • 2. Assume, for example, _j∈Jt_j=⊥ and let J′⊃J be given, then_∈J′t_j=⊥, so u_i=_j∈J′t′_j′<:t_i′ if i∈J, and u_i=t_i t_i′<:t_i′ otherwise, so τ⊕_J′τ′<:τ′. If the other disjunct is ⊥, we see that τ⊕_J′τ′<:τ by a similar argument.
  • 3. Note that _j∈J∪{l}t_j=_j∈J t_j and _j∈J∪{l}t_j′=_∈Jt_j′, so τ⊕_J∪{l}τ′<:τ⊕_Jτ′.
  • 4. The argument is similar to 2.
  • 5. Writing J′ for J ∪ {j′}, we see that the partitioning will be the same for J and J′. Now consider the ith component of τ _Jτ′. If i˜i′ for some i′ ∈ J, then i˜j′, so we have
    (τ_Jτ′)_i=(_j∈Jt_{j j∈J}t′_j) {t_k t′_k|k˜i, k∉J}=
    (_j∈Jt_{j j∈J}t′_j) (t_j′ t′_j′) {t_k t′_k|k˜i, k∉J′}=
    (t′_{j′ j∈J′}t_j t_{j′ j∈J′} t′_j) {t_k t′_k|k˜i, k∉J′}<:(_j∈J′ t_{j j∈J′} t′_j) {t_k t′_k|k˜i, k∈J′}=(τ⊕_J′ τ′)_i

Otherwise we have (τ_Jτ′)_i=(τ_J′τ′), so overall τ_Jτ′<:τ_J′τ′.

• • 6. Obvious, since the components of τ⊕_∃τ′ are obtained by forming the meet.

This suggests an improved implementation of Algorithm 2, shown in two parts as Algorithm 3 and Algorithm 4, which straightforwardly exploit the above properties to avoid performing useless consensus operations whose result would be discarded by the max_⊥ operator anyway. In the latter, we make use of an operation PART(ν, i) which returns the equivalence partition index i belongs to in TTC ν.

To show that these improvements make our type inference algorithm feasible in practice, we measure its performance on some typical programs from our intended application domain. The Datalog programs to be typed arise as an intermediate representation of programs written in a high-level object-oriented query language named .QL [14, 15], which are optimised and then compiled to one of several low-level representations for execution.

We measure the time it takes to infer types for the 89 queries in the so-called “Full Analysis” that ships with our source code analysis tool SemmleCode. These queries provide statistics and overview data on structural properties of the analysed code base, compute code metrics, and check for common mistakes or dubious coding practices.

A type inference for the query being compiled is performed at three different points during the optimisation process, giving a total of 267 time measurements. All times were measured on a machine running a Java 1.6 virtual machine under Linux 2.6.28-13 on an Intel Core2 Duo at 2.1 GHz with four gigabytes of RAM, averaging over ten runs, and for each value discarding the lowest and highest reading.

Of the 267 calls to type inference, 65% finish in 0.5 seconds or less, 71% take no more than one second, 93% no more than two, and in 98% of the cases type inference is done in under three seconds. Only two type inferences take longer than four seconds, at around 4.5 seconds each.

The size of the programs for which types are inferred varies greatly, with most programs containing between 500 and 1500 subterms, but some programs are significantly larger at more than 3000 subterms. Interestingly, program size and type inference time are only very weakly correlated (p<0.4), and the correlation between the number of stratification layers (which roughly corresponds to the number of fixpoint computations to be done) and type inference time is also not convincing (ρ<0.6).

The low correlation is evident in FIG. 6 which shows the type inference time plotted on the y-axis versus the depth (i.e., number of layers) of the program on the x-axis. In particular, the two cases in which type inference takes longer than four seconds are both of medium size and not among the deepest either.

This suggests that the asymptotic behaviour of the algorithm in terms of input size is masked by the strong influence of implementation details (at least for the kind of programs we would expect to deal with in our applications), and we can expect significant performance gains from fine tuning the implementation.

Our experiments also show that saturation, while extremely expensive in theory, is quite well-behaved in practice: in 78% of all cases, no new TTCs are added during the first iteration of Algorithm 1, so the typing is already saturated. In another 17% of all cases we need only one more iteration, and we almost never (≈0.01%) need to do more than four iterations.

Although in every individual iteration we potentially have to take the consensus over every combination of columns, our heuristics manage to exclude all combinations of more than one column in 94% of all cases, and sometimes (14%) can even show that no consensus formation is needed at all.

Since our type inference algorithm makes heavy use of BDDs, some statistics about their usage may also be of interest. We use about 300 BDD variables (one for every type symbol), with most of the BDDs constructed during type checking being of rather moderate size: the BDD to represent the type hierarchy (which encodes about 800 constraints automatically derived during translation from .QL to Datalog) occupies around 4000 nodes, while the BDDs for individual type propositions never take more than 100 nodes.

While we have anecdotal evidence to show that the optimisation techniques we have presented in earlier work [13] benefit greatly from combining them with the richer type hierarchies our type system supports, we leave the precise investigation of this matter to future work.

6. Benefits

The present invention handles negated types accurately. Also, the present invention handles inhabitation constraints (i.e., existentials in type programs) precisely.

We achieve pleasing theoretical properties and can support a rich language of type constraints by sacrificing polynomial time guarantees, although our experiments show that this is a reasonable tradeoff for our application area.

While the performance of our prototype implementation is promising, other implementation approaches certainly exist and may be worth exploring. Bachmair et al. [4] develop a decision procedure for monadic first order logic based on the superposition calculus. Since type programs can readily be expressed as monadic first order formulae, their algorithm could be used to decide type containment. Another possibility would be to use Ackermann's translation from monadic first order logic to equality logic [2], and then employ a decision procedure for this latter logic.

7. Review of High Level Flow Diagrams

FIG. 7 is a block diagram illustrating how types of database queries are inferred. Type inferencer 701 receives database schema 703 and query 704 and infers types from definitions in the program by replacing each use of a database relation in the program by the types in the database schema, resulting in type program 706 as described in the section above entitled “2. Type Programs.” The type inferencer furthermore receives type hierarchy 705 to check types in the type program for inclusion as described in the section above entitled “4. Containment Checking,” generating information about type inclusion 708. The type inferencer also uses the type hierarchy to generate emptiness information 710 indicating which types in the type program are empty as described in the section above entitled “2. Type Programs.” In one embodiment the inclusion and emptiness information is used by query optimizer 711 to generate an optimized version of the query that can be run on query engine 712 to query relational data source 713, as described in the section above entitled “1. Introduction,” subsection “Overview.”

FIG. 8 is a flow chart illustrating inclusion testing for types represented by sets of type tuple constraints (TTCs). To test inclusion of a type represented by a set S1 of TTCs 801 in a type represented by a set S2 of TTCs 802, set S2 is first saturated 803 by adding all prime implicants of all TTCs in S2 to the set, yielding a larger set S2′ 804, as described in the section above entitled “4. Containment Checking.” All TTCs in S1 are then examined in turn 805, and for every TTC 807 in S1 it is checked whether there is a larger, covering TTC in S2′ 808, as described in the section above entitled “4. Containment Checking,” subsection “Definition 7.” If no such larger TTC is found 809, the type represented by S1 is not included in S2. Otherwise the TTC is removed from S1 810 and the next TTC is examined. If a covering TTC is found for every TTC in S1 806, then the type represented by S1 is included in the type represented by S2.

FIG. 9 is a flow chart illustrating saturation of sets of type tuple constraints (TTCs) for inclusion testing. Starting with a given set S of TTCs 901, every pair of TTCs from S is considered in turn 902, and a consensus of the two TTCs is computed 903, as described in the section above entitled “3. Representing Type Programs,” subsection “Definition 4.” If S does not contain a TTC larger than the consensus 904, it is added to the set S 905. This is repeated until all pairs of TTCs have been considered 907.

8. Conclusion

We have presented a type inference procedure that assigns an upper envelope to each Datalog program. That envelope is itself a Datalog program that makes calls to monadic extensionals only. The algorithm is able to cope with complex type hierarchies, which may include statements of implication, equivalence and disjointness of entity types.

The type inference procedure is itself an extremely simple syntactic mapping. The hard work goes into an efficient method of checking containment between type programs. We achieve this via a novel adaption of Quine's algorithm for the computation of the prime implicants of a logical formula. Generalising from logical formulae to type programs, we bring types into a saturated form on which containment is easily checked.

As shown by our experiments, the algorithm for inferring a type and saturating it works well on practical examples. While it may still exhibit exponential behaviour in the worst case, such extreme cases do not seem to arise in our application area. Thus our algorithm is a marked improvement over well-known simpler constructions that always require an exponential overhead.

Many avenues for further work remain. Perhaps the most challenging of these is the production of good error messages when type errors are identified: this requires printing the Boolean formulae represented via TTCs in legible form. We have made some progress on this, employing Coudert et al.'s restrict operator on BDDs, which is another application of prime implicants [12].

There is also substantial further engineering work to be done in the implementation. Careful inspection of the statistics show that our use of BDDs is very much unlike their use in typical model checking applications [26], and we believe this could be exploited in the use of a specialised BDD package. For now we are using JavaBDD [25], which is a literal translation of a C-based BDD package into Java.

Finally, we will need to investigate how to best exploit the advanced features offered by our type system. In particular, much experience remains to be gained in how to make it easy and natural for the programmer to specify the constraints making up the type hierarchy, and which kinds of constraints benefit which kinds of programs.

9. Non-Limiting Hardware Examples

Overall, the present invention can be realized in hardware or a combination of hardware and software. The processing system according to a preferred embodiment of the present invention can be realized in a centralized fashion in one computer system, or in a distributed fashion where different elements are spread across several interconnected computer systems and image acquisition sub-systems. Any kind of computer system—or other apparatus adapted for carrying out the methods described herein—is suited. A typical combination of hardware and software is a general-purpose computer system with a computer program that, when loaded and executed, controls the computer system such that it carries out the methods described herein.

An embodiment of the processing portion of the present invention can also be embedded in a computer program product, which comprises all the features enabling the implementation of the methods described herein, and which—when loaded in a computer system—is able to carry out these methods. Computer program means or computer programs in the present context mean any expression, in any language, code or notation, of a set of instructions intended to cause a system having an information processing capability to perform a particular function either directly or after either or both of the following a) conversion to another language, code or, notation; and b) reproduction in a different material form.

FIG. 10 is a block diagram of a computer system useful for implementing the software steps of the present invention. Computer system 1000 includes a display interface 1008 that forwards graphics, text, and other data from the communication infrastructure 1002 (or from a frame buffer not shown) for display on the display unit 1010. Computer system 1000 also includes a main memory 1006, preferably random access memory (RAM), and optionally includes a secondary memory 1012. The secondary memory 1012 includes, for example, a hard disk drive 1014 and/or a removable storage drive 1016, representing a floppy disk drive, a magnetic tape drive, an optical disk drive, etc. The removable storage drive 1016 reads from and/or writes to a removable storage unit 1018 in a manner well known to those having ordinary skill in the art. Removable storage unit 1018, represents a CD, DVD, magnetic tape, optical disk, etc. which is read by and written to by removable storage drive 1016. As will be appreciated, the removable storage unit 1018 includes a computer usable storage medium having stored therein computer software and/or data. The terms “computer program medium,” “computer usable medium,” and “computer readable medium” are used to generally refer to media such as main memory 1006 and secondary memory 1012, removable storage drive 1016, a hard disk installed in hard disk drive 1014, and signals.

Computer system 1000 also optionally includes a communications interface 1024. Communications interface 1024 allows software and data to be transferred between computer system 1000 and external devices. Examples of communications interface 1024 include a modem, a network interface (such as an Ethernet card), a communications port, a PCM-CIA slot and card, etc. Software and data transferred via communications interface 1024 are in the form of signals which may be, for example, electronic, electromagnetic, optical, or other signals capable of being received by communications interface 1024. These signals are provided to communications interface 1024 via a communications path (i.e., channel) 1026. This channel 1026 carries signals and is implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, and/or other communications channels.

Although specific embodiments of the invention have been disclosed, those having ordinary skill in the art will understand that changes can be made to the specific embodiments without departing from the spirit and scope of the invention. The scope of the invention is not to be restricted, therefore, to the specific embodiments. Furthermore, it is intended that the appended claims cover any and all such applications, modifications, and embodiments within the scope of the present invention.

REFERENCES

Each of the following twenty-seven references are hereby incorporated by reference in their entirety.

• [1] Serge Abiteboul, Georg Lausen, Heinz Uphoff, and Emmanuel Waller. Methods and rules. In ACM SIGMOD International Conference on Management of Data, pages 32-41. ACM Press, 1993.
• [2] Wilhelm Ackermann. Solvable Cases of the Decision Problem. North-Holland Publishing Company, Amsterdam, 1954.
• [3] Leo Bachmair, Harald Ganzinger, and Uwe Waldmann. Set constraints are the monadic class. In Logic in Computer Science, pages 75-83, 1993.
• [4] Leo Bachmair, Harald Ganzinger, and Uwe Waldmann. Superposition with Simplification as a Decision Procedure for the Monadic Class with Equality. In KGC '93: Proceedings of the Third Kurt Gödel Colloquium on Computational Logic and Proof Theory, pages 83-96, London, UK, 1993. Springer-Verlag.
• [5] Sacha Berger, Emmanuel Coquery, Wlodzimierz Drabent, and Artur Wilk. Descriptive Typing Rules for Xcerpt and their soundness. In Francois Fages and Sylvain Soliman, editors, Principles and Practice of Semantic Web Reasoning, volume 3703 of LNCS, pages 85-100. Springer, 2005.
• [6] George S. Boolos and Richard C. Jeffrey. Computability and Logic. Cambridge University Press, 3rd edition, 1989.
• [7] Ashok K. Chandra and Philip M. Merlin. Optimal Implementation of Conjunctive Queries in Relational Data Bases. In STOC '77: Proceedings of the ninth annual ACM symposium on Theory of computing, pages 77-90, New York, N.Y., USA, 1977. ACM.
• [8] Surajit Chaudhuri. Finding nonrecursive envelopes for Datalog predicates. In Principles of Database Systems (PODS), pages 135-146, 1993.
• [9] Surajit Chaudhuri and Phokion G. Kolaitis. Can Datalog be approximated? In Principles of Database Systems (PODS), pages 86-96, 1994.
• [10] Kevin J. Compton. Stratified least fixpoint logic. Theoretical Computer Science, 131(1):95-120, 1994.
• [11] Stavros Cosmadakis, Haim Gaifman, Paris Kanellakis, and Moshe Vardi. Decidable Optimization Problems for Database Logic Programs. In Proceedings of the 20th annual ACM Symposium on Computing, pages 477-490, 1988.
• [12] Olivier Coudert, Christian Berthet, and Jean Christophe Madre. Verification of synchronous sequential machines based on symbolic execution. In Automatic Verification Methods for Finite State Systems, volume 407 of Lecture Notes in Computer Science, pages 365-373. Springer, 1989.
• [13] Oege de Moor, Damien Sereni, Pavel Avgustinov, and Mathieu Verbaere. Type Inference for Datalog and Its Application to Query Optimisation. In PODS '08: Proceedings of the twenty-seventh ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pages 291-300, New York, N.Y., USA, 2008. ACM.
• [14] Oege de Moor, Damien Sereni, Mathieu Verbaere, Elnar Hajiyev, Pavel Avgustinov, Torbjörn Ekman, Neil Ongkingco, and Julian Tibble. .QL: Object-oriented queries made easy. In Ralf Lämmel, João Saraiva, and Joost Visser, editors, Generative and Transformational Techniques in Software Engineering, LNCS. Springer, 2007.
• [15] Oege de Moor, Mathieu Verbaere, Elnar Hajiyev, Pavel Avgustinov, Torbjörn Ekman, Neil Ongkingco, Damien Sereni, and Julian Tibble. .QL for source code analysis. In Source Code Analysis and Manipulation (SCAM '07), pages 3-16. IEEE, 2007.
• [16] Thom W. Frühwirth, Ehud Y. Shapiro, Moshe Y. Vardi, and Eyal Yardeni. Logic programs as types for logic programs. In Logic in Computer Science (LICS), pages 300-309. IEEE Computer Society, 1991.
• [17] John P. Gallagher, Kim S. Henriksen, and Gourinath Banda. Techniques for Scaling Up Analyses Based on Pre-Interpretations. In M. Gabbrielli and G. Gupta, editors, International Conference on Logic Programming (ICLP '05), volume 3668 of LNCS, pages 280-296. Springer, 2005.
• [18] John P. Gallagher and Germán Puebla. Abstract Interpretation over Non-deterministic Finite Tree Automata for Set-Based Analysis of Logic Programs. In Practical Aspects of Declarative Languages (PADL), volume 2257 of LNCS, pages 243-261. Springer, 2002.
• [19] Nevin C. Heintze and Joxan Jaffar. A finite presentation theorem for approximating logic programs. In Symposium on Principles of Programming Languages (POPL), pages 197-209, 1990.
• [20] David Hilbert and Wilhelm Ackermann. Grundzüge der Theoretischen Logik. Julius Springer, Berlin, 1928.
• [21] Kim Marriott, Harald Søndergaard, and Neil D. Jones. Denotational Abstract Interpretation of Logic Programs. ACM Transactions on Programming Languages and Systems, 16(3):607-648, 1994.
• [22] Willard V. Quine. On Cores and Prime Implicants of Truth Functions. The American Mathematical Monthly, 66(9):755-760, Nov. 1959.
• [23] Oded Shmueli. Equivalence of Datalog Queries is Undecidable. Journal of Logic Programming, 15(3):231-241, 1993.
• [24] Fang Wei and Georg Lausen. Containment of Conjunctive Queries with Safe Negation. In ICDT '03: Proceedings of the 9th International Conference on Database Theory, pages 346-360, London, UK, 2002. Springer-Verlag.
• [25] John Whaley. JavaBDD. http://javabdd.sourceforge.net/, 2007.
• [26] Bwolen Yang, Randal E. Bryant, David R. O'Halloran, Armin Biere, Olivier Coudert, Geert Janssen, Rajeev K. Ranjan, and Fabio Somenzi. A Performance Study of BDD-Based Model Checking. In 2nd International Conference on Formal Methods in Computer-Aided Design, volume 1522 of Lecture Notes in Computer Science, pages 255-289. Springer, 1998.
• [27] David Zook, Emir Pasalic, and Beata Sarna-Starosta. Typed Datalog. In Practical Aspects of Declarative Languages, pages 168-182. Springer Berlin/Heidelberg, 2009.

Claims

1. A computer-implemented method comprising:

accessing a query program with, the query program containing one or more queries to a database that contains relations described by at least one a database schema, wherein the database schema specifies a column type for every column of every extensional predicate that occurs in the query program;

receiving the database schema and at least one an entity type hierarchy for the database; and

inferring a first type program from definitions in the query program by replacing each use of a database relation an extensional predicate in the query program by its type in the conjunction of the column types of the columns of the extensional predicate, the column types being provided by the database schema, and the wherein each column type is at least portion of a respective second type program,

wherein each of the first type program and the second type program is a derived program using type symbols without other extensional relation symbols, and each of the first type program and the second type program do not contain negation,

wherein the first type program and the second type program uses programs use only extensionals that are monadic extensions, and the first type program and the second type programs do not contain negation;

testing portion portions of the first type program that has been inferred for type emptiness and type inclusion; and

finding type emptiness or type inclusion for a first portion of the first type program, wherein the first portion is also a type program; and

providing at least one of error information and or optimization information regarding the type emptiness and or type inclusion being found for the type first portion of the first type program.

2. The computer-implemented method of claim 1, further comprising:

in response to type emptiness being found for a the first portion of the first type program that has been inferred during determining of the testing portion of portions of the first type program, performing at least one of:

removing the a type test from the query program a first portion of the query program that corresponds to the first portion of the first type program;

providing a notification regarding the emptiness found for the first portion of the first type program; or

providing notification on combining queries in the query program by conjunction without creating empty parts in a combined query;

in response to an empty part of a query being a conjunction, finding a smallest set of query parts that has a conjunction that is empty;

in response to a search for a smallest empty part of a database query that traverses all parts of the database query, pushing a conjunction of an approximation of the smallest empty part and a context on top of a stack of approximations of all contexts where the empty part is being used; and

eliminating empty query parts to achieve virtual method resolution in an object-oriented query language.

3. The computer-implemented method of claim 1, further comprising:

in response to type inclusion being found for a second portion of the first type program that has been inferred during determining of the testing portion of portions of the first type program, performing at least one of:

removing the a type test from the query program a second portion of the query program that corresponds to the second portion of the first type program; and or

providing a notification regarding the type inclusion being found for the second portion of the first type program.

4. The computer-implemented method of claim 1, further comprising wherein:

inferring the first type program includes inferring an inferred type for an intensional predicate in the query program that has a declared type in the query program; and

the method further comprises:

testing the type program that has been inferred for whether a database query is contained in a given portion of the type program; and in response to the database query not being contained in the given portion, performing at least one of:

removing a type test; and

determining that the inferred type does not contain the declared type; and

providing a notification regarding that the database query declared type is not being contained in the given portion inferred type.

5. The computer-implemented method of claim 1, wherein portions of the first type program that has been inferred are represented by a set respective sets of type tuple constraints (TTCs), wherein each of the TTCs includes:

a tuple of type propositions;

an equivalence relation between tuple components, each tuple component being a type proposition; and

a set of inhabitation constraints.

6. The computer-implemented method of claim 5, further comprising:

checking inclusion of a third portion of the first type program that has been inferred represented by a first set of TTCs into another, fourth portion of the first type program represented by a second set of TTCs by:

computing a set of prime implicants of the second set; and

checking each TTC in the first set and finding determining whether a larger TTC exists in the set of prime implicants of the second set of TTCs; and

determining that the third portion is included in the fourth portion if for each TTC in the first set a larger TTC exists in the set of prime implicants of the second set of TTCs.

7. The computer-implemented method of claim 6, further comprising:

computing the set of prime implicants of the second set of TTCs by saturating the second set by exhaustively applying consensus operations to ensure all relevant TTCs are included.

8. The computer-implemented method of claim 7, wherein each of the consensus operations are performed by includes generating a new TTC from two existing TTCs by

receiving two TTCs as operands two TTCs, the tuple of type propositions of each TTC having the same number of type propositions;

receiving a set of indices and, each index in the set identifying a type proposition in a tuple of type propositions by the position of the type proposition in the tuple;

equating in each operand TTC all columns whose indices type propositions at index positions that occur in the set of indices, and preserving all other equalities and inhabitation constraints of the operands each TTC operand, and then for each index position taking disjunctions over columns type propositions indexed in the set of indices and conjunctions over all other columns type propositions as the type propositions of the tuple of type propositions of the new TTC; and

taking generating a union of two or more of the equivalence relations in of the operands operand TTCs as the equivalence relation between tuple components of the new TTC; and

generating a pointwise disjunction of the inhabitation constraints of the operands operand TTCs as the inhabitation constraints of the new TTC.

9. The computer-implemented method of claim 8, further comprising:

reducing a number of consensus operations that need to be performed during saturation by omitting consensus operations where a resulting TTC will be covered by TTCs already present in the set, wherein a TTC τ is covered by a TTC τ′ if τ<:τ′.

10. The computer-implemented method of claim 8, further comprising:

receiving a logical formula that represents a type hierarchy, and representing a component type proposition of a TTC as a binary decision diagram (BDDs BDD), and choosing a BDD variable order by assigning neighboring indices of the set of indices to type symbols that appear in a same conjunct of the logical formula that represents the type hierarchy.

11. The computer-implemented method of claim 1, wherein further comprising:

using the first type program as an approximation is used to find erroneous parts of a database query in the query program that will return an empty set of results, regardless of contents stored in the database.

12. The computer-implemented method of claim 2, wherein the providing notification on combining queries in the query program by conjunction without creating empty parts in a combined query includes depicting compatible types with similar pictures in a user interface.

13. The computer-implemented method of claim 1, wherein the one or more queries to the database contain calls to other query procedures, and the method further comprises:

using the first type program as an approximation is used to optimize these called the query procedures, by and eliminating query parts from the query procedures that will return an empty set of results in a context where they are called, regardless of any contents of the database.

14. The computer-implemented method of claim 13, wherein the context of a procedure in a database query is computed by comprising:

computing the context of each query procedure called by a query program by

traversing a call graph of that query program, and

keeping a stack of approximations of all contexts where the query procedure is being used, and

when entering a procedure in the call graph, pushing a conjunction of an approximation of a body of that procedure and a top of the stack onto the stack as a new context.

15. The computer-implemented method of claim 1, wherein further comprising:

using the first type program as an approximation is used to optimize queries, in the query program by eliminating from the query program query parts that test whether a value is included in a portion of the first type program has a particular declared type in the query program, and the approximation first type program indicates at least one of:

the value will be included in this portion of the first type program have the particular declared type regardless of contents stored in the database; and

the value will not be included in the portion of the first type program have the particular declared type regardless of contents stored in the database.

16. A system comprising:

a computer system comprising one or more computers each having a memory; and a processor communicatively coupled to the memory; and

a type inferencer communicatively coupled to the memory and the processor computer program loaded in the computer system, wherein the type inferencer is adapted to cause the computer system perform the operations of:

accessing a query program with, the query program containing one or more query operations to a database that contains relations described by at least one a database schema, wherein the database schema specifies a column type for every column of every extensional predicate that occurs in the query program;

receiving the database schema and at least one an entity type hierarchy for the database; and

inferring a first type program from definitions in the query program by replacing each use of a database relation an extensional predicate in the query program by its type in the conjunction of the column types of the columns of the extensional predicate, the column types being provided by the database schema, and the wherein each column type is at least portion of a respective second type program,

wherein each of the first type program and the second type program is a derived program using type symbols without other extensional relation symbols, and each of the first type program and the second type program do not contain negation,

wherein the first type program and the second type program uses programs use only extensionals that are monadic extensions, and the first type program and the second type programs do not contain negation;

testing portion portions of the first type program that has been inferred for type emptiness and type inclusion; and

finding type emptiness or type inclusion for a first portion of the first type program, wherein the first portion is also a type program; and

providing at least one of error information and optimization information regarding the type emptiness and or type inclusion being found for the type first portion of the first type program.

17. The system of claim 16, wherein the type inferencer is further adapted to perform the operations of:

in response to type emptiness being found for a the first portion of the first type program that has been inferred during determining of the testing portion of portions of the first type program, performing at least one of:

removing a type test from the query program a first portion of the query program that corresponds to the first portion of the first type program;

providing a notification regarding the emptiness found for the first portion of the first type program; or

providing notification on combining queries in the query program by conjunction without creating empty parts in a combined query;

in response to an empty part of a query being a conjunction, finding a smallest set of query parts that has a conjunction that is empty;

in response to a search for a smallest empty part of a database query that traverses all parts of the database query, pushing a conjunction of an approximation of the smallest empty part and a context on top of a stack of approximations of all contexts where the empty part is being used; and

eliminating empty query parts to achieve virtual method resolution in an object-oriented query language.

18. The system of claim 16, wherein the type inferencer is further adapted to perform the operations of:

in response to type inclusion being found for a second portion of the first type program that has been inferred during determining of the testing portion of portions of the first type program, performing at least one of:

removing a type test from the query program a second portion of the query program that corresponds to the second portion of the first type program; and or

providing a notification regarding the type inclusion being found for the second portion of the first type program.

19. A non-transitory computer program product, the computer program product comprising instructions for to cause a computer system to perform the operations of:

accessing a query program with, the query program containing one or more query operations to a database that contains relations described by at least one a database schema, wherein the database schema specifies a column type for every column of every extensional predicate that occurs in the query program;

receiving the database schema and at least one an entity type hierarchy for the database; and

inferring a first type program from definitions in the query program by replacing each use of a database relation an extensional predicate in the query program by its type in the conjunction of the column types of the columns of the extensional predicate, the column types being provided by the database schema, and the wherein each column type is at least portion of a respective second type program,

wherein each of the first type program and the second type program is a derived program using type symbols without other extensional relation symbols, and each of the first type program and the second type program do not contain negation,

wherein the first type program and the second type program uses programs use only extensionals that are monadic extensions, and the first type program and the second type programs do not contain negation;

testing portion portions of the first type program that has been inferred for type emptiness and type inclusion; and

finding type emptiness or type inclusion for a first portion of the first type program, wherein the first portion is also a type program; and

providing at least one of error information and optimization information regarding the type emptiness and or type inclusion being found for the type first portion of the first type program.

20. The non-transitory computer program product of claim 19, further comprising instructions to cause the computer system to perform the operations of:

in response to type emptiness being found for a the first portion of the first type program that has been inferred during determining of the testing portion of portions of the first type program, performing at least one of:

removing a type test from the query program a first portion of the query program that corresponds to the first portion of the first type program;

providing a notification regarding the emptiness found for the first portion of the first type program; or

providing notification on combining queries in the query program by conjunction without creating empty parts in a combined query;

in response to an empty part of a query being a conjunction, finding a smallest set of query parts that has a conjunction that is itself empty;

in response to a search for a smallest empty part of a database query that traverses all parts of the database query, pushing a conjunction of an approximation of that part and a context on top of a stack; and

eliminating empty query parts to achieve virtual method resolution in an object-oriented query language.

21. The computer-implemented method of claim 1, wherein:

each of the column types is a monadic type found in a set of type symbols.

22. The computer-implemented method of claim 2, further comprising:

in response to finding that the first portion of the query program is a conjunction, finding a smallest set of query parts in the first portion of the query program that has a conjunction that is empty, and providing a notification regarding the emptiness found for the smallest set of query parts that has a conjunction that is empty.