Sorted dict

[amascolo]

Support for sorting and hybrid queries (i.e. using SortedDict and Range). Includes all queries ported over to SDQL.

[amascolo]

Details that came up during implementation:

1. added a Timer expression in SDQL to mark where the timer should start – since we need to exclude the initial data sorting time in benchmarks (nice aside is that now we don't need to treat load expressions as a special case)

2. added external function SortedIndices - to sort the initial data

3. added external function SortedVec - to sort the tries

4. special case to call emplace_back - when constructing @vec { <...> -> 1 } we interpret the relational form <...> -> 1 as i -> <...> and append elements to a std::vec

5. special case to call at - since in SortedDict we can't use [ ] which has special semantics for insertion (this wasn't an issue previously for vecdict since after construction we only iterated on it – i.e. set semantics – whereas here we access elements)

6. cache the last found element – to avoid calling SortedDict::find twice in situations where we check with contains then retrieve the element using at (in handwritten C++ code we could assign the result of find – i.e. an iterator - to some variable, but this isn't representable in SDQL)

[amascolo]

Have ported query FJ 3a to SDQL and checked its performance is identical to handwritten C++.

Codegen and runtime should work everywhere, just need to generate all the other queries in SDQL.

[amirsh]

Unclear why you need to return this type? Is it because later on you are looking up over it based on the index? Otherwise, this makes the type system very complicated.

[amascolo]

Yes. It's used by the "pure" sorting FJ queries in progs/job/sorting/fj.

For example, in query 3a the last line interm0(i).col0 is a lookup by index:

let interm0_unsort = sum(<mk_off, _> <- range(mk.size))
  let x0 = mk.movie_id(mk_off) in
    if (x0 ∈ t_trie0) then
      let t_trie1 = t_trie0(x0) in
      sum(<t_i, _> <- t_trie1)
        let t_off = t_offsets(t_i)
        @vec { <col0=mk.movie_id(mk_off), col1=mk.keyword_id(mk_off), col2=t.title(t_off)> -> 1 }

let interm0 = ext(`SortedVec`, 0, interm0_unsort)

let interm0_trie0 = sum(<i, _> <- range(ext(`Size`, interm0)))
  @st(ext(`Size`, interm0)) { interm0(i).col0 -> @range { i -> 1 } }

[amirsh]

What does exp contain here? The entire program that will be timed?

[amascolo]

exp contains the entire program after where the timer expression appears, which is what we want to time since we must exclude the initial load and sorting expressions.

This is simplest for now. In future, we could add more fine-grained control of the timer, e.g. timer_start / timer_elapsed / timer_stop to time different subsections of the program.

[amascolo]

@amirsh this PR is now ready to merge, subject to your approval:

• Supports all sort-based and hybrid JOB queries in SDQL ✅
• Performance within 2% of C++ (no regression on hash-based queries)