Reading the diagrams

This is an operator reference for this fork of Trill, in the style of the ReactiveX operator pages, with interactive diagrams in the manner of RxMarbles.

Every diagram is computed by Trill itself. The page first draws a recorded run of the query, and then loads Trill into your browser (row mode, under .NET WebAssembly). From then on, dragging an input event re-runs the query and redraws the output. The docs/site/DiagramTests project checks every recorded run against the engine, and a browser check requires each page’s live run to match its recording. Those checks cover the data a diagram draws; the drawing itself is checked only for clipping, so report anything that looks wrong.

Events have lifetimes

A Reactive Extensions marble is a point in time. A Trill event is not: it has a lifetime, a half-open interval [start, end) of application time, and the event is “alive” for that whole interval. Operators such as Join, Union and the windows work on how lifetimes overlap, so the diagrams draw events as bars from start to end, except for one-tick inputs, which are circles.

Bar: an event with lifetime [start, end). Its label is key:value.
Circle: an input event whose lifetime is a single tick, [t, t+1). Trill’s StreamEvent.CreatePoint makes these.
Colour: the event’s key. Joins, grouping and patterns match on keys, so colour shows which events can meet. A legend under each diagram names the keys, and every bar carries its key in its label too.

Hover over any event to see its exact lifetime.

How Trill emits an event: intervals and edges

Trill can deliver an output event in two ways, and the diagram shows which one it used:

Interval (rounded bar): the whole lifetime arrives in one message.
Start edge and end edge (square bar with black ticks): Trill first emits a start edge, an event whose end is not known yet, and later an end edge that closes it. The diagram pairs the two into one bar, with a tick at each end. A start edge with no end edge yet is drawn running off the right with →∞.

The logical lifetime is the same either way. The difference matters to code that consumes Trill’s output through ToStreamEventObservable, which sees the edges as separate messages. Operators that cannot know an output’s end when its start is decided, such as ClipEventDuration, LeftOuterJoin and WhereNotExists, emit edges.

Interacting

Drag an input event left or right to move it.
Drag a bar’s right end to make it longer or shorter. Circles (one-tick inputs) can only be moved.
The axis stays fixed while you drag, so the event stays under the pointer. If an output then lies past the right of the axis, the axis widens to show it when you let go.
The status line under the diagram says whether you are looking at the recorded run or live Trill, and how long the last run took.

The inputs use small integer times so that the diagrams stay readable. Trill’s times are long ticks, and nothing here depends on their unit.

Operators by category

Each operator has a page with an interactive diagram computed by Trill. It covers filtering and projection, joins, pattern detection, windows and aggregates, and the operators they build on.

Filtering and projection

Operator	What it does
Where	Keeps the events whose payload satisfies a predicate.
Select	Transforms each event’s payload; lifetimes are unchanged.
Select (with start time)	Transforms each event’s payload using its start time as well.
SelectMany	Turns each event into zero or more events with the same lifetime.
SelectByKey	Transforms each event’s payload using the stream’s key as well.
Distinct	Removes duplicates among the events alive at each moment.

Lifetimes and windows

Operator	What it does
AlterEventDuration	Gives every event a new duration, keeping its start time.
ClipEventDuration	Ends each left event when a right event with the same key starts.
TumblingWindowLifetime	Moves each event into a fixed, non-overlapping window, ready for an aggregate.
HoppingWindowLifetime	Moves each event into overlapping windows of a fixed size that start every period.

Combining streams

Operator	What it does
Union	Merges two streams of the same type into one.
Multicast	Lets one query use the same input stream more than once.

Joins

Operator	What it does
Join	Pairs left and right events with equal keys whose lifetimes overlap; each pair lives for the overlap.
LeftOuterJoin	Like Join, but a left event also appears, through a separate selector, for the times when nothing matches it.
WhereNotExists	Keeps each left event only for the times when no right event with the same key is alive.

Aggregates

Operator	What it does
Count (tumbling window)	Counts the events alive at each moment; after a tumbling window, that is a count per window.
Sum (hopping window)	Sums a value over the events alive at each moment; after a hopping window, that is a sum per hop.
GroupApply	Splits a stream by key, runs the same query on each group, and merges the results.

Patterns

Operator	What it does
Detect	Finds sequences of events that match a pattern within a maximum duration.

Where

Keeps the events whose payload satisfies a predicate.

Lifetimes pass through unchanged: an event either appears in the output with its own lifetime, or not at all.

In the diagram, a:1 fails Value > 1 and is dropped; b:2 and a:3 keep their lifetimes [3,9) and [6,8).

See also: Select, Distinct, SelectMany, WhereNotExists

Select

Transforms each event’s payload; lifetimes are unchanged.

Each input event produces exactly one output event with the same lifetime and a new payload.

In the diagram every value is multiplied by ten. Move an input event and its output moves with it.

Three more overloads build a payload of a new type from the old one without listing every field. You pass the new type’s constructor, such as () => new Reading(), and then either one or two pairs of a field of the new type and an expression for its value, or a dictionary from field name to expression. Public fields and properties with the same name in both types are copied, and the named fields are computed.

Select (with start time)

Transforms each event’s payload using its start time as well; lifetimes are unchanged.

This overload of Select takes a selector of the form (start, payload) => result, where start is the event’s start time, a long. Like the plain Select, it changes only the payload: each output event keeps its input’s lifetime.

In the diagram, each label gains the time its event starts: a:1 starts at 1 and becomes 1 at 1. Drag an input event and its label follows its new start time.

SelectMany and SelectByKey have the same form, with the start time as the first parameter.

See also: Select, SelectMany, SelectByKey

SelectMany

Turns each event into zero or more events, each with the input event’s lifetime.

The selector returns a sequence of payloads for each input payload. Each payload in the sequence becomes an output event with the input event’s lifetime. An empty sequence drops the event, so SelectMany can filter as well as expand.

In the diagram, each event becomes as many events as its value: a:1 gives one, b:2 two and a:3 three, all with their input’s lifetime.

An overload takes (start, payload), as Select (with start time) does. On a grouped stream, SelectManyByKey also passes the grouping key, as SelectByKey does.

See also: Select, Where, SelectByKey

SelectByKey

Transforms each event’s payload using the stream’s key as well; lifetimes are unchanged.

Every Trill stream has a key type, the TKey in IStreamable<TKey, TPayload>, and each event carries a key of that type alongside its payload. SelectByKey passes that key to the selector: (key, payload) => result. On a stream from ToStreamable() the key type is Empty, so the key says nothing. Inside a grouping operator such as GroupApply, the key holds the group: there it is a CompoundGroupKey, and its InnerGroup is the group’s key.

The diagram uses GroupApply to make each event’s Key field the stream key that the selector receives. Each output names its group: a:1 becomes 1 in a.

Two related forms take the start time as well, (start, key, payload). SelectManyByKey is the same for SelectMany, with (key, payload) or (start, key, payload).

See also: GroupApply, Select, SelectMany

Distinct

Removes duplicates among the events alive at each moment.

Distinct works on snapshots. At every moment, the output holds one event for each distinct value among the events alive then. With a selector, such as e => e.Key, two events are duplicates when the selector gives the same result for both, and the output payload is that result. Without a selector, Distinct compares whole payloads.

In the diagram, two a events overlap from 3 to 6, but the output has only one a at any moment. That a is split at 3 and at 6, where the number of a events alive changes. Distinct is built from GroupApply and Count, and each change in a group’s count starts a new output event. The output arrives as start and end edges.

See also: Where, GroupApply, Count (tumbling window)

AlterEventDuration

Gives every event a new duration, keeping its start time.

AlterEventDuration(d) sets each event’s lifetime to [start, start + d). It is the usual way to turn one-tick events into intervals, so that later operators such as joins and patterns can see them overlap.

In the diagram each one-tick input becomes a three-tick interval: a:1 at 1 becomes [1,4).

ClipEventDuration

Ends each left event when a right event with the same key starts.

For each left event, the first right event with the same key that starts after the left event’s start cuts it short: the left event ends there. Left events with no such right event keep their lifetime.

In the diagram, a:1 [1,5) is clipped by a:10 at 2 and becomes [1,2); b:2 [3,9) is clipped by b:20 at 4 and becomes [3,4); a:3 [6,8) keeps its lifetime, because the next a on the right starts at 10.

The clipped events are emitted as a start edge and an end edge: when the left event starts, Trill cannot yet know whether it will be clipped. See reading the diagrams.

See also: AlterEventDuration, WhereNotExists

TumblingWindowLifetime

Moves each event into a fixed, non-overlapping window, ready for an aggregate.

TumblingWindowLifetime(size) is HoppingWindowLifetime(size, size). It snaps each event’s start up to the next multiple of size (or keeps it, if it is already on a boundary) and gives it a lifetime of size. An event with start in (k·size − size, k·size] therefore lives in [k·size, k·size + size): everything that happened in one window becomes visible together, when that window closes.

In the diagram (size 4), the events at 1, 2 and 4 all live in [4,8); the event at 7 lives in [8,12) and the one at 9 in [12,16). Note that the event at exactly 4 lands with the events at 1 and 2, not with 7.

Follow it with an aggregate to compute per-window results: see Count.

HoppingWindowLifetime

Moves each event into overlapping windows of a fixed size that start every period.

When size is a multiple of period, as in the diagram, HoppingWindowLifetime(size, period) snaps each event’s start up to the next multiple of period (a start already on a boundary stays) and gives it a lifetime of size. The event then stays alive across several consecutive hops, so an aggregate over the result sees it in each overlapping window.

Two other cases behave differently (see HoppingWindowLifetime in LifetimeExtensions.cs):

When period is larger than size, the windows have gaps between them. Events whose start falls in a gap are dropped. The others are snapped up as above and get a lifetime of size.
When size is larger than period but not a multiple of it, Trill snaps the start and the original start plus size to the period grid separately, so lifetimes can differ from size.

In the diagram (size 4, period 2), a:1 at 1 and b:2 at 2 both live in [2,6), a:3 at 4 in [4,8), b:4 at 7 in [8,12) and a:5 at 9 in [10,14).

Follow it with an aggregate to compute per-window results: see Sum.

Union

Merges two streams of the same type into one.

Every event from both inputs appears in the output with its own lifetime; nothing is matched or deduplicated.

See also: Multicast

Multicast

Lets one query use the same input stream more than once.

Multicast(s => ...) gives the function a shared stream s. Every branch the function builds from s shares one subscription to the source, so the source is read once however many branches use it.

In the diagram, one branch keeps key a and the other keeps values above 2, and the two are unioned. a:3 passes both filters, so it appears twice in the output: Union does not deduplicate.

See also: Union, Where

Join

Pairs left and right events with equal keys whose lifetimes overlap; each pair lives for the overlap.

For every left event and right event with the same join key, if their lifetimes overlap, the output has one event whose lifetime is the intersection of the two, with a payload made by the result selector.

In the diagram, a:1 [1,5) and a:10 [2,4) overlap in [2,4); b:2 [3,9) and b:20 [4,7) overlap in [4,7). a:3 [6,8) has no overlapping a on the right, so it produces nothing. Drag a:30 left until it overlaps a:3 to see a third pair appear.

To join events that happen within some span of each other, first give the inputs that span as their lifetime with AlterEventDuration.

LeftOuterJoin

Like Join, but a left event also appears, through a separate selector, for the times when nothing matches it.

The output covers each left event’s whole lifetime. Where a matching right event overlaps it, the inner selector makes the payload; for the times when no matching right event is alive, the outer selector does. A left event is therefore cut into time slices.

In the diagram, a:1 [1,5) becomes a:1+none [1,2), a:1+10 [2,4) and a:1+none [4,5); b:2 [3,9) becomes [3,4) unmatched, [4,7) matched and [7,9) unmatched; a:3 [6,8) is unmatched throughout.

The first unmatched slice of a:1 and of b:2 is emitted as a start edge and an end edge: when it starts, Trill cannot yet know when a match will arrive.

See also: Join, WhereNotExists

WhereNotExists

Keeps each left event only for the times when no right event with the same key is alive.

An anti-semi-join: the output is the left payload, alive for the parts of its lifetime not covered by any right event with an equal key. Trill has no WhereExists counterpart. A Join that projects the left payload comes close, but it emits one copy of the left event for each overlapping right event with the same key, where a semi-join would emit one.

In the diagram, a:1 [1,5) survives as [1,2) and [4,5) around a:10 [2,4); b:2 [3,9) survives as [3,4) and [7,9) around b:20 [4,7); a:3 [6,8) survives whole.

As with LeftOuterJoin, slices whose end Trill cannot know when they start are emitted as start and end edges.

See also: Join, LeftOuterJoin, ClipEventDuration

Count (tumbling window)

Counts the events alive at each moment; after a tumbling window, that is a count per window.

Trill aggregates are snapshot aggregates: the result at each moment is computed over the events alive at that moment, and a new output event starts whenever that set changes. Windowing operators set lifetimes so that “alive at the same moment” means “in the same window”.

In the diagram, TumblingWindowLifetime(4) puts the events at 1, 2 and 4 in [4,8), so the count is 3 there; the window [8,12) holds one event and [12,16) one.

Sum (hopping window)

Sums a value over the events alive at each moment; after a hopping window, that is a sum per hop.

After HoppingWindowLifetime(4, 2) each event lives for 4 ticks starting on a 2-tick boundary, so consecutive hops overlap and an event counts in two of them.

In the diagram the result is 3 in [2,4) (a:1 and b:2), 6 in [4,6) (a:3 joins them), 3 in [6,8) (only a:3), 4 in [8,10), 9 in [10,12) (b:4 and a:5) and 5 in [12,14).

Each result is emitted as a start edge and an end edge: the aggregate cannot know when the set of alive events will next change.

GroupApply

Splits a stream by key, runs the same query on each group, and merges the results.

GroupApply(keySelector, g => query, resultSelector) runs query once per key and gives the result selector the group key, so each output can say which group it came from. This is how per-key windowed statistics are written.

In the diagram, TumblingWindowLifetime(5) puts the events at 1, 2 and 4 in [5,10) and those at 7 and 9 in [10,15). Per key: a averages 1 and 3 to 2 in [5,10); b has only 2 there; in [10,15) b is 4 and a is 5.

Detect

Finds sequences of events that match a pattern within a maximum duration.

Detect runs an automaton (Trill’s AFA) over the stream. SingleElement(predicate) steps match one event each, in order. A match is reported when its last event arrives, and it lives until the first event of the match plus maxDuration.

In the diagram the pattern is “an a, then a b, within 4”. a at 1 and b at 2 give a match [2,5): it starts at 2, when the b completes it, and ends at 1 + 4. a at 4 and b at 7 give [7,8), ending at 4 + 4. The a at 9 has no b after it.

Drag the b at 7 to 9 and the second match disappears: a match starting with the a at 4 must complete before 4 + 4. Drag it to 10, 11 or 12 and a new match forms instead, with the a at 9. From 13 on, that match has expired too, and only the first match remains.

See also: AlterEventDuration, Join

Row mode and the RowModeReporter hook

Trill runs most operators in columnar mode. It stores each batch of events as one array per field and generates code for the operator that works on those arrays. When an operator cannot do that, it falls back to row mode, which handles one event object at a time. Row mode gives the same results, and it is usually slower.

Fallback is automatic. Before this hook was added, a fallback left a trace in the operator’s ErrorMessages only when code generation failed. The hook reports each fallback as it happens, with its reason.

The diagrams on this site run every query in row mode on purpose, through Config.ForceRowBasedExecution, because the browser cannot compile the generated code. The hook is not called when that setting is on.

Setting the hook

Config.RowModeReporter is a static Action<RowModeReport>. Set it before you build your queries:

Config.RowModeReporter = report =>
    Console.WriteLine($"{report.Operator.GetType().Name}: {report.Reason} ({report.Detail})");

For example, this payload type has a read-only field:

public sealed class Reading { public readonly string Sensor; public int Value; }

Building a query with this payload type reports the ingress with the reason TypeNotColumnar and the detail field Sensor is read-only. Downstream operators generally run in row mode too, without reports of their own, because their input is already in rows.

Some operators directly after the ingress do trigger another report. For an ingress built by calling ToStreamable() on StreamEvents, Trill fuses certain operators directly after it into a new ingress. These are Where, Select and the constant-duration overload AlterEventDuration(n). The new ingress reports the same reason again. The AlterEventDuration overloads that take a function are not fused, and add no report.

What a report contains

Property	What it holds
`Operator`	The operator that falls back: the `IStreamable` instance, or, for egress, the observable the egress method returned. It is typed `object`, so match on its type name.
`Reason`	One of the four reasons below.
`NonColumnarType`	For `TypeNotColumnar`, the type that failed the check. It may be the operator’s input, its output or another type it holds. Otherwise null.
`Detail`	A plain-language reason, or null if the operator recorded none. For `TypeNotColumnar`, the rule the type broke.
`Diagnostics`	Diagnostics recorded during code generation, or null. They include the compiler’s diagnostics if compilation ran.
`Exception`	The exception that stopped code generation, or null.
`AtSubscribe`	True if the fallback decision was made during a subscription that Trill tracks: through an egress method or `Cache()`. False if it was made while building the query or during an untracked subscription. One untracked path is a direct call to `IStreamable.Subscribe`, which is intended for internal use.

report.ToString() gives a one-line summary, such as Row mode: <operator> (TypeNotColumnar): field Sensor is read-only.

The four reasons

Reason	When	Look at
`TypeNotColumnar`	A type the operator would hold in columns cannot be represented as columns.	`NonColumnarType` and `Detail`
`CodeGenerationFailed`	Trill tried to generate the operator’s columnar code and failed.	`Diagnostics` and `Exception`
`PartitionedStream`	The stream is partitioned, and the operator has no columnar form for partitioned streams.	`Detail` is “partitioned streams run in row mode”
`NotSupported`	The operator has no columnar form for this query.	`Detail`, when the operator recorded a reason; otherwise null

Types that cannot be held in columns

For TypeNotColumnar, Detail names the first rule the type breaks. Trill stops at the first failure. It follows the table from top to bottom, treating the three consecutive property checks as a group. For each property, it checks for a missing setter, a non-public getter or setter, and an init-only setter, in that order. It then moves to the next property in reflection order.

`Detail`	Typical cause
`field X has an anonymous type`	A public field whose type is anonymous.
`property X has an anonymous type`	An anonymous type with a property whose type is also anonymous. An anonymous type with no such property passes the check without applying the remaining rules.
`the type is not public`	Generated code must name the type, so it must be visible outside its assembly.
`field X is not public`	A non-public instance field other than an auto-property’s backing field. An F# record stops here, because it keeps its state in fields such as `Key@`.
`field X is read-only`	A `readonly` field, as in a `readonly struct`.
`property X has no setter`	A get-only auto-property.
`property X is not public`	An auto-property whose getter or setter is not public.
`property X has an init-only setter`	A `{ get; init; }` property. A C# positional record stops here.
`the type is abstract or an interface`	A class that is abstract, or an interface.
`the type has no public parameterless constructor`	A class that Trill cannot create.

A struct skips the last two rules. For types other than anonymous types, columnar mode requires a public type that keeps its state in public, settable fields and auto-properties. A class must also be concrete and have a public parameterless constructor.

When the hook is called

On the thread that builds or subscribes to the query, at the moment the operator decides. It is never called while data flows.
Once for each fallback decision, with no deduplication. A query built twice reports twice. Map-reduce builds its query in its constructor and again at Subscribe, and each build reports. Multicast builds its selector once to work out the stream’s properties, and that build reports too.
It is process-wide: it sees reports from every query, and it may be called on several threads at once.
Not at all when Config.ForceRowBasedExecution is set, and not for a code generation failure that throws because fallback to row mode is disabled.
If the hook throws, the exception is caught and counted in Config.RowModeReporterFailures, and the query is built as usual.

Operators that convert between rows and columns on purpose, such as Union, do not report. A binary operator also does not report when it converts one input to rows because the other input is already in rows. The operator upstream that fell back has already reported.

See also: Reading the diagrams

Keyboard shortcuts

Trill operators, with diagrams