DEV Community: SensorFlow

Go JSON Event Contracts: Unknown Fields Are Not Duplicate Fields

SensorFlow — Mon, 02 Nov 2026 18:36:20 +0000

A JSON event can decode successfully while containing two competing values for the same member. Adding DisallowUnknownFields does not make those duplicates an error. It checks a different boundary: whether an incoming name can be mapped to a field in the destination struct.

This matters when an event identifier, event name, or monetary property crosses multiple parsers. A validator and an ingestion service may accept the same bytes but assign different meanings to them. Before calling a contract strict, test both member uniqueness and field-name matching.

Disclosure: this is an AI-generated technical article published by the SensorFlow project account. The examples are synthetic, were executed locally, and do not describe a customer incident or a completed production security assessment. There is no product integration or SDK compatibility claim.

What the standard actually says

RFC 8259, section 4 says object member names SHOULD be unique. It explains that implementations differ when names are repeated. Do not translate that into a claim that every JSON parser must reject duplicates, or that every parser keeps the last value.

The encoding/json documentation describes Go's legacy behavior: duplicate members are processed in order, with later values replacing or merging with prior values depending on the destination. Struct field matching also accepts case-insensitive matches. A map with string keys does not perform that struct-field matching.

These are two separate questions. The exact duplicate event/event is a duplicate JSON member. The pair event/EVENT contains distinct JSON names, but can still target the same Go struct field under legacy matching.

A small regression test

Save the following as duplicate_test.go in an empty scratch directory. Run go test -v duplicate_test.go. This uses the standard encoding/json import, not encoding/json/v2, and makes no HTTP requests.

package contract

import (
    "encoding/json"
    "strings"
    "testing"
)

type event struct {
    Name string `json:"event"`
}

func TestMemberBoundaries(t *testing.T) {
    cases := []struct {
        name, body, want string
        wantErr bool
    }{
        {"ordinary", `{"event":"paid"}`, "paid", false},
        {"duplicate", `{"event":"first","event":"second"}`, "second", false},
        {"case variant", `{"event":"first","EVENT":"second"}`, "second", false},
        {"unknown", `{"event":"paid","extra":1}`, "", true},
        {"wrong type", `{"event":7}`, "", true},
    }
    for _, tc := range cases {
        t.Run(tc.name, func(t *testing.T) {
            var got event
            dec := json.NewDecoder(strings.NewReader(tc.body))
            dec.DisallowUnknownFields()
            err := dec.Decode(&got)
            if (err != nil) != tc.wantErr {
                t.Fatalf("error = %v, wantErr = %v", err, tc.wantErr)
            }
            if !tc.wantErr && got.Name != tc.want {
                t.Fatalf("name = %q, want %q", got.Name, tc.want)
            }
        })
    }
    t.Run("map preserves case distinction", func(t *testing.T) {
        var got map[string]string
        err := json.Unmarshal([]byte(`{"event":"first","EVENT":"second"}`), &got)
        if err != nil || len(got) != 2 || got["event"] != "first" || got["EVENT"] != "second" {
            t.Fatalf("map = %#v, error = %v", got, err)
        }
    })
}

All six synthetic subtests passed locally using Go 1.26.4. The duplicate and case-variant cases are expected successes in this legacy-behavior test. A passing test therefore does not mean the input satisfies a strict event contract; it means the observed parser behavior matches the expectations encoded here.

The test structure follows Go's testing package, while strings.NewReader supplies the input reader. The exact toolchain and import path belong in the evidence when comparing parsers.

What to enforce before materializing a map

If the event contract requires unique member names, enforce that requirement at a parser boundary that can still observe every member. Once a normal map has kept a single value for an exact duplicate, checking the map cannot recover the discarded member from thin air.

Also decide whether field names are case-sensitive. Rejecting exact duplicate names alone does not prevent two distinct names from mapping to one legacy struct field. Keep those two policies explicit rather than relying on a switch whose name sounds broadly strict.

Do not copy this test into production as a full validator. It does not test nested duplicate objects, payload size, invalid UTF-8, multiple top-level values, authorization, or application-level field constraints. Add samples at the raw JSON boundary and compare the result at every parser your real event traverses.

Version changes need contract tests

The Go team's 2025 introduction to JSON v2 discusses duplicate names and case-insensitive matching among the motivations for changing defaults. It describes an experiment introduced in Go 1.25; that historical article is not evidence of the current availability or defaults in every deployed Go version.

Before a migration, run a fixed corpus against the actual old and new toolchains. Include duplicates with string values, case variants, nested members, and unknown properties. Record accepted/rejected results and materialized values. Treat a formerly accepted payload becoming rejected as a compatibility decision to review, rather than silently dropping it from ingestion statistics.

The useful outcome is a documented contract and reproducible evidence. “JSON decoded” and “unknown fields rejected” each prove a narrow fact. Neither proves that every participant interpreted the original event identically.

Which AI song generator fits a lyrics-first workflow?

SensorFlow — Mon, 02 Nov 2026 13:20:44 +0000

AI music tools now cover very different workflows. Some start with a short prompt, some are closer to a browser DAW, and some are built around composition or licensing. Comparing them only by “song quality” hides the decision most creators make first: do you already have lyrics, or are you starting from a blank idea?

This comparison uses publicly available official product and help pages checked on November 2, 2026. Plans and features change, so treat the linked source pages as the current reference.

The short version

Suno is a broad consumer creation suite with voices, uploads, stem splitting, covers, mashups and Studio tools. Its official pricing page separates free personal use from paid commercial rights.
Udio is built around credit-based song creation and editing. Its help center documents daily and monthly free limits, paid credit limits, and the fact that subscription credits do not roll over.
MusicGPT combines text-to-song, lyrics-to-song and audio editing features, and also exposes an API for products that need music or voice workflows.
AIVA is a composition-focused option with a large style catalog, MIDI/audio influence uploads, and plan-dependent licensing. Its Pro plan is the one that assigns full copyright to the user.
SingMyLyrics is designed around a simple lyrics-first flow: paste original lyrics or start from a theme, choose a style and voice, then create a playable song page.

Comparison table

Tool	Starting point	Editing or production focus	Licensing / plan detail	Useful when
Suno	Prompt, lyrics, audio, voice	Voices, stems, covers, mashups, Studio	Official pricing lists free personal use and paid commercial rights	You want one broad creative workspace
Udio	Prompt and lyrics	Create, extend, remix, inpaint and edit	Free accounts have daily and monthly credit limits; paid credits have separate limits	You want to iterate on generated clips
MusicGPT	Text, lyrics or audio	Remix, extend, replace, stems, voice and audio tools	API and web plans are documented separately	You need an API or post-generation editing
AIVA	Prompt, composition direction, MIDI/audio influence	Style models, editing and file-format export	Free is non-commercial; Standard is limited commercial; Pro assigns full copyright	You care about composition control and licensing
SingMyLyrics	Original lyrics or a short theme	Style, voice, instrumental mode and quality tier	Credits are shown before generation; public songs can have their own playable page	You want the shortest path from words to a shareable song

What the official pages actually say

Suno: the widest creation surface

Suno’s official product pages describe Voices, Stem Splitter, Covers, Mashups, Samples, Personas and Studio 2.0. The pricing page currently lists a free plan with 50 credits per day and no commercial rights, a Pro plan with 2,500 monthly credits and commercial use rights, and a Premier plan with 10,000 monthly credits and additional Studio features.

Sources: Suno pricing, Suno products.

Udio: credit limits matter to the workflow

Udio’s help center says free accounts receive 10 daily credits plus a 100-credit monthly limit. It also explains that a pair of 32-second generations costs two credits, while a pair of approximately 130-second generations costs four credits. Standard and Pro accounts have higher monthly limits, but subscription credits do not roll over.

Source: Udio credits and credit limits.

MusicGPT: generation plus an API layer

MusicGPT’s API page lists music generation, remix, extend, replace, text-to-speech, voice changer, cover, stem splitter, sound generation, lyrics generation, BPM detection and transcription. That makes it a better comparison for developers who want to embed audio features, not only for people choosing a consumer music app.

Source: MusicGPT API.

AIVA: composition and rights are part of the product choice

AIVA describes more than 250 styles, custom style models, MIDI or audio influence uploads, and downloads in multiple formats. Its official pricing page says the free plan is for non-commercial use, Standard supports limited monetization on selected platforms, and Pro assigns full copyright to the user.

Sources: AIVA, AIVA legal terms.

Where a lyrics-first tool fits

If the important asset is the lyric, the workflow should keep that text visible and editable before audio generation. A useful sequence is:

Paste lyrics or describe a theme.
Review the draft and correct names, dates and pronunciation.
Choose style, voice, instrumental mode and quality.
Generate, listen and decide whether another version is needed.
Share a stable song page instead of sending an unexplained audio file.

That is the workflow SingMyLyrics is built to make simple. The lyrics-to-song guide explains the input side, while the comparison page covers the prompt-first versus lyrics-first distinction.

A practical decision rule

Choose the tool based on the next action, not a generic leaderboard:

Pick Suno if you want a broad set of creation and editing modes in one product.
Pick Udio if credit limits and iterative clip editing fit your process.
Pick MusicGPT if you need an API or a wider audio editing toolkit.
Pick AIVA if composition control and explicit plan-level licensing are central.
Pick SingMyLyrics if you have words, a story or a memory and want to turn that input into a playable song with minimal setup.

No comparison table replaces a test with your own lyrics. The same verse can expose differences in pronunciation, arrangement, editability and licensing that a feature list cannot show.

Sources and disclosure

This is an independent comparison, not an endorsement or affiliation with Suno, Udio, MusicGPT or AIVA. Product claims are paraphrased from the official pages linked above. The SingMyLyrics links point to the product’s public guide and comparison page.

Designing an AI song generator: the UI states beyond the prompt box

SensorFlow — Sun, 01 Nov 2026 17:42:38 +0000

Disclosure: I work on SingMyLyrics. This article was written with AI assistance. These are design recommendations, not a claim that every safeguard described below is already implemented in the product.

A song generator can start with a single text box. The difficult part is explaining what happens after someone presses Generate, especially when they have pasted lyrics they care about and the request spends credits.

The product context is straightforward: someone brings original words, chooses a musical style and voice, and wants to hear an arrangement. The lyrics-to-song walkthrough shows that user-facing task. Here is how I would break down the interface contract behind it.

1. Model writing and generation as separate actions

An idea such as "a birthday song about our late-night road trips" is not the same input as a finished verse and chorus.

Give the composer an explicit mode: start from an idea, or use my lyrics. If an AI writes a draft, keep the draft editable before sending it to audio generation. Avoid making a user regenerate the audio just to correct a name in the text.

Treat a title as optional metadata where the provider permits it. If the provider requires one, validate that requirement before submission. The same applies to length limits: count the text that will actually be submitted, including section markers, rather than guessing from the visible number of lines.

2. Keep the user's text when a request fails

A failed request should leave the lyrics, title and musical choices intact. Store the current draft in component state and consider a local draft with a clear delete option for recovery after navigation.

Local persistence has a privacy tradeoff. Lyrics may contain personal stories. Tell users when a draft is stored on their device, avoid putting the text in URLs, and do not send full lyrics to analytics just to count form errors.

3. Separate job state from the browser's patience

The following is an illustrative UI model, not a provider API schema:

type GenerationView =
  | { kind: "editing" }
  | { kind: "submitting"; requestId: string }
  | { kind: "pending"; jobId: string }
  | { kind: "ready"; jobId: string; songId: string }
  | { kind: "failed"; jobId?: string; message: string }
  | { kind: "checking"; jobId: string };

checking matters. A browser that has stopped waiting does not prove that the provider stopped working. Keep the job identifier and offer a status refresh. Starting a new generation should be an explicit action because it may create another charge.

Use provider progress only when the provider actually reports it. Otherwise, an honest "Still generating" message is more useful than an invented percentage or a countdown that repeatedly resets.

4. Make credit handling a server responsibility

Show the credit cost before submission, but resolve the authoritative price and balance on the server. A disabled Generate button improves the interface; it does not prevent duplicate requests from another tab or a network retry.

A robust design needs a stable request identity, an atomic balance update and a durable record of the resulting job. Repeating the same request identity should retrieve the existing job rather than spend again. If the provider supports idempotency, pass an appropriate identity through that boundary too. Without that support, a timeout during submission needs reconciliation before resubmission.

Also distinguish a credit reservation, a final charge and a refund in the ledger. The appropriate policy depends on the provider's billing behavior. Whatever policy you choose, explain it in the interface and apply refunds once, even if a failure event arrives twice.

5. Make playback and download recoverable

"Job completed" and "this browser can play the file" are different checks. The player needs a visible loading state and a useful media-error state. Keep the generated song page reachable if playback fails.

If a provider returns temporary audio URLs, decide whether you will copy permitted assets into durable storage or refresh the URL through the server. Do not promise permanent downloads until the storage and retention policy supports that promise.

6. Test transitions, not just the happy path

Before polishing the waveform, try these scenarios:

Double-click Generate and confirm that one intended request creates one job and one credit operation.
Refresh while a job is pending and confirm that its status can be recovered.
Let a browser request time out, then let the provider finish, and confirm that the finished song remains discoverable.
Deliver the same completion or failure event twice and confirm that neither the charge nor refund is duplicated.
Return a completed job with an unavailable media URL and confirm that the UI preserves the song record and offers recovery.
Fail lyric generation and confirm that the original idea is still present.

These checks do not establish musical quality. They establish whether the application respects the user's input, time and credits. That is a useful foundation for any product that turns a prompt into a slower, paid generation job.

Why Yesterday's Event Report Changed Today: A Snapshot Test for Late Arrivals

SensorFlow — Mon, 19 Oct 2026 16:56:32 +0000

A dashboard can show a higher count for yesterday without either query being wrong. An event may have happened yesterday, arrived after the first report ran, and been included in a later run. To investigate, keep event time, arrival time, and the report's as-of cutoff separate.

This AI-generated engineering note is published from the SensorFlow project account. Its synthetic code was executed locally, and the linked documentation was checked. The example is synthetic JavaScript. It is not an SDK, pipeline, or production reliability test, and it does not claim that any named analytics product loses events.

A report needs two boundaries

The event-time window answers which day an action belongs to. The as-of cutoff answers which records were available to that particular report run. For this example, both are explicit UTC instants. A record occurring at 23:59Z but arriving at 00:10Z belongs to the earlier day, yet a snapshot made at 00:05Z cannot include it.

Apache Beam's time model distinguishes event time from processing time and discusses watermarks as estimates of completeness. The idea is useful even when no Beam runner is involved: a clock passing midnight does not prove that every event for the day has arrived.

Run a small snapshot test

Save this as snapshot.mjs, then run node snapshot.mjs. The rows are made-up records, not captured customer data.

import assert from 'node:assert/strict';

const rows = [
  { id: 'a', occurred: '2026-10-14T10:00:00Z', arrived: '2026-10-14T10:01:00Z' },
  { id: 'b', occurred: '2026-10-14T23:59:00Z', arrived: '2026-10-15T00:10:00Z' },
  { id: 'c', occurred: '2026-10-15T00:02:00Z', arrived: '2026-10-15T00:03:00Z' }
];

function instant(value) {
  assert.match(value, /Z$/);
  const time = Date.parse(value);
  assert.ok(Number.isFinite(time));
  return time;
}

function dailyCount(input, asOf) {
  const cutoff = instant(asOf);
  const start = instant('2026-10-14T00:00:00Z');
  const end = instant('2026-10-15T00:00:00Z');
  const seen = new Map();
  for (const event of input) {
    const arrived = instant(event.arrived);
    const occurred = instant(event.occurred);
    if (arrived >= cutoff || occurred < start || occurred >= end) continue;
    if (seen.has(event.id)) assert.equal(seen.get(event.id), occurred);
    else seen.set(event.id, occurred);
  }
  return seen.size;
}

assert.equal(dailyCount(rows, '2026-10-15T00:05:00Z'), 1);
assert.equal(dailyCount(rows, '2026-10-15T00:11:00Z'), 2);
assert.equal(dailyCount([...rows, rows[0]], '2026-10-15T00:11:00Z'), 2);
assert.equal(dailyCount(rows, '2026-10-15T00:10:00Z'), 1);
assert.equal(dailyCount([...rows].reverse(), '2026-10-15T00:11:00Z'), 2);
assert.throws(() => dailyCount([...rows, { ...rows[0], occurred: '2026-10-14T11:00:00Z' }], '2026-10-15T00:11:00Z'));
console.log('PASS: six synthetic snapshot checks');

The first snapshot counts only a. The later one counts a and late arrival b. Event c belongs to the next UTC day. A repeated a does not increase the logical count, while a conflicting occurrence time for the same ID is rejected by this teaching rule. The cutoff uses a half-open interval: a row arriving at the cutoff is excluded.

This example assumes the ID is stable and unique in its intended scope. A real system needs a scoped event key, conflict handling, versioned event definitions, and a defined arrival timestamp. It also needs the business's chosen calendar timezone. JavaScript Date stores instants; local and UTC calendar interpretations differ, while Date.parse can be unreliable for nonstandard strings. The explicit Z strings here avoid machine-local date parsing. The test uses Node's strict assertions.

Explain revisions instead of hiding them

A useful daily report records its date window, timezone, as-of cutoff, and whether it is provisional. When a count changes, compare the two snapshots by event ID: which rows arrived later, which were duplicates, and which changed definition or were rejected? That makes “late data” a testable explanation rather than a guess.

The six checks above verify only the synthetic rule. They do not establish a universal wait period, an exactly-once delivery guarantee, or end-to-end event completeness. A real acceptance test must compare application outcomes, capture attempts, receiver responses, stored rows, and report filters under one declared contract.

Event Timestamps Need Units, Not Guesswork

SensorFlow — Mon, 19 Oct 2026 01:04:33 +0000

An event can have a valid name, a stable identifier, and a timestamp that lands decades outside the reporting period. The request may still be valid JSON. A successful HTTP response would not tell you whether the timestamp meant seconds or milliseconds.

Event timestamp validation should start with a declared unit, not a guess based on the number of digits. This tutorial builds a small, offline Node.js check for that boundary. All inputs are synthetic; no live SDK or analytics backend is required.

1. Make the unit part of the contract

Require either a field with a defined unit, such as occurred_at_ms, or a versioned source contract that explicitly declares the unit. Convert once at that boundary, then validate the normalized value. Do not silently infer seconds or milliseconds from the magnitude of an unfamiliar input.

JavaScript's Date representation uses milliseconds since the Unix epoch. That does not mean every system producing a numeric timestamp uses milliseconds. Your adapter needs to know the producer's contract before it constructs a Date.

For a concrete boundary, this example accepts a numeric value and the separate unit s or ms. A real API could instead fix the unit in its schema; clients would not need to send redundant metadata on every event.

The contract deliberately rejects numeric strings and fractional values. That is an example policy, not a claim that fractional-second formats are inherently invalid. If a source uses them, define precision and rounding in its own adapter rather than allowing accidental coercion here.

2. Run an explicit conversion check

Save the following as timestamp-check.cjs, then run node timestamp-check.cjs. It needs only Node's built-in assertion module, with no packages, account, or network access.

The fixture uses a fixed UTC acceptance window: the start of 2026 inclusive through the start of 2027 exclusive. This is a repeatable teaching boundary, not a suggested retention or backfill policy. Historical imports need a window that matches their actual contract.

const assert = require('node:assert/strict');

const START = Date.UTC(2026, 0, 1);
const END = Date.UTC(2027, 0, 1);

function normalizeEventTime(value, unit) {
  if (unit !== 's' && unit !== 'ms') {
    throw new TypeError('unit must be s or ms');
  }
  if (!Number.isSafeInteger(value)) {
    throw new TypeError('time must be a numeric safe integer');
  }
  const ms = unit === 's' ? value * 1000 : value;
  if (!Number.isSafeInteger(ms)) {
    throw new RangeError('normalized time is not a safe integer');
  }
  if (ms < START || ms >= END) {
    throw new RangeError('time is outside the fixture window');
  }
  return ms;
}

const checks = [
  ['milliseconds unchanged', () =>
    assert.equal(normalizeEventTime(START + 123, 'ms'), START + 123)],
  ['seconds converted once', () =>
    assert.equal(normalizeEventTime(START / 1000, 's'), START)],
  ['lower bound included', () =>
    assert.equal(normalizeEventTime(START, 'ms'), START)],
  ['last millisecond included', () =>
    assert.equal(normalizeEventTime(END - 1, 'ms'), END - 1)],
  ['before window rejected', () =>
    assert.throws(() => normalizeEventTime(START - 1, 'ms'), RangeError)],
  ['upper bound excluded', () =>
    assert.throws(() => normalizeEventTime(END, 'ms'), RangeError)],
  ['seconds mislabeled ms', () =>
    assert.throws(() => normalizeEventTime(START / 1000, 'ms'), RangeError)],
  ['milliseconds mislabeled s', () =>
    assert.throws(() => normalizeEventTime(START, 's'), RangeError)],
  ['string rejected', () =>
    assert.throws(() => normalizeEventTime(String(START), 'ms'), TypeError)],
  ['null rejected', () =>
    assert.throws(() => normalizeEventTime(null, 'ms'), TypeError)],
  ['fraction rejected', () =>
    assert.throws(() => normalizeEventTime(START / 1000 + 0.5, 's'), TypeError)],
  ['NaN rejected', () =>
    assert.throws(() => normalizeEventTime(NaN, 'ms'), TypeError)],
  ['Infinity rejected', () =>
    assert.throws(() => normalizeEventTime(Infinity, 'ms'), TypeError)],
  ['unknown unit rejected', () =>
    assert.throws(() => normalizeEventTime(START, 'minutes'), TypeError)],
  ['unsafe integer rejected', () =>
    assert.throws(() => normalizeEventTime(2 ** 53, 'ms'), TypeError)],
  ['JSON non-finite round trip rejected', () => {
    const received = JSON.parse(JSON.stringify({ time: NaN }));
    assert.equal(received.time, null);
    assert.throws(() => normalizeEventTime(received.time, 'ms'), TypeError);
  }],
];

for (const [name, check] of checks) {
  check();
  console.log(`PASS: ${name}`);
}
console.log(`${checks.length} checks passed`);
console.log(new Date(normalizeEventTime(START / 1000, 's')).toISOString());

The expected final lines are 16 checks passed and 2026-01-01T00:00:00.000Z. The assertion behavior is documented in Node's assertion module. The checks exercise our example policy; they do not certify an SDK integration or production ingestion service.

Notice the two integer checks. Number.isSafeInteger checks the input without turning a numeric string into a number. Checking again after multiplication protects the normalized result, not just the original value.

The two mislabeled-unit cases fail because the converted result is outside this fixture's window. That is useful defense in depth, not a universal unit detector: an input can have the wrong meaning and still fit a broad window. The producer's declared contract remains essential.

3. Keep validation separate from display and transport

An ISO string is a useful inspection aid. toISOString emits UTC with a Z suffix, which avoids depending on the machine's local display timezone. It does not verify the device clock or prove when an action happened.

Validate on both sides of serialization. JSON.stringify converts NaN and Infinity to null; an undefined property on an object is omitted. The final test demonstrates why validating only the in-memory producer value is not the same as checking what a receiver sees.

Keep distinct fields for event occurrence and receipt. An offline client can report an earlier action after reconnecting. Replacing its occurrence time with receipt time changes the meaning of time-based reports; preserving it does not eliminate clock skew either.

For a real ingestion system, define separate policies for:

Unknown or unsupported units: reject or quarantine with a reason.
Events outside the allowed range: distinguish historical replay from malformed input.
Future-looking device clocks: use an explicit tolerance rather than silent correction.
Invalid records: retain only the diagnostic fields permitted by your privacy and retention rules.

Do not turn a validation failure into a zero timestamp or today's date. Both are plausible-looking values that can hide the original problem.

4. Test the source boundary before changing old data

For a browser producer, inspect the value immediately before serialization and the parsed request at the receiving boundary. For a server adapter, test the upstream contract and the normalized result independently. For a historical import, use the source's documented period rather than this tutorial's 2026 window.

Before a migration, add a fixture with one known timestamp and its exact expected UTC instant. Preserve the source, schema version, and declared unit in whatever minimal diagnostic context your policy allows. A repaired dataset should be reproducible from a documented transformation, not from another round of digit guessing.

The acceptance criterion is narrow: declared unit, correct conversion, permitted precision, and an explicit time range. It says nothing about whether the business action was genuine, the identity was correct, or the event was delivered exactly once. Test those properties separately.

Start with one timestamp from each producer and add the boundary tests to that producer's contract checks. A green count is useful only when the accepted meaning is written down beside it.

Affiliation and AI disclosure: this account is maintained by the SensorFlow team. This article and its synthetic checks were AI-assisted; the fixture was executed locally. This is an independent validation exercise, not a production SDK test.

Missing, Null, and Empty Are Different Event Properties

SensorFlow — Thu, 15 Oct 2026 14:41:16 +0000

Affiliation: this account is maintained by the SensorFlow team. This tutorial was drafted with AI assistance; its synthetic JavaScript fixture was executed locally. It is a standalone validation exercise, not a product capability claim or a production test.

An event arrives without campaign_id. A convenience fallback turns it into direct, and a chart now reports a business conclusion that the event never established. A missing property, an explicit null, an empty string, and a real identifier should not collapse into the same value simply because they are all inconvenient to group.

Event property validation should establish two separate facts: whether the property exists, and whether its value satisfies a documented contract. Only after those checks should a reporting rule assign a business meaning. A transport success response does not answer either question.

1. Write down what each state means

For a synthetic SignupCompleted event, this exercise uses the following policy:

Absent campaign_id: campaign context was not supplied.
Explicit null: the sender supplied an explicit no-value marker.
Empty or whitespace-only string: a supplied identifier violates this contract.
A nonblank string: syntactically valid campaign identifier.
Any other type: invalid campaign identifier.

These are deliberately narrow labels. Explicit null does not prove consent was denied, and absence does not prove traffic was direct. If your source can distinguish “not applicable,” “not collected,” and “collection failed,” define separate reviewed fields or states. Do not reconstruct that evidence later from a generic null.

The JSON Schema null reference distinguishes a null value from an absent property. Its object reference also separates declared properties, required properties, and additional-property rules. Listing a property under properties alone does not make it required.

Our example accepts an absent or null campaign value but records its state. It requires an account identifier. That is a policy choice for this exercise, not a universal analytics schema.

2. Validate before choosing defaults

Save the complete example as property-contract.cjs and run node property-contract.cjs with a current Node.js installation. There are no dependencies or network calls. Every identifier below is synthetic.

const assert = require('node:assert/strict');

function fieldState(object, key) {
  if (!Object.hasOwn(object, key)) return 'absent';
  const value = object[key];
  if (value === null) return 'explicit_null';
  if (typeof value !== 'string') return 'invalid_type';
  return value.trim() === '' ? 'empty' : 'value';
}

function validate(event) {
  if (event === null || typeof event !== 'object' || Array.isArray(event)) {
    throw new Error('event must be an object');
  }
  const allowed = new Set(['event', 'account_id', 'campaign_id']);
  for (const key of Object.keys(event)) {
    if (!allowed.has(key)) throw new Error(`unexpected field: ${key}`);
  }
  if (!Object.hasOwn(event, 'event') || event.event !== 'SignupCompleted') {
    throw new Error('unexpected event');
  }
  if (fieldState(event, 'account_id') !== 'value') {
    throw new Error('account_id must be a nonblank own string property');
  }
  const campaign = fieldState(event, 'campaign_id');
  if (campaign === 'empty' || campaign === 'invalid_type') {
    throw new Error(`invalid campaign_id: ${campaign}`);
  }
  return { account_id: event.account_id, campaign_state: campaign };
}

const base = { event: 'SignupCompleted', account_id: 'synthetic-account-a' };
const valid = [base, { ...base, campaign_id: null },
  { ...base, campaign_id: 'synthetic-summer' }];
assert.deepEqual(valid.map(e => validate(e).campaign_state),
  ['absent', 'explicit_null', 'value']);
assert.throws(() => validate({ ...base, campaign_id: '' }));
assert.throws(() => validate({ ...base, campaign_id: '  ' }));
assert.throws(() => validate({ ...base, campaign_id: 0 }));
assert.throws(() => validate({ ...base, campaign_id: false }));
assert.throws(() => validate({ ...base, campaign_id: undefined }));
assert.throws(() => validate({ ...base, account_id: null }));
assert.throws(() => validate({ ...base, account_id: '' }));
assert.throws(() => validate({ event: 'SignupCompleted' }));
assert.throws(() => validate({ ...base, event: 'SignupClicked' }));
assert.throws(() => validate({ ...base, campain_id: 'typo' }));
assert.throws(() => validate([]));
assert.equal(fieldState(Object.create({ campaign_id: 'inherited' }), 'campaign_id'), 'absent');
assert.equal(validate(JSON.parse(JSON.stringify({ ...base, campaign_id: undefined })))
  .campaign_state, 'absent');
assert.equal(validate(JSON.parse(JSON.stringify({ ...base, campaign_id: null })))
  .campaign_state, 'explicit_null');
console.log(valid.map(validate));
console.log('PASS: 15 synthetic acceptance checks');

The three accepted samples retain three different campaign states. An empty string, wrong type, misspelled extra field, missing account, or wrong event name fails loudly instead of becoming a plausible-looking chart category.

The validator does not silently trim and rewrite the identifier. Whitespace-only strings fail, while other nonblank strings remain unchanged. A production contract may impose additional length or character restrictions; add them explicitly with matching tests.

3. Test the value that actually crosses the wire

One assertion is intentionally surprising: an own property containing JavaScript undefined fails this validator before serialization, but becomes absent after a JSON round trip. That is not a contradiction. These are two different objects at two different boundaries.

MDN documents JSON.stringify's conversion rules: undefined-valued object properties are omitted; array elements behave differently. A sender-side object inspection therefore cannot replace checking the serialized request and the parsed receiver-side event.

Test both boundaries when your application builds a payload in JavaScript. This does not mean every SDK uses JSON.stringify directly; SDK transformations, batching, and encoding must be verified separately. The fixture is not an SDK transport payload or a receiver integration test.

The helper uses Object.hasOwn rather than truthiness or prototype-chain membership. That keeps absent fields separate from present nulls and avoids treating inherited properties as collected data. It is not a substitute for parsing untrusted input safely or for application authorization.

4. Decide where strictness belongs

Rejecting unexpected fields helps expose the typo campain_id. It can also reject a legitimate new field during a rolling client deployment. This example chooses a closed, version-specific contract to make the failure observable; it does not recommend abruptly applying that policy to every production event.

For production, decide how old and new producer versions coexist before tightening the receiver. A quarantine path, a versioned schema, and error counters may be more appropriate than dropping a whole batch. Preserve enough safe diagnostic context to investigate, but do not copy arbitrary personal data into validation logs.

The fixture uses Node's strict assertion module so a broken expectation stops the run. Its tests verify the listed rules, not semantic truth. A nonblank account ID can still identify the wrong account; a well-formed campaign ID can still have incorrect attribution. Those need trusted business context and integration tests.

5. Keep unknown data visible in the report

After validation, report missing context separately from known campaign values. If you later group absent and explicit-null states into a single “unknown” display category, preserve the original state for investigation. Do not rename “unknown” to “direct” unless separate attribution evidence supports that interpretation.

For a practical review, check one valid value, one absent field, one explicit null, one empty value, one wrong type, and one misspelled property. Compare the in-memory event, serialized request, parsed event, and final reporting category. A small fixture with a stated contract is more useful than a chart whose defaults quietly invent meaning.

Design Events That Answer Product Questions: A Practical Tracking Plan

SensorFlow — Thu, 01 Oct 2026 06:43:59 +0000

When someone asks why signup conversion dropped, a generic button_click event rarely answers the question. The team needs to agree on where the journey starts, what counts as completion, how users are counted, and how failures and retries are represented.

An event tracking plan records those agreements before instrumentation. It connects a business question to event names, trigger conditions, identity rules, properties and acceptance tests. A working collector cannot resolve a disagreement about what an event means.

This guide uses a fictional SaaS signup flow. The examples are teaching fixtures, not customer results. I work on SensorFlow, a self-hosted event analytics project; the design method below applies independently of the analytics tool you choose. AI assisted the preparation of this article, and its references and product claims were checked before publication.

Start with a question you can calculate

Instead of asking for "signup conversion," write a more precise question:

Of the distinct visitors who opened the signup page during a calendar day in our reporting timezone, how many completed account creation within 24 hours of their first qualifying page visit?

This defines the starting population, completion condition, identity unit, reporting timezone and conversion window. You still need to define how an anonymous visitor maps to an account, and whether repeated visits restart the window. Those decisions belong in the metric definition.

Amplitude's event selection guide works back from the questions a product team wants to answer. Capturing more interactions does not automatically produce a better measurement plan. Autocapture can help explore interface behavior, but a click or form submission is not proof that an account was created. PostHog documents automatic capture and custom events separately; custom business events let you express the outcome your system can actually confirm.

Define a small event contract

For each event, write down the following fields. The table describes a suggested contract, rather than a required schema for any vendor.

Field	Question to resolve	Signup example
Event name	What action or state occurred?	`signup_submitted`, `account_created`
Trigger	At what exact point is it emitted?	Emit `account_created` after the account transaction commits
Source	Which component owns that state?	Browser for intent; backend for confirmed creation
Identity	Which stable identifier is used?	Anonymous visitor before signup; account ID after creation
Time	When did it occur, and which timezone does the report use?	Preserve occurrence time and define the reporting timezone
Required properties	Which dimensions are essential?	`signup_method`, `app_version`
Optional properties	How is missing context represented?	Missing `campaign_id` is unknown, not silently "direct"
Retry rule	Can one operation produce multiple events?	Reuse a stable operation ID and define deduplication
Owner and version	Who approves a changed meaning?	Product and engineering owners, plus a contract version

Snowplow's tracking design documentation combines event purpose, trigger conditions and data structures in event specifications. Amplitude's tracking plan documentation also records sources, descriptions and property rules. Both are useful references when turning a short product request into something an engineer can implement and a tester can verify.

Keep names stable and put context in properties

Avoid separate event names such as signup_from_ad and signup_from_email when the action is the same. A stable signup_submitted event with a signup_source property is easier to extend when another channel appears. Segment's tracking plan guidance similarly advises against dynamic event names and property keys.

Choose a naming convention and apply it consistently. Title Case or snake_case can both work; the problem is a team alternating between them without a defined mapping. Give each property a type, allowed values, and a rule for missing data. Keep email addresses, phone numbers and full form contents out of a generic analytics payload unless a separately reviewed use case requires them.

Do not merge intent and outcome just to reduce the event count. signup_submitted means a user tried. account_created means the business state changed. If both become a vague signup, a validation error can look like a successful conversion.

Some tools provide recommended events, such as those described in Google Analytics' event guide. Define your own business meaning first, then map it to each destination. Similar event names do not guarantee equivalent semantics across tools.

Instrument at the agreed trigger

For a project already using the official Sensors Data JavaScript SDK, a custom intent event could have this shape after SDK initialization:

// Record the attempt. This is not proof of account creation.
sensors.track('signup_submitted', {
  signup_method: 'email',
  signup_source: 'website',
  contract_version: '1',
});

This is an instrumentation fragment, not a standalone application or a live ingestion test. Initialize the SDK, identity handling and destination for your own environment; SensorFlow's SDK integration guide provides project-specific setup references. Consult the official documentation for your exact SDK version.

Let the component that can confirm the final business state own the success event. An HTTP 200 response might only mean a request was accepted. If the browser and backend both emit account_created, define how they are reconciled, and where a stable operation ID is generated. A property called operation_id does not create deduplication by itself: the receiving and analytical layers must use it according to the contract.

Test the contract, then test the metric

Prepare a small set of non-sensitive fixtures with expected event sequences:

An anonymous visitor opens the signup page and leaves. Expect the start event and no creation event.
Form validation fails. Specify whether signup_submitted means every submit attempt or only a valid request; expect no creation event in either case.
An account is created. Expect one confirmed creation with the correct property types.
A network failure causes a retry. Check how the same operation is counted, not just how many requests were sent.
A visitor signs in or continues on another device. Verify the documented anonymous-to-account mapping instead of assuming two tools merge identities identically.

Check raw records against the contract first. Then recalculate the metric using the same identity, timezone, window and retry rules as the report. Receiving an event and calculating an interpretable conversion rate are separate acceptance criteria.

Keep expected and actual results with the release record. Those fixtures become regression tests when the SDK, application workflow or analytics destination changes.

Choose an analytics system against the contract

An open-source or self-hosted analytics system should be evaluated against the events and questions your team needs. Check whether it receives your existing instrumentation, exposes raw records, explains identity and time behavior, and provides the analyses your users need. Also account for the deployment, access control, monitoring and upgrade work your team will own.

SensorFlow is one option for teams retaining an existing Sensors Data SDK while controlling the event data pipeline. The open-source repository supports deployment and demonstration data; real SDK ingestion requires license activation. It is a focused self-hosted path rather than a feature-equivalent replacement for a mature all-in-one analytics suite, and it has no official affiliation with Sensors Data.

Start by writing one event contract and checking these five fixtures. Once the team can explain the result by hand, it has a concrete basis for selecting an analytics system and reviewing its reports.

Your Demo Dashboard Is Not a Funnel: Auditing Synthetic Events in ClickHouse

SensorFlow — Thu, 01 Oct 2026 04:49:25 +0000

A newly installed analytics stack can show a convincing sequence of "app open → product view → add to cart → purchase" before it has ingested a single real user event. That is useful for testing whether the database and dashboard start, but it is not evidence of conversion. Four non-empty bars may represent four disjoint groups of users.

This is a practical audit of SensorFlow's public demo-data generator. SensorFlow is an independent, open-source, self-hosted event pipeline built around an existing Sensors Data SDK integration, a Go receiver, ClickHouse, and Apache Superset. It does not claim to replace every feature of a full product-analytics suite. Its installation deliberately shows synthetic data first; live SDK ingestion requires a separate license activation. That boundary makes the demo a useful example of why a chart needs a data-contract check.

Does a four-stage dashboard prove a conversion funnel?

No. Stage counts answer "did these event names occur?" A user funnel answers "did the same identified user perform these events in the required order, within the chosen time window?" Counts, distinct-user counts, user overlap, and ordered conversion are four different measures.

For a real funnel, define the identity key, the start and end events, a time window, timezone, duplicate handling, and whether anonymous and logged-in IDs are merged. Without those decisions, a percentage can look precise while measuring the wrong thing.

Inspect the generator before interpreting the chart

The current demo SQL inserts rows from numbers(240). It selects the event name with number % 12, and the demo user with number % 48. Because 48 is divisible by 12, each demo-user-* stays in one event category throughout this generated batch. A demo user is not walking through four stages.

The script also spaces event timestamps three hours apart and assigns sample revenue to purchase rows. Its is_first_day value is part of the synthetic generator, not a production calculation after identity stitching. These fields are good for rendering example charts, not for validating business conversion or GMV.

The underlying ClickHouse table is sensors.event, with time, event, distinct_id, and app_id. The demo rows are marked app_id = 'sensorflow-demo'. If you have installed the bundled demo, first compare event rows with distinct users:

SELECT
    event,
    count() AS event_rows,
    uniqExact(distinct_id) AS users
FROM sensors.event
WHERE app_id = 'sensorflow-demo'
GROUP BY event
ORDER BY event;

On a fresh import of one batch from the current generator, this returns 100 app-open rows from 20 IDs, 80 product-view rows from 16 IDs, 40 add-to-cart rows from 8 IDs, and 20 purchase rows from 4 IDs. Those numbers come from the generator, not from measured customer behavior. Existing installations or changed scripts may produce different results.

Test whether the viewers and buyers are even the same people

Before reporting 4 / 16 = 25% as a conversion rate, ask whether a single distinct_id appears in both groups:

WITH per_user AS
(
    SELECT
        distinct_id,
        countIf(event = 'demo_product_view') > 0 AS viewed,
        countIf(event = 'demo_purchase') > 0 AS purchased
    FROM sensors.event
    WHERE app_id = 'sensorflow-demo'
    GROUP BY distinct_id
)
SELECT
    countIf(viewed) AS view_users,
    countIf(purchased) AS purchase_users,
    countIf(viewed AND purchased) AS both_users
FROM per_user;

Recreating the published generator's 240 rows with ClickHouse Local produced 16 / 4 / 0: sixteen viewers, four buyers, and zero IDs in both sets. Therefore 4 / 16 is not a view-to-purchase conversion rate for this demo. Even a non-zero overlap would establish only that the events occurred for the same ID, not that they happened in the right order.

The query uses ClickHouse's documented countIf aggregate combinator and uniqExact. Exact distinct counts are useful when checking a small test set; ClickHouse notes that uniqExact can consume more memory than approximate alternatives as cardinality grows. Do not infer production query performance from this 240-row sample.

What a production funnel must add

First, choose a stable identity rule. Decide whether distinct_id represents a logged-in account, an anonymous browser ID, or a merged identity. Cross-device behavior and login transitions can change the denominator.

Second, enforce sequence and time. A user who purchases before the product-view event must not count as a view-to-purchase conversion under an ordered definition. ClickHouse documents windowFunnel for ordered chains inside a sliding window. Its window units depend on the timestamp argument; SensorFlow's time column is DateTime64(3), so do not paste a seconds-based example into a millisecond conversion without testing it against your ClickHouse version.

Third, isolate environments and reconcile the dashboard. Filter out sensorflow-demo, staging traffic, and test accounts when calculating production KPIs. A Superset bar chart should use the same event names, identity rule, date boundaries, and filters as a SQL check against raw events.

Finally, test the real pipeline with a small, manually checkable set of SDK events. Confirm the client request, receiver response, ClickHouse row, and Superset metric separately. SensorFlow's SDK integration notes describe the receiver URL and compatibility boundaries. The README makes clear that installing the demo does not start live ingestion; activating it is a separate licensed step.

A five-step acceptance test

Read the demo or seed-data generator; determine whether it represents a real user journey at all.
Compare raw rows and distinct IDs for every event stage.
Check overlap between stages before calculating any conversion percentage.
Add ordered-event, time-window, identity, and deduplication rules to the real-event test.
Match the dashboard filters to the SQL query and exclude synthetic traffic from production reports.

The takeaway is not that synthetic dashboards are bad. They are excellent smoke tests for installation and visualization. The mistake is treating their stage distribution as evidence that real users converted. SensorFlow's transparent ClickHouse path lets a team inspect raw events and define its own metrics, but it also leaves that metric design and infrastructure operation with the team. If your organization needs a ready-made, no-SQL product-analytics interface instead, evaluate one on that basis.

SensorFlow vs Countly in 2026: Analytics Application or SDK-to-ClickHouse Pipeline?

SensorFlow — Tue, 29 Sep 2026 08:00:00 +0000

Countly and SensorFlow both matter to teams that want to operate analytics infrastructure, but they solve different starting problems. Countly is an analytics application with its own SDKs, event collection, reporting interface, and engagement features. SensorFlow is a narrower ingestion path for compatible existing Sensors Data SDK events: Go receiver → team-owned ClickHouse → SQL or Apache Superset.

Short answer: Start with Countly if you need an integrated web, mobile, or desktop analytics product and can instrument with Countly SDKs. Test SensorFlow if you already use Sensors Data SDK, want to keep that instrumentation, and specifically need events in your own ClickHouse. Neither choice removes the need to check edition terms, actual payload compatibility, privacy duties, and operational cost.

This comparison uses the projects' public documentation checked in September 2026. It is not a benchmark, legal opinion, or price quote. SensorFlow is independent of Countly and Sensors Data.

First decide what you are replacing

Countly's public server README describes mobile, web, and desktop collection, sessions, views, events, dashboards, APIs, and plugins. Countly supplies its own SDKs across Lite, Flex, and Enterprise. A team adopting it can instrument an app with a Countly SDK and use Countly's reporting interface. It is not merely a pageview counter.

SensorFlow starts from another constraint: an application already sends events through a compatible Sensors Data SDK and the team does not want to rewrite every client just to change where data lands. The SensorFlow repository documents this route:

Existing Sensors Data SDK clients send requests to a Go ingestion service.
The service writes event rows to the team's ClickHouse.
Analysts query ClickHouse directly or build Apache Superset dashboards.

The repository offers a license-free demo with sample data and Superset dashboards. Real SDK ingestion requires a SensorFlow license. SensorFlow is not a replacement for Countly's built-in analytics and engagement UI, nor a promise that every Sensors Data SDK version or payload mode works automatically.

Countly's write API can accept data from other sources, but that is not the same as accepting Sensors Data SDK wire requests. Pointing an existing SDK server URL at Countly's collector is not a demonstrated migration; it would require an adapter or re-instrumentation and identity testing. Countly SDK calls likewise are not Sensors Data SDK calls that SensorFlow can ingest as-is.

Compare the edition you would actually run

Countly Lite is self-hosted and includes core event collection and dashboards. Countly's current feature matrix lists dashboards, event management, and mobile/web/desktop analytics for Lite. The same matrix marks funnels, retention, cohorts, and user profiles for Flex/Enterprise, not Lite. It would be misleading either to say “Countly has no funnels” or to promise that Lite includes them. Check the exact version and plan you are evaluating; packaging can change.

Countly's server license states AGPL-3.0 with modified Section 7 and branding restrictions. The official licensing FAQ explains an important nuance: organizations can use Lite internally and track their own paid apps, but offering Countly as a service to customers, giving those customers dashboard access for their apps, or rebranding it has different licensing requirements. Do not reduce this to “all commercial use is forbidden,” and read the current terms for your deployment. SensorFlow's public repository uses Apache-2.0, while its real SDK-ingestion path depends on a separately supplied license.

A compact decision matrix

SDK boundary: Countly uses Countly SDKs or its documented write API. SensorFlow is built around compatible existing Sensors Data SDK traffic.
Data store: Countly's server architecture uses MongoDB. SensorFlow's event rows go to your ClickHouse.
Reporting: Countly supplies a product interface. SensorFlow supplies SQL/Superset workflow; your team owns metric definitions.
Advanced analysis: Countly's current matrix lists built-in funnels and retention for Flex/Enterprise. With SensorFlow, you must implement those definitions in SQL or another analysis layer.
Activation: Countly Lite has public source with its stated license conditions. SensorFlow's public demo works without a license; real ingestion does not.

Raw data and operational trade-offs

Countly can be self-hosted, and its architecture documentation says stored raw data can be retrieved through REST APIs or MongoDB commands. It would be wrong to imply that only SensorFlow permits raw-data access. The real distinction is whether you prefer Countly's MongoDB model and application workflows or need events to land directly in an existing ClickHouse environment.

ClickHouse is not automatically faster or cheaper for your workload. SensorFlow puts analytical work on the operator: event schema, identity stitching, deduplication, time zones, retention, backups, access control, and SQL definitions. Countly supplies more application functionality out of the box, but requires its own deployment, SDK, and edition decisions. Test on your own representative event volume and business questions before ranking reliability or total cost.

A useful proof of concept

Ask one question: “Of users who started registration, how many completed it within 24 hours?” Agree on event names, stable user identity, time zone, window, and denominator before comparing dashboards.

Countly path: Instrument a staging app with the relevant Countly SDK. Send registration-start and registration-complete events. Check that their properties appear in Countly's UI or read API. If you need a built-in funnel, verify your chosen edition includes one; do not assume Lite does.
SensorFlow path: Launch the demo, inspect sample rows, then activate licensed ingestion for a meaningful test. Send events from your actual Sensors Data SDK version. Query the resulting ClickHouse rows and verify event name, identity, timestamp, and property types. An HTTP success alone does not prove correct storage.
Recovery path: In a test environment, interrupt the store and inspect retries, duplicates, buffering, and restore behavior. A one-event demo is not a production reliability claim.
Analyst path: Have the person who will own the metric answer the registration question. Count the work required to build and maintain the answer, not only the time required to start containers.

The SensorFlow quick start documents the demo-first and activation sequence.

Decision

Choose Countly when you want a self-hostable analytics application across web, mobile, or desktop, a built-in reporting workflow, and can adopt Countly instrumentation. Evaluate the Lite versus Flex/Enterprise boundary if funnels, retention, profiles, or engagement features are decisive.

Evaluate SensorFlow when retaining compatible Sensors Data SDK events and landing them directly in owned ClickHouse are the hard requirements, and a SQL/Superset workflow plus licensed ingestion are acceptable. If you need Countly's integrated UI or a production ingestion path without a separate license, SensorFlow is a poor fit.

The right question is not “Which has more features?” It is “Which preserves the instrumentation we have, stores data where we need it, and gives our team a reporting workflow it can actually operate?”

SensorFlow vs Umami in 2026: Website Analytics or SDK-to-ClickHouse?

SensorFlow — Mon, 28 Sep 2026 08:00:00 +0000

Umami and SensorFlow both let a team operate analytics infrastructure, but they solve different starting problems. Umami is a self-hostable web analytics application with its own tracker and reporting interface. SensorFlow is a narrower route for compatible existing Sensors Data SDK events to reach a team-owned ClickHouse database, with SQL and Apache Superset used for analysis.

Short answer: Choose Umami first if you want to add website analytics, events, campaigns, and conversion reporting without building a SQL-first workflow. Evaluate SensorFlow if you already have Sensors Data SDK instrumentation and the main requirement is to land those events in your own ClickHouse. Umami is an MIT-licensed open-source project. SensorFlow publishes Apache-2.0 source and a license-free demo, but real SDK ingestion requires a SensorFlow license.

This comparison uses public documentation as of September 2026. It is not a benchmark, pricing quote, or claim that one product is generally better.

Start with the event source, not the dashboard

Umami's normal website setup is to add a website, install its tracker, and read the resulting reports. Its documentation covers traffic sources, visitor behavior, conversions, and revenue. It also documents custom events, funnels, and retention; describing Umami as pageviews only would be wrong. The event-tracking guide shows HTML data attributes and JavaScript calls.

For a site with the Umami tracker installed, a named event with properties can be sent like this:

umami.track('signup_completed', { plan: 'pro', step: 3 });

That call is appropriate for Umami's instrumentation model. It is not a drop-in replacement for a Sensors Data SDK call. Umami also documents a POST /api/send endpoint, but that endpoint has its own payload shape, website ID, and User-Agent requirement. Pointing an existing Sensors Data SDK server URL at it would not translate the protocol. A migration needs an adapter or client-side changes, plus identity testing.

SensorFlow's starting point is different: retain compatible SDK instrumentation and change the receiver. The public repository documents this path:

Sensors Data SDK events → SensorFlow Go ingestion → ClickHouse → SQL / Superset

The README separates a license-free demo from licensed real ingestion. The demo starts supporting services, sample events, and Superset dashboards. To process actual SDK traffic, a team must obtain and activate a license and test its exact SDK versions and payloads. No blanket compatibility claim is justified.

Where the systems differ

Instrumentation. Umami uses its own website tracker, JavaScript calls, or documented API. SensorFlow is evaluated as a receiver for compatible Sensors Data SDK requests. If you have no existing Sensors Data SDK dependency, that SensorFlow advantage may be irrelevant.

Storage. Umami's current installation guide specifies PostgreSQL and offers Docker Compose, a prebuilt image, or source installation. SensorFlow writes event rows to ClickHouse. This is an architectural distinction, not proof that either database wins on speed or cost.

Analysis interface. Umami offers a built-in analytics interface. Its funnel documentation describes ordered URL or event steps and conversion windows, and its retention documentation describes cohorts of returning visitors. SensorFlow uses ClickHouse SQL and Superset dashboards; it does not offer an equivalent native product-analytics interface. Check feature availability in the Umami version and deployment you actually select.

Terms. Umami's repository has a public MIT license. SensorFlow's repository is Apache-2.0, but its real SDK ingestion path requires a separate license. Treat the cost and support terms as part of the decision, not as a footnote.

When Umami is the better first test

For a website team that needs traffic sources, campaigns, custom events, funnels, and retention in one interface, Umami is a more direct workflow. Install its tracker on a staging site, send an event, and ask the actual marketer or analyst to answer a question in the UI. Adding a SQL-first architecture may create work without value if the real need is a usable analytics application.

Self-hosting does not remove operations. The team still needs HTTPS, access control, PostgreSQL backups, upgrades, tracker delivery, and consent handling appropriate to its own jurisdiction and data. Privacy-focused product design is not a universal legal exemption.

When SensorFlow is worth a proof of concept

Suppose a team has many Sensors Data SDK calls and cannot rewrite clients all at once. It wants the event rows in a ClickHouse database it operates and has people who can own SQL metrics. Then SensorFlow's receiver boundary is worth testing: can traffic from the actual SDK versions arrive in ClickHouse with the expected identity, timestamp, and property types?

That is narrower than replacing Umami. Superset can visualize a funnel or retention query, but the team must define the identities, steps, windows, and denominators, maintain the SQL, and explain the metric to users. Owning raw rows also means owning schema changes, sensitive-property handling, retention, permissions, backups, and incidents.

The SensorFlow quick start lets a developer launch the demo first, then activate licensed ingestion. If the requirement is a fully usable no-license self-hosted collection path, Umami's public terms may fit better. If the requirement is existing Sensors Data SDK traffic in ClickHouse, test SensorFlow's licensed path before deciding whether the migration value justifies it.

A useful evaluation for both

List current SDKs, event names, properties, and anonymous-to-authenticated identity transitions. Do not assume instrumentation compatibility from a feature list.
Define what a registration means, the time zone, and the funnel window before opening a dashboard.
For Umami, install the tracker on a non-production page, send a custom event, and confirm the event and properties in its reporting interface. A JavaScript call alone is not proof of storage.
For SensorFlow, start the demo and inspect sample ClickHouse rows. If real ingestion is needed, activate the license, send one event from the actual SDK, and query it back. Check event name, distinct ID, timestamp, and property types.
Let the intended users answer the same business question. Compare the product-manager workflow in Umami with the SQL/Superset workflow the team would maintain in SensorFlow.
In a test environment, exercise database outage, retries, duplicates, backup restore, and credential rotation. An HTTP success response is not a reconciliation report.

Do not generalize a small test into a throughput or cost ranking. Measure a representative volume and operational workload before making a production decision.

Decision

Choose Umami when the main task is adding a self-hostable website analytics application with its own instrumentation and reports. Evaluate SensorFlow when retaining compatible Sensors Data SDK traffic and landing it in owned ClickHouse are the actual constraints, and the team accepts licensed ingestion and a SQL/Superset workflow. They can coexist as separate website and product-event pipelines, but that adds governance work rather than eliminating it.

SensorFlow is an independent project. It is not affiliated with, endorsed by, or certified by Umami or Sensors Data.

I Built an Open-Source, Self-Hosted Alternative to PostHog on ClickHouse

SensorFlow — Sun, 27 Sep 2026 21:54:50 +0000

Per-event pricing can make analytics costs rise with traffic. I built an alternative: SensorFlow — an open-source, self-hosted event analytics stack from collection to SQL and BI, with data staying on your own servers.

Website: https://sensorflow.site/
GitHub: https://github.com/data-analyze-bi/sensorflow (Apache-2.0)
Live demo: https://superset.sensorflow.site

The problem with per-event pricing

Managed analytics charges by event volume. The more successful your product, the bigger the bill — growth gets taxed. Self-hosting flips the model: you pay for servers, not for success.

Architecture

Deliberately boring:

SDK → Go collector → ClickHouse → SQL / Apache Superset

Collect: Go ingestion service, wire-compatible with Sensors Data SDKs — existing SDK instrumentation may be reused after checking SDK version, encryption, identity, property types, and timestamps.
Store: ClickHouse, built for analytical queries (funnels, retention) at scale.
Analyze: Raw SQL when you want control, Superset dashboards when you want clicks.

Run ./install.sh for the local demo, then obtain a license and run ./activate.sh to enable real SDK ingestion. No Kubernetes required.

SensorFlow vs PostHog

	SensorFlow	PostHog
Deployment	Self-hosted only	Cloud or self-hosted
Query model	SQL-first, ClickHouse-native	Product UI-first
Pricing	Annual license, $699/yr list price	Usage-based
Suite extras	None (no replay/flags)	Replay, flags, experiments

PostHog is the broader suite. SensorFlow is the narrower, SQL-first option for teams that want ClickHouse and full ownership of their stack.

Pricing

Annual license starts at $699/yr list price, with the first order currently discounted to $349. It is not billed per event. The first month is free. You bring the servers.

Try it

Play with the live demo for five minutes, then run it locally. If it's useful, a star helps: https://github.com/data-analyze-bi/sensorflow

Independent open-source project, not affiliated with Sensors Data.

SensorFlow vs Plausible in 2026: Two Very Different Ways to Own Analytics

SensorFlow — Sun, 27 Sep 2026 08:00:00 +0000

Both Plausible and SensorFlow can put analytics data on infrastructure you operate. Both involve ClickHouse. That is where the easy equivalence ends. Plausible is a privacy-first web analytics product with a tracking script, goals, and a purpose-built dashboard. SensorFlow is a narrower ingestion route for compatible, already-deployed Sensors Data SDK events: SDK → Go receiver → your ClickHouse → Apache Superset.

Short answer: If your problem is “I need readable website traffic and conversion reports,” start with Plausible. If your problem is “I already have Sensors Data SDK instrumentation and need to test landing its events in my own ClickHouse,” evaluate SensorFlow. Do not choose SensorFlow because someone says Plausible cannot self-host or cannot access raw ClickHouse data: Plausible's own repository documents both. And do not assume SensorFlow's open-source demo means production SDK ingestion is free: its getting-started guide separates demo deployment from licensed activation.

This comparison is based on vendor documentation available in September 2026. It contains no speed, cost, or privacy-compliance benchmark.

The first distinction is the event contract

Plausible's official Events API accepts POST /api/event with fields including domain, name, and url. Its JavaScript tracker can collect pageviews and custom events, and custom properties give those events context. That is a useful model for website teams. The dashboard organizes traffic sources, pages, goals, and conversions without asking an analyst to write SQL.

SensorFlow's documented starting point is different: existing Sensors Data SDK calls. It attempts to receive compatible SDK payloads through a Go service and write resulting events into ClickHouse. Its core value is preserving a tested subset of existing instrumentation while changing the server-side destination. That does not make its receiver a drop-in Plausible Events API endpoint. Nor does Plausible's Events API mean it automatically understands a Sensors Data SDK payload. A migration between the two needs an explicit payload mapping, identity decision, and verification of timestamps and properties.

Consider a practical example: a product already has mobile and web SDK calls for SignupStarted and SignupCompleted, including a stable account identifier. If the team wants the original events for joins and custom SQL, it should test the SensorFlow path with its exact SDK version. If a marketing team chiefly needs privacy-oriented website acquisition and goal reports, it may be faster to instrument Plausible's tracker and use its interface. The event contract, not the word “open source,” decides the implementation work.

Plausible also uses ClickHouse

It would be misleading to frame this as “Plausible's closed database versus SensorFlow's ClickHouse.” Plausible's repository states that its self-hosted Community Edition uses ClickHouse for analytics and PostgreSQL for other application data. It also explicitly says self-hosting allows direct access to raw data in ClickHouse. Its Community Edition deployment repository provides a Docker Compose route.

The more useful contrast is data model and default workflow. Plausible's ClickHouse supports Plausible's web analytics product; users normally work in its dashboard and API. SensorFlow's target is a team-operated event store whose rows are queried with ClickHouse SQL and visualized in Superset. If your organization wants to own metric definitions and join events to other warehouse data, that SQL-first approach may suit it. If the organization wants a finished, simple reporting interface, operating ClickHouse alone does not provide one.

Decision	Plausible	SensorFlow
Primary task	Website and conversion analytics	Compatible Sensors Data SDK event ingestion into ClickHouse
Cloud option	Managed Plausible Cloud	Self-hosted project; check current commercial license terms for ingestion
Self-host option	Plausible Community Edition	Docker-based demo and licensed real-ingestion path
Analytics store	ClickHouse in self-hosted CE; PostgreSQL for application data	Team-operated ClickHouse
Default interface	Plausible dashboard, goals, reports, APIs	ClickHouse SQL and Apache Superset
Existing Sensors Data SDK	Requires integration or mapping to Plausible's event model	The principal compatibility use case, subject to testing
Built-in funnel UI	Available in eligible managed plans, not Plausible CE	No equivalent built-in no-code UI; model it in SQL/Superset
Raw-data access	Plausible CE allows direct ClickHouse access; Cloud has export/API options	Direct ClickHouse queries are the normal workflow

The Plausible README distinguishes Cloud from CE. In particular, it lists marketing funnels, ecommerce revenue goals, SSO, and Sites API among premium features absent from CE. Its funnel documentation labels the feature as a Business-plan feature. This edition boundary matters, but it is not a claim that SensorFlow has a better ready-made funnel: it does not.

Privacy is not a one-word score

Plausible's product is intentionally privacy-first and designed around cookie-free website measurement. Its repository describes how it avoids storing personal data and persistent visitor identifiers. The Events API documentation adds an operational nuance: User-Agent and client IP information are used to derive unique visitor counts, and a misconfigured server-side X-Forwarded-For can silently drop an event even if the API returns HTTP 202. A production implementation should follow Plausible's exact guidance rather than equate “202” with a verified stored event.

SensorFlow can retain event identifiers and properties sent by an existing SDK in a ClickHouse database operated by the customer. That can enable deeper event-level analysis, but it also puts more privacy, consent, access-control, retention, and deletion responsibility on the operator. It would be wrong to market that as inherently more privacy-preserving than Plausible. The two systems are deliberately optimizing different analytics models.

The license and operations boundary

Plausible CE is AGPL-licensed and self-hostable without a Plausible subscription, while Plausible Cloud is a paid managed service. CE operators still pay for compute, storage, backups, maintenance, and incident response. Cloud includes a managed product and features based on the chosen plan.

SensorFlow's README documents an install-first workflow: supporting services and sample dashboards can run before a license is purchased, but receiving real Sensors Data SDK traffic requires a license and activation. This is a material difference from saying “clone the repository and all production ingestion works for free.” Evaluate the license alongside infrastructure and engineering costs.

Do not confuse running a demo with a production validation. For SensorFlow, inspect the sample events, then activate an appropriate license and send a real event from the SDK version you actually use. For Plausible, verify the chosen deployment edition, track an event with its documented API or tracker, create the required goal, and confirm that it appears in the dashboard. For both, test proxies, bot filtering, retries, backups, and access to the stored data.

A fair proof of concept

Choose one concrete question such as “How many visitors reach signup, and what properties distinguish the people who complete it?” Then:

Write down the source events and identities. Website sessions and authenticated product users are not interchangeable units.
Track the same business outcome in both tools. In Plausible, use its documented custom event and goal configuration. In SensorFlow, send a compatible SDK event and query the resulting ClickHouse row.
Check actual data arrival. Inspect Plausible's dropped-event header where relevant, and query ClickHouse directly for SensorFlow. Do not stop at an HTTP success response.
Ask the intended users to answer the question. A marketer may prefer Plausible's ready-made dashboard; a data engineer may prefer raw rows and SQL. Neither preference is universal.
Record setup and maintenance work, product entitlements, and the cost of changing existing SDK code. Do not compare a managed Plausible Cloud plan against an unoperated SensorFlow demo as though they provide the same service.

Which should you choose?

Use Plausible when you want privacy-first website analytics with a focused dashboard and tracker. Its Community Edition is a real self-hosted option with ClickHouse and raw-data access, although the CE feature set differs from paid Cloud plans.

Test SensorFlow when you already have compatible Sensors Data SDK instrumentation, explicitly want that event stream in your own ClickHouse, and have the engineering capacity to define and operate the analyses. Its narrower scope and licensed real-ingestion boundary should be stated upfront. If you need a mature out-of-the-box website analytics application, Plausible is usually the better starting point.

SensorFlow is an independent project. It is not affiliated with, endorsed by, or certified by Plausible or Sensors Data.