Logs, Diagnostics, Telemetry and Safety

This chapter covers the four interrelated systems that form the operational backbone of the Mikrofab semiconductor/TFT/PV measurement and analysis software: the logging infrastructure, the diagnostics tools, the usage report (telemetry) and hardware safety. All of these systems are designed to reliably answer the question "what happened when something went wrong?", to protect privacy, and to bring the instrument to a safe state under all circumstances.

The shared design principle is this: secrets are never written to disk or sent over the network; personal data (PII) is masked; the hardware is brought to a safe state on every error, abort and shutdown. The subsections below describe each system individually, staying faithful to the actual behavior in the code.

1. Logging System

The software has two entirely separate log streams, and it is critically important not to confuse them:

1. Local diagnostic log — written to disk, stays on the machine, never sent anywhere over the network. Used for troubleshooting and support.
2. Output Log console — a color-coded, live, operator-facing stream in the interface. Shows what is happening during a measurement in human-readable form.

These two serve different purposes; both are entirely independent of the telemetry/usage report (Section 3).

1.1. Local log file and rotating retention (rotation)

The local log is produced by a single logger named tft_measurement_app and is written both to a file and to the standard console (stderr). To prevent unbounded growth, the file is kept on a rotating basis.

Rotation logic: when the active file reaches 2 MB, ...log is renamed to ...log.1 and the oldest backup (...log.5) is deleted. This way total disk usage always stays within the upper bound.

The log line format is fixed and machine-parseable:

2026-06-29T14:32:07 | INFO | tft_measurement_app | Keithley 2612A baglandi
<timestamp (UTC)> | <LEVEL> | <logger name> | <message>

1.2. Location of the log file

The log file is kept in the logs folder under the user-specific data root. The data root is determined by selecting the first writable option in the following order:

1. The MIKROFAB_APPDATA environment variable (if set) — for portable/custom installations.
2. %APPDATA%\<Vendor>\<Application>\ (the typical path on Windows).
3. %LOCALAPPDATA%\<Vendor>\<Application>\ (fallback).
4. ~/.mikrofab/tft_measurement_app/ (home directory fallback).
5. <working directory>/user_data/ (last resort).

So the typical full path on Windows is:

%APPDATA%\Mikrofab\TFT Measurement App\logs\tft_measurement_app.log

1.3. PII masking and secret protection (in the local log)

Every line written to the local log passes through a PII filter (PiiFilter) before it lands in the file/console. This filter applies two rules:

Rule A — A line that may contain a secret is fully redacted

The message text is converted to lowercase and searched for any of the keys below; if one is found, the entire line is replaced with <redacted: possible secret> (the content is never written):

Rule B — The user name in the file path is masked

If the line does not contain a secret, the user-name component within the path is replaced with <user>:

In addition, uncaught exceptions are logged at the CRITICAL level together with their full traceback. Because this mechanism is active both while the GUI is running and in headless execution, the cause of a crash is always recorded to the file.

1.4. Output Log console panel (Output Log)

The Output Log located at the bottom of the interface (title: "OUTPUT LOG") is an operator-facing, read-only, color-coded live stream. It is placed inside a side/bottom panel (dock) with the title "Output Log" and is tabbed in the same area as the Reading Values panel.

Panel features:

• Location: A detachable/floatable dock anchored to the bottom edge of the window.
• Visibility: Can be toggled via the View > Log / Console menu.
• Content: Each line is a local timestamp in [HH:MM:ss] format + the message.
• Simultaneous local log: Every message written to the console is also written to the local log file at the same time via logger.info(...) (single call, dual target).

Color coding. The console automatically classifies a line into one of four colors based on the message content (it looks at keywords in the content):

2. Diagnostics

The diagnostics tools let you diagnose a problem without remote support, or by preparing a clean package that can be sent to the developer. There are three layers: diagnostic package export, hardware self-test, and step-by-step troubleshooting (troubleshoot) with one-click fixes.

2.1. Exporting a diagnostic package

The diagnostic package gathers everything needed for support/debugging into a single zip file. The operation is designed to be independent of the GUI (pure), best-effort, and to never crash the application.

What is included in the package:

Contents of system_info.json:

Secret redaction (in settings). Before settings.json is written, every setting field whose name contains one of the hints below is replaced with <redacted> (nested dictionaries are scanned recursively):

api_key · api-key · apikey · token · secret · password · passwd
license_key · license-key · private · signing · authorization · bearer

If this limit is exceeded, the largest log files are skipped (the package is still produced); this way the zip file stays a reasonable size.

2.2. Hardware self-test (Self-test)

The self-test verifies that an instrument is "alive and reachable" using read-only checks. No output or current is applied — so it can be run safely even when a DUT (device under test) is connected.

The self-test is triggered from the action menu of the device card on the Hardware page. The test runs in the background (on a non-blocking thread); the panel is modeless (it does not block your other work) and is updated via a signal when the test finishes.

Checks run (in order):

The overall result (roll-up) calculation is determined by the worst individual check:

input  : status of each check {OK, WARN, FAIL, SKIP}
formula: any FAIL  -> overall = FAIL
         (no FAIL) any WARN -> overall = WARN
         (only OK)            -> overall = PASS
output : PASS / WARN / FAIL  (SKIP is ignored in the roll-up)
basis  : IEEE-488.2 self-test (*TST?) + device error queue (SYST:ERR?) contract

Panel view and status band:

Panel elements:

• Status band: A large, color-coded PASS/WARN/FAIL label.
• Busy bar: An indeterminate (marquee) animation while the test runs; it keeps the panel from looking frozen during a slow real check (the *TST?, SYST:ERR? loop). It is hidden once the result arrives.
• Intro line: "This self-test runs the following read-only checks on the device:" — explains what will be done before the test.
• Check lines: An icon + name + message for each check. Icons: ✓ (OK), ⚠ (WARN), ✗ (FAIL), – (SKIP / not done).
• Raw error section: The raw text returned from the device error queue (monospace); hidden if the queue is empty.

2.3. The diagnostic band, troubleshooting and one-click fixes

When a device cannot connect or does not respond, the Hardware page shows a diagnostic band. The band explains the likely cause of the problem in human-readable form and offers one-click fix actions. Example causes and actions:

Step-by-step Troubleshoot window (Troubleshoot). The "Troubleshoot" action opens a device-specific modeless checklist. The window lists five steps, each with a Retry button next to it; "Retry" re-runs the relevant *IDN? probe off the GUI thread:

At the bottom of the window, the raw error text of the last error (monospace) is shown, along with a Help link (https://mikrofab.com/docs/troubleshoot) that goes to the online troubleshooting guide.

3. Telemetry / Usage Report

The usage report (telemetry) collects anonymous, PII-free event signals to understand how the software behaves under which conditions and to proactively fix bugs. The system works offline-first and is tightly constrained by a consent model.

3.1. Offline-first queue

Telemetry calls never stall or crash the application. Events that cannot be sent (when there is no internet/endpoint) are written to a queue file and sent in a batch on the next attempt.

Workflow:

1. When an event is produced, it is appended to the telemetry_queue.jsonl file (append).
2. A background worker (daemon thread) periodically sends the queue by taking a snapshot (a .sending file); new events are not blocked during this.
3. If the send succeeds, the events are removed from the queue; if it fails, they stay in the queue (no data loss).
4. A .sending file left over from a previous run is recovered back into the queue at startup (crash resilience).
5. On exit (shutdown), one final attempt is made to flush the queue.

3.2. What is sent, what is not

The core principle of telemetry: raw scientific data, personal data and secrets are never sent. Everything sent is either a flag, a number, or a low-cardinality class/bucket. Fields such as address/IP are reduced to only the bus type.

Masking helpers. Two layers are applied before sending:

• mask_pii: Masks in-path user names in the text (C:\Users\X, /home/X → <user>) and the bare session-user name where it appears.
• sanitize_params: Strips PII keys (name, operator, email, note, title, etc.) from the parameter dictionary by word-fragment matching. An important subtlety: the match is on the exact word fragment; this is why non-PII settings such as sample_rate or address are not stripped by mistake, but sample_name / operator_name / numune_adi are stripped.

Fingerprint reduction — bucketing. Raw hardware/screen values are reduced into buckets so as not to make the machine unique:

3.3. Consent model and geo-gate

Which telemetry categories are enabled starts with a geo-gate given by the user's region; the final say is always the user's.

Categories and their legal classes:

• notice (P): Can be bundled with a notice; notice-based default-on is recommended in non-EU regions.
• consent (R): Requires explicit opt-in everywhere; default off.

Geo-gate (sets the initial default):

Event → category mapping (gate). Before being sent, every event is passed through a gate that checks whether the category it maps to is enabled. If the category is off, the event is never produced. An event that is not in the list falls into the most restrictive usage category (a safe default instead of a silent leak). Feedback that the user sends deliberately (feedback) is exempt from this gate.

3.4. Feedback (feedback)

Feedback is a message the user sends intentionally and is evaluated independently of the telemetry consent gates.

4. Safety (Hardware Safe-state)

This system guarantees that on every path that talks to the instrument, the highest priority is to bring the hardware to a safe state. There are three pillars: pre-measurement validation + current compliance, the safe-state sequence, and E-STOP (emergency stop).

4.1. Pre-measurement validation and current compliance

No sweep starts before its parameters are validated. Validation (validate_common_limits) applies the following rules:

In addition, every voltage value to be applied is checked individually against max_abs_voltage (ensure_voltage_values_safe): if any value exceeds it, the measurement is rejected before it starts.

The safety limits come from the safety_limits block in config/default_config.json:

Compliance hit (compliance hit). If, at a measurement point, the instrument reaches its set current compliance, this is marked as a compliance_hit, written to the data, and the measurement is ended safely. On the telemetry side, these hits are reported as an aggregate at the end of the sweep (total hit count + first-hit step) — the hot loop is not touched.

4.2. Safe-state sequence

On an error, abort, or application shutdown, the hardware is brought to a safe state in a definite sequence. This sequence is based on the principle of cutting the source first and then switching (de-energize first, then open its relay).

1. SMU output OFF              (Keithley/Keysight/R&S SMU output OFF)
2. Switch Matrix relay OFF     (relay OFF — e.g. switch matrix command `a`)
3. Safe closing of connections

At the code level, this is provided by safe_shutdown:

• First smu.output_all_off() is called (in a try block).
• Then (in finally), if the context is None or relay_enabled is true, relay.all_off() is called. So the output is turned off in every case; the relay is only opened when relevant.

Crash/exit coverage. The same safe-state callback runs on interpreter exit (atexit) and on an uncaught exception too (device_safety). The order is: hardware safe first, then the existing hook chain (telemetry/log). This complements the Qt closeEvent path: non-Qt, headless or abnormal terminations are also covered. The safe-state hook swallows its own errors and never crashes the exit/reporting chain.

4.3. E-STOP (Emergency Stop)

The E-STOP button ("ACİL DURDUR" / EMERGENCY STOP) is a prominent control that brings all hardware to a safe state instantly. Unlike other hardware widgets, E-STOP does not emit an intent and wait for its owner to act; the moment it is clicked, it directly runs the safe-all operation, but it emits the result (output per target) afterward.

Trigger sequence (H3 guarantee): trigger() runs in two phases and does not short-circuit in any phase (one target's error does not stop the others):

Phase 1: All worker-stoppers   (QThread workers are stopped and joined first)
Phase 2: All device safe-functions (hardware output-off comes second)

This order prevents the GUI thread and a lagging worker from accessing the same SMU concurrently (stop the worker first, then apply output-off on the hardware).

Safe-state method selection. The controller calls the first found safe-state method of each device in priority order:

When a RoleBinding (role binding) is registered, the order matters: smu → relay → thermometer. That is, the source (SMU) is secured before the relay. A device that offers none of the known safe methods (for example a temperature controller that has not yet gained safe_off) is skipped and its name is returned to the caller; this way the gap surfaces in advance instead of being a surprise on the bench.

Result (EStopResult):

The interface shows a message based on the result: if all are safe, "Emergency stop: all devices safe."; if some could not be secured, "Emergency stop: some devices could not be secured."

Other important behaviors:

• Idempotent: After E-STOP has fired once, repeated triggers (e.g. a top-bar click followed by the closeEvent's emergency-stop call) are no-ops; the worker-stoppers/safe-functions do not run twice. It is set up again with reset() for a new measurement.
• Accessibility (a11y): The button's accessible name is updated according to its state: "EMERGENCY STOP — armed (outputs ready to stop)" (armed) or "EMERGENCY STOP — active (stop in progress)" (live). Screen readers announce this state.
• Visual pulse (pulse): The button blinks with an opacity pulse to draw attention — slow in the armed state (1800 ms, 1.0→0.70), fast when active/during measurement (600 ms, 1.0→0.45). If the user prefers reduced motion (Windows accessibility setting / Qt), the animation is not created at all.
• Crash coverage: The same safe-all is bound to device_safety at startup; this way it runs on a crash / uncaught exception / exit too.

4.4. Calibration reminder

Taking the reminder feature of bench tools (e.g. BenchVue) as a model, the software logs a warning at startup when recalibration is overdue. The calculation is pure date arithmetic (no I/O, does not block startup).

Days-remaining calculation (days_until_calibration_due):

input  : last_date (ISO), interval_days, today (today)
formula: remaining = (last_date.ordinal + interval_days) - today.ordinal
output : remaining (days)   — a negative value = overdue
special: last_date empty/invalid OR interval_days <= 0  ->  None (no reminder)

At startup, if the reminder is enabled and remaining < 0, the following warning is written to the local log:

WARNING | Calibration overdue by <N> day(s)

Setting location: Settings > Hardware > Calibration.

Summary: Quick Reference