Reporting and Export

This chapter covers the full set of report and export outputs produced by the Mikrofab measurement and analysis software: how a measurement becomes a traceable, machine-readable and tamper-evident record, which presentation forms (HTML/PDF) are derived from that record, which table formats (CSV/TXT/XLSX/JSON) are written, and how the files are named.

1. Overview: Three Report Paths

The software produces reports/outputs from three different places. They all share the same discipline (header + sha256 integrity + "missing != zero" + deterministic output), but they serve different purposes.

Architecturally, the flow is as follows:

                          +- write_json  (canonical, machine-readable, checksum source)
   report dict -----------+- write_csv   (table)             -- DATA format
   (build_report / compose)
                          +- build_report_html (escaped)        -- PRESENTATION (derived from data)
                                                                    PDF: Qt QTextDocument

The report engine lives inside this repository, under app/reporting/; there is no shared external reporting package. The core files:

• app/reporting/report_model.py — canonical measurement report model + JSON/CSV writers.
• app/reporting/report_builder.py — pure HTML generator (build_report_html).
• app/reporting/analysis_report.py — analysis report (header + payload, same sealing discipline).
• app/reporting/report_compose.py — the Report Builder engine that merges multiple sources.
• app/reporting/report_index.py — the index that scans and lists generated reports (database-free).
• app/storage/data_writer.py — the layer that writes raw + report files during/after a measurement.
• app/gui/analyze/exporters.py — Analysis workspace export writers.

2. Measurement Report (Automatic, on Every Save)

When a measurement completes, DataWriter.save(...) is called and all files for that single run are written into the subfolder belonging to that run. This is the software's "default" report generator: without the user pressing any extra button, every measurement lands on disk as a traceable record.

2.1 Run folder and file naming

Every measurement run is written to its own subfolder under the output folder. The folder name is generated by run_folder_name(...):

<output_folder>/YYYY-MM-DD-<Day>-<Type>-<HHMMSS>/
example: measurements/2026-06-12-Wed-PV-103045/

The base name (base_name) of the files inside the folder is:

<prefix>_<TFT>_<MOD>_<YYYYMMDD_HHMMSS>_<suffix>

If the prefix (filename_prefix) and suffix (filename_suffix) are empty they are skipped; forbidden file characters (<>:"/\|?* and control characters) are cleaned with sanitize_token.

2.2 Files written

DataWriter.save writes the following formats by default (configurable via export_formats; the default is ("csv", "txt", "xlsx") + always the report + metadata):

The raw-data CSV/TXT/XLSX carry these 13 columns (CSV_COLUMNS):

timestamp, measurement_mode, tft_id, sweep_direction, step_index,
vds_set, vgs_set, vds_measured, vgs_measured, ids, igs, compliance_hit, elapsed_s

2.3 Canonical report (*_report.json) — Header / Traceability

The canonical report is produced by report_model.build_report(...) and carries a three-level traceability header. The table below shows the top-level fields of the report and their source.

2.4 Column families — the right schema per measurement

An important design decision: the columns of the report table are chosen by measurement family. In older versions there was a single fixed schema, TFT-shaped (Vds/Vgs/Ids/Igs); a solar-cell report therefore looked like a "transistor report", with Vgs/Igs always coming out as 0. Now each family picks its own columns; because all writers (JSON/CSV/HDF5) use report["columns"], the correct schema is applied automatically.

Common head/tail columns across all families: point (integer), step_index, sweep_direction, elapsed_s (s), status. The unit and type live in the column definition and are not repeated in each row (single source); the label locale comes from i18n via label_key.

Derived-column formulas:

2.5 Rows and status — "Missing != zero"

_point_to_row(...) converts each MeasurementPoint into a row. Non-numeric / NaN / Inf values map to null (never to 0). Each row's status field:

"rows": [
  { "point": 1, "vds_set": 0.0, "ids": 1.2e-6, "igs": 1e-9, "status": "ok"   },
  { "point": 2, "vds_set": 0.1, "ids": null,   "igs": null, "status": "na"   },  // missing != 0
  { "point": 3, "vds_set": 0.2, "ids": 2.0e-3, "igs": 1e-9, "status": "fail" }   // compliance
]

2.6 Integrity (checksum)

Every canonical report carries a sha256 digest, so that if the file is later modified it can be detected (tamper-evident).

input  : report dict (with integrity.value set to "")
formula: sha256( canonical_json(report) )   // sort_keys=True, ensure_ascii=False, compact
output : integrity = { "algo": "sha256", "value": "<64-hex>" }

Before writing the file, write_json calls seal_integrity(...); this is why the JSON on disk carries the sealed value. The checksum is computed without referencing itself (the value is left empty first). Same report → byte-identical output → same checksum.

3. Measurement Report — HTML and PDF Presentation ("Generate Report")

The canonical JSON/CSV is for machines; the human-facing single-file HTML report (and, where possible, PDF) is produced from the measurement result screen with the "Generate report" button (_generate_report). This presentation is the visible form of the canonical object.

3.1 Report structure (sections)

The generated HTML/PDF consists of the following sections in order (build_report_html):

1. Title block — Report title (e.g. "TFT Measurement Report"), subtitle (<TFT> — <Mode>) and generation time (generated_at).
2. Plot — The current measurement plot is embedded as PNG (markers included). In HTML as a base64 data: URI; in PDF via QTextDocument.addResource as a plot.png resource.
3. Measurement Conditions table — Mode, TFT ID, instrument IDN, VISA resource, channels (drain/gate), relay state, point count, software version; followed by the sweep parameters and (if present) outdoor environment rows.
4. Extracted Parameters table — Mode-aware polished labels (Voc/Isc/FF/PCE, Vth, Ion/Ioff…). If there is no dedicated generator, it falls back to a raw summary dump. This table uses the single source (summary_format) shared with the Result page; the numbers you see in the report are exactly the same as those on screen.
5. User Note — A free-text box if user_comment is present.
6. Footer — Software version stamp.

3.2 Output files and naming

"Generate report" writes two files (directly into the output folder, not the run subfolder):

<TFT>_<MOD>_<YYYYMMDD_HHMMSS>_report.html
<TFT>_<MOD>_<YYYYMMDD_HHMMSS>_report.pdf

• HTML: Because the plot is base64-embedded, the single file is portable and opens in a browser.
• PDF: Produced via Qt QTextDocument → QPrinter; there are no extra dependencies. The plot is embedded as a plot.png resource.

4. Analysis Report (Analysis Workspace — "Export" Menu)

After an analysis is solved in the analysis workspace, the "Export" button (analyze2.header.export) in the top bar opens a menu. The menu contents are the intersection of the formats supported by the active analysis module's descriptor and the formats that are implemented.

4.1 Export menu options

4.2 Analysis report (report_json / report_pdf) structure

The analysis report is produced by build_analysis_report(...); it carries the same header discipline as the measurement report and reuses report_model's sealing/checksum/write_json functions (i.e. the JSON is byte-stable and integrity-sealed). The top-level fields:

The analysis payload (Qt-free, language-independent; labels are kept as i18n keys):

• descriptor — Analysis module identifier, version, category, title/summary keys, engine.
• metrics[] — For each metric, raw GUM fields + formatted display: {key, label_key, unit, value, u_std, k, n, r2, display, primary}. If there is no value, value=null, display="—".
• method — {description_key, equations[], conventions[], fit_method}. Equations are carried as plain text (plain) + a note; conventions are i18n keys.
• standards[] — Cited standards: {id, designation, title, edition, scope} (resolved from the registry).
• reliability — Quality verdict: ✓ Reliable (reliable) / ✕ Unreliable (unreliable) / ⚠ Non-STC (non_stc).
• provenance — Origin/trace envelope: {data_file, file_hash, column_mapping, parameters, software_version, timestamp_utc}.
• narrative, notes[], conditions[] — Narrative, notes, and the condition tokens required by the descriptor.

4.3 Uncertainty notation (Quantity / GUM)

Analysis metrics are formatted through a Quantity object along the lines of GUM (Guide to the Expression of Uncertainty in Measurement) (format_quantity). The display rules:

• Expanded uncertainty: U = k · u_std (k = coverage factor).
• In data files (JSON/Excel) the raw fields (value, u_std, k, n) are kept separately; the formatted display is for convenience only.

4.4 Excel export (write_metrics_xlsx)

The analysis Excel output contains two sheets:

1. "Metrics" — Columns: metric, value, unit, u_std, n, display, reliability. One row per descriptor metric; if there is no Quantity, the value is empty and display="—".
2. "Provenance" — Origin envelope as key/value: data_file, file_hash, software_version, timestamp_utc, plus column_mapping and parameters as JSON.

5. Composed Report (Data and Reports Workspace)

Beyond individual measurement/analysis reports, the Data and Reports workspace offers a Report Builder that combines multiple sources into a single professional report. The engine is report_compose.py: pure, Qt-free; it does not add a new writer on top of the writers above.

5.1 Workspace tabs

5.2 Report Builder — three-column flow

The Report Builder consists of three columns (there is no drag-and-drop field well; the column families are deterministic):

Column 1 — Setup:

Column 2 — Content: Depending on the selected source, either a checkbox list of selected measurements or the Lab Notebook sample selector (fabrication steps + linked measurements). Below it are the section checkboxes.

Column 3 — Live preview: As selections change, the HTML produced by report_compose.compose(...) is previewed instantly (the preview never crashes; on error it shows a red message).

5.3 Report sections

5.4 Lab Notebook bridge (linking without a schema change)

The Report Builder links measurements to Lab Notebook samples without adding a new table. The link fields: samples.sample_id == measurements.tft_id and sample_steps.linked_measurement_id. The three-hop bridge:

1. get_sample(sample_db_id) → sample_id.
2. Wide pool: measurement_db.query(tft_id=sample_id).
3. The "called" set: the linked_measurement_id values of the steps (union, unique by id).

5.5 Output files and canonical JSON

"Build" → report_compose.write(...) writes in the selected formats. The file root name: <sample_id or template>_rapor (forbidden characters cleaned).

The canonical JSON additionally carries these traceability fields: report_type="composed", source (type, sample_id, measurement_ids, step_ids), source_sha256 and measurements_sha256 (the staleness sub-digest), sections[], and a summarized sample and measurements. Deterministic: a fixed created_at + report_id → byte-identical output.

5.6 Reports library and "staleness" (stale)

The "Reports" tab scans the output folder without a database: report_index.list_reports(...) reads every *_rapor.json (only report_type=="composed") and lists the header + the generated sibling formats (pdf/html/csv). Double-clicking a row or "Open" opens the report in the default application (priority order: PDF > HTML > JSON).

5.7 Statistics tab (aggregation)

The "Database / Statistics" tab aggregates the extracted parameters across all measurements of a selected sample (tft_id) (the benchtop SPC/distribution equivalent):

Mean = Σxᵢ / n
σ (sample std, n-1) = sqrt( Σ(xᵢ − Mean)² / (n − 1) )   // n>1; if n=1 then σ=0
min, max, n

None/NaN values are excluded; if n=0, all statistics are — (missing != 0). The "Report from selection" button opens the Report Builder with that sample.

6. Export Formats (Reference)

This section gathers all the formats in one place.

6.1 Canonical CSV (machine format)

A locale-independent, single, documented format (report_model.write_csv / exporters.write_dataset_csv):

• Encoding UTF-8 + BOM (utf-8-sig — so Excel opens it correctly), dot decimal, comma separator, line ending \n.
• Headers are stable key + unit (e.g. ids_A, vds_set_V), and are never localized.
• Missing point → empty cell (""); 0 is never written. The na/fail/ok distinction lives in the status column.
• The CSV carries only the table; the header + checksum live in the companion JSON file.

point,step_index,sweep_direction,vds_set_V,vgs_set_V,vds_measured_V,vgs_measured_V,ids_A,igs_A,elapsed_s_s,status
1,0,Forward,0.0,0.0,0.0,0.0,1.2e-6,1e-09,0.01,ok
2,0,Forward,0.1,0.0,,,,,0.02,na

6.2 Excel-compatible CSV (convenience copy)

write_dataset_csv_excel — for Turkish/European-locale Excel:

6.3 TXT (tab-separated)

<base>.txt — the tab-separated version of the raw measurement table; the same 13 columns, UTF-8. Practical for quickly pasting into tools like Gnuplot/Origin.

6.4 XLSX (Excel workbook)

There are two paths:

• Raw measurement (DataWriter._write_xlsx): a "measurement" sheet (13 columns) + a mode-aware "Summary" sheet (label/value). If openpyxl is missing, a dependency-free minimal XLSX (direct zip + XML) is written (_write_minimal_xlsx) — i.e. the Excel output is produced in every environment.
• Analysis metrics (write_metrics_xlsx): the "Metrics" + "Provenance" sheets (Section 4.4).

6.5 JSON metadata vs canonical JSON

Two different JSONs must not be confused:

metadata.json contains the following fields: device_identity (IDN/VISA/switch_port), relay_enabled, selected_tft, sweep_parameters, date_time, started_at, user_comment, software_version, smu_mapping (drain/gate), point_count, planned_point_count, stopped_early, compliance_hit and (if present) <mode>_summary.

6.6 PNG (plot image)

Plots are exported as PNG:

• Measurement report: The plot is rendered to PNG at 150 DPI and embedded into the HTML as base64 and into the PDF as a plot.png resource.
• Analysis: Via "Export → PNG", the active plot is written directly to a PNG file (plot_canvas.export_png).

The generated report files are named so that their source is evident: measurement reports follow <TFT>_<MOD>_<date>_<time>_report.json / _report.csv, while composed reports follow the *_rapor.json pattern. This way each output reveals from its name which measurement or session it was derived from.

6.7 File naming convention — summary

7. Determinism, the Locale Trap and Levels

7.1 Determinism (the rule every writer obeys)

• Same input → byte-identical output. _canonical_json uses sort_keys=True, ensure_ascii=False and compact separators; there is no hidden wall-clock in the payload — created_at comes from the caller. A caller wanting bit-identical output passes a fixed created_at + fixed report_id.
• Locale-independent data. JSON/CSV numbers are of numeric type and use dot-decimal. Only the human HTML/PDF presentation is localized.
• Escaped HTML. build_report_html passes every dynamic value through html.escape.

7.2 Level → required-field mapping

7.3 What is currently PLANNED

The following are defined in the contract but not yet implemented; the report does not claim them:

• A native .xlsx report writer and an archival PDF/A profile (currently PDF is produced from HTML via Qt).
• A GUM Type-B uncertainty column for measurement metrics (the measurement DB carries bare scalars; uncertainty lives on the analysis-kit Quantity path).
• A Level-2 uncertainty budget, a calibration certificate and a signature chain (e.g. PAdES for PDF); calibration and signoff are currently stubs.
• Background-thread generation for large/bulk reports.

8. Standard References

The behaviors in this chapter are based on the in-repo contracts and standards:

• docs/standards/06-reporting.md — The reporting standard (determinism, checksum, three-level header, locale trap, "missing != zero", auto-transmission ban).
• docs/guides/reporting-guide.md — The concrete contract: canonical JSON skeleton, column definitions, CSV convention, writer contract, composed report (report_compose).
• docs/standards/03-measurement-and-analysis.md — Result + R² + uncertainty.
• docs/standards/05-data-and-database.md — Stored data / NULL discipline.
• docs/standards/12-i18n-l10n.md — The locale/language distinction.
• Analysis standards (citation): analysis reports carry the relevant measurement standard (designation/title/edition/scope) via the standards[] in the module descriptor.

9. Quick Reference and FAQ

Which file is the official record of a measurement?

<base>_report.json (canonical, sha256-sealed). The human-facing presentation is derived from it.

My Excel shows comma-formatted numbers as broken — what should I do?

Use the "CSV (Excel-compatible)" variant (Section 6.2) or prefer XLSX. For archiving, always keep the canonical CSV (dot-decimal).

Why does a value appear empty/— in the report?

That point could not be measured (status="na") or the metric could not be computed. The software never fills a missing value with 0 (missing != zero).

What does the "stale" badge mean?

The source measurements changed/were deleted after the report was produced. Re-generate the report for current numbers; the existing report remains as a sealed snapshot.

How do I verify that two files came from the same measurement?

Via integrity.value (sha256). The same report dict always yields the same digest; if the file changed, the digest does not match.

Why is the PDF sometimes not produced?

The PDF is produced via Qt (QTextDocument → QPrinter); if Qt is missing or there is a write error, the writer silently skips it and at least the HTML/JSON is preserved (robust-generation principle).