spec/archive
Archive (.txtab.zip) and column storage
Storage per table
Where a table's data lives is declared on its P table line (tables.md):
P table |
Data |
|---|---|
no reference= |
inline records after the table block |
reference=<file> |
records in an external CSV/TSV file |
reference=<dir>/ storage=columns |
a directory with one .col file per column |
storage=isrows(default) orcolumns. Any other value isvalue-invalid(FAIL).storage=columnsMUST havereference=ending in/(storage-columns-without-dir, FAIL).section-size=is allowed only withstorage=columns(section-size-without-columns, WARN, ignored).- Column storage is designed for archives, but a
storage=columnsdirectory MAY also sit next to a.txtabon a file system.
Column storage
text
orders/
├── 00-id.col
├── 01-customer_id.col
└── 02-amount.col- One file per physical column (stored and materialized; virtual computed columns have none), named
<index>-<name>.colwith a zero-padded zero-based index;<index>.colwhen the column has no name (header=0). - Each file holds one value per line, in record order, in the table's encoding; an empty line is an empty field; the table's
nullmarker applies. Values are not quoted and contain no line breaks. - Every
.colfile MUST have exactlyrowslines (col-line-count-mismatch, FAIL). A missing file iscol-file-missing(FAIL); the number of files MUST equalcolumns=when declared (col-count-mismatch, FAIL). - The directory holds only
.colfiles (or section directories). There is no per-table header file: the table's metadata is its block in the main file. - Column names come from the file names and the table's
columnlines; there is no header row.
Sections
With section-size=N, each column is a directory <index>-<name>/ holding files <start>.col, where <start> is the zero-padded first record of the section.
- All columns share the same section boundaries (
section-boundary-mismatch, FAIL). - Sections partition
[0, rows); line counts sum torows(section-partition-error, FAIL). - Record
nis linen − startof sectionstart = ⌊n / N⌋ · N. - Recommended:
section-size=10000.
.txtab.zip
A standard ZIP archive holding a main file and, optionally, the external data its tables reference.
text
sales.txtab.zip
├── sales.txtab ← main file, first entry
├── orders/ ← storage=columns
│ ├── 00-id.col
│ └── 01-amount.col
└── customers.csv ← reference=customers.csvMain file
- The main file MUST be the first central-directory entry (
zip-main-not-first, FAIL). Readers MUST NOT scan past it. - Its name MUST be the archive name without
.zip(sales.txtab.zip→sales.txtab), ordata.txtab(zip-main-bad-name, FAIL). - It is an ordinary
.txtaband describes every table in the archive. There is no separate manifest. reference=in the main file resolves against archive entries, relative to the main file's directory; it MUST NOT escape the archive (reference-escapes-dir, FAIL). If no entry matches, a reader MAY look on the file system next to the archive; if that fails too,reference-not-found(WARN).- Entries not referenced by any table MAY be present (attachments); readers ignore them.
- There is no
original-size=: ZIP stores every entry's size and CRC.
Compression and encryption
- Entries SHOULD use Deflate. Store, Deflate64, BZIP2, LZMA and Zstandard MAY be used; others are
zip-unsupported-compression(FAIL). ZIP64 MUST be supported. - Archives MAY use standard ZIP encryption (traditional PKZIP or AES). The password comes from the tool (for example
--password) and MUST NOT be stored in any Text Tables line or in the comment. An encrypted archive without a password iszip-encrypted(FAIL, operational).
ZIP comment
The archive comment (end-of-central-directory, at most 65535 bytes) lets tools see the schema without extracting.
- Writers SHOULD write it. It MUST be UTF-8 and MUST be a valid Text Tables prefix: the main file's lines up to and excluding data records — first line, file block, every
P tableand its block — with the same prefix P. - If it does not fit, the writer drops lines, never reorders them:
- Always kept: the first line and every
P tableline. - The remaining budget is filled by priority level, each level added to all blocks at once; the first level that does not fit completely ends the comment, and later levels are not added: file metadata (
meta-source,meta-author,meta-created,meta-exported,meta-as-of,meta-license,meta-tool) →column→agg-*→sql-pre-data→sql-post-data→ othermeta-*→sql-dql→chart/chart-<engine>→note/link. Comments are never included. - If anything was dropped, the line
P meta-comment-truncated(a flag, no value) follows the first line.
- Always kept: the first line and every
- The comment is advisory; the main file wins. A comment that is not UTF-8 (
zip-comment-not-utf8), not a valid prefix (zip-comment-not-prefix), or disagrees with the main file beyond truncation (zip-comment-disagree) is a WARN and MUST NOT block reading. txtab peek ARCHIVEreads only the comment; summingdata-size=over itsP tablelines estimates the unpacked size of the data.
Verification
txtab verify ARCHIVE.txtab.zip checks:
- the main file is first and correctly named;
- every
reference=resolves to an entry; column-stored tables have the right files, line counts and sections; - everything
txtab verifychecks on a.txtab, withrows-mismatchandcolumns-mismatchescalated to FAIL; - the comment, if present (WARN only).