Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Add a format

A format has two parts: a descriptor in datui-cli, with the facts about it that need no file, and a reader in datui-lib, with the code that reads it. Both are matched exhaustively, so a format missing either part does not compile.

StepWhere
A variantFileFormat in crates/datui-cli/src/formats.rs, and its place in FileFormat::ALL
A descriptorA const beside the others, spread from BASE, DELIMITED, READ_INTO or MODEL, and its line in FileFormat::descriptor
A parser and a READERA module of its own in crates/datui-lib/src/formats/; a format Polars reads goes in formats/readers/polars.rs
Its line in the registryreaders::of in crates/datui-lib/src/formats/readers/mod.rs
Its docs pageA heading for it on a family page in docs/formats/, and its line in format_page in crates/datui-cli/src/docgen.rs
Generated docscargo run -p datui-cli --bin gen_docs -- write: the format table, the format count, --format’s and --table’s help

The open, the home screen, --help and the docs all read the descriptor:

#![allow(unused)]
fn main() {
const ELF: Descriptor = Descriptor {
    name: "elf",
    title: "ELF",
    extensions: &["elf", "axf"],
    tables: Some(Tables {
        opens: Some("its symbols"),
        by_name: false,
        help: "symbols (default) or sections",
    }),
    summary: Summary::Tab("ELF"),
    ..BASE
};
}
FieldSays
nameWhat --format takes and the home screen counts (12 parquet). The dataset cache stores it, so never rename one
titleIts name in a sentence and in the docs
extensions, name_endingsThe file names that identify it. An extension may belong to only one format
read, compressed, streamHow a local file is read: Lazy, Converted or InMemory, and how a compressed file is read
http, bucket_object, bucket_prefixHow a remote file, object or prefix is read
many_filesWhether several files of it are one table
linesFor text read a line at a time: what --follow reads
tablesThe tables a file of it holds, and what --table takes
summaryIts Info panel tab, or why it has none
declares_types, conversionWhether a file declares its column types; what the loading screen says while it converts

The reader starts from readers::BASE:

#![allow(unused)]
fn main() {
pub(crate) const READER: crate::formats::readers::Reader = crate::formats::readers::Reader {
    scan,
    signatures: &[crate::formats::readers::Signature {
        says: |head, _| looks_like(head),
        kind: crate::formats::readers::Kind::Magic,
        trusted: crate::formats::readers::EVERYWHERE,
    }],
    tables: Some(|_| Ok(tables())),
    ..crate::formats::readers::BASE
};
}
FieldHolds
scanOpens the files and returns a frame, or a Scan that the load turns into one
convertConverts a file into files datui can scan, when the scan returns Scan::ReadInto
signaturesThe leading bytes that identify it, how they match (Magic, Structure, Text), and where they are trusted (Trusted: pipes, unnamed files, listings, files of tables)
tables, table_schemaThe tables, and their columns, that the home screen lists
facts, previewCheap reads for the Info panel and the home screen’s preview
python, exportThe Polars call that Copy as Python writes; the default export format

The module docs of formats/readers/mod.rs list the places outside a format’s own module that still name it, and why.

Tests that catch a miss

TestFails when
every_format_is_listed (datui-cli)A variant is missing from ALL
names_and_extensions_say_one_formatTwo formats claim an extension or a name
help_and_refusals_come_from_the_descriptors--format’s or --table’s help leaves it out
conversions_say_what_they_readA converted format has no loading-screen text, or shares another format’s
read_mode_by_format_and_storage (datui-cli)Its read modes differ from the test’s list; update the list on purpose
readers_agree_with_their_descriptors (datui-lib)The reader lists tables, converts or scans a prefix where the descriptor says otherwise
the_docs_tab_table_agrees_with_the_descriptorsdocs/user-guide/dataset-info.md’s table of tabs has no row for it
the_copy_docs_name_each_format_s_readerdocs/user-guide/copying.md’s reader table has no row for it, in --format’s order
the_generated_docs_are_currentgen_docs write was not run

A hand-written parser of untrusted bytes also gets a fuzz target and a seed corpus; see Fuzzing. Give the format’s docs page an example that opens a public file or a file the block creates, so the doc-example runner tests it.

scripts/dev/test.sh cli
scripts/dev/test.sh unit readers::