datui documentation

Open, query, chart and export tabular data in your terminal. datui reads local files and cloud storage, and works with Polars frames in Python.

Parquet, CSV, JSON, Arrow, Excel, SQLite, logs, audio, model files and more, including binary formats you describe in a format spec.

Start here: Install datui, then follow the quick start with a small public dataset.

Find a task

Help while you work

Press ? for the current screen’s keys. The footer says what is in effect, and the keys of whatever mode is active. Esc backs out; Ctrl+Q quits. The search button above searches this manual.

Choose another version, watch the demos, or report a problem.

Install datui

datui runs on Linux, macOS and Windows. Pick one method, check it with datui --version, then go on to the quick start.

Linux and macOS, one line

curl -fsSL https://raw.githubusercontent.com/derekwisong/datui/main/scripts/install/install.sh | sh

The script installs the latest release for your platform:

It checks each download against the release’s SHA256SUMS, asks before it changes apt or runs an AUR helper (with no terminal to ask on, it goes ahead), and runs the installed binary before it reports success. Options go after sh -s --:

Package-manager and binary-download options are below; Uninstall says how to remove each.

Without root

Where there is no sudo, as in Azure Cloud Shell, the script installs the binary into ~/.local/bin (or $XDG_BIN_HOME) instead, and says how to put that on your PATH if it is not there. Pass --user to do the same on a machine that has sudo:

curl -fsSL https://raw.githubusercontent.com/derekwisong/datui/main/scripts/install/install.sh | sh -s -- --user

Package managers

On Fedora and RHEL, the one-line script installs the .rpm; or install it from the release, as below.

Homebrew needs brew trust before it will install from a third-party tap. The pip package installs the datui command and the Python module. cargo install compiles datui, which takes a C compiler and some minutes; with cargo-binstall installed, cargo binstall datui fetches the release archive for your platform instead.

Apt repository

Add the signing key and source once:

curl -fsSL https://derekwisong.github.io/datui-apt/public.key | sudo gpg --dearmor -o /usr/share/keyrings/datui-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/datui-archive-keyring.gpg] https://derekwisong.github.io/datui-apt/ ./" | sudo tee /etc/apt/sources.list.d/datui.list
sudo apt update
sudo apt install datui

After that, apt upgrade keeps datui current.

Pre-built binaries

Every release on GitHub carries these, with a SHA256SUMS file and its Sigstore signature. <VERSION> is the release’s version, such as 0.4.0:

An archive holds datui, LICENSE, man/ and completions/. Unpack one and put datui somewhere on your PATH, or install a package:

sudo apt install ./datui_<VERSION>-1_amd64.deb
sudo dnf install https://github.com/derekwisong/datui/releases/download/v<VERSION>/datui-<VERSION>-1.x86_64.rpm

From source

Needs a Rust toolchain, 1.95 or newer.

git clone https://github.com/derekwisong/datui.git
cd datui
cargo build --release --locked

The binary is target/release/datui. To build a specific release, check out its tag first (git tag --list, then git checkout vX.Y.Z). To install into ~/.cargo/bin instead, run cargo install --path . --locked from the checkout.

Five features are on by default. --no-default-features leaves them all out; add back the ones you want with --features:

cargo build --release --locked --no-default-features --features sql,streaming

Example datasets lists only what the build can open.

Manual pages

The .deb, .rpm, AUR package, Homebrew formula and release archives install the manual pages: man datui, man datui-config, man 5 datui-config, man datui-keys. pip install datui puts them in the environment’s share/man, which man searches while its bin is on your PATH.

After cargo install or a build from source, datui man shows them, and this installs them where man finds them for your user:

datui man --dir ~/.local/share/man

datui man --list lists the pages.

Shell completions

The packages, Homebrew and the release archives (completions/) install the completion scripts for bash, zsh and fish. Otherwise datui completions SHELL prints the script for the flags, commands and format names. Set it up once per shell:

Windows

winget install derekwisong.datui

Uninstall

None of these touch your config, themes, saved views, cache or log. datui config path names the config files; the directories are:

Quick start

Open public penguin measurements from the built-in catalog, compare three species, chart them, and copy the result. Install datui first.

1. Open the data

datui

Type penguins to narrow the home screen, select Palmer penguins under Example datasets, and press Enter. A catalog file this small downloads without a question.

Type penguins: one match, its details beside it. Enter opens it.

The table has 344 penguins. Empty fields in the file are null, shown as ∅; rownames is the row number the host, Rdatasets, adds. The data is CC0, from Palmer Station LTER; credit Horst, Hill and Gorman (2020).

To skip the home screen, give the URL; datui asks before it downloads a URL you type, and Enter says yes:

datui https://vincentarelbundock.github.io/Rdatasets/csv/palmerpenguins/penguins.csv

2. Group by species

Which species is heaviest? Press :; the command line opens at sql:. Type this, then press Enter:

SELECT species, AVG(body_mass_g) AS mean_mass_g, COUNT(*) AS penguins
FROM df
GROUP BY species
ORDER BY mean_mass_g DESC

Type it on one line, or press Alt+Enter for a new line.

Which species is heaviest? Gentoo, 5,076 g on average over 124 penguins. :, the query, Enter.

AVG skips the two penguins with no mass; COUNT(*) counts every row. The table is named df. Press Enter on Gentoo to see its 124 penguins, and Esc to come back.

The same summary is one line in q, a subset of q that evaluates right to left. Press :, then Ctrl+T for q::

select mean_mass_g: avg body_mass_g by species

A new query starts a fresh view, clearing sidebar filters and sort. See querying for more SQL and q.

3. Chart the measurements

Do heavier penguins have longer flippers? Press R to return to the original rows, put the column cursor on body_mass_g (g, type body_mass_g, Enter), then press c for a chart and 2 for Scatter. Y starts on the cursor’s column.

Do heavier penguins have longer flippers? Yes. c 2, then X flipper_length_mm.

↓ ↑ move between rows of the panel; Space on X opens its picker. Type part of the name and press Enter. The two penguins with no measurements are left out.

Now press 3 for Bar, choose species for X, and on Aggregate press → for count: Adelie 152, Gentoo 124, Chinstrap 68.

Press e in the chart to export a PNG, SVG or PDF. Press Esc to return to the table. More chart options.

4. Copy or export

Run the species summary again, then choose an output:

Export and copy use the current rows and columns. They do not overwrite the input file unless you export to that path and confirm.

Use your own data

Replace <FILE>, <DIRECTORY> and <BUCKET>/<PREFIX> with your own:

datui <FILE>
datui <DIRECTORY>/
datui s3://<BUCKET>/<PREFIX>/

datui alone starts at the home screen.

Next: open files and directories, formats, cloud access, or all keyboard shortcuts.

Demos

Two recordings, each of a dataset from Example datasets on the home screen, opened as its publisher serves it. Every step they show is written out in the guides below.

When should I fly out of JFK?

NYC flights (2013): 336,776 departures from New York’s three airports. Sorted by dep_delay, inspected, then charted as the mean departure delay by hour, one line per origin. At JFK it climbs from 0.5 minutes at 5:00 to 26.1 at 21:00. The data is 2013’s; this is not a forecast.

Do it yourself: Query data and Make a chart.

A public bucket, opened in place

NOAA daily weather (GHCN-D) on S3, with no login. Central Park’s 155 years of daily highs, from the catalog’s bookmark, charted by year; then one year of the whole bucket, s3://noaa-ghcn-pds/parquet/by_year/YEAR=2024/, opened as one table and scrolled to its last row. The waits are real; nothing is downloaded whole.

Recorded 2026-10-05 with datui 0.4.0-dev on an 8-core Ryzen 7 9800X3D, over a wired home connection, with a cold cache: YEAR=2024 opened its 135 files as one table of 38,466,379 rows in about a second. Your times depend on your connection and the bucket.

Do it yourself: Connect to cloud storage.

More examples

Home screen

datui with no path opens the home screen, where you find and open a dataset: recent files, directories, cloud storage, your catalog and a catalog of public data.

datui

Ctrl+O returns here from anywhere, on the dataset you left. A filter typed before comes back selected: typing replaces it, and ~ opens the path prompt. Typing narrows the list; Esc clears the filter and selects the first dataset; Enter does what the footer names for the row. Every key is in the keyboard reference.

Open a file or directory

1. Type part of a name to narrow the list.
2. Select the row and press Enter, or double-click it.
3. To browse a directory rather than read it as one table, press →.

To type a path or URL, press ~ with the filter empty. The list shows the directory being typed, narrowed by the name after the last /, with the first name picked.

s3://, gs:// and az:// complete from names datui already knows (listed sources and prefixes, recents, the example datasets); nothing is asked of the store, so s3://noaa Tab gives s3://noaa-ghcn-pds/.

The footer names where the list is, how many rows the filter matches and the order, and at the right Enter, named for what it does on the selected row (Open, Open all, Inside, Look), what Ctrl+D does there (^D Add to catalog.toml, or ^D Forget on its own rows), ^E Docs on a catalog row, then ? keys (F1 keys once a filter is typed, since ? then types). Letters type into the filter, so q types q; Ctrl+C quits.

Sections

Folds last between runs. A heading says why its section is listed and how it stands: catalog.toml, catalog or built in for a catalog; nfs4, listing (then 1,200 so far as a slow share answers), unavailable, or first 5,000 when a listing stops there.

Recent

Recent ranks datasets by frecency, as zoxide ranks directories: an open counts four times within the hour, twice within the day, half within the week and a quarter after. A place (a directory) ranks with its best dataset, and entering it shows all its files, opened or not. The cursor starts on the dataset opened last, so Enter reopens it. Places fill up to a third of the list at first; … more in … places shows the rest.

Add to your catalog

Opening a dataset adds its directory to Recent. To keep a dataset or a directory on the home screen, press Ctrl+D on its row: it goes into catalog.toml, listed under MY DATASETS. Ctrl+D again on that row forgets it. A directory there is a row to step into.

Catalogs has the file’s keys, team catalogs and datui catalog check. [home] desktop_recents = false drops ELSEWHERE; datui never writes the desktop’s file, and lists a place there only when you enter it.

Search below the current directory

Typing narrows every section and searches below the working directory in the background. Found lists matches by their path from there.

[home.search] sets the depth, time, results and exclusions; cross_filesystems = false stays off network shares and automounts.

The details pane

What is NOAA daily weather, and who publishes it? ↓ to it: an S3 dataset from NOAA, CC0, no login, with notes on its columns. Ctrl+E shows them all.

Below about 100 columns the pane hides, and the first rows show in a strip at the bottom when the list leaves four lines free.

What a row’s label says

A row reads its name, / for a directory, two spaces, and a label: data/ 3 dirs, events/ hive, Palmer penguins csv.

A SQLite database with several tables lists them inside, a row each. A workbook, an NMEA log or an ELF file opens its first worksheet, fixes or symbols on Enter, and a file a format spec reads as several record types opens whole; → lists the record types, and Enter on one opens it alone. A Hugging Face cache lists its splits, as tables, above its files.

Names starting _ or . and _$folder$ markers are passed over, except partitions such as _date=2025-01-01.

A file not read lazily says how, dim beside its name:

The … files with no reader row, or Ctrl+A, shows files no reader takes, dimmed; [home] show_unreadable = true shows them always. Enter on a local one, or Ctrl+X on any local file, shows its bytes in the hex view. Inside a SQLite database the same row shows its internal tables.

Opening a directory

→ always goes inside. Enter does what the bar says: Open all (one table), Inside, Open (a file), or Look (find out first). Inside, the first row reads the directory as one table and says how:

On a mixed directory the pane names what is read and what is skipped. Delta, Iceberg and Hudi files are read without the transaction log, so deleted rows and old versions may show. How files combine is in Open files and directories.

Storage markers

Catalogs

A catalog is a file of named datasets, local and remote, shown as a section under its label: catalog.toml (MY DATASETS), each file in catalogs/ or listed in catalogs, and EXAMPLE DATASETS. Catalogs has the keys.

▾ MY DATASETS  5   catalog.toml  ─────────────────────────────────
  ▪ Sales                                          6.8 KB   now
  ▪ Archive/ 1 csv                                          now
  ◦ Gone missing
  ≈ Weather/ dataset
  ≈ Penguins                                      ~16.1 KB

Nothing else remote is asked for until you open or enter a dataset. Inside one, the title reads My datasets › Weather › by_year, and the pane gives its description, publisher, license, homepage, URL and login.

Documentation view

Ctrl+E on a catalog row, on a place inside one, or on a file whose format spec documents it, shows what the catalog and the spec say of it, full screen. The Info panel’s Documentation tab shows the same page for the open dataset.

What does ELEMENT hold? Ctrl+E on NOAA daily weather, ↓ to ELEMENT, Enter: TMAX is the maximum temperature in tenths of a degree C.

When a catalog lists a file a spec documents, the catalog’s description and documentation link stand. Where both note a column, the catalog’s description, unit and legend each stand when it gives one, and the spec’s fill the rest: a catalog description of price keeps the spec’s unit. The columns only the spec notes, and its record types, stay.

o opens only the page’s links (homepage, documentation, url), and only http:// and https:// ones, without a user name or password. It first asks Open <URL>? with the whole URL, a host that is not ASCII in its xn-- form; Enter opens it, Esc does not. Over SSH, or on Linux and the BSDs without DISPLAY or WAYLAND_DISPLAY, the browser would not open in front of you, so o is not offered: y copies the link.

Ctrl+E takes the place of readline’s end of line: the filter has no cursor, and is edited at its end.

Example datasets

Don’t want them? [home] hide = ["examples"] in config.toml hides them for good; Delete on the heading hides them until datui cache clear; an examples.toml of your own, in catalogs/, replaces them.

Example datasets is the catalog that comes with datui: data its publishers host, read with no login, listed after your own. datui ships none of the data; datui catalog show examples prints the catalog. Selecting its heading shows how many datasets it lists, where it comes from, and how to hide it; a heading of your own catalog shows its file and its [home] hide id.

• A web file’s row gives its format and size, ~ until measured. One under 50 MB downloads without a question; if it passes 50 MB while downloading, it stops and asks once. A URL typed at ~ is always asked about.
• Once opened, a dataset comes back under Recent by its catalog name.
• The pane gives the publisher, license and homepage; check the license before you use the data.
• NYC flights, NOAA daily weather, NYC yellow taxis and Earthquakes carry their publisher’s documentation: the pane lists what the columns mean under COLUMNS, Ctrl+E shows the whole Documentation view, and the Info panel and the inspector explain them once the data is open.
• NOAA daily weather lists two bookmarks under it, Daily highs, 2024 and Central Park, NY. Enter opens one whole; the dataset’s own row still steps inside.
• A build without the http or cloud feature leaves out the rows it cannot open.
• A catalog file named examples.toml, in catalogs/ or listed, replaces this one, even after Delete hid it; [home] hide = ["examples"] hides either, and ["examples/nyc-taxis"] one entry. An empty examples.toml hides the section.

The guides use them:

Cloud sources

CLOUD lists a row per store: logins found on this machine and connections you configure. For private storage, sign in first; for a first try with no login, use Example datasets.

▾ CLOUD  5  ──────────────────────────────────────────────────────
  ≈ Amazon S3         s3      3 buckets     datui config
  ≈ Google Cloud      gcs     4 projects    project: example-project · gcloud
  ≈ Lab MinIO         s3      1 bucket      127.0.0.1:9000 · datui config
  ≈ onprem            s3      403           minio.corp.example:9000 · datui config

Enter goes down a level: S3 source › bucket › directory › object; Google Cloud source › project › bucket › …; Azure source › account › container › …. The title shows the trail (cloud › Lab MinIO › data › 2024); Backspace goes up one level and Esc back to where you started.

Loading

The rows show at once, with the buckets an earlier run listed. Nothing is listed, and no credential command (aws, gcloud, az, a credential_process) runs, until you ask:

A slow endpoint holds up only its own row. A level lists 1,000 names at a time (3,000 so far) and stops at 5,000 (first 5,000); typing past them asks the bucket for the names the filter starts, and the heading adds + 1,907 STATION=USW*. Backspace or Esc stops a listing. Typing also matches bucket names listed before, from every source.

When listing fails, the row says why in a word and the pane gives the whole message:

Delete on a source hides it until datui cache clear. To hide one for good:

[cloud]
hide = ["gcs-default"]

Which sources appear

datui adds the logins it finds and the connections you configure; detected sources lists where each is found, its id and the discover setting. Listing a bucket does not mean its objects can be read. Catalogs, public or yours, have sections of their own.

What a cloud row shows

A listing shows name, size and modification time, and labels directories as local ones are (hive, 12 parquet, dir); job files such as _SUCCESS and empty folder objects are left out. Row counts and schemas are read when a dataset opens.

Loading

Enter shows the load’s progress; Ctrl+O cancels it. A file that fails to open shows the error here, and Esc returns to the dataset open before. A network location that does not answer reads unavailable; Ctrl+R tries again.

What datui remembers

The cache holds recent paths, how often and how lately each was opened, what was measured (counts, column names, size, modification time), query history, section folds, bucket listings, hidden cloud sources and each terminal’s last answer about its background, never the data. Your catalog is in the config directory, not the cache. Local facts are measured again when a file’s size or time changes.

datui cache clear --recents

Limits

Narrow and plain terminals

Without UTF-8, markers and borders are ASCII; display.unicode overrides the guess.

Linux packages install a desktop entry for application menus and file managers’ “Open with”; launched from a menu, datui opens here.

Open files and directories

Pass datui a file, several files, a directory, a glob or a URL; files of one shape open as one table.

printf 'id,amount\n1,9.50\n2,3.25\n' > jan.csv
printf 'id,amount\n3,4.00\n' > feb.csv
datui jan.csv
datui jan.csv feb.csv
mkdir -p exports && cp jan.csv feb.csv exports/ && datui exports/

During a load, Ctrl+O cancels and returns home and Ctrl+Q quits. Every flag is in Command-line options; defaults for most are settings.

Directories

datui DIR/ needs no flag:

--hive reads a glob, or a layout that does not say so itself, as partitioned. Reading flags take part in the decision: datui --no-header exports/ keeps the first rows of headerless CSVs from being taken as headers, which would make the files look like separate tables.

Hive-partitioned data

A tree of key=value directories (year=2024/month=01/...) opens as one table, its partition columns first. Pass the root, or a glob with --hive:

datui --hive 's3://noaa-ghcn-pds/parquet/by_year/YEAR=2024/ELEMENT=T*/*.parquet'

• Only Parquet is read as a hive tree; for anything else, open one partition.
• A path that exists is never a glob: d[1].parquet opens that file.
• The Partitions tab of the Info panel lists the keys and values.
• Local and remote directories and remote globs read the schema from the Parquet footers; a local glob is handed to Polars.

Files that disagree

The schema is the union of the files’ Parquet footers; no data is scanned to find it.

Above 20,000 files the footers are sampled evenly across the list; above 64 the table may open on a partial schema while the rest load. The Info panel says what was read; Large datasets has the details.

When every file’s row count is known, an empty cell says why:

Without every row count, all three show as ∅; the column name’s marker still shows. Queries, pivots and exports write all three as null.

Compression

.gz, .zst, .bz2 and .xz files are decompressed as they open; --compression gzip|zstd|bzip2|xz names it when the extension does not.

printf 'id,amount\n1,9.50\n2,3.25\n' | gzip > sales.csv.gz
datui sales.csv.gz

Compressed CSV, TSV and PSV are decompressed once to a temporary file in --temp-dir, then scanned; -c read.decompress_in_memory=true reads them into memory instead. Which formats open compressed is in the formats table.

Temporary files

Decompressed, converted and downloaded files live in the temp directory while datui uses them.

On Windows a file still mapped cannot be removed; it is tried again as datui quits.

Binary columns

A binary column shows a dim ‹binary› instead of its bytes. Exports and analysis still read the bytes, and the inspector shows them. The color is binary_col in the theme.

Remote data

These are public and open with no login:

datui s3://noaa-ghcn-pds/parquet/by_year/YEAR=2024/ELEMENT=TMAX/
datui https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.csv

Connect to cloud storage covers logins and what is downloaded; the home screen’s cloud sources find data without typing a URL.

The table

A dataset opens in the table. ? lists its keys; Keyboard shortcuts lists every screen’s.

Only the table’s own keys act there: a letter with Ctrl or Alt held does nothing, beyond the paging keys above.

The footer

Under a thin rule at the bottom of every screen, one line says where you are and what is in effect, in pipeline order, with the cursor’s place at the right:

 weather/daily.parquet › query › prcp > 0 · date ▼        41,208 / 1,204,331  ? keys

Which day of 2013 left latest? The daily query, then ] on delay: 2013-03-08, at 83.5 minutes. The footer says query › delay ▼; the header marks delay▼.

At rest only ? keys is offered. A mode shows its two or three keys while it is active: after the column cursor moves, +/- Filter [/] Sort F Counts; with a find in effect, n/N Next Esc Clear; while following a file, t Pause; on a by view, Enter Drill. A message, such as Copied 3 rows, takes the room of the dataset’s name for a moment and never adds a line. On a narrow terminal the line gives up, in order: the dataset’s name (cut in the middle first), the filters and sort (counted), background work (its spinner alone), the query, and the position (41,208 / 1.2M). The mode’s keys and help stay.

The footer grows, up to three lines, only for something ongoing: a prompt being typed (find, the command line) or a job with a count, such as a find reading past the rows on hand (rows 1,200,000 / 3,475,226 with a bar and Esc Stop). It takes the rows from the bottom of the table; the top stays put.

Go to a row

: opens the command line; digits alone make it row:, and Enter goes there (0 is the top).

Another table of the file

T lists the other tables of the file on screen; Enter opens the one picked in place of this one, as --table would.

The one open is marked opened. Type to narrow, ↑ ↓ to move, Esc to keep the table. The query, filters and sort stay with the table they were on; recents record the new table, and a saved view for it applies. The footer offers T only on a file of several tables; on any other, T says Only one table here.

Row numbers

# shows or hides row numbers.

display.row_numbers turns them on or off whatever the format (true, false) or leaves them to the format ("auto"); display.row_numbers_start is the first row’s number. A sorted or filtered view of a scanned file numbers its rows only while # is on, because the numbering keeps the filter from being pushed into the scan. A dataset in a store, or of many files, is not numbered that way at all: there # counts the view’s rows under a sort or filter, and the footer says # counts the view.

Sort by a column

[ sorts by the cursor’s column ascending, ] descending, replacing the sort in effect; the same key again on that column takes the sort away. The header carries ▲ or ▼, and the footer names the sort. The Sort & Filter sidebar adds secondary sorts.

Empty cells and marked columns

An empty cell shows which kind of empty it is: ∅ (ASCII ~) for a null, · (ASCII .) for a column the row’s file does not have, ≠ (ASCII !) for a column its file holds in another type. A column name marked * is not in every file, or the files disagree on its type; the Info panel’s Schema tab says where the schema came from. Files that disagree has the details.

Long values, control characters and column widths: Column widths and In the table.

The mouse

Dialogs, tabs and the cell’s menu: The mouse. Shift+drag selects text in most terminals. mouse = false under [display], or --mouse=false, leaves the mouse to the terminal: Mouse and text selection.

Keys typed while datui works

While a load, query or other read runs, the footer shows a spinner, and keys typed meanwhile are held and replayed in order once the work is done.

At most 32 keys are held, and held keys are dropped with the screen they were typed at. At the loading screen nothing is held: the keys above act, the rest are dropped. The mouse is never held: the wheel across, the footer’s keys and a width drag act as their keys do, and the wheel down, a click on the table, a dropped header and the cell’s menu are dropped.

The mouse

Every mouse action is a shortcut to keys: a click or a drag does what the keys it stands for do, and nothing the mouse does needs it.

At the table

The cell’s menu

╭───────────────────────────────╮
│ ▎+      Filter to this value  │
│  -      Filter out this value │
│  F      Value counts          │
│  [      Sort ascending        │
│  ]      Sort descending       │
│  y      Copy                  │
│  Space  Inspect row           │
╰───────────────────────────────╯

Each line names its key, and running it presses that key on the cell.

In dialogs and sidebars

In the footer

While datui works

The mouse is never held for later. A click acts where its keys would act at once and is dropped where they would wait (Keys typed while datui works): a width drag acts at a busy table, as < > do, and a dropped header, a menu line or a click on a field is dropped.

Selecting text

Shift+drag selects text in most terminals. To leave the mouse to the terminal, set mouse = false under [display], or run with --mouse=false: Mouse and text selection.

Pipes and growing files

datui reads data piped to it, shows a file or a pipe as it grows, and records a stream while you view it.

Standard input

datui - reads the data piped to it, and so does datui with no path when something is piped in. Keys still come from the terminal.

printf 'id,amount\n1,9.50\n2,3.25\n' | datui
journalctl -o json -n 500 | datui
printf 'id,amount\n1,9.50\n' > sales.csv && datui - < sales.csv
(echo 1,2; echo 3,4) | datui --no-header -

• The first rows show once a thousand lines have arrived, or fewer when the producer is slower than that, and datui reads on to the end of the stream. The view stays where you put it; the footer says reading stdin and the bytes so far, and the row count reads 1,234+ until the stream ends. Value counts, analysis and charts opened meanwhile keep the rows they opened on, and t there reads the new ones, as for --follow.
• What cannot be read before it is finished (Parquet, an Arrow IPC file, Excel, a compressed stream) is read to the end first; the loading screen counts the bytes.
• NDJSON and journal JSON show the entries whose lines have ended; an object still being written waits for its newline. The journal’s columns are the fields of every line that had arrived when the table opened; NDJSON’s, those of its first 100 lines. A field first seen later joins as a column at the right once the stream ends, so the final columns cover every line. Filters and the sort stay; under a query, a reshape, a group or a drill-down, the column joins once that is cleared.
• The data is written to a temporary file in spool under the cache directory as it arrives, not the system temp directory (memory on many Linux systems); --temp-dir puts it elsewhere. The file is removed when the dataset goes; one left by a crash is removed after a week. Ctrl+O stops the read and removes the file.
• The format comes from the first bytes, unless --format or --compression names it: see Detected by content.
• The delimited text flags apply.
• The dataset is named stdin. It is not added to recents, and views match it by its columns only.

Following a growing file

--follow (-f) shows rows as they are appended: to a local CSV, TSV, PSV, NDJSON or text file, or record batches to an Arrow IPC stream. With -, it shows standard input as it arrives instead of waiting for it to end.

printf 'time,value\n1,10\n2,20\n' > readings.csv && datui -f readings.csv

(echo time,value; while sleep 0.2; do echo "$(date +%s),$RANDOM"; done) | datui -f -

The footer says following and how long ago rows last arrived, or paused and how many have arrived since, and offers t and Esc.

An Arrow IPC stream shows a record batch once the batch is whole; one with dictionary-encoded columns cannot be followed. --follow refuses Parquet, Arrow IPC files, Excel and the other formats written with a footer, compressed files, and remote data, with a message.

Recording standard input

--tee FILE records standard input to FILE while you view it: the bytes as they came, except a WAV stream’s sizes, filled in when it ends (below). The table reads FILE itself; there is no second copy.

(echo n; seq 1 1000) | datui --tee numbers.csv -

(echo time,value; while sleep 0.2; do echo "$(date +%s),$RANDOM"; done) | datui -f --tee run1.csv -

Without -f, the whole stream is recorded before the table opens. With -f, a format that cannot be followed (WAV) shows what had arrived when it opened, and the recording goes on. FILE is never removed.

--tee - passes the stream on to standard output instead, as tee does, and draws on the terminal (/dev/tty, or the console on Windows). Standard output must go to a pipe or a file. The table reads a temporary copy in --temp-dir; the bar says sent once the stream ends, and a reader downstream that stops reading stops the copy, saying so.

(echo n; seq 1 1000) | datui --tee - - | gzip > numbers.csv.gz

To record a serial device or a sound card, replace <DEVICE> with yours:

cat <DEVICE> | datui -f --tee capture.log -
arecord -D <DEVICE> -f S16_LE -r 48000 -c 2 -t wav - | datui -f --tee take1.wav -

Connect to cloud storage

Pass datui an s3://, gs://, abfss:// or https:// URL, or open a cloud source on the home screen.

datui s3://noaa-ghcn-pds/parquet/by_year/YEAR=2024/ELEMENT=TMAX/
datui gs://cloud-samples-data/bigquery/us-states/us-states.parquet
datui https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.csv

One remote path opens per run; a directory, prefix or glob can hold many files. This page is the one home for cloud logins; the variables are listed in Environment variables.

Amazon S3

The examples below use your names: replace <PROFILE>, <BUCKET> and <PREFIX> with yours.

AWS_PROFILE=<PROFILE> datui s3://<BUCKET>/<PREFIX>/

Each bucket’s region is found on its own.

AWS profiles

With no keys in the environment or the config, datui uses the profile the AWS tools would.

The files are AWS_CONFIG_FILE and AWS_SHARED_CREDENTIALS_FILE, else ~/.aws/config and ~/.aws/credentials. A profile’s region and endpoint_url (or an s3 endpoint_url under its services section) apply, after AWS_ENDPOINT_URL_S3 and AWS_ENDPOINT_URL. Every other profile that can log in is a source of its own on the home screen, aws-<profile>; one with an endpoint is S3-compatible, and its URLs are s3://aws-<profile>@bucket/key.

S3-compatible storage (MinIO, R2, Ceph)

Point datui at the endpoint with the AWS variables. Replace the endpoint, keys and <BUCKET>:

AWS_ENDPOINT_URL=http://localhost:9000 AWS_ACCESS_KEY_ID=<KEY_ID> AWS_SECRET_ACCESS_KEY=<SECRET> AWS_REGION=us-east-1 datui s3://<BUCKET>/sales.parquet

The endpoint is the first of AWS_ENDPOINT_URL_S3, AWS_ENDPOINT_URL and AWS_ENDPOINT that is set. There are no flags or config keys for keys: on the command line they show in ps and the shell history.

Several stores at once

Name each store in the config, with the environment variables that hold its keys; the config holds the names, never the values:

[[cloud.connections]]
name = "lab"
kind = "s3"
endpoint_url = "http://localhost:9000"
access_key_id_env = "LAB_KEY"
secret_access_key_env = "LAB_SECRET"

[[cloud.connections]]
name = "onprem"
label = "On-prem MinIO"
kind = "s3"
endpoint_url = "https://minio.corp.example:9000"
access_key_id_env = "ONPREM_KEY"
secret_access_key_env = "ONPREM_SECRET"

A name is lowercase letters, digits and -. Put it before the bucket to say which store you mean; replace <BUCKET> and <KEY>:

datui s3://lab@<BUCKET>/<KEY>

• Servers set up in the MinIO client (mc alias set, MC_HOST_<alias>) or s3cmd need no config: they are the sources mc-<alias> and s3cfg.
• A connection of kind = "s3" without endpoint_url is a second AWS login; its URLs stay s3://bucket/key.
• A catalog keeps a dataset of one of these stores on the home screen with connection = "onprem".
• Cloud connections has every field.

Google Cloud Storage

Sign in, then open a file; replace <BUCKET> and <KEY>:

gcloud auth application-default login
datui gs://<BUCKET>/<KEY>

Every project the login can see is listed on the home screen. A project named in DATUI_GCP_PROJECT or GOOGLE_CLOUD_PROJECT is listed first, and is the one listed when the login cannot search for projects.

Azure Blob Storage

Sign in with the Azure CLI (or Connect-AzAccount in Azure PowerShell), then open a file; replace <CONTAINER>, <ACCOUNT> and <PATH>:

az login
datui abfss://<CONTAINER>@<ACCOUNT>.dfs.core.windows.net/<PATH>

With Azure tools installed and nobody signed in, the Azure row says not signed in and names the command to run. In Azure Cloud Shell, az is signed in already; the install script puts datui in ~/.local/bin there (without root).

Reading blobs as a sign-in needs the Storage Blob Data Reader role; Owner or Contributor on the subscription is not enough, except on an account with hierarchical namespace where your login owns the container. When a read is refused for that reason and the login may fetch the account’s keys, datui reads with the key instead, as the Azure Portal does; the details pane says access key, and the key stays in memory. An account with shared-key access disabled is never read that way. To read only as your sign-in:

[cloud]
use_azure_account_keys = false

Public data

Public buckets and containers open with no login:

datui s3://noaa-ghcn-pds/parquet/by_year/YEAR=2024/ELEMENT=TMAX/

A container of many datasets opens on the home screen, to browse:

datui abfss://release@overturemapswestus2.dfs.core.windows.net/

The home screen’s Example datasets lists public data with publishers and licenses. A catalog of your own reads its datasets with no login, even on a machine that has one, with auth = "anonymous". GBIF’s occurrence snapshots are CC BY-NC 4.0 (https://www.gbif.org/terms):

label = "GBIF"

[occurrences]
name = "Occurrences"
url = "s3://gbif-open-data-us-east-1/occurrence/"
auth = "anonymous"
license = "CC BY-NC 4.0"

Parquet part files with no extension, like GBIF’s occurrence.parquet/000001, open as Parquet, as does a local file with no extension that starts and ends with PAR1.

Examples on public data

NOAA’s daily weather for 2024 is one hive partition, YEAR=2024, with an ELEMENT= directory per measurement. On the home screen: Example datasets, Enter on NOAA daily weather (GHCN-D), → on by_year, Enter on YEAR=2024. Or:

datui s3://noaa-ghcn-pds/parquet/by_year/YEAR=2024/

38,466,379 rows, with YEAR and ELEMENT as columns from the directory names.

Central Park from the catalog’s bookmark, then q, ~, the path, Enter twice and G. Recorded on a wired home connection with a cold cache; the waits are real.

The most common measurements:

SELECT ELEMENT, COUNT(*) AS observations, COUNT(DISTINCT ID) AS stations
FROM df
GROUP BY ELEMENT
ORDER BY observations DESC

PRCP (precipitation) leads with 11,456,946 observations from 42,758 stations, then SNOW, TMAX and TMIN. The query reads every file.

Which measurements are most common in 2024? :, the query, Enter: 74 elements, PRCP first.

In ELEMENT=TMAX, USW00094728 is Central Park and DATA_VALUE is tenths of a degree Celsius:

SELECT CAST(STRPTIME(DATE, '%Y%m%d') AS DATE) AS day, DATA_VALUE / 10.0 AS high_c
FROM df
WHERE ID = 'USW00094728'
ORDER BY day

366 days, from −6.0 to 35.0 °C: chart day against high_c as a line.

Bitcoin blocks are partitioned by day. On the home screen: Bitcoin and Ethereum, → on btc, Enter on blocks. Or:

datui s3://aws-public-blockchain/v1.0/btc/blocks/

A filter on the partition column reads only the matching days:

SELECT EXTRACT(MONTH FROM mediantime) AS month, COUNT(*) AS blocks,
       SUM(transaction_count) AS txs
FROM df
WHERE date >= '2024-01-01' AND date < '2025-01-01'
GROUP BY month
ORDER BY month

12 rows, 4,179 to 4,761 blocks a month. The table opens before every footer is read; the bar counts them, Reading footers: …, while you work.

HTTP and HTTPS

datui https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.csv
datui --format csv 'https://earthquake.usgs.gov/fdsnws/event/1/query?format=csv&starttime=2024-01-01&endtime=2024-01-02'

The file is downloaded to the temp directory (--temp-dir), then opened; --format names a format the URL does not. The copy is removed when datui exits, a quit mid-download included (temporary files). A model file’s header is read by range instead (Model files).

Every request datui makes, to a web server or a cloud store, sends User-Agent: datui/VERSION (+https://github.com/derekwisong/datui): datui and its version, nothing about you or the machine. http.user_agent replaces it:

datui -c 'http.user_agent=<NAME/VERSION (CONTACT)>' <URL>

A file that is not there (404) or a host that does not answer says so on its home row, in place of the size, before you open it.

What gets read

The formats table has every format. Paging reads ahead of the screen; queries, sorting and analysis may read the whole input (Large datasets).

Building without cloud support

A build without the cloud and http features (from source) refuses remote URLs with a message saying so, and a bucket in a catalog reads cloud support not in this build.

Info panel

i at the table opens the Info panel: the dataset’s schema, what its format records, how it is stored and read, and what datui noticed about it. i or Esc closes it. While the dataset has unread notes, i opens on the Notes tab; after that, on Model, Audio, MIDI or VCD for those files, and on Schema for anything else.

The panel’s footer names the keys that work now. ← → (or Tab Shift+Tab) switch tabs from anywhere, and the body always has the keys: the rail marks the schema’s current row. When the schema is taller than the panel, the footer counts the columns out of view.

The file size, and a Parquet, Arrow IPC, Avro or ORC file’s tab, are read in the background the first time the panel opens for a dataset: the few KB at an end of the file the open read too, and none of its rows. Until they arrive the size and the tab read reading...; a file that cannot be read shows why in their place. Remote sources, directories, globs, datasets of several files, and compressed or streamed copies have no file size and no such tab.

Column types

Enter on the Schema tab, or Change type… in the cell menu (a right click on a cell), changes the column’s type for the view: the same names, formats and rules a format spec’s type takes. Combine into datetime… in the cell menu, on a text, date or time column, makes a column as a spec’s derived column does.

• The type row shows a changed type in the accent, and the footer says typed zip (or typed 3) before the filters and sort.
• The first time the change is made, one pass counts the values it made null, and the Notes tab says how many: code: 1 value not i64, read as null.
• Filters, charts, analysis, find, value counts and exports see the new type; a Parquet export writes it.
• A saved view keeps each change as a spec’s [columns] entry says it, { "name": "zip", "type": "str" }. On data without the column, the step is left out with a note.
• A query, a pivot or a melt starts from the data as read, without the changes.

Tabs by format

Each format has its own tab beside Schema, or none, and a file that holds several tables lists them on the home screen as places inside it (shop.db/orders, book.xlsx/Sales) that recents record and Enter opens.

CSV, TSV, PSV, JSON, NDJSON and plain text have no tab of their own: text holds its rows and nothing else. An .xlsx or .xlsm workbook lists its worksheets from its directory, without reading them; an .xls or .xlsb file keeps its worksheet names where only reading the workbook finds them, so it opens its first worksheet and --table or T names another. On the Excel and SQLite tabs, ↑ ↓ move a cursor over the worksheets or tables and Enter opens the one under it in place of this one.

A Hugging Face cache directory lists its splits inside it the same way (cache/test), above the files they are made of.

Every key of the panel is in the keyboard reference. The row count is the dataset’s, not the page’s; D at the table shows the column types in a second header row.

On a CSV, TSV or PSV file, H on the Schema tab reads the first row as data, under column_1, column_2, …, and again as column names. It reads the file again, so the query, filters and sort are cleared, and the panel closes. The footer offers it only for those files.

Model

For a SafeTensors or GGUF file, i opens on the Model tab:

Values are shown whole up to 64 KiB, so a chat template wraps over as many lines as it takes; a longer value, such as a whole tokenizer.json, ends with how much more there is. Arrays of up to 16 items are listed; longer ones, such as a tokenizer’s vocabulary, show their length ([128,256 strings]). Across several files, the first file to name a key gives its value.

Audio

For a WAV, BWF, RF64 or AIFF file, i opens on the Audio tab:

A data size of 0 or a placeholder, as a recorder leaves it, says so: the frames are counted from the file’s size.

MIDI

For a MIDI file, i opens on the MIDI tab, or on Notes first when a note never ends or a file could not be read:

For a directory of songs, the lines are totals and the tempo range, and the list is the files that could not be read, with why.

File format tabs

For a VCD dump, i opens on the VCD tab; for every other format the tab sits beside Schema.

A list of more than 10,000 shows the first 10,000 and how many more there are.

Notes

Notes explain what datui found while listing files and reading metadata:

Each note states its scope, such as in all 6,541 footers or in 20,000 of 200,000 footers (sample). Background metadata reads can update these findings. Notes reuse information gathered during loading; they do not scan the data values. For null rates, duplicates and other content checks, use Data Quality.

When all footers are available, a missing-column note may identify the first partition containing a column, or a single partition where it appears. Datui omits these patterns when metadata is sampled or partition names cannot be reliably ordered, such as part=2 and part=10.

Lake tables: datui reads their plain files without applying table metadata. The displayed row count may include deleted rows and superseded versions. See lake table directories.

Each note is one sentence and a line beneath it saying what it is based on, so in 1 of 3 files never stands for files datui has not looked at. The list shows whole notes only, never a claim without its basis; when it is taller than the panel, the corner counts the notes out of view.

Enter on a type-conflict note offers read as text: the column is read from the files that disagree too, at the type each wrote, so the values the conflict hid show, and the marks and the note go. Nothing is listed or read from the footers again. A filter or sort on the column then compares text, and a note says so. See files that disagree.

Most notes describe the dataset as opened. Filter/sort exclusion notes follow the active view and disappear when those controls are cleared. Queries, pivots and drill-downs hide dataset notes until you reset or return to the original level.

Unread notes accent the i key and open on the Notes tab. Set notes_accent = false under [display] in the config to disable the accent; the notes are still collected and the tab still appears.

Measurements

The Resources tab reports work datui can measure:

Listing and Footers appear for paths handled by datui’s metadata reader, including Parquet directories and remote sources. Paths delegated directly to Polars may omit those measurements. Last page is available on either route, unless the dataset is already known to be empty.

Footer reads can exceed the number of files: schema and row-count passes may read the same footer more than once. Use the Listing count for dataset size. A dataset opened from cached metadata reads no footers and shows no Footers row.

Inspect a row

Press Space at the table to see every field of the current row, and the focused field’s whole value. It opens on the field of the column cursor’s column. Esc or Space closes it.

Enter, or a double-click on the row, opens it too, except on a row of a by query or a SQL GROUP BY, where Enter drills down into the group and Space inspects. Where Enter drills, the footer says Enter Drill.

The title counts the rows: Row 3 of 60. Inside a drill-down it counts the group’s rows and names the group, Row 3 of 20 · region=north, since the inspector covers the breadcrumb.

The list takes the rows its fields need and the value the rest; when they do not all fit, the value keeps the lines it needs, up to half the height, and the list scrolls, with … 5 above and … 12 more at its ends. At 140 columns and wider the fields and the value sit side by side at full height, and the fields flow into as many columns as fit: a row with more fields than the screen has rows narrows the value to make room for them. At 240 columns and wider, a row with a binary field gives the value room for a hex dump of 32 bytes a line. The fields’ rule counts them, 14 · 2 null · 1 empty, and the footer offers only the keys that act now.

The table cuts long text at the edge of its column and rounds floats to its preview. The inspector shows what is stored:

Exact means the value as Polars stored it, not the spelling in a CSV file: a 1.50 read from CSV is the float 1.5.

A dataset a catalog documents says more under the value: what the field means, and what the value stands for when the documentation lists it. In NOAA’s weather data, AWDR under ELEMENT reads AWDR = Average daily wind direction (degrees), and a blank Q_FLAG reads blank = did not fail any quality assurance check.

Keys

The keyboard reference has every key.

Read a long value

Tab or Shift+Tab moves the focus, and the ▎ rail, into the value. Nothing moves: the list and the value are split by what they hold, so a long value already has its room.

Only what is on screen is wrapped, so the end of a 2 MiB string or a 1 MiB binary is as near as its start: Tab End. The rule says where the pane is: lines 41-73 of 4,000 · 1% for text, 0x0-0x1ff of 0x100000 for bytes.

Word wrap breaks between words and after / & ? , ; | -, so a URL wraps at its separators. A long run with no spaces, such as base64, fills each row instead. w switches to hard wrap, at the pane’s edge, for every value until pressed again.

Views

e cycles the views that apply to the value, and the rule names the one shown. The footer offers e only where there is more than one.

JSON text over 64 KB is indented in the background; until then it shows raw, with json, indenting... on the rule. gzip and zstd are decompressed in the background when their Text view is chosen, with decompressing... on the rule until then; bytes that turn out to hold no text go back to Hex, and the rule says not text. Escaped text tells a line break (\n) from a backslash followed by n (\\n), and shows invisible characters such as a no-break space as \u{a0}.

Compare two rows

c adds a column with the next row’s values, and Δ (* in an ASCII terminal) marks each field whose values differ. The rule counts them, 14 · 5 differ, and the title names the other row: Row 2 of 60 · compare with 3. At 240 columns and wider the row before is shown as well, in row order (previous, this, next), each named over its column, and the title says compare with 1 and 3; a field is marked when it differs from either. f then lists only the fields that differ.

m pins the current row: Compare then shows it beside each row you move to with ← →, and the title says compare with pinned 3. m on the pinned row lets it go. Esc or c leaves Compare; the next Esc closes the inspector.

Drill down into nested values

Enter on a struct, a list or an array opens it one level down: a struct’s fields, or a list’s items as [0], [1], … with their types and previews. Text that holds a JSON object or array opens the same way, its keys in the document’s order. The footer says Enter Open where it applies.

The title is the path from the row: Row 42 of 60 › customer › address. On a narrow terminal the middle steps give way to …. A list of structs, or a JSON array of objects, shows as a table with a column per field (the first object’s keys); +3 after the header counts the columns that do not fit, and the focused item’s whole value is under the table.

Text that does not parse stays where it is, and the footer says why: Not JSON: key must be a string at line 1 column 2. From then on Enter leaves it as text, read in the value like any other.

Copy a field or the row

y copies the focused value through the same clipboard as the copy dialog, as its view shows it: numbers exact, lists and structs as JSON, indented JSON in the JSON view, the text bytes hold in their Text view, other bytes as base64 (the footer says y Copy base64), a null as nothing.

Y copies the whole row as one JSON object, field names as keys and values exact, without leaving the inspector. Hidden and binary fields are included once read; otherwise the message counts them: Copied row 3: 12 fields, 2 not read.

A copy over an osc52 clipboard’s osc52_limit_kb is refused.

Open a value elsewhere

o writes the focused value, as its view shows it, to a read-only temporary file named for the field, the row and its kind (.json, .xml, .txt, .png, .pdf, .bin, …), and opens it:

A program named with arguments, such as code --wait, runs with them. Nothing is read back into the dataset.

Hidden and binary columns

The table reads only the columns it shows, and reads a binary column as a ‹binary› placeholder. In the inspector, columns hidden in Sort & Filter are listed after the others with a ⊘ mark, and they and binary columns read not read. Enter on one reads those fields for this row, in the background. From then on, while the focus stays on that field, each row moved to is read too, without holding the keys. The read also checks the row’s other fields: if a query’s sort with ties put another row there on reading again, the pane says so instead of showing that row’s values. A sort from Sort & Filter or a SQL ORDER BY keeps tied rows in order, and a SQL result comes back in one order, so a second read finds the same row.

In the table

The table keeps a row on one line. A line break in a value shows as ¶, a tab as » and another control character or a direction mark (such as U+202E, which would turn the rest of the row around) as ¤ ($, > and ? in an ASCII terminal), so line1\nline2 reads line1¶line2 instead of line1line2.

A date or datetime past the calendar’s range, such as a sentinel of i64::MIN + 1 microseconds, shows its stored number: -9223372036854775807 us since 1970-01-01 UTC, 2147483647 days since 1970-01-01. So do Describe, Data Quality and copies.

Hex view

The hex view shows a file as its bytes: the offset, the bytes in hex in groups of four, and the same bytes as ASCII. Use it to look inside a file datui has no reader for, and to work out the layout of a format spec.

The file is memory-mapped, never read whole: drawing reads only the rows on screen, and a find reads the file on a worker. A file of many gigabytes opens at once.

Find a record’s length

This makes feed.bin, 500 records of 25 bytes that each start with SYNC, and opens it:

make_feed.py

import ctypes


class Record(ctypes.LittleEndianStructure):
    _layout_ = "ms"
    _pack_ = 1  # no padding: 25 bytes a record
    _fields_ = [
        ("sync", ctypes.c_char * 4),
        ("seq", ctypes.c_uint64),
        ("price", ctypes.c_uint32),
        ("change", ctypes.c_int16),
        ("size", ctypes.c_int16),
        ("flags", ctypes.c_int32),
        ("check", ctypes.c_uint8),
    ]


with open("feed.bin", "wb") as f:
    for i in range(500):
        f.write(bytes(Record(b"SYNC", i, 3 * i, -i, i, 0, i % 256)))

python3 make_feed.py
datui --hex feed.bin

Press f, type SYNC, Enter. The status line says the matches are 25 bytes apart; R makes that the bytes per row, and the records line up:

Hex · feed.bin · 12,500 bytes · 25 bytes/row (fixed)
offset    00 01 02 03  04 05 06 07   08 09 0a 0b  0c 0d 0e 0f   10 11 12 13  14 15 16 17   18
00000000  53 59 4e 43  00 00 00 00   00 00 00 00  00 00 00 00   00 00 00 00  00 00 00 00   00  SYNC·····················
00000019  53 59 4e 43  01 00 00 00   00 00 00 00  03 00 00 00   ff ff 01 00  00 00 00 00   01  SYNC·····················
00000032  53 59 4e 43  02 00 00 00   00 00 00 00  06 00 00 00   fe ff 02 00  00 00 00 00   02  SYNC·····················
0x19 of 0x30d4 · 0.2% · found SYNC · every 25 bytes · format unknown

Bytes 4 to 11 count up in each record: a little-endian u8 field, in a format spec’s types. The byte inspector reads the bytes at the cursor every way at once, which is how the rest of the layout is found.

Layout

The byte inspector sits beside the bytes when there is room for it and 16 bytes per row; elsewhere i opens it under them, in at most half the rows, counting the readings that do not fit. r or --hex-width N fixes the bytes per row (1 to 4096) so that records line up; a row wider than the screen shows the part the cursor is in.

Bytes are colored by class: 0x00, printable ASCII, whitespace, other control bytes, 0x80 to 0xFE, and 0xFF. The colors are the hex_* slots in the color settings. In the ASCII column a byte that is not printable is · (. on a terminal without Unicode).

Keys

Moving takes vim’s keys (h j k l, w b, 0 $, g G); the keyboard reference has every key.

Go to an offset

Find

Ctrl+U in the prompt finds text as UTF-16 little-endian. A match may span rows. Every match on screen is marked, and Esc stops a find still reading a large file.

The byte inspector

At the cursor, little-endian and big-endian side by side:

A marked range (v) shows its length under the readings.

Query data

: opens the command line in the footer: a row number, or a query in SQL or q over the table. The prefix says what Enter will do: row: while the line is digits alone, else sql: or q:. Ctrl+T switches between SQL and q, keeping what is typed, and the line opens in that language next time. With a query in effect the line opens on its text, in its language, selected: typing replaces it, the arrows edit it.

To keep the rows that hold some text, find it with / and press Ctrl+G.

Running a query, or clearing one, starts a fresh view: sidebar filters, sort, frozen columns and pivot/melt are dropped. Apply them after the query. A SQL ORDER BY on columns marks their headers ▲ or ▼, as a sort does, until a sort from the sidebar replaces it. The command line stays open until the query’s first rows are in. A query that fails on the data is not applied: the reason shows under it, the table keeps what it showed, and the query stays to fix.

To start in q, set query.default_mode:

[query]
default_mode = "q"

A build without the sql feature has q alone.

Run a query

Open NYC flights (2013) from Example datasets on the home screen: 336,776 departures from JFK, LaGuardia and Newark, delays in minutes. When does a JFK departure leave late? At sql::

SELECT hour, AVG(dep_delay) AS mean_delay, COUNT(dep_delay) AS flights
FROM df
WHERE origin = 'JFK'
GROUP BY hour
ORDER BY hour

Statements are split over lines here to read; type them on one line, or press Alt+Enter for a new line. 19 rows, one per scheduled hour. The mean delay climbs from 0.5 minutes at 5:00 to 26.1 at 21:00. In q the same query is one line:

select mean_delay: avg dep_delay, flights: count dep_delay by hour where origin = "JFK"

Which airlines arrive late?

SELECT carrier, AVG(arr_delay) AS delay, COUNT(*) AS flights
FROM df
GROUP BY carrier
ORDER BY delay DESC

F9 is last at +21.9 minutes; AS, at −9.9, arrives early on average. Press Enter on AS to see its 714 flights: every one is Newark to Seattle. Esc comes back. See Drill down a GROUP BY.

Which airlines arrive late? F9, by 21.9 minutes on average; AS arrives early. :, the query, Enter, then G to AS.

Where does AS fly? Enter on AS: its 714 flights, Newark (EWR) to Seattle (SEA). Esc goes back.

SQL

SQL mode runs the SQL that the bundled version of Polars supports, not the full SQL standard. A statement reads one table, df:

Sidebar filters and sort are not part of df, and neither is the previous statement’s result: each statement starts from df again.

A statement’s rows come back in one order on every read: joins, unions, DISTINCT and groupings keep the order of the rows they read (a grouping that drills down, with no ORDER BY or LIMIT, is sorted by its keys instead), and rows that an ORDER BY ranks equal keep the order they come in. Paging through the result never repeats a row or skips one, and a LIMIT without ORDER BY keeps the same rows.

Write SQL

• The columns of df are listed under the input in their types’ colors, narrowed to the name being typed. Names with spaces complete in double quotes: "Team 1"; in q, as col["Team 1"].
• A long statement wraps onto a second line, then scrolls within the two.
• An empty input shows an example to start from: SELECT * FROM df WHERE ....

When a statement fails on a value that will not convert, the reason names the column, the values that failed and SQL that gets past them:

The count is exact when the whole column was checked before the run stopped. Otherwise it says “At least N”, from the rows read so far.

Dates and messy text

Premier League (2020-21) writes dates as Sat Sep 12 2020, and twelve postponed matches as Tue Jan 12 2021(P). Scores are text such as 4–3, with an en dash. SQL takes both apart:

SELECT Round,
       CAST(STRPTIME(SUBSTR(Date, 1, 15), '%a %b %d %Y') AS DATE) AS match_date,
       "Team 1" AS home, "Team 2" AS away,
       CAST(SPLIT_PART(FT, '–', 1) AS INT) + CAST(SPLIT_PART(FT, '–', 2) AS INT) AS goals
FROM df
ORDER BY goals DESC, match_date, home

SUBSTR drops the (P). Two matches had nine goals: Aston Villa 7–2 Liverpool on 2020-10-04 and Manchester Utd 9–0 Southampton on 2021-02-02.

Which matches had the most goals? :, the query, Enter: match_date is a date and goals a number, two nine-goal matches first.

NYC flights has a time_hour timestamp, but it is in UTC, so a late-evening departure lands on the next day. The local date is in year, month and day:

SELECT DATE(CONCAT_WS('-', year, month, day)) AS flight_date,
       COUNT(*) AS flights, AVG(dep_delay) AS delay
FROM df
GROUP BY flight_date
ORDER BY flight_date

365 rows, one per day of 2013. Chart flight_date against delay as a line and 2013-03-08 stands out at 83.5 minutes.

A date or datetime past the calendar’s range, such as a sentinel of i64::MIN + 1 microseconds, has no calendar text:

A nanosecond datetime only spans 1677-09-21 to 2262-04-11. Near those ends, such as pandas’ Timestamp.max:

Drill down a GROUP BY

Press Enter on a row of a GROUP BY result to see the rows behind it: the rows of df that passed the WHERE and share the row’s keys, with every column of df, a key that is a column first. A null key shows the rows whose key is null. Esc comes back to the grouped rows, cursor and frozen columns as they were.

On NYC yellow taxis (January 2025), 3.5 million trips, group by a computed key:

SELECT EXTRACT(HOUR FROM tpep_pickup_datetime) AS pickup_hour,
       COUNT(*) AS trips, AVG(tip_amount) AS avg_tip, AVG(fare_amount) AS avg_fare
FROM df
GROUP BY pickup_hour
ORDER BY pickup_hour

24 rows: 18:00 is the busiest hour with 267,951 trips, 4:00 the quietest with 20,033. Enter on hour 4 shows those 20,033 trips.

Where a statement does not drill, Enter inspects the row instead. Where it drills, the footer says Enter Drill. A result that drills has the keys that lead it frozen, as a q by does. Without ORDER BY or LIMIT it comes back sorted by its keys, since Polars returns groups in no fixed order.

q

q is a subset of the q language, not a complete q or q-sql, and it evaluates right to left. Use it where it is shorter than the SQL:

from df is optional and accepted after the columns and by, as in q and like SQL’s FROM df: select mean_delay: avg dep_delay by hour from df where origin = "JFK".

Right to left means a * b + c is a * (b + c), and (a + b) * 2 > 100 is (a + b) * (2 > 100): put the comparison first, 100 < (a + b) * 2. Query syntax has the grammar, every accessor and more examples on the built-in datasets.

A by clause without aggregates gives one row per group; with aggregates, one summary row per group. Press Enter on a group to drill down to its rows; a row above the table names the group, and Esc comes back. A group without aggregates shows the columns you selected; an aggregated one shows every column of the rows behind it, after the query’s where, key columns first. The cursor, frozen columns and column order come back with Esc.

Save a query

A view saves the active query, in its language, with the filters and sort, to replay on the next file of the same shape.

Find in the table

/ (or f) moves the cursor to a cell holding text, a regex or letters in order, without changing the rows; Ctrl+G keeps only the rows that match.

Type what to find. As you type, the matching cells among the rows on hand light up, and the prompt says how many are on screen (3 on screen); nothing is read for that. Press Enter: the cursor, column cursor and all, goes to the first matching cell at or after its row, highlighted until the cursor leaves it. Find reads the view as it stands (query, filters, sort, the columns shown) and never changes it.

n and N start from the cursor’s cell, as in vim: move the column cursor along a found row and n finds the next match to its right. Past the last match n comes round to the first, and the footer says Wrapped to the top; N comes round the other way. Each n typed while a find reads runs in turn: five move five matches. Esc stops them all.

In the prompt

Case is ignored until the pattern has a capital letter: oslo finds Oslo, Oslo does not find oslo. In a regex, an escape such as \S is not a capital. Each value is matched as text, so 05-17 finds a date and 2.5 a number; binary and nested columns are skipped.

Keep the matches

Ctrl+G in the prompt keeps only the rows with a match, as a filter: the footer shows it (has "chicken", has /^Chick/, name has letters "chkn" when limited to a column), and so does the Sort & Filter sidebar, where it is removed like any other filter; R resets the view. The find stays in effect, so n walks the matches among the rows kept.

The footer

While a find is in effect the footer offers n/N and Esc, and its status line shows the pattern: find "chicken", or find /^ch/ for a regex, find ~chkn for letters in order, with in name when it is limited to a column. match 3 follows when the finds so far walked from the top of the view to the cursor. Nothing counts every match: a find reads only as far as the next one.

Large data

A find starts in the rows already read. Past them it reads the view on, a window at a time, and the footer counts the rows read on a line of its own (rows 1,200,000 / 3,475,226, with a bar when the view’s rows are counted); Esc or Ctrl+O stops it. A filtered view, a CSV or NDJSON cannot skip to a window, so N there reads the rows before the cursor in one pass, and n from deep in the view reads windows no smaller than the rows above them. The order of a sorted view, a SQL result included, is the order on screen, so n never skips or repeats a row.

Sort, filter and arrange columns

s opens the Sort & Filter sidebar, where you sort, filter, hide, move, freeze and size columns.

It opens on what is in effect: the sort and the filters, one row each. The Columns tab beside it lists every column. The tab bar is the first row: ← → there switch tabs. The sidebar takes the keys every dialog takes:

Filter, sort and hide columns

Open Food nutrition (fast food) from Example datasets: 515 menu items from eight chains, nutrients per item. Find the chicken dishes with at least 40 g of protein, heaviest first:

1. Press / to find, then Ctrl+T for letters in order. Type chicken and press Ctrl+G to keep the 178 of 515 that match.
2. Press s: the sidebar opens on add sort…. Press ↓ twice, past the find’s filter, to add filter… and Space. Type protein and press Enter, >= and Enter, 40 and Enter.
3. Press ↑ three times to add sort… and Space. Type calories and press Enter, then Space on the new sort for descending.
4. Press ↑ to the tab bar, → for Columns and ↓ into the find field. Type vit, then ↓ v ↓ v to hide vit_a and vit_c.
5. Press Enter to apply.

The table shows 30 of 515, led by McDonald’s 20 piece Buttermilk Crispy Chicken Tenders at 2,430 calories. The calories header carries ▼.

Filter on a cell

At the table, + and - filter on the cell under the cursor, where the row cursor and the column cursor cross.

Each press adds a filter to the Sort & Filter tab, joined to the others with and, so you can edit or delete it there and R clears it. The value is the cell’s exactly as stored: a float to its last digit, so 0.1 + 0.2 and 0.3 are two values even where the table draws both as 0.3, and a date and time to its last fraction of a second, in its zone. A list, struct or binary cell has no value to compare; the bar says so.

Apply or cancel changes

Sidebar filters and sort apply to the current query or reshape result. Running a new query clears them, so apply the query first and the sidebar settings afterward. The footer names the filters and the sort, and counts the rows kept beside the cursor’s: 1 / 30. Info shows the dataset’s total. Canceling the sidebar discards whatever was staged; reopening it shows what is actually applied.

Columns tab

Which chicken dishes have the most protein, heaviest first? The five steps above: 30 of 515, McDonald’s 20 piece Buttermilk Crispy Chicken Tenders first. s again, ↑ → to Columns, ↓ and vit show the two hidden columns.

One row per column, with its lock, its place and direction in the sort (1▲, 2▼), a width set by hand, and a ⊘ when it is hidden. Each sorted column’s header in the table carries its own direction mark (▲/▼), so the sort and r reversing it are visible at a glance. ↓ from the tab bar reaches the find field; type in it to narrow the list, and ↓ again goes to the list, on the table’s column cursor (or the first match), then:

Column widths

Each column keeps the width it was first drawn at, so paging, scrolling, reordering, hiding and opening a sidebar move nothing. A longer value on a later page ends in … (ASCII ...).

The last column on screen also takes the room left at the right edge, so a long text there shows more of each value. A number column, which sits flush right, and a width set by hand keep their width.

A number is cut only when it is the first scrolling column and has no room; otherwise it waits, whole, for the scroll.

A query, a pivot or melt, drilling down or back up, a new sort, r and a new filter change the rows, so automatic widths are learned again from the first page they show.

At the table, the column cursor’s column takes < > (4 cells narrower or wider), = (fit to the rows on screen) and w (automatic) at once, so you see each change as you make it. The Columns tab stages the same keys for any column until Enter.

A width set with < >, = or f is kept through paging, resizing and reordering until w, C or R. Text gets exactly that width; a number column is never narrower than its numbers. Applying only width changes leaves the table on the page you were on.

The space between columns is the display.cell_padding setting: "comfortable" (2 cells, the default), "compact" (1) or a number. See the settings reference.

Frozen columns

Frozen columns stay at the left edge, left of a │, while the rest scroll. When the window is too narrow for all of them beside a usable scrolling column, the separator turns dashed (┆, ASCII :) and the frozen columns that do not fit scroll after it, so every column stays reachable. The freeze is kept: a wider window shows them all frozen again. A value or name cut short at the edge of its column ends in … (ASCII ...).

Every column carries its own direction, so calories can run descending while restaurant runs ascending. Nulls go last in either direction.

Back in the main view, r reverses every direction at once and R resets everything: query, filters, sort, column order, hidden columns and widths, frozen columns, pivot/melt, drill-down and the applied view.

Move across a wide table

The table has a column cursor as well as a row cursor: the cursor’s column is tinted from header to last row, and the cell where it crosses the current row stands out from both. On a 16-color terminal, or with NO_COLOR, its header and that cell are drawn reversed.

• Shift+→ starts the next page at the first column not shown whole, so a column cut at the edge is read whole there. A column wider than the window still moves one at a time.
• The last page is full: it ends with the last column.
• Shift+← right after Shift+→ goes back to the page it left; otherwise it ends the page before with the column left of the first one shown.
• g leaves a column already whole on screen where it is; another becomes the first after the frozen ones, or lands on the last page.
• Frozen columns stay put, and the cursor walks them too: h from the first scrolling column goes to the last frozen one, and l back goes to the first scrolling column, scrolling back to it.
• At the last page, Shift+→ takes the cursor to the last column; at the first, Shift+← takes it to the page’s first column, then the first.
• The cursor stays on its column when columns are hidden, moved or frozen in the sidebar; when its own column is hidden, the column that takes its place takes the cursor.
• g lists the columns the table shows, in its order. Hidden columns are not listed: show them with v on the Columns tab. A frozen column is on screen already, so choosing it moves nothing.

H and L move the cursor’s column itself one place left or right, the cursor with it: the same column order the Columns tab’s + - set, and R puts it back. A frozen column moves among the frozen ones, and a scrolling column among the scrolling ones.

The keys that act on one column act on the cursor’s:

Once the column cursor moves, the footer offers those keys: +/- Filter [/] Sort F Counts. It says where the cursor is too: col 43/300 before the row, while there is room. It counts the columns the table shows, frozen first; hidden columns are not counted.

Sort & Filter tab

What is in effect, under two rules: Sort, one row per key in order (1 ▲ restaurant), then add sort…; Filters, one row per filter (column, operator, value, and how it joins the row above: and/or), then add filter….

Space on add sort… opens a list of the columns not sorted yet, on the table’s column cursor: type to narrow, Enter adds the column as the last key, ascending. Space on add filter… starts a new filter on the column cursor’s column. C removes every sort and filter. While something is staged, the footer’s Apply is in the accent.

Editing a filter walks three steps on the row: pick the column (type to narrow, ↑ ↓ move, Enter chooses), pick the operator the same way, then type the value — Enter saves the row, and Esc abandons the edit and only the edit. Then Enter applies.

The operator list offers what the column’s type takes: a number, date or time has no contains, and a flag only =, != and the null tests.

The value is read as the column’s type, so > 1000 on a number column is a numeric comparison and >= 2024-01-01 on a date column compares dates:

A value the column cannot read keeps the sidebar open, and the line above the keys says why, such as day: "2024-13-01" is not a date written YYYY-MM-DD. A clock a time zone skips or repeats (the night clocks change) asks for its offset. Filters stay in place while you chart, analyze or export, and are saved in views.

From the command line

For anything more involved, a SQL WHERE or the where clause of a q query takes expressions, OR groups and date arithmetic.

Pivot and melt

p reshapes the table: Pivot turns values into columns, Melt turns columns into rows. It reshapes the rows the query and filters leave.

p opens the builder full screen: the form on the left, a live preview of the result on the right. Below 100 columns the preview sits under the form.

Under them, the first rows of the result, typed and colored as the table draws them. Each change to the form runs the preview again in the background; Enter applies the reshape to the whole view.

Pivot

Open US baby names (1880-2017) from Example datasets: 1.9 million rows of year, sex, name, n and prop. Keep three girls’ names:

SELECT year, name, n FROM df WHERE sex = 'F' AND name IN ('Emma', 'Jennifer', 'Olivia')

376 rows. Press p and choose these settings on Pivot:

Use ↓ to move through settings. Space opens a column picker; type to narrow, Space to select and Enter to close it. ← → step Aggregate.

One column per name, a row per year? The preview says 138 rows × 4 columns before anything is applied. p, then Index year, Columns name, Values n.

The status footer under the builder reads <dataset> › pivot & melt and names the keys that act on the focused field, here Enter Apply Space Open Esc Close.

Press Enter to apply.

The result has 138 rows, one per year, and the columns year, Emma, Jennifer and Olivia. A year with no count for a name is null: Jennifer is ∅ before 1916, and in 1917 and 1918. Chart it as three lines; see the examples.

Output columns are sorted alphabetically. Pivot reads the rows once, in the background, and keeps one value per index and column pair in memory; filter large datasets first. A pivot that would make more than 10,000 columns is refused, with the count it would make; so is a view whose pivot would.

A date or datetime past the calendar’s range, such as a sentinel of i64::MIN + 1 microseconds, names its column by its stored number, as the table shows it: -9223372036854775807 us since 1970-01-01 UTC.

Choose an aggregate

Available functions are last (default), first, min, max, avg, med, std and count. String values support only first and last. Those two functions are positional and retain nulls. After sorting with nulls last, last returns null for any group ending in a null.

Count with a pivot

count turns rows into tallies. On Space launches (1957-2018), with no query, press p and set:

62 rows of launches per year: O for those that reached orbit, F for failures. 1967 has 127 and 12. Rows come in the order the years first appear in the file; a line chart draws them in X order regardless.

Melt

To turn the names pivot back into rows, press p, select Melt, and set:

The name fields start as variable and value, selected: typing replaces them. Apply with Enter. The result has 414 rows with columns year, name and n: the 376 counts, plus a null for each of the 38 years Jennifer has none.

Other strategies select columns by regex (By pattern), data type (By type) or an Explicit list. The form shows how many columns match, and the preview the rows they make, before it runs.

Melting dates together with text makes the values text. A date past the calendar’s range becomes its stored number, as in a pivot.

Press R from the table to clear the reshape and other view changes.

Keys

The builder takes the keys every dialog takes. It opens on its first row, Pivot or Melt.

Views

A view saved after a pivot or melt stores the reshape and the query, filters and sort it ran over. When it is applied, the order is that query, filters and sort, the pivot or melt, then any SQL, filters and sort on its result, then column order.

Value counts

F at the table counts the rows holding each value of the column cursor’s column, with a summary of the column above them. Esc goes back.

Move the cursor with h l or g. ← → on the counts move to the previous or next column, and the table’s cursor goes with them.

Which carrier flies most

Open NYC flights (2013) from Example datasets on the home screen, press g, type carrier, Enter, then F:

Value Counts · carrier · all 336,776 rows
 Rows 336776   Distinct 16   Nulls 0

 carrier   Count▼        %  Cum %
▎UA         58665    17.4%  17.4%   ████████████████████████████████████████████
 B6         54635    16.2%  33.6%   █████████████████████████████████████████
 EV         54173    16.1%  49.7%   ████████████████████████████████████████▋
 DL         48110    14.3%  64.0%   ████████████████████████████████████▏

Four carriers fly almost two thirds of the flights. Enter on B6 shows its 54,635 flights as a drill-down, Group: carrier=B6; Esc comes back to the counts.

→ moves to flight, a number: it opens as a histogram of its counts, under a summary that answers whether the column can be added up:

 Rows 336776   Distinct 3844   Nulls 0   Sum 664096549   Mean 1971.9236   Min 1
 Max 8500

c turns a number’s histogram into the listing of its values, and back.

Histogram

A number column’s counts show as a histogram: 40 bins from the least value to the greatest, or a bin per value when an integer column spans fewer than 40. When the tails reach ten times past the 1st to 99th percentile, the bins span that range instead and the values outside it are counted under the plot: 1,207 values outside p1-p99. The bins are made from the counts, so they are exact wherever the counts are. c shows the listing; another column opens as its own type says.

What the screen shows

Values are written as the table writes them: , groups digits here too.

Keys

What is read

The counts are of the view: the query, the filters and a drill-down apply. One pass over the column counts every row, in the background; Esc or another column stops it.

A remote view, or one over 100 times the sample size (10,000,000 rows at the default), held in one Parquet or IPC file, is sampled first instead: a few of its row groups are read, and the header says sample of. a then counts every row. A view the sampler would have to read whole anyway, such as a directory of files or a CSV, is counted exactly from the start. The sample size is [analysis] sample_rows; 0 always counts every row.

Make a chart

c charts the rows the query and filters leave, starting from a chart chosen for the column cursor’s column. Esc returns to the table.

The panel says suggested for f64 under Type until the type is changed. c again from the same column on the same dataset brings back the chart as it was left; from another column it suggests again.

The panel

The panel on the left holds four shelves, the same four for every type, then the options. A shelf the type does not use stays in place, dimmed, with why (density, same as X); focus passes over it.

The plot’s title row says how the chart is made of the rows: mean by month, running sum, colored by carrier, count per bin, one box per carrier. The columns are named at their axes, the Y column over the y axis and X under the x axis at the right, so the title row is blank for a plain line or scatter. At its right end, dimmed, the chart says what it read: sample of 10,000 of 337k rows · seed 42891, 1,207 values outside p1-p99. The seed draws the same sample again; a chart of every row names none. When the row is too narrow for both, the notes are cut with …, or left out.

Plot two columns

Open Palmer penguins from Example datasets, as in the quick start, then:

1. Press c, then 2 for Scatter.
2. Choose flipper_length_mm for X and body_mass_g for Y.

Use ↓ to move through the rows and Space to open a shelf’s picker. Type part of a name to narrow it; on Y, Space toggles a column and Enter closes the picker. The picker leaves out the X column. The two penguins with no measurements are left out.

Points are drawn in X order, so a line runs left to right whatever order the table is in. Nulls are dropped per series: a row with no X is left out, and a missing Y drops that series’ point and breaks its line. Other series keep their points.

Esc keeps the chart as you left it: sort or filter the table, press c on the same column, and the same shelves chart the new view. A column the view no longer has is dropped from the chart. Opening another dataset starts over.

Aggregate

The Aggregate row under Y, on a line, scatter or bar: ← → step through none, count, distinct, sum, mean, median, stdev, quantile, min, max, first and last. With one, the rows that share an X (and a color) are made one point, or one bar, over every row of the view, in one group-by in the background; the chart says how many rows in its title row (all 336,776 rows, or rows in the groups shown when a color leaves values out without Other), and the footer Grouping 337k rows... while it runs. Without one, a chart samples, and says so.

An X of more than about 200,000 values is refused before any row is grouped, judged from a sample of X, with the advice to bucket it.

On NYC flights (2013), with no query: press c on carrier, pick arr_delay for Y and step the aggregate to mean. Sixteen bars, F9 longest at 21.92 minutes; HA and AS arrive early on average, so their bars grow left of zero.

Which airlines arrive late? F9, by 21.92 minutes on average. c on carrier, Y arr_delay, → on Aggregate to mean.

Color

Color splits a chart into one series per value of a category column, each in its own palette color. Without a pick, the ten values with the most rows are drawn (all of them, when the view is filtered to fewer); the line under Color says which. Space on that line lists every value with its rows, most first: type to narrow, Space toggles a value (up to ten), Enter charts the ones picked. The values are counted over the whole view.

Which New York airport leaves latest, and when? Newark in the evening, 31.1 minutes on average at 19:00. A line of dep_delay by hour, Aggregate mean, Color origin.

Other gathers every value without a series of its own into one more series, the legend’s last entry, in dimmed. ← → on the line under Color turn it on or off; the line then reads top 10 + 6 other or 3 picked + 4,809 other. It starts on for a scatter and off for the other types. When every value has a series there is no Other.

A null value is a series of its own, named null.

Chart types

Axes, grid and legend

Ticks fall on round values, steps of 1, 2 or 5 times a power of ten (or 25, 250 and so on), as many as the plot has room for: about one label per 15 columns and one per 4 rows. The y axis runs from the round value below the data to the one above it. Smaller unlabeled ticks mark the steps between labels where there is room.

Without UTF-8 the grid is . and : and the tick marks +. Exports and the Distribution plots write their numbers the same way.

Count rows per category

With Palmer penguins open, press c on species: a bar of its counts. The bars read Adelie 152, Gentoo 124, Chinstrap 68: no query needed.

Counts are exact. The whole view is counted in one pass that keeps a count per category and no rows, whatever the sample size, and the chart says so under the plot: all 344 rows.

Chart one value per category

With the aggregate none, a bar chart takes a result that is already one row per category: the bar is the row’s value. Group first with a query, then chart it. On NYC flights (2013):

1. Press : and run SELECT carrier, AVG(arr_delay) AS delay, COUNT(*) AS flights FROM df GROUP BY carrier ORDER BY delay DESC.
2. Press c on carrier, pick delay for Y, and step the aggregate to none.

The same sixteen bars as the mean above, F9 longest at 21.92 minutes.

A chart with no aggregate reads at most Rows rows, so a grouped result with more categories than that is sampled, and says so.

Examples on the built-in datasets

Each starts from the dataset of that name under Example datasets.

When does a JFK departure leave late? In the evening, 26.1 minutes on average at 21:00. The JFK query, then c 1 on mean_delay, X hour.

How did three names rise and fall? Jennifer peaks at 63,604 in 1972; Emma and Olivia rise after 2000. The pivot, then c 1, X year, Y all three.

Where does the earth shake? Around the Pacific. c 2 on latitude, X longitude. A rolling feed: your month draws its own points.

Large tables

On Rows:

Nothing is read while the row is being changed: the chart stays as drawn, and the line under the row says Enter to read. A size that is not one (12x, 0) says why there and is not read. Sample keeps its size while Every row is chosen.

The sample is drawn as the analysis tools draw theirs, with the same seed: 50 runs of one Parquet or IPC file, or one streamed pass over anything else. Another option, or another chart of columns already read, draws from the same rows without reading the table again. An exported chart carries the same notes under the plot. The default size comes from chart_rows in the [analysis] config section and is 10,000.

A Line chart with no aggregate and no color, over more rows than the sample size, draws an envelope instead: X is cut into half as many steps as the sample size, and each step draws its lowest and highest value, so every peak of a waveform or a long time series stays on the plot where a sample would miss it. Two streamed passes read the view: the rows and X’s range, then each step. The chart says so in its title row: min and max of 192M rows in 5,000 steps. Scatter keeps the sample, and so does a Line over Parquet read in place from S3, GCS or Azure, which the envelope would download whole twice.

Keys

In a picker: type part of a name to narrow, ↑ ↓ move, Enter or Space chooses (on a line or scatter chart’s Y and on the Color values Space toggles one in or out), Tab or Shift+Tab chooses and moves to the next or previous row, and Esc backs out of the picker alone.

Export

e in the chart view opens the export dialog, on its path. Type a path and press Enter; the keys are those of every dialog, and Ctrl+P recalls a path exported to before. A path ending .png, .svg or .pdf takes that format; any other path gets Format’s extension after it (chart.v2 writes chart.v2.png). You are asked before an existing file is overwritten, and a failed export leaves it as it was (Overwriting). A blank path, or a failed write, keeps the dialog open with the reason under the fields.

The recipe is a JSON document that reads back as a view:

datui -c chart.export_recipe=false

Text is set in IBM Plex Sans, bundled with datui, so a chart comes out the same on every machine; a character it lacks falls back to a system font. Its size follows the page: smaller on a journal column, larger on a slide. A bar chart exports up to its first 100 bars and counts the rest.

Colors

Series take chart_1 through chart_10 from the theme, in order; bars and histograms take chart_1, Other dimmed, and the grid chart_grid. The Dark export takes the same slots, its background background, and its text text_primary and text_secondary.

Two series never share a color. A slot the theme gives the same color as an earlier one is skipped, and on a 256- or 16-color terminal, or under NO_COLOR, the slots are counted as the terminal shows them: a chart draws one series per distinct color, so a 16-color terminal draws fewer (top 4 of 16 by rows), with Other for the rest when it is on. A Dark export skips a repeated slot too, and otherwise always has all ten.

Analysis

a opens Analysis: Describe, Distribution, Correlation Matrix and Data Quality, over a sample of the table.

A list of tools sits on the right: ↑ ↓ pick a tool and Enter opens it, moving into its pane. Tab or Shift+Tab moves between the list and the result. Esc in the result goes back to the list, and Esc there returns to the table. Results last while the table shows the same rows: a again shows them as you left them.

Analysis runs on the data as you see it, after any query and filters, unless the sample is set to read the source.

Describe

Summary statistics per column, like Polars’ describe: count, nulls, mean, standard deviation, min, 25th percentile, median, 75th percentile and max. Date, datetime, time and duration columns get all of them but the standard deviation, written as the table writes them; text columns get min and max. When they do not all fit, the header counts those out of view (+4 →) and ← → scroll to the last. Distribution scrolls its columns the same way.

On NYC yellow taxis (January 2025), 3,475,226 trips: press a, Enter on Describe, set Random seed to 1 in the Sample form and press Enter. The header reads Describe · sample of 100,000 of 3,475,226 rows. fare_amount has a mean of 16.92 and a median of 12.47, from −595.20 to 950.00: refunds and typos are part of the data. passenger_count and four other columns have 16,000 nulls.

Distribution

Compares each numeric column against fourteen distributions — Normal, Log-Normal, Uniform, Power Law, Exponential, Beta, Gamma, Chi-Squared, Student’s t, Poisson, Bernoulli, Binomial, Geometric and Weibull — and names the one the values are consistent with, along with the Shapiro-Francia normality statistic and p-value, coefficient of variation, outlier count (past 1.5 IQR from the quartiles or 3 standard deviations from the mean, over every value in the sample), skewness and kurtosis.

A p-value is how surprising the values would be if they came from the fitted distribution, not the probability that they did. The tests assume independent draws: a time series such as a price over years is dependent, and its histogram need not match any family.

Press Enter on a column for the detail view: the verdict, then a Q-Q plot and a histogram comparing the values with the family chosen in the list, drawn with that family’s fitted parameters. ↑ ↓ choose another family to compare with, which does not change the verdict; s toggles the histogram between linear and log scale; Esc returns to the table.

The list’s last row names the histogram’s scale. Log needs positive values: asked for on others, the histogram stays linear and the scale reads Linear in the warning color.

On the same taxi sample, every column reads No clear fit, which is honest for fares, distances and tips. fare_amount has 10,459 outliers (10.5%), skewness 6.92 and kurtosis 279.05; its detail view shows the median, 12.47, as Describe does.

Correlation matrix

Pairwise correlations between every numeric column, colored by strength. Move around with the arrow keys and press Enter on a cell for the pair: the coefficient with a plain reading of it, R², the p-value, and how many row pairs it was computed from, out of the total rows. Enter on a diagonal cell does nothing. Correlation is undefined for a constant column. r draws a new sample from the matrix, not from inside the pair’s detail.

Spearman is Pearson’s r of the ranks: it reads any relation that only rises or only falls, where Pearson reads a straight line. Both use the rows where the two columns hold a value, and both come from the one run, so m reads nothing. Spearman ranks at most 64 Mi values (rows times numeric columns); past that, as when every row of a large table is read, the matrix has Pearson only and s chooses a smaller sample.

Cells show three decimal places and the pair four. A value that would round to 1 but is not exactly 1 shows as 0.999 (0.9999 in the pair), so 1.000 always means a perfect relation.

On Palmer penguins, first hide rownames, the host’s row number, so it is not treated as a measurement: s, Tab, ↓, v, Enter. Then a, Correlation Matrix, Enter to read all 344 rows. flipper_length_mm against body_mass_g is r = 0.871, strong positive, from 342 of 344 rows.

Data quality

Choose Data Quality to check the rows in scope and get a report: what is likely wrong, what depends on intent, and which columns are clean. It opens on its Setup, which starts from the same sample as every other tool and reads nothing until you run it.

See Check data quality for the workflow and the reference for Setup, keys and metric definitions.

Sampling

Every analysis tool reads the same sample: which rows, how they are picked, how many, and the seed. The first tool you run on a dataset shows the Sample form in its pane with the cursor in it: change a setting or press Enter to run with the form as it stands; Esc goes back to the tool list. After that, every tool you pick runs at once on the same sample. A value the data does not hold, or rows that match nothing, is refused with what the data does hold. s opens the form again from any tool; Enter applies it and runs the tool on screen again, and Esc discards the edit. The other tools’ results go with the old sample, so switching tools compares like with like. The header says what was read: Describe · sample of 100,000 of 36,839,175 rows · source year=2020..2022.

Each kind of rows brings its own settings, with what it needs to know:

Partitions, files and time ranges read the source, ignoring the query and filters.

How a spread sample is read depends on the source:

The sort is left out of an analysis read: no statistic depends on it. While a cancelled run is still finishing, no tool starts another read: a, r, v and a new run wait. Data Quality reads the same sample, at the same size, as every other tool.

A view with its own sample (S at the table) is read whole by every tool: the header says Reads the view's sample 100,000 of 3.48M, and s edits the view’s sample, drawing it again before the tool runs; Every row there takes it away.

The sample’s starting size is analysis.sample_rows; 0 starts at every row. For one run, --sample-rows N:

[analysis]
sample_rows = 100000

Check data quality

Data Quality reports missing values, repeated values and changes across files or time: a, Data Quality, Enter.

The pane is Setup: every setting a run takes, before anything is read. Press Enter to run it as it stands, or change it first. Nothing in Setup reads the data; Enter does, once.

s opens the Sample form over Setup; its Enter applies the sample to Setup and returns there. Esc discards everything changed in Setup. After a run, e opens Setup again.

Check missing values

Open Food nutrition (fast food) from Example datasets, then:

1. Press a, choose Data Quality and press Enter.
2. Press Enter in Setup. The table’s 515 rows are fewer than the sample size, so every row is read.

Press 2 for Columns: each column’s missing count and findings. A filter changes the rows checked; to ignore it, press s, set Rows from to the unfiltered source, and press Enter twice: once to apply the sample, once to run.

On a large file the run reads a sample. On NYC yellow taxis (January 2025) with Random seed 1, the header reads Data Quality · sample of 100,000 of 3,475,226 rows, and the one note is passenger_count, RatecodeID and three more columns missing together in 16,000 rows (16.0%) of the sample.

Inspect a finding

On Overview, select a finding and press Enter: its numbers, the evidence and what to check. Press Enter again to open the matching rows; on a sampled run these are the sample’s rows, the ones the finding counted. Esc returns to the finding.

The rows come from the ones the run kept, so opening them reads nothing. A full scan keeps no rows, and an older sample may have been released: then Enter shows what reading the rows would take, and only Enter there reads. Esc reads nothing.

To see fewer findings, press c for one column’s or t for one type’s; o orders them by rows affected, then by rate. Problems stay above notes, the line above the list says what is narrowed, and Esc shows every finding again. None of it reads or measures anything.

← → move between the pages.

Compare parts of a dataset

Press e for Setup. Move to Grain and press ← →, or Space for the list: by file, by a partition column, by day, week or month of a date column, or in chunks of rows. Set Compare to the segment before or a baseline the same way, then press Enter to run.

Segments lists each segment’s rows, its share of null cells and its largest change: a row count that halved or doubled, or a column’s rate that moved past sampling noise. o puts the largest changes first, Enter shows a segment’s columns beside the one it is compared with, and b makes the selected segment the baseline.

Trends draws each column across the whole range, the rows per segment first, pooling consecutive segments into bars; m changes the measure. A sample thin per segment, such as 100,000 rows over years of days, names only large changes on Segments; Trends shows the smaller ones. To judge single days, sample Equal per value of the date, or read every row.

Read a trend

On Trends, a sampled report draws the rows the sample drew under the exact rows, and says how many segments it reached thinly or not at all: a segment with rows and no sampled row is a bar of its own mark (·, ? in ASCII), never a short bar. Select a line and press Enter for its bars; ↑ ↓ walk them.

When segments are thin, press w: Setup opens with the next coarser window staged (days to weeks, weeks to months). Its Read says what that costs, often nothing, since days sum into weeks; Enter runs it and Esc keeps the grain you had. Nothing on these pages reads.

Find gaps in time windows

A window with no rows is a gap only once you say rows belong in it:

1. With a time-window grain, move to Expected in Setup and press Space.
2. Choose Windows: every window of the grain, or for hours and days, weekdays only.
3. Optionally type From and Before, such as 2024-01-01 and 2025-01-01. Blank, the range runs from the first window found to the last.
4. Press Enter, then Enter to run. With a report on screen and nothing else changed, this reads nothing: the windows are checked against the counts the report holds.

Trends then sums up the expected windows, and g lists each run of them with no rows to show:

A range is checked over at most 20,000 windows; past that, Trends says so and asks for a shorter range or a coarser grain.

Measure the time between dates

1. In Setup, move to Time roles and press Space. Give each role its column: datui does not infer a column’s meaning from its name.
2. Intervals lists the pairs measured, such as event to received. Press Space on it to choose any start and end. Setup names a role that is in no interval.
3. Optionally set Latency over to count breaches, and with a daily grain, Window by to put each delay on the day it started or ended.
4. Press Enter to run, then 5 for Intervals.

A breach is duration > threshold: exactly an hour is not over an hour. Enter on an interval opens its detail at any terminal size; Enter on a count there shows its rows; on a sample, they are cut from the rows the run kept rather than read from the file again. Valid from to valid to reads as a validity period: no end is open, and an end before its start ends first.

Times stored as text

A column of times written as text, such as 01/31/2024 08:15:00, is read as time only when you say how:

1. In Setup, move to Text as time and press Space.
2. Choose the column, then its format. Each format says how many of the values on screen it reads, and the one that reads the most comes first.
3. The column now splits by day, week or month under Grain, and can take a Time role.

The format applies to this study only: every other check still sees the text. A format with an offset, such as 2024-01-31T08:15:00Z or +05:00, reads each value as an instant in UTC; a time with no zone beside it is read as UTC, and Setup says so. Values the format does not read are reported as Unparsed times, not as missing values, and Enter on the finding opens their rows. A role or grain on text with no format is named in Setup before you run.

Declare what a column must hold

Declare what a column must hold, and the run reports the rows that break it:

1. In Setup, move to Column intent and press Space.
2. Choose a column and press Space for its form.
3. Tick Key for the columns whose values together name one row, Required for a column every row must fill, type the Allowed values separated by commas (quote one that holds a comma: "a, b"), or a Minimum and Maximum. Text can be Read as a whole number or decimal; a range then compares the number. Enter applies.
4. Enter on the list returns to Setup; Enter there runs.

Each is a problem, with its rows one Enter away. Intent reads nothing of its own: it is measured on the rows the run reads, and a change to it reuses the rows a run already read. On a sample, a key finds repeats among the sampled rows only: a repeat there is a repeat in the data, and no repeat says nothing about the rest. Setup’s Read says so before you run; set the sample’s method to Every row to check every row, which adds one pass over the key’s columns.

Export the report

On any report page, press x. Type a path, choose JSON or Markdown with Tab and ← →, and press Enter.

The report is written from what is on screen: nothing is read, and the data need not still be there. A file that exists is overwritten only after you confirm, and only once the whole report is written; see Overwriting. A failed write keeps the dialog open, the path as typed and the reason under it.

Check the read size

Setup’s Read section says what Enter will read before it reads: one sampling pass (which also counts the grain’s segments when it streams), a count of the grain’s column for exact segment totals, rows a run already read, or nothing when the report is already on screen or in the session cache. After a sampled run, changing roles, text formats, column intent, row chunks or a coarser window of a counted grain reads nothing; after any run, so does changing the comparison or expected windows. A new seed, size or scope reads a new sample. The Read rule names the rows runs have kept for reuse, such as 100,000 rows kept · 12.4 MiB; d releases them, and the next run that would have used them reads its sample again.

A full scan of a remote dataset (S3, GCS, Azure) reads its scope once per check. When the whole dataset fits [analysis] quality_local_copy (2GiB by default) and the free disk, Read says 1 fetch of 8 objects (16.5 MiB) to a local copy · up to 7 passes over it: each object is fetched once into the cache directory, and every pass, and every later full scan of the dataset, reads the copy. The Read rule then adds local copy · 16.5 MiB, and d releases it with the rows. Otherwise Read says the passes go to the source, and why, such as No local copy: 3.0 GiB over the 2.0 GiB limit. See Local copy of a remote source. p shows the access plan in full. Reading every row asks for confirmation first; Esc there leaves the sample and the report as they were.

While a run reads, the progress names its stage, whether that stage reads the source, and the rows seen where the read can count them. Esc cancels: a sampling pass, or a full scan’s passes, stop at the next batch, and a local copy being fetched stops at its next chunk and is removed. A read that cannot stop is shown as finishing, in the header and in Setup, until it ends; until then Run waits, nothing else reads beside it, and the last report stays.

Under the verdict, every report says what it covers:

✓ No problems found  12 of 12 columns clean
Checks  7 sampled · 2 metadata · 1 unavailable
Rows    100,000 of 36,839,175 sampled (0.27%) · 36,839,175 traversed
Limits  Nearly unique: needs every row checked

Checks counts what ran, over what, and what could not run; Limits says why, and names thin segments and unread footers. Rows gives the rows behind the numbers, and the rows the run’s reads passed through to get them.

The header says what each result was measured on: Data Quality · sample of 100,000 of 36,839,175 rows. Missing-column and type-conflict findings describe the loaded source’s footers, whatever rows the sample covers.

See the reference for the metric definitions, and Keyboard shortcuts for every key.

Large datasets

datui reads only the rows on screen to show a table, but a query, sort, aggregation or analysis may read the whole input.

datui --sample-rows 50000 https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-01.parquet

[performance] streaming (on by default) runs what Polars can in batches; it is not a memory limit on every query. Performance has measured times to first rows and memory.

How large datasets open

A directory of more than 64 Parquet files, local or in the cloud, opens on the first and last files by name and reads the other footers in the background; the footer counts them on a line of its own. Until they are in:

• The total row count is estimated from a random sample of 2,000 footers, read first: ~4.12B (est.) in the footer and on the Info panel.
• Every empty cell shows as ∅.
• New columns join the end of the table as they are found; Notes says how much was read.
• A query, pivot or drill-down leaves the new columns out until you return to the data as opened.