twilight venturesthe exports
Every export writes a new file and leaves the original alone. The file that is open is refused as a destination, a canceled export leaves nothing behind, not even the scratch file the writer keeps beside the destination while it works, and when the export finishes a balloon offers to open the result: a JSONL file lands in JSONL Lens and a Parquet file in Parquet Lens when those are installed. Export is a Pro action on the JetBrains plugins and on the VS Code extensions. The copy-outs are free where a product has them: a row as JSON and the page as CSV in Parquet Lens, SQLite Lens and DuckDB Lens, the page as TSV in XLSX Lens, the visible rows as CSV in the VS Code extensions.
what each product writes
| product | scope | formats |
|---|---|---|
| JSONL Lens | the current page, the search matches or the filter matches | .jsonl, CSV, Parquet |
| Log Lens | the current page, the filtered view, the search matches or the query matches | .log, .jsonl, CSV, Parquet |
| Parquet Lens | all rows, the search matches or the filter matches | CSV, JSONL, Parquet; the profile report |
| SQLite Lens | a table, the search matches or any query result | CSV, JSONL, Parquet; the profile report |
| XLSX Lens | the sheet, the search matches or the current filtered view | CSV, JSONL, Markdown, Parquet; the profile report |
| Notebook Lens | the source of every cell, outputs stripped; or the whole notebook | .py (percent format), Markdown; HTML for the whole notebook |
| DuckDB Lens | a table, the search matches or any query result | CSV, JSONL, Parquet |
| Heap Dump Lens | no export | |
| JSONL Lens for VS Code | all records, the filter matches or the search matches | JSONL, CSV |
| Parquet Lens for VS Code | all rows, the filter matches or the search matches | CSV, JSONL |
csv
CSV follows RFC 4180 with a comma. Parquet Lens also offers the Excel EU dialect, semicolons and a decimal comma, which is what Excel and LibreOffice expect on the locales that use the semicolon as their list separator, and remembers the choice across files. Log Lens writes $level, $time and $message first and then one column per parsed field in the order the fields first appear, so a plain log comes out as a table without any setup.
jsonl
JSONL is the hand-off format inside the family. A Pro export from Log Lens, Parquet Lens, SQLite Lens, DuckDB Lens or XLSX Lens writes the rows out as JSON Lines, and the finished-export balloon opens the file in JSONL Lens as a browsable, searchable grid when it is installed. The filter language is shared, so an expression written for JSONL Lens works in Parquet Lens, Log Lens and the two VS Code extensions.
parquet
An Avro, TFRecord, Arrow or Feather dataset becomes one snappy-compressed Parquet file in one step from the Parquet Lens plugin for JetBrains IDEs, and a SQLite table, a DuckDB table, an Excel sheet, a log or a JSONL file goes out the same way from its own plugin, which makes Parquet the conversion path of the family; the two VS Code extensions export CSV and JSONL. Where the source declares no column types, the columns are typed from their values: whole numbers become int64, other numbers double, true and false a boolean column, a nested object or array is written as JSON text, a column that mixes kinds is written as the text the grid shows, and a field a record does not have is null. Parquet Lens keeps the type the file declares where the writer has it, and XLSX Lens writes a column of dates as dates, or as timestamps when the cells carry a time.
the profile report
The Parquet Lens, SQLite Lens and XLSX Lens plugins have a second Pro export, the column profile report. It is asked for by name: end the export's file name in .profile.html, .profile.md or .profile.json and the rows you chose are read once and turned into a report instead of a copy. The report takes every column in turn. The counts of rows, nulls and empties come first, then the kinds of value the column holds. A numeric column also gets its min, max and mean, quartiles, a histogram and an outlier count. A text column gets its distinct count and the values that appear most. A date or timestamp column gets its earliest and latest value and the widest gap between two neighbors. The table as a whole gets a duplicate-row count, exact up to two million rows, and a correlation matrix over its numeric columns, forty at most. Any figure that came from a sample rather than from every row says so next to the number. The HTML version opens in the browser from the finished balloon and the Markdown pastes into a ticket as it is. The JSON keeps the numbers at full precision.
notebooks
Notebook Lens exports differently, since a notebook is cells rather than rows. Its export takes the source of every cell, outputs stripped, and writes a .py file in the percent format that VS Code and Spyder read, or a Markdown file with the code fenced, with every cell present and the cell count of the export matching the notebook's. The other export takes the whole notebook, outputs and images included, and writes one HTML page that opens offline. Either way the file is written as a sibling part file first and renamed into place once it is complete, so canceling halfway leaves whatever was there before untouched.