twilight venturesparquet lens
JetBrains Marketplaceplugin 33510free viewer, Pro subscription
Install from the JetBrains Marketplacefree to read; the Pro trial runs 30 days
The columns go down further than the schema admits. Take the lantern that reads the footer first: the schema comes off the metadata with no scan of the rows, and a 10 GB dataset pages the way a 10 MB one does.
Parquet Lens opens Parquet, Avro, TFRecord, Arrow and Feather files in the editor tab of IntelliJ IDEA and the other JetBrains IDEs, straight off disk, with no notebook and no conversion step. The file is read in the pieces its own format has (row groups, record batches, blocks or records), so nothing is loaded whole and a 10 GB dataset pages like a small one. The schema tab shows the types, the codecs and the per-column null counts and min and max the Parquet footer carries, read from the metadata alone, before any row is decoded. Pro adds a filter, evaluated across every row group with min/max pruning where the writer stored those statistics, a sort of the whole dataset by any column rather than the visible page, exact stats over every row, an export of the rows as CSV, JSONL or Parquet, and a profile report on every column.
- opens
- .parquet, .pq, .parq, .avro, .tfrecord, .tfrecords and .tfrec (plain, gzip or zlib compressed), .arrow, .arrows, .ipc, .feather and .ftr, and .h5, .hdf5 and .he5 HDF5 files (a .nc file that carries the HDF5 signature opens the same way)
- runs on
- PyCharm, IntelliJ IDEA and every IntelliJ-based IDE from 2024.2 on, with no upper build
- store
- JetBrains Marketplace, plugin 33510
- newest version
- 1.9.11, published 2026-10-22
- pro
- a subscription with its own 30-day trial; the price is quoted below as the store shows it
free
- Any size: the file opens on its metadata, rows arrive in pages of up to 5000 in constant memory, go-to-row lands on any row, and there is no row cap
- Arrow and Feather: Arrow IPC file and stream formats, Feather v2 with lz4 and zstd compression and dictionary-encoded columns
- Avro containers: blocks decoded only when paged into view; null, deflate, snappy and zstandard codecs; nested records, arrays and maps as readable cells, logical types decoded to their values
- TFRecord datasets: tf.Example and SequenceExample records paged by their framing, every feature key a column, byte features as text or as an image label with its size
- Schema view: logical and physical types, nullability, codec, encodings, and per-column null counts and min/max from the footer, no scan needed
- Find in file: streaming text search across every row, at multi-gigabyte scale
- Column stats on the visible page: nulls, distinct, min, max and mean, always labeled with their scope
- Full types: timestamps, dates, times, decimals, uuid, float16, binary, and nested lists, structs and maps rendered as JSON with a pretty detail pane
- Bytes as: a raw binary column read as text in the charset you pick, in the grid, the detail pane, find, copy and export alike
- Copy out: a row as JSON, the page as CSV
- Named errors: a corrupt or truncated file reports a clear reason, and a codec the reader does not have is refused by name
- Themes: eight looks for the grid; trying one is free for the open window
pro
- Filter queries: keep only the rows where columns match, for example
lang == "en" && score >= 0.5, evaluated across every row group with min/max pruning - Export: write all rows, the search matches or the filter matches to CSV, JSONL or a new Parquet file; an Avro, TFRecord, Arrow or Feather dataset becomes Parquet in one step
- Column profile: a report on every column, with ranges, histograms, outliers, the largest gap in a timestamp column, a duplicate-row count and a correlation matrix, written as HTML, Markdown or JSON
- Global sort: sort the whole dataset by any column, not just the visible page
- Exact stats: distinct counts and true distributions over every row, beyond what the metadata can tell you
- Kept themes: a chosen look stays set from file to file
price, as the store shows it
The viewer is free and stays free. Pro is a subscription for this plugin: USD 2.90 a month or USD 29 a year for individual use. For organizations it is, per user, USD 5.80 a month or USD 58 a year.
where it stops
- No editing: files are never written to; a Pro export creates a new file, a CSV, JSONL or Parquet copy or a profile report
- Not a query engine: the Pro filter predicates and the whole-file sort, no SQL, no joins across files
We wrote the Arrow reader ourselves against the format specification, and the test suite checks every decode path against files the reference implementation wrote.
what's new
- 1.9.11published 2026-10-22
Four kinds of damaged or changing file now get a message that says what is wrong. An Arrow or Feather file that another program cuts short while its rows are on screen says the file changed on disk and offers Reload. An Arrow schema nested deeper than 64 levels is refused at open as nested too deep, where a stack overflow raised the error. An Avro file whose block record count is damaged is refused with the block named, where another block's records showed at the wrong row numbers. A gzip or zlib TFRecord file that inflates more than 100 times past 128 MB is refused as a decompression bomb, with the advice to decompress it outside the IDE, where a 1.6 MB file of zeros wrote 1 GB into the temp folder.
- 1.9.10published 2026-10-17
Feather files that pandas or pyarrow write at their defaults, and Arrow files polars writes under lz4, showed their columns and then no rows. The first page, Find, the filter, the sort and exact stats stopped on "LZ4 decompression failed: Dependent block stream is unsupported", and an export wrote no rows. For pandas and pyarrow files this began once a column passed one 64 KB lz4 block, about 8,000 rows of 64-bit numbers; polars files failed at any size. The lz4 library these tools use links each block to the one before it, and the Arrow reader now follows those links the way the Parquet reader already did, so these files read in full.
- 1.9.9published 2026-10-15
The profile report kept up to 100,000 of a column's distinct strings to count them, hundreds of megabytes on a column of long values; it keeps a 64-bit hash of each now, about 2 MB a column at the cap. A buyer whose subscription has lapsed sees Get Pro in the footer, and the gate says the trial has ended; 1.9.8 offered a second free trial there, which JetBrains does not grant. The footer reads the license key eight times on a refresh and used to check its certificate chain every time; the verdict is kept per key now.
more
- every format, every product
- opening a large file
- the exports and the profile report
- the VS Code edition
- support
Pricing, invoices and licensing questions go to sales@twilightventures.dev; anything else about the product to support@twilightventures.dev.