Quey is available on desktop

The Capture Tool: a field guide
Technical7 min read

The Capture Tool: a field guide

How region capture actually works: the drag, the screenshot handoff, the element-scoring algorithm behind every crop, multi-format clipboard output, and freehand capture.

Capture is the odd one out among Quey's tools — it doesn't select an element, it selects a rectangle. Here's what happens between the drag and the clipboard.

The drag

Opening Capture switches the cursor to a crosshair and dims the page. Click-drag defines a rectangle; release under an 8px threshold in either dimension and nothing happens — the drag is treated as a stray click, not a zero-size capture. Selection and hover visuals from the Select tool are hidden for the duration, since the selection overlay renders above the capture dim shade and would otherwise float over it looking like a stray selection.

What actually gets screenshotted

The content script can't call Chrome's tab-capture API directly — only the extension's background service worker can — so capture sends a CAPTURE_VISIBLE_TAB message across that boundary and waits for a data URL back. Before that message goes out, Quey hides its own UI and waits two animation frames (a requestAnimationFrame nested inside another) to guarantee the hide has actually painted before the screenshot fires — a single frame isn't a reliable guarantee that a style change has hit the screen yet. The full-tab screenshot is then cropped to your rectangle on a canvas, scaled by devicePixelRatio so the output stays sharp on high-DPI displays.

The rectangle isn't just pixels — it's scored

This is the part that isn't visible in the UI at all. Behind every region capture, Quey scores every element on the page against your rectangle and picks a primary element plus up to four related elements, not just the crop. The scoring weighs, per candidate:

  • overlap ratio (how much of the element's own area falls inside your rectangle) — weighted heaviest
  • coverage ratio (how much of your rectangle the element itself covers)
  • whether the element's center point falls inside the rectangle
  • how deep the element sits in the semantic component tree (capped, so this never dominates)
  • a boost for interactive elements — links and buttons score higher than a generic wrapping div

Candidates are deduplicated by containment — if a scored element's parent is already picked, the child is skipped, so you don't end up with both a card and the card's inner wrapper counted separately — then the top five survive, highest-scored first. Structurally meaningless wrappers are filtered out before scoring even starts, so a bare layout div never outscores the button sitting inside it.

The result travels with the capture as primaryElement and relatedElements — real CapturedElement objects, source location and all, not just a bounding box. A region capture hands an agent a starting point for the thing in that area, not just a picture of it.

What lands on the clipboard

A region capture writes four formats simultaneously:

  • image/png — the cropped screenshot
  • text/plain — a one-line caption built from the primary element's semantic path (Semantics: Header › Nav › ActionRow), or Semantics: Unknown if nothing scored
  • text/html — the image wrapped in a <figure>/<figcaption>, for tools that paste rich content
  • application/x-quey+json — the full CapturedRegionSelection: the rect, the page profile, primary and related elements, all of it

Skipping the clipboard entirely

Capture has a second mode that doesn't touch the clipboard at all: trigger it from the Agent panel instead of the dock, and the same crop-and-score pipeline runs, but the result posts straight to the agent session instead of waiting for a paste. Same drag, same scoring, different destination — useful when you already know you're about to describe what should happen to that region rather than just wanting a picture of it.

Freehand, when a rectangle isn't the shape you need

Capture's rectangle-drag isn't the only way to grab part of the page — a separate freehand draw mode lets you sketch an arbitrary region, sharing the same screenshot-and-crop machinery underneath. It's covered in the Annotate guide, since its two destinations are the Agent panel or a comment, not the clipboard.

Tips

  • Don't fight for pixel-perfect edges — the scoring system is forgiving about exactly where your rectangle's border falls, since it weighs overlap and coverage rather than requiring an exact bounding match.
  • Drag loosely around a section rather than trying to trace one element exactly — capturing a card plus a little surrounding whitespace still resolves that card as primaryElement thanks to the interactive/depth weighting.
  • If you want an agent to act on the region instead of just seeing it, start Capture from the Agent panel — it skips the clipboard round-trip entirely.
  • Check the caption before you paste somewhere that only accepts plain text — Semantics: Unknown means nothing in your rectangle scored highly enough to identify, which usually means the drag was too loose or landed on genuinely generic markup.