Apply a universal 1.5x playback speed to computer-use recordings - #14970
Draft
warp-agent-staging[bot] wants to merge 2 commits into
Draft
Apply a universal 1.5x playback speed to computer-use recordings#14970warp-agent-staging[bot] wants to merge 2 commits into
warp-agent-staging[bot] wants to merge 2 commits into
Conversation
Wires the server-owned playback speed multiplier forward to both substrates instead of only macOS, and switches macOS's live uniform-speed default from 4x to 1.5x to match. - crates/ai: read the new fractional playback_speed wire field (crates/ai/src/agent/action/convert.rs); widen AIAgentActionType::StartRecording::playback_speed_multiplier from Option<u32> to Option<f32>. f32 isn't Eq, so AIAgentActionType and every type that transitively wraps it (AIAgentAction, AIAgentOutputMessageType, AIAgentOutputMessage in the warp crate; TuiAIBlockSection in warp_tui) drop the Eq derive and keep PartialEq, which is sufficient everywhere they're used. - crates/computer_use: bump the client fallback default from 4.0 to 1.5 (crates/computer_use/src/lib.rs). macOS live setpts filter already supported fractional values, so no mac/recording.rs change is needed beyond the default and tests. Linux gets a new final ffmpeg speed pass (apply_playback_speed in crates/computer_use/src/linux/recording.rs) applied strictly after the smart cut and overlay burn-in, so QUALITY-1112 overlay remap and 1x-master invariant are preserved; the live x11grab capture stays unaffected. - app/src/ai: resolve the fractional multiplier in execute/start_recording.rs, thread it through ActiveRecording (recording_controller.rs) into recording_finalize.rs call to post_process_recording. - Updated the QUALITY-1112 spec invariant 11 and Playback multiplier resolution to describe the new final pass instead of the old ignored-on-Linux behavior. - Wired up crates/ai/src/agent/action/convert_tests.rs into the module tree (it existed but was never included via a path mod declaration, so it never actually compiled or ran) and added coverage for fractional/real-time playback_speed conversion. - Bumped the warp_multi_agent_api git dependency to the warp-proto-apis feature branch commit that adds the new playback_speed field. This must be re-pinned to a commit on warp-proto-apis main once that PR merges, before this PR merges. Co-Authored-By: Warp Agent <agent@warp.dev>
Addresses an adversarial review of the universal-1.5x-playback-speed change with four fixes, plus real ffmpeg/X11 verification evidence. 1. Explicit real-time request no longer collapses to the default. playback_speed is now wrapped in a proto oneof (see the warp-proto-apis commit bump below) so wire presence survives the Rust conversion as a true Option<f32>: None means the server said nothing (client default applies); Some(raw) is an explicit request, including <= 1.0 for real-time, resolved by a new resolve_playback_speed_multiplier (execute/start_recording.rs) instead of being coerced back to the default. Added start_recording_tests.rs and new convert_tests.rs cases covering absent/0/1/1.5. 2. Non-finite/absurd multipliers no longer reach ffmpeg. Added computer_use::sanitize_playback_speed_multiplier (treats NaN/+-Infinity/<=1.0 as real-time, clamps to MAX_PLAYBACK_SPEED_MULTIPLIER=100.0) and applied it at every command-building site: mac new_ffmpeg_capture_command, Linux new_playback_speed_command, and the executor's resolver. Added tests for NaN/Infinity/f32::MAX on both platforms. 3. A failed final speed pass no longer strands the burned-overlay MP4. Extracted finalize_playback_speed in linux/recording.rs, which now removes overlay_path on both the success and the error path. Added failure- and success-path cleanup tests. 4. warp-proto-apis dependency bumped again to the commit that wraps playback_speed in a oneof for wire-safe presence tracking, which fix 1 depends on. Verification: installed ffmpeg and started a real Xvfb display in this environment (previously unavailable, so the ffmpeg-gated tests silently no-op'd). All 87 computer_use tests now execute for real, including a new full-pipeline test that runs the actual production pipeline (cut, overlay burn-in, speed pass) end to end and verifies the output is playable, its duration reflects the cut+speedup, and the burned-in overlay pill is visible during the action window and absent before it. Co-Authored-By: Warp Agent <agent@warp.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Client-side half of applying a universal 1.5x playback speed to computer-use screen recordings on both macOS and Linux. Previously the speed multiplier (4x) was a hardcoded client constant applied only on macOS; the server never sent a value and Linux deliberately ignored it (QUALITY-1112's smart cut needs a 1x master).
Depends on warp-proto-apis#356 (new
playback_speedfield, wrapped in a oneof) and warp-server#14366 (server now sends it). Merge order: proto-apis → warp-server → this PR.Wire-compat note
See warp-proto-apis#356 for the field design. The new
playback_speedfield is wrapped in a one-memberoneofrather than a plain scalar, so wire presence survives into the Rust binding asOption<f32>: this is required to distinguish "the server explicitly requested real-time (sent 0 or 1)" from "the server said nothing" — a plain proto3 scalar can't make that distinction (both decode to 0.0), and editions 2023 rejects an explicitoptionallabel on a field (confirmed directly against protoc 30.2). The deprecateduint32 playback_speed_multiplierfield is untouched, so old clients/servers keep their existing behavior unchanged.macOS behavior change
macOS's default drops from 4x to 1.5x (the server now sends 1.5x on every call, and the client's own fallback default also changed to 1.5x). This makes macOS recordings longer than before — the requester accepted this tradeoff explicitly.
Linux behavior
Linux's x11grab capture still records at 1x (unchanged — required for the QUALITY-1112 smart cut and ASS overlay remap to work). The playback multiplier is applied as a final ffmpeg pass, strictly after the smart cut (
cut_to_segments) and overlay burn-in (burn_overlays_into_cut), so the overlays (authored/remapped onto the cut's own timeline) are burned in before their presentation timestamps get rescaled.Changes
crates/ai: read the presence-preservingplayback_speedoneof;AIAgentActionType::StartRecording::playback_speed_multiplierisOption<f32>whereNonemeans "server said nothing" andSome(raw)is an explicit request (which may be<= 1.0for real-time).f32isn'tEq, soAIAgentActionTypeand every type that transitively wraps it (AIAgentAction,AIAgentOutputMessageType,AIAgentOutputMessageinwarp;TuiAIBlockSectioninwarp_tui) drop theEqderive and keepPartialEq(verified nothing needs theEqbound).crates/computer_use: client fallback default bumped4.0→1.5. Newsanitize_playback_speed_multiplier/MAX_PLAYBACK_SPEED_MULTIPLIER(100.0) validate every value immediately before it reaches an ffmpeg command on both platforms: non-finite (NaN/±Infinity) or<= 1.0values become real-time (no filter); larger values are clamped, so a pathological input (e.g.f32::MAX) can never collapsesetptsto zero via floating-point precision loss. Linux'sfinalize_playback_speednow removes the burned-overlay intermediate on both the success and the failure path of the speed pass (previously a failure short-circuited before cleanup, stranding a potentially large temp MP4).app/src/ai:resolve_playback_speed_multiplierinexecute/start_recording.rsis the single place that turns the wire value into what the recorder uses:None→ client default;Some(raw)→sanitize_playback_speed_multiplier(raw)(this fixes a real-time-request-collapses-to-default regression found in review). Threaded throughActiveRecording→recording_finalize.rs→post_process_recordingas before.agents/specs/QUALITY-1112-*.md: updated invariant 11 and the "Playback multiplier" resolution to describe the final pass instead of "ignored on Linux".crates/ai/src/agent/action/convert_tests.rs, which existed but was never wired into the module tree (missing#[path]mod declaration) and silently never compiled or ran; wired it up.Cargo.toml/Cargo.lock: bumpedwarp_multi_agent_apito the warp-proto-apis commit that adds the oneof-wrapped field. Must be re-pinned to a commit on warp-proto-apismainonce DUP: Duplicate of #069 - Warp should be a Drag Destination #356 merges.Verification
Local
cargo build/test/clippy -D warnings/fmt --checkall pass forcomputer_use,ai,warp(lib),warp_tui.Real ffmpeg + X11 evidence (installed ffmpeg and started a real Xvfb display in the sandbox for this revision, since the ffmpeg-gated tests previously silently no-op'd without them):
computer_usetests execute for real (not skipped), including the full X11 capture tests (records_window_target_via_native_x11grab_after_raise,records_full_display_for_screen_target).post_process_recording_runs_the_full_cut_overlay_speed_pipeline) runs the actual production pipeline — cut → overlay burn-in → 1.5x speed pass — against a real fixture through real ffmpeg, and asserts: the output is playable, its duration reflects both the cut and the 1.5x speedup, and the burned-in overlay pill is visible during the action's display window and absent beforehand. Screenshots of the two probed frames and the produced MP4 are attached to this PR (see below).NaN/±Infinity/f32::MAXon both platforms, and for the real-time-collapse regression (Some(0.0)/Some(1.0)must resolve to real-time, not the 1.5x default).finalize_playback_speed_removes_input_on_success/_on_failure).CI status of all three PRs (checked directly against each PR's workflow config):
Lintsjob unrelated to any file this PR touches, and agqlgencodegen job killed bySIGTERM/timeout — both passed cleanly on rerun and are unrelated to this change). This also re-confirmed thego.mod/go.sumtidiness fix..github/workflows/ci.yml'sparamsjob (which everything elseneeds:) hasif: github.event.pull_request.draft == false && ..., with no other condition applicable here. No CI signal is obtainable while the PR stays a draft.Conversation: link
Run: link
Computer-use video recordings
View video recording - Real recording produced end-to-end by the production post_process_recording pipeline (cut -> overlay burn-in -> 1.5x speed pass) using real ffmpeg. Duration 1.30s (expected ~1.17s for a 1.75s 1x cut at 1.5x).
This PR was generated with Oz.