mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-04-28 06:51:16 +08:00
Add ASCII video skill to creative category
This commit is contained in:
SHL0MS
250
skills/creative/ascii-video/SKILL.md
Normal file
250
skills/creative/ascii-video/SKILL.md
Normal file
@@ -0,0 +1,250 @@
|
||||
---
|
||||
name: ascii-video
|
||||
description: "Production pipeline for ASCII art video — any format. Converts video/audio/images/generative input into colored ASCII character video output (MP4, GIF, image sequence). Covers: video-to-ASCII conversion, audio-reactive music visualizers, generative ASCII art animations, hybrid video+audio reactive, text/lyrics overlays, real-time terminal rendering. Use when users request: ASCII video, text art video, terminal-style video, character art animation, retro text visualization, audio visualizer in ASCII, converting video to ASCII art, matrix-style effects, or any animated ASCII output."
|
||||
---
|
||||
|
||||
# ASCII Video Production Pipeline
|
||||
|
||||
Full production pipeline for rendering any content as colored ASCII character video.
|
||||
|
||||
## Modes
|
||||
|
||||
| Mode | Input | Output | Read |
|
||||
|------|-------|--------|------|
|
||||
| **Video-to-ASCII** | Video file | ASCII recreation of source footage | `references/inputs.md` § Video Sampling |
|
||||
| **Audio-reactive** | Audio file | Generative visuals driven by audio features | `references/inputs.md` § Audio Analysis |
|
||||
| **Generative** | None (or seed params) | Procedural ASCII animation | `references/effects.md` |
|
||||
| **Hybrid** | Video + audio | ASCII video with audio-reactive overlays | Both input refs |
|
||||
| **Lyrics/text** | Audio + text/SRT | Timed text with visual effects | `references/inputs.md` § Text/Lyrics |
|
||||
| **TTS narration** | Text quotes + TTS API | Narrated testimonial/quote video with typed text | `references/inputs.md` § TTS Integration |
|
||||
|
||||
## Stack
|
||||
|
||||
Single self-contained Python script per project. No GPU.
|
||||
|
||||
| Layer | Tool | Purpose |
|
||||
|-------|------|---------|
|
||||
| Core | Python 3.10+, NumPy | Math, array ops, vectorized effects |
|
||||
| Signal | SciPy | FFT, peak detection (audio modes only) |
|
||||
| Imaging | Pillow (PIL) | Font rasterization, video frame decoding, image I/O |
|
||||
| Video I/O | ffmpeg (CLI) | Decode input, encode output segments, mux audio, mix tracks |
|
||||
| Parallel | concurrent.futures / multiprocessing | N workers for batch/clip rendering |
|
||||
| TTS | ElevenLabs API (or similar) | Generate narration clips for quote/testimonial videos |
|
||||
| Optional | OpenCV | Video frame sampling, edge detection, optical flow |
|
||||
|
||||
## Pipeline Architecture (v2)
|
||||
|
||||
Every mode follows the same 6-stage pipeline. See `references/architecture.md` for implementation details, `references/scenes.md` for scene protocol, and `references/composition.md` for multi-grid composition and tonemap.
|
||||
|
||||
```
|
||||
┌─────────┐ ┌──────────┐ ┌───────────┐ ┌──────────┐ ┌─────────┐ ┌────────┐
|
||||
│ 1.INPUT │→│ 2.ANALYZE │→│ 3.SCENE_FN │→│ 4.TONEMAP │→│ 5.SHADE │→│ 6.ENCODE│
|
||||
│ load src │ │ features │ │ → canvas │ │ normalize │ │ post-fx │ │ → video │
|
||||
└─────────┘ └──────────┘ └───────────┘ └──────────┘ └─────────┘ └────────┘
|
||||
```
|
||||
|
||||
1. **INPUT** — Load/decode source material (video frames, audio samples, images, or nothing)
|
||||
2. **ANALYZE** — Extract per-frame features (audio bands, video luminance/edges, motion vectors)
|
||||
3. **SCENE_FN** — Scene function renders directly to pixel canvas (`uint8 H,W,3`). May internally compose multiple character grids via `_render_vf()` + pixel blend modes. See `references/composition.md`
|
||||
4. **TONEMAP** — Percentile-based adaptive brightness normalization with per-scene gamma. Replaces linear brightness multipliers. See `references/composition.md` § Adaptive Tonemap
|
||||
5. **SHADE** — Apply post-processing `ShaderChain` + `FeedbackBuffer`. See `references/shaders.md`
|
||||
6. **ENCODE** — Pipe raw RGB frames to ffmpeg for H.264/GIF encoding
|
||||
|
||||
## Creative Direction
|
||||
|
||||
**Every project should look and feel different.** The references provide a vocabulary of building blocks — don't copy them verbatim. Combine, modify, and invent.
|
||||
|
||||
### Aesthetic Dimensions to Vary
|
||||
|
||||
| Dimension | Options | Reference |
|
||||
|-----------|---------|-----------|
|
||||
| **Character palette** | Density ramps, block elements, symbols, scripts (katakana, Greek, runes, braille), dots, project-specific | `architecture.md` § Character Palettes |
|
||||
| **Color strategy** | HSV (angle/distance/time/value mapped), discrete RGB palettes, monochrome, complementary, triadic, temperature | `architecture.md` § Color System |
|
||||
| **Color tint** | Warm, cool, amber, matrix green, neon pink, sepia, ice, blood, void, sunset | `shaders.md` § Color Grade |
|
||||
| **Background texture** | Sine fields, noise, smooth noise, cellular/voronoi, video source | `effects.md` § Background Fills |
|
||||
| **Primary effects** | Rings, spirals, tunnel, vortex, waves, interference, aurora, ripple, fire | `effects.md` § Radial / Wave / Fire |
|
||||
| **Particles** | Energy sparks, snow, rain, bubbles, runes, binary data, orbits, gravity wells | `effects.md` § Particle Systems |
|
||||
| **Shader mood** | Retro CRT, clean modern, glitch art, cinematic, dreamy, harsh industrial, psychedelic | `shaders.md` § Design Philosophy |
|
||||
| **Grid density** | xs(8px) through xxl(40px), mixed per layer | `architecture.md` § Grid System |
|
||||
| **Font** | Menlo, Monaco, Courier, SF Mono, JetBrains Mono, Fira Code, IBM Plex | `architecture.md` § Font Selection |
|
||||
| **Mirror mode** | None, horizontal, vertical, quad, diagonal, kaleidoscope | `shaders.md` § Mirror Effects |
|
||||
| **Transition style** | Crossfade, wipe (directional/radial), dissolve, glitch cut | `shaders.md` § Transitions |
|
||||
|
||||
### Per-Section Variation
|
||||
|
||||
Never use the same config for the entire video. For each section/scene/quote:
|
||||
- Choose a **different background effect** (or compose 2-3)
|
||||
- Choose a **different character palette** (match the mood)
|
||||
- Choose a **different color strategy** (or at minimum a different hue)
|
||||
- Vary **shader intensity** (more bloom during peaks, more grain during quiet)
|
||||
- Use **different particle types** if particles are active
|
||||
|
||||
### Project-Specific Invention
|
||||
|
||||
For every project, invent at least one of:
|
||||
- A custom character palette matching the theme
|
||||
- A custom background effect (combine/modify existing ones)
|
||||
- A custom color palette (discrete RGB set matching the brand/mood)
|
||||
- A custom particle character set
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1: Determine Mode and Gather Requirements
|
||||
|
||||
Establish with user:
|
||||
- **Input source** — file path, format, duration
|
||||
- **Mode** — which of the 6 modes above
|
||||
- **Sections** — time-mapped style changes (timestamps → effect names)
|
||||
- **Resolution** — default 1920x1080 @ 24fps; GIFs typically 640x360 @ 15fps
|
||||
- **Style direction** — dense/sparse, bright/dark, chaotic/minimal, color palette
|
||||
- **Text/branding** — easter eggs, overlays, credits, themed character sets
|
||||
- **Output format** — MP4 (default), GIF, PNG sequence
|
||||
|
||||
### Step 2: Detect Hardware and Set Quality
|
||||
|
||||
Before building the script, detect the user's hardware and set appropriate defaults. See `references/optimization.md` § Hardware Detection.
|
||||
|
||||
```python
|
||||
hw = detect_hardware()
|
||||
profile = quality_profile(hw, target_duration, user_quality_pref)
|
||||
log(f"Hardware: {hw['cpu_count']} cores, {hw['mem_gb']:.1f}GB RAM")
|
||||
log(f"Render: {profile['vw']}x{profile['vh']} @{profile['fps']}fps, {profile['workers']} workers")
|
||||
```
|
||||
|
||||
Never hardcode worker counts, resolution, or CRF. Always detect and adapt.
|
||||
|
||||
### Step 3: Build the Script
|
||||
|
||||
Write as a single Python file. Major components:
|
||||
|
||||
1. **Hardware detection + quality profile** — see `references/optimization.md`
|
||||
2. **Input loader** — mode-dependent; see `references/inputs.md`
|
||||
3. **Feature analyzer** — audio FFT, video luminance, or pass-through
|
||||
4. **Grid + renderer** — multi-density character grids with bitmap cache; `_render_vf()` helper for value/hue field → canvas
|
||||
5. **Character palettes** — multiple palettes chosen per project theme; see `references/architecture.md`
|
||||
6. **Color system** — HSV + discrete RGB palettes as needed; see `references/architecture.md`
|
||||
7. **Scene functions** — each returns `canvas (uint8 H,W,3)` directly. May compose multiple grids internally via pixel blend modes. See `references/scenes.md` + `references/composition.md`
|
||||
8. **Tonemap** — adaptive brightness normalization with per-scene gamma; see `references/composition.md`
|
||||
9. **Shader pipeline** — `ShaderChain` + `FeedbackBuffer` per-section config; see `references/shaders.md`
|
||||
10. **Scene table + dispatcher** — maps time ranges to scene functions + shader/feedback configs; see `references/scenes.md`
|
||||
11. **Parallel encoder** — N-worker batch clip rendering with ffmpeg pipes
|
||||
12. **Main** — orchestrate full pipeline
|
||||
|
||||
### Step 4: Handle Critical Bugs
|
||||
|
||||
#### Font Cell Height (macOS Pillow)
|
||||
|
||||
`textbbox()` returns wrong height. Use `font.getmetrics()`:
|
||||
|
||||
```python
|
||||
ascent, descent = font.getmetrics()
|
||||
cell_height = ascent + descent # correct
|
||||
```
|
||||
|
||||
#### ffmpeg Pipe Deadlock
|
||||
|
||||
Never use `stderr=subprocess.PIPE` with long-running ffmpeg. Redirect to file:
|
||||
|
||||
```python
|
||||
stderr_fh = open(err_path, "w")
|
||||
pipe = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL, stderr=stderr_fh)
|
||||
```
|
||||
|
||||
#### Brightness — Use `tonemap()`, Not Linear Multipliers
|
||||
|
||||
ASCII on black is inherently dark. This is the #1 visual issue. **Do NOT use linear `* N` brightness multipliers** — they clip highlights and wash out the image. Instead, use the **adaptive tonemap** function from `references/composition.md`:
|
||||
|
||||
```python
|
||||
def tonemap(canvas, gamma=0.75):
|
||||
"""Percentile-based adaptive normalization + gamma. Replaces all brightness multipliers."""
|
||||
f = canvas.astype(np.float32)
|
||||
lo = np.percentile(f, 1) # black point (1st percentile)
|
||||
hi = np.percentile(f, 99.5) # white point (99.5th percentile)
|
||||
if hi - lo < 1: hi = lo + 1
|
||||
f = (f - lo) / (hi - lo)
|
||||
f = np.clip(f, 0, 1) ** gamma # gamma < 1 = brighter mids
|
||||
return (f * 255).astype(np.uint8)
|
||||
```
|
||||
|
||||
Pipeline ordering: `scene_fn() → tonemap() → FeedbackBuffer → ShaderChain → ffmpeg`
|
||||
|
||||
Per-scene gamma overrides for destructive effects:
|
||||
- Default: `gamma=0.75`
|
||||
- Solarize scenes: `gamma=0.55` (solarize darkens above-threshold pixels)
|
||||
- Posterize scenes: `gamma=0.50` (quantization loses brightness range)
|
||||
- Already-bright scenes: `gamma=0.85`
|
||||
|
||||
Additional brightness best practices:
|
||||
- Dense animated backgrounds — never flat black, always fill the grid
|
||||
- Vignette minimum clamped to 0.15 (not 0.12)
|
||||
- Bloom threshold lowered to 130 (not 170) so more pixels contribute to glow
|
||||
- Use `screen` blend mode (not `overlay`) when compositing dark ASCII layers — overlay squares dark values: `2 * 0.12 * 0.12 = 0.03`
|
||||
|
||||
#### Font Compatibility
|
||||
|
||||
Not all Unicode characters render in all fonts. Validate palettes at init:
|
||||
```python
|
||||
for c in palette:
|
||||
img = Image.new("L", (20, 20), 0)
|
||||
ImageDraw.Draw(img).text((0, 0), c, fill=255, font=font)
|
||||
if np.array(img).max() == 0:
|
||||
log(f"WARNING: char '{c}' (U+{ord(c):04X}) not in font, removing from palette")
|
||||
```
|
||||
|
||||
### Step 4b: Per-Clip Architecture (for segmented videos)
|
||||
|
||||
When the video has discrete segments (quotes, scenes, chapters), render each as a separate clip file. This enables:
|
||||
- Re-rendering individual clips without touching the rest (`--clip q05`)
|
||||
- Faster iteration on specific sections
|
||||
- Easy reordering or trimming in post
|
||||
|
||||
```python
|
||||
segments = [
|
||||
{"id": "intro", "start": 0.0, "end": 5.0, "type": "intro"},
|
||||
{"id": "q00", "start": 5.0, "end": 12.0, "type": "quote", "qi": 0, ...},
|
||||
{"id": "t00", "start": 12.0, "end": 13.5, "type": "transition", ...},
|
||||
{"id": "outro", "start": 208.0, "end": 211.6, "type": "outro"},
|
||||
]
|
||||
|
||||
from concurrent.futures import ProcessPoolExecutor, as_completed
|
||||
with ProcessPoolExecutor(max_workers=hw["workers"]) as pool:
|
||||
futures = {pool.submit(render_clip, seg, features, path): seg["id"]
|
||||
for seg, path in clip_args}
|
||||
for fut in as_completed(futures):
|
||||
fut.result()
|
||||
```
|
||||
|
||||
CLI: `--clip q00 t00 q01` to re-render specific clips, `--list` to show segments, `--skip-render` to re-stitch only.
|
||||
|
||||
### Step 5: Render and Iterate
|
||||
|
||||
Performance targets per frame:
|
||||
|
||||
| Component | Budget |
|
||||
|-----------|--------|
|
||||
| Feature extraction | 1-5ms |
|
||||
| Effect function | 2-15ms |
|
||||
| Character render | 80-150ms (bottleneck) |
|
||||
| Shader pipeline | 5-25ms |
|
||||
| **Total** | ~100-200ms/frame |
|
||||
|
||||
**Fast iteration**: render single test frames to check brightness/layout before full render:
|
||||
```python
|
||||
canvas = render_single_frame(frame_index, features, renderer)
|
||||
Image.fromarray(canvas).save("test.png")
|
||||
```
|
||||
|
||||
**Brightness verification**: sample 5-10 frames across video, check `mean > 8` for ASCII content.
|
||||
|
||||
## References
|
||||
|
||||
| File | Contents |
|
||||
|------|----------|
|
||||
| `references/architecture.md` | Grid system, font selection, character palettes (library of 20+), color system (HSV + discrete RGB), `_render_vf()` helper, compositing, v2 effect function contract |
|
||||
| `references/inputs.md` | All input sources: audio analysis, video sampling, image conversion, text/lyrics, TTS integration (ElevenLabs, voice assignment, audio mixing) |
|
||||
| `references/effects.md` | Effect building blocks: 12 value field generators (`vf_sinefield` through `vf_noise_static`), 8 hue field generators (`hf_fixed` through `hf_plasma`), radial/wave/fire effects, particles, composing guide |
|
||||
| `references/shaders.md` | 38 shader implementations (geometry, channel, color, glow, noise, pattern, tone, glitch, mirror), `ShaderChain` class, full `_apply_shader_step()` dispatch, audio-reactive scaling, transitions, tint presets |
|
||||
| `references/composition.md` | **v2 core**: pixel blend modes (20 modes with implementations), multi-grid composition, `_render_vf()` helper, adaptive `tonemap()`, per-scene gamma, `FeedbackBuffer` with spatial transforms, `PixelBlendStack` |
|
||||
| `references/scenes.md` | **v2 scene protocol**: scene function contract, `Renderer` class, `SCENES` table structure, `render_clip()` loop, beat-synced cutting, parallel rendering + pickling constraints, 4 complete scene examples, scene design checklist |
|
||||
| `references/troubleshooting.md` | NumPy broadcasting traps, blend mode pitfalls, multiprocessing/pickling issues, brightness diagnostics, ffmpeg deadlocks, font issues, performance bottlenecks, common mistakes |
|
||||
| `references/optimization.md` | Hardware detection, adaptive quality profiles (draft/preview/production/max), CLI integration, vectorized effect patterns, parallel rendering, memory management |
|
||||
528
skills/creative/ascii-video/references/architecture.md
Normal file
528
skills/creative/ascii-video/references/architecture.md
Normal file
@@ -0,0 +1,528 @@
|
||||
# Architecture Reference
|
||||
|
||||
## Grid System
|
||||
|
||||
### Multi-Density Grids
|
||||
|
||||
Pre-initialize multiple grid sizes. Switch per section for visual variety.
|
||||
|
||||
| Key | Font Size | Grid (1920x1080) | Use |
|
||||
|-----|-----------|-------------------|-----|
|
||||
| xs | 8 | 400x108 | Ultra-dense data fields |
|
||||
| sm | 10 | 320x83 | Dense detail, rain, starfields |
|
||||
| md | 16 | 192x56 | Default balanced, transitions |
|
||||
| lg | 20 | 160x45 | Quote/lyric text (readable at 1080p) |
|
||||
| xl | 24 | 137x37 | Short quotes, large titles |
|
||||
| xxl | 40 | 80x22 | Giant text, minimal |
|
||||
|
||||
**Grid sizing for text-heavy content**: When displaying readable text (quotes, lyrics, testimonials), use 20px (`lg`) as the primary grid. This gives 160 columns -- plenty for lines up to ~50 chars centered. For very short quotes (< 60 chars, <= 3 lines), 24px (`xl`) makes them more impactful. Only init the grids you actually use -- each grid pre-rasterizes all characters which costs ~0.3-0.5s.
|
||||
|
||||
Grid dimensions: `cols = VW // cell_width`, `rows = VH // cell_height`.
|
||||
|
||||
### Font Selection
|
||||
|
||||
Don't hardcode a single font. Choose fonts to match the project's mood. Monospace fonts are required for grid alignment but vary widely in personality:
|
||||
|
||||
| Font | Personality | Platform |
|
||||
|------|-------------|----------|
|
||||
| Menlo | Clean, neutral, Apple-native | macOS |
|
||||
| Monaco | Retro terminal, compact | macOS |
|
||||
| Courier New | Classic typewriter, wide | Cross-platform |
|
||||
| SF Mono | Modern, tight spacing | macOS |
|
||||
| Consolas | Windows native, clean | Windows |
|
||||
| JetBrains Mono | Developer, ligature-ready | Install |
|
||||
| Fira Code | Geometric, modern | Install |
|
||||
| IBM Plex Mono | Corporate, authoritative | Install |
|
||||
| Source Code Pro | Adobe, balanced | Install |
|
||||
|
||||
**Font detection at init**: probe available fonts and fall back gracefully:
|
||||
|
||||
```python
|
||||
import platform
|
||||
|
||||
def find_font(preferences):
|
||||
"""Try fonts in order, return first that exists."""
|
||||
for name, path in preferences:
|
||||
if os.path.exists(path):
|
||||
return path
|
||||
raise FileNotFoundError(f"No monospace font found. Tried: {[p for _,p in preferences]}")
|
||||
|
||||
FONT_PREFS_MACOS = [
|
||||
("Menlo", "/System/Library/Fonts/Menlo.ttc"),
|
||||
("Monaco", "/System/Library/Fonts/Monaco.ttf"),
|
||||
("SF Mono", "/System/Library/Fonts/SFNSMono.ttf"),
|
||||
("Courier", "/System/Library/Fonts/Courier.ttc"),
|
||||
]
|
||||
FONT_PREFS_LINUX = [
|
||||
("DejaVu Sans Mono", "/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf"),
|
||||
("Liberation Mono", "/usr/share/fonts/truetype/liberation/LiberationMono-Regular.ttf"),
|
||||
("Noto Sans Mono", "/usr/share/fonts/truetype/noto/NotoSansMono-Regular.ttf"),
|
||||
("Ubuntu Mono", "/usr/share/fonts/truetype/ubuntu/UbuntuMono-R.ttf"),
|
||||
]
|
||||
FONT_PREFS = FONT_PREFS_MACOS if platform.system() == "Darwin" else FONT_PREFS_LINUX
|
||||
```
|
||||
|
||||
**Multi-font rendering**: use different fonts for different layers (e.g., monospace for background, a bolder variant for overlay text). Each GridLayer owns its own font:
|
||||
|
||||
```python
|
||||
grid_bg = GridLayer(find_font(FONT_PREFS), 16) # background
|
||||
grid_text = GridLayer(find_font(BOLD_PREFS), 20) # readable text
|
||||
```
|
||||
|
||||
### Collecting All Characters
|
||||
|
||||
Before initializing grids, gather all characters that need bitmap pre-rasterization:
|
||||
|
||||
```python
|
||||
all_chars = set()
|
||||
for pal in [PAL_DEFAULT, PAL_DENSE, PAL_BLOCKS, PAL_RUNE, PAL_KATA,
|
||||
PAL_GREEK, PAL_MATH, PAL_DOTS, PAL_BRAILLE, PAL_STARS,
|
||||
PAL_BINARY, PAL_MUSIC, PAL_BOX, PAL_CIRCUIT, PAL_ARROWS,
|
||||
PAL_HERMES]: # ... all palettes used in project
|
||||
all_chars.update(pal)
|
||||
# Add any overlay text characters
|
||||
all_chars.update("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789 .,-:;!?/|")
|
||||
all_chars.discard(" ") # space is never rendered
|
||||
```
|
||||
|
||||
### GridLayer Initialization
|
||||
|
||||
Each grid pre-computes coordinate arrays for vectorized effect math:
|
||||
|
||||
```python
|
||||
class GridLayer:
|
||||
def __init__(self, font_path, font_size):
|
||||
self.font = ImageFont.truetype(font_path, font_size)
|
||||
asc, desc = self.font.getmetrics()
|
||||
bbox = self.font.getbbox("M")
|
||||
self.cw = bbox[2] - bbox[0] # character cell width
|
||||
self.ch = asc + desc # CRITICAL: not textbbox height
|
||||
|
||||
self.cols = VW // self.cw
|
||||
self.rows = VH // self.ch
|
||||
self.ox = (VW - self.cols * self.cw) // 2 # centering
|
||||
self.oy = (VH - self.rows * self.ch) // 2
|
||||
|
||||
# Index arrays
|
||||
self.rr = np.arange(self.rows, dtype=np.float32)[:, None]
|
||||
self.cc = np.arange(self.cols, dtype=np.float32)[None, :]
|
||||
|
||||
# Polar coordinates (aspect-corrected)
|
||||
cx, cy = self.cols / 2.0, self.rows / 2.0
|
||||
asp = self.cw / self.ch
|
||||
self.dx = self.cc - cx
|
||||
self.dy = (self.rr - cy) * asp
|
||||
self.dist = np.sqrt(self.dx**2 + self.dy**2)
|
||||
self.angle = np.arctan2(self.dy, self.dx)
|
||||
|
||||
# Normalized (0-1 range) -- for distance falloff
|
||||
self.dx_n = (self.cc - cx) / max(self.cols, 1)
|
||||
self.dy_n = (self.rr - cy) / max(self.rows, 1) * asp
|
||||
self.dist_n = np.sqrt(self.dx_n**2 + self.dy_n**2)
|
||||
|
||||
# Pre-rasterize all characters to float32 bitmaps
|
||||
self.bm = {}
|
||||
for c in all_chars:
|
||||
img = Image.new("L", (self.cw, self.ch), 0)
|
||||
ImageDraw.Draw(img).text((0, 0), c, fill=255, font=self.font)
|
||||
self.bm[c] = np.array(img, dtype=np.float32) / 255.0
|
||||
```
|
||||
|
||||
### Character Render Loop
|
||||
|
||||
The bottleneck. Composites pre-rasterized bitmaps onto pixel canvas:
|
||||
|
||||
```python
|
||||
def render(self, chars, colors, canvas=None):
|
||||
if canvas is None:
|
||||
canvas = np.zeros((VH, VW, 3), dtype=np.uint8)
|
||||
for row in range(self.rows):
|
||||
y = self.oy + row * self.ch
|
||||
if y + self.ch > VH: break
|
||||
for col in range(self.cols):
|
||||
c = chars[row, col]
|
||||
if c == " ": continue
|
||||
x = self.ox + col * self.cw
|
||||
if x + self.cw > VW: break
|
||||
a = self.bm[c] # float32 bitmap
|
||||
canvas[y:y+self.ch, x:x+self.cw] = np.maximum(
|
||||
canvas[y:y+self.ch, x:x+self.cw],
|
||||
(a[:, :, None] * colors[row, col]).astype(np.uint8))
|
||||
return canvas
|
||||
```
|
||||
|
||||
Use `np.maximum` for additive blending (brighter chars overwrite dimmer ones, never darken).
|
||||
|
||||
### Multi-Layer Rendering
|
||||
|
||||
Render multiple grids onto the same canvas for depth:
|
||||
|
||||
```python
|
||||
canvas = np.zeros((VH, VW, 3), dtype=np.uint8)
|
||||
canvas = grid_lg.render(bg_chars, bg_colors, canvas) # background layer
|
||||
canvas = grid_md.render(main_chars, main_colors, canvas) # main layer
|
||||
canvas = grid_sm.render(detail_chars, detail_colors, canvas) # detail overlay
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Character Palettes
|
||||
|
||||
### Design Principles
|
||||
|
||||
Character palettes are the primary visual texture of ASCII video. They control not just brightness mapping but the entire visual feel. Design palettes intentionally:
|
||||
|
||||
- **Visual weight**: characters sorted by the amount of ink/pixels they fill. Space is always index 0.
|
||||
- **Coherence**: characters within a palette should belong to the same visual family.
|
||||
- **Density curve**: the brightness-to-character mapping is nonlinear. Dense palettes (many chars) give smoother gradients; sparse palettes (5-8 chars) give posterized/graphic looks.
|
||||
- **Rendering compatibility**: every character in the palette must exist in the font. Test at init and remove missing glyphs.
|
||||
|
||||
### Palette Library
|
||||
|
||||
Organized by visual family. Mix and match per project -- don't default to PAL_DEFAULT for everything.
|
||||
|
||||
#### Density / Brightness Palettes
|
||||
```python
|
||||
PAL_DEFAULT = " .`'-:;!><=+*^~?/|(){}[]#&$@%" # classic ASCII art
|
||||
PAL_DENSE = " .:;+=xX$#@\u2588" # simple 11-level ramp
|
||||
PAL_MINIMAL = " .:-=+#@" # 8-level, graphic
|
||||
PAL_BINARY = " \u2588" # 2-level, extreme contrast
|
||||
PAL_GRADIENT = " \u2591\u2592\u2593\u2588" # 4-level block gradient
|
||||
```
|
||||
|
||||
#### Unicode Block Elements
|
||||
```python
|
||||
PAL_BLOCKS = " \u2591\u2592\u2593\u2588\u2584\u2580\u2590\u258c" # standard blocks
|
||||
PAL_BLOCKS_EXT = " \u2596\u2597\u2598\u2599\u259a\u259b\u259c\u259d\u259e\u259f\u2591\u2592\u2593\u2588" # quadrant blocks (more detail)
|
||||
PAL_SHADE = " \u2591\u2592\u2593\u2588\u2587\u2586\u2585\u2584\u2583\u2582\u2581" # vertical fill progression
|
||||
```
|
||||
|
||||
#### Symbolic / Thematic
|
||||
```python
|
||||
PAL_MATH = " \u00b7\u2218\u2219\u2022\u00b0\u00b1\u2213\u00d7\u00f7\u2248\u2260\u2261\u2264\u2265\u221e\u222b\u2211\u220f\u221a\u2207\u2202\u2206\u03a9" # math symbols
|
||||
PAL_BOX = " \u2500\u2502\u250c\u2510\u2514\u2518\u251c\u2524\u252c\u2534\u253c\u2550\u2551\u2554\u2557\u255a\u255d\u2560\u2563\u2566\u2569\u256c" # box drawing
|
||||
PAL_CIRCUIT = " .\u00b7\u2500\u2502\u250c\u2510\u2514\u2518\u253c\u25cb\u25cf\u25a1\u25a0\u2206\u2207\u2261" # circuit board
|
||||
PAL_RUNE = " .\u16a0\u16a2\u16a6\u16b1\u16b7\u16c1\u16c7\u16d2\u16d6\u16da\u16de\u16df" # elder futhark runes
|
||||
PAL_ALCHEMIC = " \u2609\u263d\u2640\u2642\u2643\u2644\u2645\u2646\u2647\u2648\u2649\u264a\u264b" # planetary/alchemical symbols
|
||||
PAL_ZODIAC = " \u2648\u2649\u264a\u264b\u264c\u264d\u264e\u264f\u2650\u2651\u2652\u2653" # zodiac
|
||||
PAL_ARROWS = " \u2190\u2191\u2192\u2193\u2194\u2195\u2196\u2197\u2198\u2199\u21a9\u21aa\u21bb\u27a1" # directional arrows
|
||||
PAL_MUSIC = " \u266a\u266b\u266c\u2669\u266d\u266e\u266f\u25cb\u25cf" # musical notation
|
||||
```
|
||||
|
||||
#### Script / Writing System
|
||||
```python
|
||||
PAL_KATA = " \u00b7\uff66\uff67\uff68\uff69\uff6a\uff6b\uff6c\uff6d\uff6e\uff6f\uff70\uff71\uff72\uff73\uff74\uff75\uff76\uff77" # katakana halfwidth (matrix rain)
|
||||
PAL_GREEK = " \u03b1\u03b2\u03b3\u03b4\u03b5\u03b6\u03b7\u03b8\u03b9\u03ba\u03bb\u03bc\u03bd\u03be\u03c0\u03c1\u03c3\u03c4\u03c6\u03c8\u03c9" # Greek lowercase
|
||||
PAL_CYRILLIC = " \u0430\u0431\u0432\u0433\u0434\u0435\u0436\u0437\u0438\u043a\u043b\u043c\u043d\u043e\u043f\u0440\u0441\u0442\u0443\u0444\u0445\u0446\u0447\u0448" # Cyrillic lowercase
|
||||
PAL_ARABIC = " \u0627\u0628\u062a\u062b\u062c\u062d\u062e\u062f\u0630\u0631\u0632\u0633\u0634\u0635\u0636\u0637" # Arabic letters (isolated forms)
|
||||
```
|
||||
|
||||
#### Dot / Point Progressions
|
||||
```python
|
||||
PAL_DOTS = " \u22c5\u2218\u2219\u25cf\u25c9\u25ce\u25c6\u2726\u2605" # dot size progression
|
||||
PAL_BRAILLE = " \u2801\u2802\u2803\u2804\u2805\u2806\u2807\u2808\u2809\u280a\u280b\u280c\u280d\u280e\u280f\u2810\u2811\u2812\u2813\u2814\u2815\u2816\u2817\u2818\u2819\u281a\u281b\u281c\u281d\u281e\u281f\u283f" # braille patterns
|
||||
PAL_STARS = " \u00b7\u2727\u2726\u2729\u2728\u2605\u2736\u2733\u2738" # star progression
|
||||
```
|
||||
|
||||
#### Project-Specific (examples -- invent new ones per project)
|
||||
```python
|
||||
PAL_HERMES = " .\u00b7~=\u2248\u221e\u26a1\u263f\u2726\u2605\u2295\u25ca\u25c6\u25b2\u25bc\u25cf\u25a0" # mythology/tech blend
|
||||
PAL_OCEAN = " ~\u2248\u2248\u2248\u223c\u2307\u2248\u224b\u224c\u2248" # water/wave characters
|
||||
PAL_ORGANIC = " .\u00b0\u2218\u2022\u25e6\u25c9\u2742\u273f\u2741\u2743" # growing/botanical
|
||||
PAL_MACHINE = " _\u2500\u2502\u250c\u2510\u253c\u2261\u25a0\u2588\u2593\u2592\u2591" # mechanical/industrial
|
||||
```
|
||||
|
||||
### Creating Custom Palettes
|
||||
|
||||
When designing for a project, build palettes from the content's theme:
|
||||
|
||||
1. **Choose a visual family** (dots, blocks, symbols, script)
|
||||
2. **Sort by visual weight** -- render each char at target font size, count lit pixels, sort ascending
|
||||
3. **Test at target grid size** -- some chars collapse to blobs at small sizes
|
||||
4. **Validate in font** -- remove chars the font can't render:
|
||||
|
||||
```python
|
||||
def validate_palette(pal, font):
|
||||
"""Remove characters the font can't render."""
|
||||
valid = []
|
||||
for c in pal:
|
||||
if c == " ":
|
||||
valid.append(c)
|
||||
continue
|
||||
img = Image.new("L", (20, 20), 0)
|
||||
ImageDraw.Draw(img).text((0, 0), c, fill=255, font=font)
|
||||
if np.array(img).max() > 0: # char actually rendered something
|
||||
valid.append(c)
|
||||
return "".join(valid)
|
||||
```
|
||||
|
||||
### Mapping Values to Characters
|
||||
|
||||
```python
|
||||
def val2char(v, mask, pal=PAL_DEFAULT):
|
||||
"""Map float array (0-1) to character array using palette."""
|
||||
n = len(pal)
|
||||
idx = np.clip((v * n).astype(int), 0, n - 1)
|
||||
out = np.full(v.shape, " ", dtype="U1")
|
||||
for i, ch in enumerate(pal):
|
||||
out[mask & (idx == i)] = ch
|
||||
return out
|
||||
```
|
||||
|
||||
**Nonlinear mapping** for different visual curves:
|
||||
|
||||
```python
|
||||
def val2char_gamma(v, mask, pal, gamma=1.0):
|
||||
"""Gamma-corrected palette mapping. gamma<1 = brighter, gamma>1 = darker."""
|
||||
v_adj = np.power(np.clip(v, 0, 1), gamma)
|
||||
return val2char(v_adj, mask, pal)
|
||||
|
||||
def val2char_step(v, mask, pal, thresholds):
|
||||
"""Custom threshold mapping. thresholds = list of float breakpoints."""
|
||||
out = np.full(v.shape, pal[0], dtype="U1")
|
||||
for i, thr in enumerate(thresholds):
|
||||
out[mask & (v > thr)] = pal[min(i + 1, len(pal) - 1)]
|
||||
return out
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Color System
|
||||
|
||||
### HSV->RGB (Vectorized)
|
||||
|
||||
All color computation in HSV for intuitive control, converted at render time:
|
||||
|
||||
```python
|
||||
def hsv2rgb(h, s, v):
|
||||
"""Vectorized HSV->RGB. h,s,v are numpy arrays. Returns (R,G,B) uint8 arrays."""
|
||||
h = h % 1.0
|
||||
c = v * s; x = c * (1 - np.abs((h*6) % 2 - 1)); m = v - c
|
||||
# ... 6 sector assignment ...
|
||||
return (np.clip((r+m)*255, 0, 255).astype(np.uint8),
|
||||
np.clip((g+m)*255, 0, 255).astype(np.uint8),
|
||||
np.clip((b+m)*255, 0, 255).astype(np.uint8))
|
||||
```
|
||||
|
||||
### Color Mapping Strategies
|
||||
|
||||
Don't default to a single strategy. Choose based on the visual intent:
|
||||
|
||||
| Strategy | Hue source | Effect | Good for |
|
||||
|----------|------------|--------|----------|
|
||||
| Angle-mapped | `g.angle / (2*pi)` | Rainbow around center | Radial effects, kaleidoscopes |
|
||||
| Distance-mapped | `g.dist_n * 0.3` | Gradient from center | Tunnels, depth effects |
|
||||
| Frequency-mapped | `f["cent"] * 0.2` | Timbral color shifting | Audio-reactive |
|
||||
| Value-mapped | `val * 0.15` | Brightness-dependent hue | Fire, heat maps |
|
||||
| Time-cycled | `t * rate` | Slow color rotation | Ambient, chill |
|
||||
| Source-sampled | Video frame pixel colors | Preserve original color | Video-to-ASCII |
|
||||
| Palette-indexed | Discrete color lookup | Flat graphic style | Retro, pixel art |
|
||||
| Temperature | Blend between warm/cool | Emotional tone | Mood-driven scenes |
|
||||
| Complementary | `hue` and `hue + 0.5` | High contrast | Bold, dramatic |
|
||||
| Triadic | `hue`, `hue + 0.33`, `hue + 0.66` | Vibrant, balanced | Psychedelic |
|
||||
| Analogous | `hue +/- 0.08` | Harmonious, subtle | Elegant, cohesive |
|
||||
| Monochrome | Fixed hue, vary S and V | Restrained, focused | Noir, minimal |
|
||||
|
||||
### Color Palettes (Discrete RGB)
|
||||
|
||||
For non-HSV workflows -- direct RGB color sets for graphic/retro looks:
|
||||
|
||||
```python
|
||||
# Named color palettes -- use for flat/graphic styles or per-character coloring
|
||||
COLORS_NEON = [(255,0,102), (0,255,153), (102,0,255), (255,255,0), (0,204,255)]
|
||||
COLORS_PASTEL = [(255,179,186), (255,223,186), (255,255,186), (186,255,201), (186,225,255)]
|
||||
COLORS_MONO_GREEN = [(0,40,0), (0,80,0), (0,140,0), (0,200,0), (0,255,0)]
|
||||
COLORS_MONO_AMBER = [(40,20,0), (80,50,0), (140,90,0), (200,140,0), (255,191,0)]
|
||||
COLORS_CYBERPUNK = [(255,0,60), (0,255,200), (180,0,255), (255,200,0)]
|
||||
COLORS_VAPORWAVE = [(255,113,206), (1,205,254), (185,103,255), (5,255,161)]
|
||||
COLORS_EARTH = [(86,58,26), (139,90,43), (189,154,91), (222,193,136), (245,230,193)]
|
||||
COLORS_ICE = [(200,230,255), (150,200,240), (100,170,230), (60,130,210), (30,80,180)]
|
||||
COLORS_BLOOD = [(80,0,0), (140,10,10), (200,20,20), (255,50,30), (255,100,80)]
|
||||
COLORS_FOREST = [(10,30,10), (20,60,15), (30,100,20), (50,150,30), (80,200,50)]
|
||||
|
||||
def rgb_palette_map(val, mask, palette):
|
||||
"""Map float array (0-1) to RGB colors from a discrete palette."""
|
||||
n = len(palette)
|
||||
idx = np.clip((val * n).astype(int), 0, n - 1)
|
||||
R = np.zeros(val.shape, dtype=np.uint8)
|
||||
G = np.zeros(val.shape, dtype=np.uint8)
|
||||
B = np.zeros(val.shape, dtype=np.uint8)
|
||||
for i, (r, g, b) in enumerate(palette):
|
||||
m = mask & (idx == i)
|
||||
R[m] = r; G[m] = g; B[m] = b
|
||||
return R, G, B
|
||||
```
|
||||
|
||||
### Compositing Helpers
|
||||
|
||||
```python
|
||||
def mkc(R, G, B, rows, cols):
|
||||
"""Pack 3 uint8 arrays into (rows, cols, 3) color array."""
|
||||
o = np.zeros((rows, cols, 3), dtype=np.uint8)
|
||||
o[:,:,0] = R; o[:,:,1] = G; o[:,:,2] = B
|
||||
return o
|
||||
|
||||
def layer_over(base_ch, base_co, top_ch, top_co):
|
||||
"""Composite top layer onto base. Non-space chars overwrite."""
|
||||
m = top_ch != " "
|
||||
base_ch[m] = top_ch[m]; base_co[m] = top_co[m]
|
||||
return base_ch, base_co
|
||||
|
||||
def layer_blend(base_co, top_co, alpha):
|
||||
"""Alpha-blend top color layer onto base. alpha is float array (0-1) or scalar."""
|
||||
if isinstance(alpha, (int, float)):
|
||||
alpha = np.full(base_co.shape[:2], alpha, dtype=np.float32)
|
||||
a = alpha[:,:,None]
|
||||
return np.clip(base_co * (1 - a) + top_co * a, 0, 255).astype(np.uint8)
|
||||
|
||||
def stamp(ch, co, text, row, col, color=(255,255,255)):
|
||||
"""Write text string at position."""
|
||||
for i, c in enumerate(text):
|
||||
cc = col + i
|
||||
if 0 <= row < ch.shape[0] and 0 <= cc < ch.shape[1]:
|
||||
ch[row, cc] = c; co[row, cc] = color
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Section System
|
||||
|
||||
Map time ranges to effect functions + shader configs + grid sizes:
|
||||
|
||||
```python
|
||||
SECTIONS = [
|
||||
(0.0, "void"), (3.94, "starfield"), (21.0, "matrix"),
|
||||
(46.0, "drop"), (130.0, "glitch"), (187.0, "outro"),
|
||||
]
|
||||
|
||||
FX_DISPATCH = {"void": fx_void, "starfield": fx_starfield, ...}
|
||||
SECTION_FX = {"void": {"vignette": 0.3, "bloom": 170}, ...}
|
||||
SECTION_GRID = {"void": "md", "starfield": "sm", "drop": "lg", ...}
|
||||
SECTION_MIRROR = {"drop": "h", "bass_rings": "quad"}
|
||||
|
||||
def get_section(t):
|
||||
sec = SECTIONS[0][1]
|
||||
for ts, name in SECTIONS:
|
||||
if t >= ts: sec = name
|
||||
return sec
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Parallel Encoding
|
||||
|
||||
Split frames across N workers. Each pipes raw RGB to its own ffmpeg subprocess:
|
||||
|
||||
```python
|
||||
def render_batch(batch_id, frame_start, frame_end, features, seg_path):
|
||||
r = Renderer()
|
||||
cmd = ["ffmpeg", "-y", "-f", "rawvideo", "-pix_fmt", "rgb24",
|
||||
"-s", f"{VW}x{VH}", "-r", str(FPS), "-i", "pipe:0",
|
||||
"-c:v", "libx264", "-preset", "fast", "-crf", "18",
|
||||
"-pix_fmt", "yuv420p", seg_path]
|
||||
|
||||
# CRITICAL: stderr to file, not pipe
|
||||
stderr_fh = open(os.path.join(workdir, f"err_{batch_id:02d}.log"), "w")
|
||||
pipe = subprocess.Popen(cmd, stdin=subprocess.PIPE,
|
||||
stdout=subprocess.DEVNULL, stderr=stderr_fh)
|
||||
|
||||
for fi in range(frame_start, frame_end):
|
||||
t = fi / FPS
|
||||
sec = get_section(t)
|
||||
f = {k: float(features[k][fi]) for k in features}
|
||||
ch, co = FX_DISPATCH[sec](r, f, t)
|
||||
canvas = r.render(ch, co)
|
||||
canvas = apply_mirror(canvas, sec, f)
|
||||
canvas = apply_shaders(canvas, sec, f, t)
|
||||
pipe.stdin.write(canvas.tobytes())
|
||||
|
||||
pipe.stdin.close()
|
||||
pipe.wait()
|
||||
stderr_fh.close()
|
||||
```
|
||||
|
||||
Concatenate segments + mux audio:
|
||||
|
||||
```python
|
||||
# Write concat file
|
||||
with open(concat_path, "w") as cf:
|
||||
for seg in segments:
|
||||
cf.write(f"file '{seg}'\n")
|
||||
|
||||
subprocess.run(["ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", concat_path,
|
||||
"-i", audio_path, "-c:v", "copy", "-c:a", "aac", "-b:a", "192k",
|
||||
"-shortest", output_path])
|
||||
```
|
||||
|
||||
## Effect Function Contract
|
||||
|
||||
### v2 Protocol (Current)
|
||||
|
||||
Every scene function: `(renderer, features_dict, time_float, state_dict) -> canvas_uint8`
|
||||
|
||||
```python
|
||||
def fx_example(r, f, t, S):
|
||||
"""Scene function returns a full pixel canvas (uint8 H,W,3).
|
||||
Scenes have full control over multi-grid rendering and pixel-level composition.
|
||||
"""
|
||||
# Render multiple layers at different grid densities
|
||||
canvas_a = _render_vf(r, "md", vf_plasma, hf_angle(0.0), PAL_DENSE, f, t, S)
|
||||
canvas_b = _render_vf(r, "sm", vf_vortex, hf_time_cycle(0.1), PAL_RUNE, f, t, S)
|
||||
|
||||
# Pixel-level blend
|
||||
result = blend_canvas(canvas_a, canvas_b, "screen", 0.8)
|
||||
return result
|
||||
```
|
||||
|
||||
See `references/scenes.md` for the full scene protocol, the Renderer class, `_render_vf()` helper, and complete scene examples.
|
||||
|
||||
See `references/composition.md` for blend modes, tone mapping, feedback buffers, and multi-grid composition.
|
||||
|
||||
### v1 Protocol (Legacy)
|
||||
|
||||
Simple scenes that use a single grid can still return `(chars, colors)` and let the caller handle rendering, but the v2 canvas protocol is preferred for all new code.
|
||||
|
||||
```python
|
||||
def fx_simple(r, f, t, S):
|
||||
g = r.get_grid("md")
|
||||
val = np.sin(g.dist * 0.1 - t * 3) * f.get("bass", 0.3) * 2
|
||||
val = np.clip(val, 0, 1); mask = val > 0.03
|
||||
ch = val2char(val, mask, PAL_DEFAULT)
|
||||
R, G, B = hsv2rgb(np.full_like(val, 0.6), np.full_like(val, 0.7), val)
|
||||
co = mkc(R, G, B, g.rows, g.cols)
|
||||
return g.render(ch, co) # returns canvas directly
|
||||
```
|
||||
|
||||
### Persistent State
|
||||
|
||||
Effects that need state across frames (particles, rain columns) use the `S` dict parameter (which is `r.S` — same object, but passed explicitly for clarity):
|
||||
|
||||
```python
|
||||
def fx_with_state(r, f, t, S):
|
||||
if "particles" not in S:
|
||||
S["particles"] = initialize_particles()
|
||||
update_particles(S["particles"])
|
||||
# ...
|
||||
```
|
||||
|
||||
State persists across frames within a single scene/clip. Each worker process (and each scene) gets its own independent state.
|
||||
|
||||
### Helper Functions
|
||||
|
||||
```python
|
||||
def hsv2rgb_scalar(h, s, v):
|
||||
"""Single-value HSV to RGB. Returns (R, G, B) tuple of ints 0-255."""
|
||||
h = h % 1.0
|
||||
c = v * s; x = c * (1 - abs((h * 6) % 2 - 1)); m = v - c
|
||||
if h * 6 < 1: r, g, b = c, x, 0
|
||||
elif h * 6 < 2: r, g, b = x, c, 0
|
||||
elif h * 6 < 3: r, g, b = 0, c, x
|
||||
elif h * 6 < 4: r, g, b = 0, x, c
|
||||
elif h * 6 < 5: r, g, b = x, 0, c
|
||||
else: r, g, b = c, 0, x
|
||||
return (int((r+m)*255), int((g+m)*255), int((b+m)*255))
|
||||
|
||||
def log(msg):
|
||||
"""Print timestamped log message."""
|
||||
print(msg, flush=True)
|
||||
```
|
||||
476
skills/creative/ascii-video/references/composition.md
Normal file
476
skills/creative/ascii-video/references/composition.md
Normal file
@@ -0,0 +1,476 @@
|
||||
# Composition & Brightness Reference
|
||||
|
||||
The composable system is the core of visual complexity. It operates at three levels: pixel-level blend modes, multi-grid composition, and adaptive brightness management. This document covers all three.
|
||||
|
||||
## Pixel-Level Blend Modes
|
||||
|
||||
### The `blend_canvas()` Function
|
||||
|
||||
All blending operates on full pixel canvases (`uint8 H,W,3`). Internally converts to float32 [0,1] for precision, blends, lerps by opacity, converts back.
|
||||
|
||||
```python
|
||||
def blend_canvas(base, top, mode="normal", opacity=1.0):
|
||||
af = base.astype(np.float32) / 255.0
|
||||
bf = top.astype(np.float32) / 255.0
|
||||
fn = BLEND_MODES.get(mode, BLEND_MODES["normal"])
|
||||
result = fn(af, bf)
|
||||
if opacity < 1.0:
|
||||
result = af * (1 - opacity) + result * opacity
|
||||
return np.clip(result * 255, 0, 255).astype(np.uint8)
|
||||
```
|
||||
|
||||
### 20 Blend Modes
|
||||
|
||||
```python
|
||||
BLEND_MODES = {
|
||||
# Basic arithmetic
|
||||
"normal": lambda a, b: b,
|
||||
"add": lambda a, b: np.clip(a + b, 0, 1),
|
||||
"subtract": lambda a, b: np.clip(a - b, 0, 1),
|
||||
"multiply": lambda a, b: a * b,
|
||||
"screen": lambda a, b: 1 - (1 - a) * (1 - b),
|
||||
|
||||
# Contrast
|
||||
"overlay": lambda a, b: np.where(a < 0.5, 2*a*b, 1 - 2*(1-a)*(1-b)),
|
||||
"softlight": lambda a, b: (1 - 2*b)*a*a + 2*b*a,
|
||||
"hardlight": lambda a, b: np.where(b < 0.5, 2*a*b, 1 - 2*(1-a)*(1-b)),
|
||||
|
||||
# Difference
|
||||
"difference": lambda a, b: np.abs(a - b),
|
||||
"exclusion": lambda a, b: a + b - 2*a*b,
|
||||
|
||||
# Dodge / burn
|
||||
"colordodge": lambda a, b: np.clip(a / (1 - b + 1e-6), 0, 1),
|
||||
"colorburn": lambda a, b: np.clip(1 - (1 - a) / (b + 1e-6), 0, 1),
|
||||
|
||||
# Light
|
||||
"linearlight": lambda a, b: np.clip(a + 2*b - 1, 0, 1),
|
||||
"vividlight": lambda a, b: np.where(b < 0.5,
|
||||
np.clip(1 - (1-a)/(2*b + 1e-6), 0, 1),
|
||||
np.clip(a / (2*(1-b) + 1e-6), 0, 1)),
|
||||
"pin_light": lambda a, b: np.where(b < 0.5,
|
||||
np.minimum(a, 2*b), np.maximum(a, 2*b - 1)),
|
||||
"hard_mix": lambda a, b: np.where(a + b >= 1.0, 1.0, 0.0),
|
||||
|
||||
# Compare
|
||||
"lighten": lambda a, b: np.maximum(a, b),
|
||||
"darken": lambda a, b: np.minimum(a, b),
|
||||
|
||||
# Grain
|
||||
"grain_extract": lambda a, b: np.clip(a - b + 0.5, 0, 1),
|
||||
"grain_merge": lambda a, b: np.clip(a + b - 0.5, 0, 1),
|
||||
}
|
||||
```
|
||||
|
||||
### Blend Mode Selection Guide
|
||||
|
||||
**Modes that brighten** (safe for dark inputs):
|
||||
- `screen` — always brightens. Two 50% gray layers screen to 75%. The go-to safe blend.
|
||||
- `add` — simple addition, clips at white. Good for sparkles, glows, particle overlays.
|
||||
- `colordodge` — extreme brightening at overlap zones. Can blow out. Use low opacity (0.3-0.5).
|
||||
- `linearlight` — aggressive brightening. Similar to add but with offset.
|
||||
|
||||
**Modes that darken** (avoid with dark inputs):
|
||||
- `multiply` — darkens everything. Only use when both layers are already bright.
|
||||
- `overlay` — darkens when base < 0.5, brightens when base > 0.5. Crushes dark inputs: `2 * 0.12 * 0.12 = 0.03`. Use `screen` instead for dark material.
|
||||
- `colorburn` — extreme darkening at overlap zones.
|
||||
|
||||
**Modes that create contrast**:
|
||||
- `softlight` — gentle contrast. Good for subtle texture overlay.
|
||||
- `hardlight` — strong contrast. Like overlay but keyed on the top layer.
|
||||
- `vividlight` — very aggressive contrast. Use sparingly.
|
||||
|
||||
**Modes that create color effects**:
|
||||
- `difference` — XOR-like patterns. Two identical layers difference to black; offset layers create wild colors. Great for psychedelic looks.
|
||||
- `exclusion` — softer version of difference. Creates complementary color patterns.
|
||||
- `hard_mix` — posterizes to pure black/white/saturated color at intersections.
|
||||
|
||||
**Modes for texture blending**:
|
||||
- `grain_extract` / `grain_merge` — extract a texture from one layer, apply it to another.
|
||||
|
||||
### Multi-Layer Chaining
|
||||
|
||||
```python
|
||||
# Pattern: render layers -> blend sequentially
|
||||
canvas_a = _render_vf(r, "md", vf_plasma, hf_angle(0.0), PAL_DENSE, f, t, S)
|
||||
canvas_b = _render_vf(r, "sm", vf_vortex, hf_time_cycle(0.1), PAL_RUNE, f, t, S)
|
||||
canvas_c = _render_vf(r, "lg", vf_rings, hf_distance(), PAL_BLOCKS, f, t, S)
|
||||
|
||||
result = blend_canvas(canvas_a, canvas_b, "screen", 0.8)
|
||||
result = blend_canvas(result, canvas_c, "difference", 0.6)
|
||||
```
|
||||
|
||||
Order matters: `screen(A, B)` is commutative, but `difference(screen(A,B), C)` differs from `difference(A, screen(B,C))`.
|
||||
|
||||
---
|
||||
|
||||
## Multi-Grid Composition
|
||||
|
||||
This is the core visual technique. Rendering the same conceptual scene at different grid densities (character sizes) creates natural texture interference, because characters at different scales overlap at different spatial frequencies.
|
||||
|
||||
### Why It Works
|
||||
|
||||
- `sm` grid (10pt font): 320x83 characters. Fine detail, dense texture.
|
||||
- `md` grid (16pt): 192x56 characters. Medium density.
|
||||
- `lg` grid (20pt): 160x45 characters. Coarse, chunky characters.
|
||||
|
||||
When you render a plasma field on `sm` and a vortex on `lg`, then screen-blend them, the fine plasma texture shows through the gaps in the coarse vortex characters. The result has more visual complexity than either layer alone.
|
||||
|
||||
### The `_render_vf()` Helper
|
||||
|
||||
This is the workhorse function. It takes a value field + hue field + palette + grid, renders to a complete pixel canvas:
|
||||
|
||||
```python
|
||||
def _render_vf(r, grid_key, val_fn, hue_fn, pal, f, t, S, sat=0.8, threshold=0.03):
|
||||
"""Render a value field + hue field to a pixel canvas via a named grid.
|
||||
|
||||
Args:
|
||||
r: Renderer instance (has .get_grid())
|
||||
grid_key: "xs", "sm", "md", "lg", "xl", "xxl"
|
||||
val_fn: (g, f, t, S) -> float32 [0,1] array (rows, cols)
|
||||
hue_fn: callable (g, f, t, S) -> float32 hue array, OR float scalar
|
||||
pal: character palette string
|
||||
f: feature dict
|
||||
t: time in seconds
|
||||
S: persistent state dict
|
||||
sat: HSV saturation (0-1)
|
||||
threshold: minimum value to render (below = space)
|
||||
|
||||
Returns:
|
||||
uint8 array (VH, VW, 3) — full pixel canvas
|
||||
"""
|
||||
g = r.get_grid(grid_key)
|
||||
val = np.clip(val_fn(g, f, t, S), 0, 1)
|
||||
mask = val > threshold
|
||||
ch = val2char(val, mask, pal)
|
||||
|
||||
# Hue: either a callable or a fixed float
|
||||
if callable(hue_fn):
|
||||
h = hue_fn(g, f, t, S) % 1.0
|
||||
else:
|
||||
h = np.full((g.rows, g.cols), float(hue_fn), dtype=np.float32)
|
||||
|
||||
# CRITICAL: broadcast to full shape and copy (see Troubleshooting)
|
||||
h = np.broadcast_to(h, (g.rows, g.cols)).copy()
|
||||
|
||||
R, G, B = hsv2rgb(h, np.full_like(val, sat), val)
|
||||
co = mkc(R, G, B, g.rows, g.cols)
|
||||
return g.render(ch, co)
|
||||
```
|
||||
|
||||
### Grid Combination Strategies
|
||||
|
||||
| Combination | Effect | Good For |
|
||||
|-------------|--------|----------|
|
||||
| `sm` + `lg` | Maximum contrast between fine detail and chunky blocks | Bold, graphic looks |
|
||||
| `sm` + `md` | Subtle texture layering, similar scales | Organic, flowing looks |
|
||||
| `md` + `lg` + `xs` | Three-scale interference, maximum complexity | Psychedelic, dense |
|
||||
| `sm` + `sm` (different effects) | Same scale, pattern interference only | Moire, interference |
|
||||
|
||||
### Complete Multi-Grid Scene Example
|
||||
|
||||
```python
|
||||
def fx_psychedelic(r, f, t, S):
|
||||
"""Three-layer multi-grid scene with beat-reactive kaleidoscope."""
|
||||
# Layer A: plasma on medium grid with rainbow hue
|
||||
canvas_a = _render_vf(r, "md",
|
||||
lambda g, f, t, S: vf_plasma(g, f, t, S) * 1.3,
|
||||
hf_angle(0.0), PAL_DENSE, f, t, S, sat=0.8)
|
||||
|
||||
# Layer B: vortex on small grid with cycling hue
|
||||
canvas_b = _render_vf(r, "sm",
|
||||
lambda g, f, t, S: vf_vortex(g, f, t, S, twist=5.0) * 1.2,
|
||||
hf_time_cycle(0.1), PAL_RUNE, f, t, S, sat=0.7)
|
||||
|
||||
# Layer C: rings on large grid with distance hue
|
||||
canvas_c = _render_vf(r, "lg",
|
||||
lambda g, f, t, S: vf_rings(g, f, t, S, n_base=8, spacing_base=3) * 1.4,
|
||||
hf_distance(0.3, 0.02), PAL_BLOCKS, f, t, S, sat=0.9)
|
||||
|
||||
# Blend: A screened with B, then difference with C
|
||||
result = blend_canvas(canvas_a, canvas_b, "screen", 0.8)
|
||||
result = blend_canvas(result, canvas_c, "difference", 0.6)
|
||||
|
||||
# Beat-triggered kaleidoscope
|
||||
if f.get("bdecay", 0) > 0.3:
|
||||
result = sh_kaleidoscope(result.copy(), folds=6)
|
||||
|
||||
return result
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adaptive Tone Mapping
|
||||
|
||||
### The Brightness Problem
|
||||
|
||||
ASCII characters are small bright dots on a black background. Most pixels in any frame are background (black). This means:
|
||||
- Mean frame brightness is inherently low (often 5-30 out of 255)
|
||||
- Different effect combinations produce wildly different brightness levels
|
||||
- A spiral scene might be 50 mean, while a fire scene is 9 mean
|
||||
- Linear multipliers (e.g., `canvas * 2.0`) either leave dark scenes dark or blow out bright scenes
|
||||
|
||||
### The `tonemap()` Function
|
||||
|
||||
Replaces linear brightness multipliers with adaptive per-frame normalization + gamma correction:
|
||||
|
||||
```python
|
||||
def tonemap(canvas, target_mean=90, gamma=0.75, black_point=2, white_point=253):
|
||||
"""Adaptive tone-mapping: normalizes + gamma-corrects so no frame is
|
||||
fully dark or washed out.
|
||||
|
||||
1. Compute 1st and 99.5th percentile (ignores outlier pixels)
|
||||
2. Stretch that range to [0, 1]
|
||||
3. Apply gamma curve (< 1 lifts shadows, > 1 darkens)
|
||||
4. Rescale to [black_point, white_point]
|
||||
"""
|
||||
f = canvas.astype(np.float32)
|
||||
lo = np.percentile(f, 1)
|
||||
hi = np.percentile(f, 99.5)
|
||||
if hi - lo < 10:
|
||||
hi = max(hi, lo + 10) # near-uniform frame fallback
|
||||
f = np.clip((f - lo) / (hi - lo), 0.0, 1.0)
|
||||
f = np.power(f, gamma)
|
||||
f = f * (white_point - black_point) + black_point
|
||||
return np.clip(f, 0, 255).astype(np.uint8)
|
||||
```
|
||||
|
||||
### Why Gamma, Not Linear
|
||||
|
||||
Linear multiplier `* 2.0`:
|
||||
```
|
||||
input 10 -> output 20 (still dark)
|
||||
input 100 -> output 200 (ok)
|
||||
input 200 -> output 255 (clipped, lost detail)
|
||||
```
|
||||
|
||||
Gamma 0.75 after normalization:
|
||||
```
|
||||
input 0.04 -> output 0.08 (lifted from invisible to visible)
|
||||
input 0.39 -> output 0.50 (moderate lift)
|
||||
input 0.78 -> output 0.84 (gentle lift, no clipping)
|
||||
```
|
||||
|
||||
Gamma < 1 compresses the highlights and expands the shadows. This is exactly what we need: lift dark ASCII content into visibility without blowing out the bright parts.
|
||||
|
||||
### Pipeline Ordering
|
||||
|
||||
The pipeline in `render_clip()` is:
|
||||
|
||||
```
|
||||
scene_fn(r, f, t, S) -> canvas
|
||||
|
|
||||
tonemap(canvas, gamma=scene_gamma)
|
||||
|
|
||||
FeedbackBuffer.apply(canvas, ...)
|
||||
|
|
||||
ShaderChain.apply(canvas, f=f, t=t)
|
||||
|
|
||||
ffmpeg pipe
|
||||
```
|
||||
|
||||
Tonemap runs BEFORE feedback and shaders. This means:
|
||||
- Feedback operates on normalized data (consistent behavior regardless of scene brightness)
|
||||
- Shaders like solarize, posterize, contrast operate on properly-ranged data
|
||||
- The brightness shader in the chain is no longer needed (tonemap handles it)
|
||||
|
||||
### Per-Scene Gamma Tuning
|
||||
|
||||
Default gamma is 0.75. Scenes that apply destructive post-processing need more aggressive lift because the destruction happens after tonemap:
|
||||
|
||||
| Scene Type | Recommended Gamma | Why |
|
||||
|------------|-------------------|-----|
|
||||
| Standard effects | 0.75 | Default, works for most scenes |
|
||||
| Solarize post-process | 0.50-0.60 | Solarize inverts bright pixels, reducing overall brightness |
|
||||
| Posterize post-process | 0.50-0.55 | Posterize quantizes, often crushing mid-values to black |
|
||||
| Heavy difference blending | 0.60-0.70 | Difference mode creates many near-zero pixels |
|
||||
| Already bright scenes | 0.85-1.0 | Don't over-boost scenes that are naturally bright |
|
||||
|
||||
Configure via the scene table:
|
||||
|
||||
```python
|
||||
SCENES = [
|
||||
{"start": 9.17, "end": 11.25, "name": "fire", "gamma": 0.55,
|
||||
"fx": fx_fire, "shaders": [("solarize", {"threshold": 200}), ...]},
|
||||
{"start": 25.96, "end": 27.29, "name": "diamond", "gamma": 0.5,
|
||||
"fx": fx_diamond, "shaders": [("bloom", {"thr": 90}), ...]},
|
||||
]
|
||||
```
|
||||
|
||||
### Brightness Verification
|
||||
|
||||
After rendering, spot-check frame brightness:
|
||||
|
||||
```python
|
||||
# In test-frame mode
|
||||
canvas = scene["fx"](r, feat, t, r.S)
|
||||
canvas = tonemap(canvas, gamma=scene.get("gamma", 0.75))
|
||||
chain = ShaderChain()
|
||||
for sn, kw in scene.get("shaders", []):
|
||||
chain.add(sn, **kw)
|
||||
canvas = chain.apply(canvas, f=feat, t=t)
|
||||
print(f"Mean brightness: {canvas.astype(float).mean():.1f}, max: {canvas.max()}")
|
||||
```
|
||||
|
||||
Target ranges after tonemap + shaders:
|
||||
- Quiet/ambient scenes: mean 30-60
|
||||
- Active scenes: mean 40-100
|
||||
- Climax/peak scenes: mean 60-150
|
||||
- If mean < 20: gamma is too high or a shader is destroying brightness
|
||||
- If mean > 180: gamma is too low or add is stacking too much
|
||||
|
||||
---
|
||||
|
||||
## FeedbackBuffer Spatial Transforms
|
||||
|
||||
The feedback buffer stores the previous frame and blends it into the current frame with decay. Spatial transforms applied to the buffer before blending create the illusion of motion in the feedback trail.
|
||||
|
||||
### Implementation
|
||||
|
||||
```python
|
||||
class FeedbackBuffer:
|
||||
def __init__(self):
|
||||
self.buf = None
|
||||
|
||||
def apply(self, canvas, decay=0.85, blend="screen", opacity=0.5,
|
||||
transform=None, transform_amt=0.02, hue_shift=0.0):
|
||||
if self.buf is None:
|
||||
self.buf = canvas.astype(np.float32) / 255.0
|
||||
return canvas
|
||||
|
||||
# Decay old buffer
|
||||
self.buf *= decay
|
||||
|
||||
# Spatial transform
|
||||
if transform:
|
||||
self.buf = self._transform(self.buf, transform, transform_amt)
|
||||
|
||||
# Hue shift the feedback for rainbow trails
|
||||
if hue_shift > 0:
|
||||
self.buf = self._hue_shift(self.buf, hue_shift)
|
||||
|
||||
# Blend feedback into current frame
|
||||
result = blend_canvas(canvas,
|
||||
np.clip(self.buf * 255, 0, 255).astype(np.uint8),
|
||||
blend, opacity)
|
||||
|
||||
# Update buffer with current frame
|
||||
self.buf = result.astype(np.float32) / 255.0
|
||||
return result
|
||||
|
||||
def _transform(self, buf, transform, amt):
|
||||
h, w = buf.shape[:2]
|
||||
if transform == "zoom":
|
||||
# Zoom in: sample from slightly inside (creates expanding tunnel)
|
||||
m = int(h * amt); n = int(w * amt)
|
||||
if m > 0 and n > 0:
|
||||
cropped = buf[m:-m or None, n:-n or None]
|
||||
# Resize back to full (nearest-neighbor for speed)
|
||||
buf = np.array(Image.fromarray(
|
||||
np.clip(cropped * 255, 0, 255).astype(np.uint8)
|
||||
).resize((w, h), Image.NEAREST)).astype(np.float32) / 255.0
|
||||
elif transform == "shrink":
|
||||
# Zoom out: pad edges, shrink center
|
||||
m = int(h * amt); n = int(w * amt)
|
||||
small = np.array(Image.fromarray(
|
||||
np.clip(buf * 255, 0, 255).astype(np.uint8)
|
||||
).resize((w - 2*n, h - 2*m), Image.NEAREST))
|
||||
new = np.zeros((h, w, 3), dtype=np.uint8)
|
||||
new[m:m+small.shape[0], n:n+small.shape[1]] = small
|
||||
buf = new.astype(np.float32) / 255.0
|
||||
elif transform == "rotate_cw":
|
||||
# Small clockwise rotation via affine
|
||||
angle = amt * 10 # amt=0.005 -> 0.05 degrees per frame
|
||||
cy, cx = h / 2, w / 2
|
||||
Y = np.arange(h, dtype=np.float32)[:, None]
|
||||
X = np.arange(w, dtype=np.float32)[None, :]
|
||||
cos_a, sin_a = np.cos(angle), np.sin(angle)
|
||||
sx = (X - cx) * cos_a + (Y - cy) * sin_a + cx
|
||||
sy = -(X - cx) * sin_a + (Y - cy) * cos_a + cy
|
||||
sx = np.clip(sx.astype(int), 0, w - 1)
|
||||
sy = np.clip(sy.astype(int), 0, h - 1)
|
||||
buf = buf[sy, sx]
|
||||
elif transform == "rotate_ccw":
|
||||
angle = -amt * 10
|
||||
cy, cx = h / 2, w / 2
|
||||
Y = np.arange(h, dtype=np.float32)[:, None]
|
||||
X = np.arange(w, dtype=np.float32)[None, :]
|
||||
cos_a, sin_a = np.cos(angle), np.sin(angle)
|
||||
sx = (X - cx) * cos_a + (Y - cy) * sin_a + cx
|
||||
sy = -(X - cx) * sin_a + (Y - cy) * cos_a + cy
|
||||
sx = np.clip(sx.astype(int), 0, w - 1)
|
||||
sy = np.clip(sy.astype(int), 0, h - 1)
|
||||
buf = buf[sy, sx]
|
||||
elif transform == "shift_up":
|
||||
pixels = max(1, int(h * amt))
|
||||
buf = np.roll(buf, -pixels, axis=0)
|
||||
buf[-pixels:] = 0 # black fill at bottom
|
||||
elif transform == "shift_down":
|
||||
pixels = max(1, int(h * amt))
|
||||
buf = np.roll(buf, pixels, axis=0)
|
||||
buf[:pixels] = 0
|
||||
elif transform == "mirror_h":
|
||||
buf = buf[:, ::-1]
|
||||
return buf
|
||||
|
||||
def _hue_shift(self, buf, amount):
|
||||
"""Rotate hues of the feedback buffer. Operates on float32 [0,1]."""
|
||||
rgb = np.clip(buf * 255, 0, 255).astype(np.uint8)
|
||||
hsv = np.zeros_like(buf)
|
||||
# Simple approximate RGB->HSV->shift->RGB
|
||||
r, g, b = buf[:,:,0], buf[:,:,1], buf[:,:,2]
|
||||
mx = np.maximum(np.maximum(r, g), b)
|
||||
mn = np.minimum(np.minimum(r, g), b)
|
||||
delta = mx - mn + 1e-10
|
||||
# Hue
|
||||
h = np.where(mx == r, ((g - b) / delta) % 6,
|
||||
np.where(mx == g, (b - r) / delta + 2, (r - g) / delta + 4))
|
||||
h = (h / 6 + amount) % 1.0
|
||||
# Reconstruct with shifted hue (simplified)
|
||||
s = delta / (mx + 1e-10)
|
||||
v = mx
|
||||
c = v * s; x = c * (1 - np.abs((h * 6) % 2 - 1)); m = v - c
|
||||
ro = np.zeros_like(h); go = np.zeros_like(h); bo = np.zeros_like(h)
|
||||
for lo, hi, rv, gv, bv in [(0,1,c,x,0),(1,2,x,c,0),(2,3,0,c,x),
|
||||
(3,4,0,x,c),(4,5,x,0,c),(5,6,c,0,x)]:
|
||||
mask = ((h*6) >= lo) & ((h*6) < hi)
|
||||
ro[mask] = rv[mask] if not isinstance(rv, (int,float)) else rv
|
||||
go[mask] = gv[mask] if not isinstance(gv, (int,float)) else gv
|
||||
bo[mask] = bv[mask] if not isinstance(bv, (int,float)) else bv
|
||||
return np.stack([ro+m, go+m, bo+m], axis=2)
|
||||
```
|
||||
|
||||
### Feedback Presets
|
||||
|
||||
| Preset | Config | Visual Effect |
|
||||
|--------|--------|---------------|
|
||||
| Infinite zoom tunnel | `decay=0.8, blend="screen", transform="zoom", transform_amt=0.015` | Expanding ring patterns |
|
||||
| Rainbow trails | `decay=0.7, blend="screen", transform="zoom", transform_amt=0.01, hue_shift=0.02` | Psychedelic color trails |
|
||||
| Ghostly echo | `decay=0.9, blend="add", opacity=0.15, transform="shift_up", transform_amt=0.01` | Faint upward smearing |
|
||||
| Kaleidoscopic recursion | `decay=0.75, blend="screen", transform="rotate_cw", transform_amt=0.005, hue_shift=0.01` | Rotating mandala feedback |
|
||||
| Color evolution | `decay=0.8, blend="difference", opacity=0.4, hue_shift=0.03` | Frame-to-frame color XOR |
|
||||
| Rising heat haze | `decay=0.5, blend="add", opacity=0.2, transform="shift_up", transform_amt=0.02` | Hot air shimmer |
|
||||
|
||||
---
|
||||
|
||||
## PixelBlendStack
|
||||
|
||||
Higher-level wrapper for multi-layer compositing:
|
||||
|
||||
```python
|
||||
class PixelBlendStack:
|
||||
def __init__(self):
|
||||
self.layers = []
|
||||
|
||||
def add(self, canvas, mode="normal", opacity=1.0):
|
||||
self.layers.append((canvas, mode, opacity))
|
||||
return self
|
||||
|
||||
def composite(self):
|
||||
if not self.layers:
|
||||
return np.zeros((VH, VW, 3), dtype=np.uint8)
|
||||
result = self.layers[0][0]
|
||||
for canvas, mode, opacity in self.layers[1:]:
|
||||
result = blend_canvas(result, canvas, mode, opacity)
|
||||
return result
|
||||
```
|
||||
893
skills/creative/ascii-video/references/effects.md
Normal file
893
skills/creative/ascii-video/references/effects.md
Normal file
@@ -0,0 +1,893 @@
|
||||
# Effect Catalog
|
||||
|
||||
Effect building blocks that produce visual patterns. In v2, these are used **inside scene functions** that return a pixel canvas directly. The building blocks below operate on grid coordinate arrays and produce `(chars, colors)` or value/hue fields that the scene function renders to canvas via `_render_vf()`. See `composition.md` for the v2 rendering pattern and `scenes.md` for scene function examples.
|
||||
|
||||
## Design Philosophy
|
||||
|
||||
Effects are the creative core. Don't copy these verbatim for every project -- use them as **building blocks** and **combine, modify, and invent** new ones. Every project should feel distinct.
|
||||
|
||||
Key principles:
|
||||
- **Layer multiple effects** rather than using a single monolithic function
|
||||
- **Parameterize everything** -- hue, speed, density, amplitude should all be arguments
|
||||
- **React to features** -- audio/video features should modulate at least 2-3 parameters per effect
|
||||
- **Vary per section** -- never use the same effect config for the entire video
|
||||
- **Invent project-specific effects** -- the catalog below is a starting vocabulary, not a fixed set
|
||||
|
||||
---
|
||||
|
||||
## Background Fills
|
||||
|
||||
Every effect should start with a background. Never leave flat black.
|
||||
|
||||
### Animated Sine Field (General Purpose)
|
||||
```python
|
||||
def bg_sinefield(g, f, t, hue=0.6, bri=0.5, pal=PAL_DEFAULT,
|
||||
freq=(0.13, 0.17, 0.07, 0.09), speed=(0.5, -0.4, -0.3, 0.2)):
|
||||
"""Layered sine field. Adjust freq/speed tuples for different textures."""
|
||||
v1 = np.sin(g.cc*freq[0] + t*speed[0]) * np.sin(g.rr*freq[1] - t*speed[1]) * 0.5 + 0.5
|
||||
v2 = np.sin(g.cc*freq[2] - t*speed[2] + g.rr*freq[3]) * 0.4 + 0.5
|
||||
v3 = np.sin(g.dist_n*5 + t*0.2) * 0.3 + 0.4
|
||||
v4 = np.cos(g.angle*3 - t*0.6) * 0.15 + 0.5
|
||||
val = np.clip((v1*0.3 + v2*0.25 + v3*0.25 + v4*0.2) * bri * (0.6 + f["rms"]*0.6), 0.06, 1)
|
||||
mask = val > 0.03
|
||||
ch = val2char(val, mask, pal)
|
||||
h = np.full_like(val, hue) + f.get("cent", 0.5)*0.1 + val*0.08
|
||||
R, G, B = hsv2rgb(h, np.clip(0.35+f.get("flat",0.4)*0.4, 0, 1) * np.ones_like(val), val)
|
||||
return ch, mkc(R, G, B, g.rows, g.cols)
|
||||
```
|
||||
|
||||
### Video-Source Background
|
||||
```python
|
||||
def bg_video(g, frame_rgb, pal=PAL_DEFAULT, brightness=0.5):
|
||||
small = np.array(Image.fromarray(frame_rgb).resize((g.cols, g.rows)))
|
||||
lum = np.mean(small, axis=2) / 255.0 * brightness
|
||||
mask = lum > 0.02
|
||||
ch = val2char(lum, mask, pal)
|
||||
co = np.clip(small * np.clip(lum[:,:,None]*1.5+0.3, 0.3, 1), 0, 255).astype(np.uint8)
|
||||
return ch, co
|
||||
```
|
||||
|
||||
### Noise / Static Field
|
||||
```python
|
||||
def bg_noise(g, f, t, pal=PAL_BLOCKS, density=0.3, hue_drift=0.02):
|
||||
val = np.random.random((g.rows, g.cols)).astype(np.float32) * density * (0.5 + f["rms"]*0.5)
|
||||
val = np.clip(val, 0, 1); mask = val > 0.02
|
||||
ch = val2char(val, mask, pal)
|
||||
R, G, B = hsv2rgb(np.full_like(val, t*hue_drift % 1), np.full_like(val, 0.3), val)
|
||||
return ch, mkc(R, G, B, g.rows, g.cols)
|
||||
```
|
||||
|
||||
### Perlin-Like Smooth Noise
|
||||
```python
|
||||
def bg_smooth_noise(g, f, t, hue=0.5, bri=0.5, pal=PAL_DOTS, octaves=3):
|
||||
"""Layered sine approximation of Perlin noise. Cheap, smooth, organic."""
|
||||
val = np.zeros((g.rows, g.cols), dtype=np.float32)
|
||||
for i in range(octaves):
|
||||
freq = 0.05 * (2 ** i)
|
||||
amp = 0.5 / (i + 1)
|
||||
phase = t * (0.3 + i * 0.2)
|
||||
val += np.sin(g.cc * freq + phase) * np.cos(g.rr * freq * 0.7 - phase * 0.5) * amp
|
||||
val = np.clip(val * 0.5 + 0.5, 0, 1) * bri
|
||||
mask = val > 0.03
|
||||
ch = val2char(val, mask, pal)
|
||||
h = np.full_like(val, hue) + val * 0.1
|
||||
R, G, B = hsv2rgb(h, np.full_like(val, 0.5), val)
|
||||
return ch, mkc(R, G, B, g.rows, g.cols)
|
||||
```
|
||||
|
||||
### Cellular / Voronoi Approximation
|
||||
```python
|
||||
def bg_cellular(g, f, t, n_centers=12, hue=0.5, bri=0.6, pal=PAL_BLOCKS):
|
||||
"""Voronoi-like cells using distance to nearest of N moving centers."""
|
||||
rng = np.random.RandomState(42) # deterministic centers
|
||||
cx = (rng.rand(n_centers) * g.cols).astype(np.float32)
|
||||
cy = (rng.rand(n_centers) * g.rows).astype(np.float32)
|
||||
# Animate centers
|
||||
cx_t = cx + np.sin(t * 0.5 + np.arange(n_centers) * 0.7) * 5
|
||||
cy_t = cy + np.cos(t * 0.4 + np.arange(n_centers) * 0.9) * 3
|
||||
# Min distance to any center
|
||||
min_d = np.full((g.rows, g.cols), 999.0, dtype=np.float32)
|
||||
for i in range(n_centers):
|
||||
d = np.sqrt((g.cc - cx_t[i])**2 + (g.rr - cy_t[i])**2)
|
||||
min_d = np.minimum(min_d, d)
|
||||
val = np.clip(1.0 - min_d / (g.cols * 0.3), 0, 1) * bri
|
||||
# Cell edges (where distance is near-equal between two centers)
|
||||
# ... second-nearest trick for edge highlighting
|
||||
mask = val > 0.03
|
||||
ch = val2char(val, mask, pal)
|
||||
R, G, B = hsv2rgb(np.full_like(val, hue) + min_d * 0.005, np.full_like(val, 0.5), val)
|
||||
return ch, mkc(R, G, B, g.rows, g.cols)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Radial Effects
|
||||
|
||||
### Concentric Rings
|
||||
Bass/sub-driven pulsing rings from center. Scale ring count and thickness with bass energy.
|
||||
```python
|
||||
def eff_rings(g, f, t, hue=0.5, n_base=6, pal=PAL_DEFAULT):
|
||||
n_rings = int(n_base + f["sub_r"] * 25 + f["bass"] * 10)
|
||||
spacing = 2 + f["bass_r"] * 7 + f["rms"] * 3
|
||||
ring_cv = np.zeros((g.rows, g.cols), dtype=np.float32)
|
||||
for ri in range(n_rings):
|
||||
rad = (ri+1) * spacing + f["bdecay"] * 15
|
||||
wobble = f["mid_r"]*5*np.sin(g.angle*3 + t*4) + f["hi_r"]*3*np.sin(g.angle*7 - t*6)
|
||||
rd = np.abs(g.dist - rad - wobble)
|
||||
th = 1 + f["sub"] * 3
|
||||
ring_cv = np.maximum(ring_cv, np.clip((1 - rd/th) * (0.4 + f["bass"]*0.8), 0, 1))
|
||||
# Color by angle + distance for rainbow rings
|
||||
h = g.angle/(2*np.pi) + g.dist*0.005 + f["sub_r"]*0.2
|
||||
return ring_cv, h
|
||||
```
|
||||
|
||||
### Radial Rays
|
||||
```python
|
||||
def eff_rays(g, f, t, n_base=8, hue=0.5):
|
||||
n_rays = int(n_base + f["hi_r"] * 25)
|
||||
ray = np.clip(np.cos(g.angle*n_rays + t*3) * f["bdecay"]*0.6 * (1-g.dist_n), 0, 0.7)
|
||||
return ray
|
||||
```
|
||||
|
||||
### Spiral Arms (Logarithmic)
|
||||
```python
|
||||
def eff_spiral(g, f, t, n_arms=3, tightness=2.5, hue=0.5):
|
||||
arm_cv = np.zeros((g.rows, g.cols), dtype=np.float32)
|
||||
for ai in range(n_arms):
|
||||
offset = ai * 2*np.pi / n_arms
|
||||
log_r = np.log(g.dist + 1) * tightness
|
||||
arm_phase = g.angle + offset - log_r + t * 0.8
|
||||
arm_val = np.clip(np.cos(arm_phase * n_arms) * 0.6 + 0.2, 0, 1)
|
||||
arm_val *= (0.4 + f["rms"]*0.6) * np.clip(1 - g.dist_n*0.5, 0.2, 1)
|
||||
arm_cv = np.maximum(arm_cv, arm_val)
|
||||
return arm_cv
|
||||
```
|
||||
|
||||
### Center Glow / Pulse
|
||||
```python
|
||||
def eff_glow(g, f, t, intensity=0.6, spread=2.0):
|
||||
return np.clip(intensity * np.exp(-g.dist_n * spread) * (0.5 + f["rms"]*2 + np.sin(t*1.2)*0.2), 0, 0.9)
|
||||
```
|
||||
|
||||
### Tunnel / Depth
|
||||
```python
|
||||
def eff_tunnel(g, f, t, speed=3.0, complexity=6):
|
||||
tunnel_d = 1.0 / (g.dist_n + 0.1)
|
||||
v1 = np.sin(tunnel_d*2 - t*speed) * 0.45 + 0.55
|
||||
v2 = np.sin(g.angle*complexity + tunnel_d*1.5 - t*2) * 0.35 + 0.55
|
||||
return v1 * 0.5 + v2 * 0.5
|
||||
```
|
||||
|
||||
### Vortex (Rotating Distortion)
|
||||
```python
|
||||
def eff_vortex(g, f, t, twist=3.0, pulse=True):
|
||||
"""Twisting radial pattern -- distance modulates angle."""
|
||||
twisted = g.angle + g.dist_n * twist * np.sin(t * 0.5)
|
||||
val = np.sin(twisted * 4 - t * 2) * 0.5 + 0.5
|
||||
if pulse:
|
||||
val *= 0.5 + f.get("bass", 0.3) * 0.8
|
||||
return np.clip(val, 0, 1)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Wave Effects
|
||||
|
||||
### Multi-Band Frequency Waves
|
||||
Each frequency band draws its own wave at different spatial/temporal frequencies:
|
||||
```python
|
||||
def eff_freq_waves(g, f, t, bands=None):
|
||||
if bands is None:
|
||||
bands = [("sub",0.06,1.2,0.0), ("bass",0.10,2.0,0.08), ("lomid",0.15,3.0,0.16),
|
||||
("mid",0.22,4.5,0.25), ("himid",0.32,6.5,0.4), ("hi",0.45,8.5,0.55)]
|
||||
mid = g.rows / 2.0
|
||||
composite = np.zeros((g.rows, g.cols), dtype=np.float32)
|
||||
for band_key, sf, tf, hue_base in bands:
|
||||
amp = f.get(band_key, 0.3) * g.rows * 0.4
|
||||
y_wave = mid - np.sin(g.cc*sf + t*tf) * amp
|
||||
y_wave += np.sin(g.cc*sf*2.3 + t*tf*1.7) * amp * 0.2 # harmonic
|
||||
dist = np.abs(g.rr - y_wave)
|
||||
thickness = 2 + f.get(band_key, 0.3) * 5
|
||||
intensity = np.clip((1 - dist/thickness) * f.get(band_key, 0.3) * 1.5, 0, 1)
|
||||
composite = np.maximum(composite, intensity)
|
||||
return composite
|
||||
```
|
||||
|
||||
### Interference Pattern
|
||||
6-8 overlapping sine waves creating moire-like patterns:
|
||||
```python
|
||||
def eff_interference(g, f, t, n_waves=5):
|
||||
"""Parametric interference -- vary n_waves for complexity."""
|
||||
# Each wave has different orientation, frequency, and feature driver
|
||||
drivers = ["mid_r", "himid_r", "bass_r", "lomid_r", "hi_r"]
|
||||
vals = np.zeros((g.rows, g.cols), dtype=np.float32)
|
||||
for i in range(min(n_waves, len(drivers))):
|
||||
angle = i * np.pi / n_waves # spread orientations
|
||||
freq = 0.06 + i * 0.03
|
||||
sp = 0.5 + i * 0.3
|
||||
proj = g.cc * np.cos(angle) + g.rr * np.sin(angle)
|
||||
vals += np.sin(proj * freq + t * sp) * f.get(drivers[i], 0.3) * 2.5
|
||||
return np.clip(vals * 0.12 + 0.45, 0.1, 1)
|
||||
```
|
||||
|
||||
### Aurora / Horizontal Bands
|
||||
```python
|
||||
def eff_aurora(g, f, t, hue=0.4, n_bands=3):
|
||||
val = np.zeros((g.rows, g.cols), dtype=np.float32)
|
||||
for i in range(n_bands):
|
||||
freq_r = 0.08 + i * 0.04
|
||||
freq_c = 0.012 + i * 0.008
|
||||
sp_r = 0.7 + i * 0.3
|
||||
sp_c = 0.18 + i * 0.12
|
||||
val += np.sin(g.rr*freq_r + t*sp_r) * np.sin(g.cc*freq_c + t*sp_c) * (0.6 / n_bands)
|
||||
return np.clip(val * (f.get("lomid_r", 0.3)*3 + 0.2), 0, 0.7)
|
||||
```
|
||||
|
||||
### Ripple (Point-Source Waves)
|
||||
```python
|
||||
def eff_ripple(g, f, t, sources=None, freq=0.3, damping=0.02):
|
||||
"""Concentric ripples from point sources. Sources = [(row_frac, col_frac), ...]"""
|
||||
if sources is None:
|
||||
sources = [(0.5, 0.5)] # center
|
||||
val = np.zeros((g.rows, g.cols), dtype=np.float32)
|
||||
for ry, rx in sources:
|
||||
dy = g.rr - g.rows * ry
|
||||
dx = g.cc - g.cols * rx
|
||||
d = np.sqrt(dy**2 + dx**2)
|
||||
val += np.sin(d * freq - t * 4) * np.exp(-d * damping) * 0.5
|
||||
return np.clip(val + 0.5, 0, 1)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Particle Systems
|
||||
|
||||
### General Pattern
|
||||
All particle systems use persistent state:
|
||||
```python
|
||||
S = state # dict persisted across frames
|
||||
if "px" not in S:
|
||||
S["px"]=[]; S["py"]=[]; S["vx"]=[]; S["vy"]=[]; S["life"]=[]; S["char"]=[]
|
||||
|
||||
# Emit new particles (on beat, continuously, or on trigger)
|
||||
# Update: position += velocity, apply forces, decay life
|
||||
# Draw: map to grid, set char/color based on life
|
||||
# Cull: remove dead, cap total count
|
||||
```
|
||||
|
||||
### Particle Character Sets
|
||||
|
||||
Don't hardcode particle chars. Choose per project/mood:
|
||||
|
||||
```python
|
||||
# Energy / explosive
|
||||
PART_ENERGY = list("*+#@\u26a1\u2726\u2605\u2588\u2593")
|
||||
PART_SPARK = list("\u00b7\u2022\u25cf\u2605\u2736*+")
|
||||
# Organic / natural
|
||||
PART_LEAF = list("\u2740\u2741\u2742\u2743\u273f\u2618\u2022")
|
||||
PART_SNOW = list("\u2744\u2745\u2746\u00b7\u2022*\u25cb")
|
||||
PART_RAIN = list("|\u2502\u2503\u2551/\\")
|
||||
PART_BUBBLE = list("\u25cb\u25ce\u25c9\u25cf\u2218\u2219\u00b0")
|
||||
# Data / tech
|
||||
PART_DATA = list("01{}[]<>|/\\")
|
||||
PART_HEX = list("0123456789ABCDEF")
|
||||
PART_BINARY = list("01")
|
||||
# Mystical
|
||||
PART_RUNE = list("\u16a0\u16a2\u16a6\u16b1\u16b7\u16c1\u16c7\u16d2\u16d6\u16da\u16de\u16df\u2726\u2605")
|
||||
PART_ZODIAC = list("\u2648\u2649\u264a\u264b\u264c\u264d\u264e\u264f\u2650\u2651\u2652\u2653")
|
||||
# Minimal
|
||||
PART_DOT = list("\u00b7\u2022\u25cf")
|
||||
PART_DASH = list("-=~\u2500\u2550")
|
||||
```
|
||||
|
||||
### Explosion (Beat-Triggered)
|
||||
```python
|
||||
def emit_explosion(S, f, center_r, center_c, char_set=PART_ENERGY, count_base=80):
|
||||
if f.get("beat", 0) > 0:
|
||||
for _ in range(int(count_base + f["rms"]*150)):
|
||||
ang = random.uniform(0, 2*math.pi)
|
||||
sp = random.uniform(1, 9) * (0.5 + f.get("sub_r", 0.3)*2)
|
||||
S["px"].append(float(center_c))
|
||||
S["py"].append(float(center_r))
|
||||
S["vx"].append(math.cos(ang)*sp*2.5)
|
||||
S["vy"].append(math.sin(ang)*sp)
|
||||
S["life"].append(1.0)
|
||||
S["char"].append(random.choice(char_set))
|
||||
# Update: gravity on vy += 0.03, life -= 0.015
|
||||
# Color: life * 255 for brightness, hue fade controlled by caller
|
||||
```
|
||||
|
||||
### Rising Embers
|
||||
```python
|
||||
# Emit: sy = rows-1, vy = -random.uniform(1,5), vx = random.uniform(-1.5,1.5)
|
||||
# Update: vx += random jitter * 0.3, life -= 0.01
|
||||
# Cap at ~1500 particles
|
||||
```
|
||||
|
||||
### Dissolving Cloud
|
||||
```python
|
||||
# Init: N=600 particles spread across screen
|
||||
# Update: slow upward drift, fade life progressively
|
||||
# life -= 0.002 * (1 + elapsed * 0.05) # accelerating fade
|
||||
```
|
||||
|
||||
### Starfield (3D Projection)
|
||||
```python
|
||||
# N stars with (sx, sy, sz) in normalized coords
|
||||
# Move: sz -= speed (stars approach camera)
|
||||
# Project: px = cx + sx/sz * cx, py = cy + sy/sz * cy
|
||||
# Reset stars that pass camera (sz <= 0.01)
|
||||
# Brightness = (1 - sz), draw streaks behind bright stars
|
||||
```
|
||||
|
||||
### Orbit (Circular/Elliptical Motion)
|
||||
```python
|
||||
def emit_orbit(S, n=20, radius=15, speed=1.0, char_set=PART_DOT):
|
||||
"""Particles orbiting a center point."""
|
||||
for i in range(n):
|
||||
angle = i * 2 * math.pi / n
|
||||
S["px"].append(0.0); S["py"].append(0.0) # will be computed from angle
|
||||
S["vx"].append(angle) # store angle as "vx" for orbit
|
||||
S["vy"].append(radius + random.uniform(-2, 2)) # store radius
|
||||
S["life"].append(1.0)
|
||||
S["char"].append(random.choice(char_set))
|
||||
# Update: angle += speed * dt, px = cx + radius * cos(angle), py = cy + radius * sin(angle)
|
||||
```
|
||||
|
||||
### Gravity Well
|
||||
```python
|
||||
# Particles attracted toward one or more gravity points
|
||||
# Update: compute force vector toward each well, apply as acceleration
|
||||
# Particles that reach well center respawn at edges
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rain / Matrix Effects
|
||||
|
||||
### Column Rain (Vectorized)
|
||||
```python
|
||||
def eff_matrix_rain(g, f, t, state, hue=0.33, bri=0.6, pal=PAL_KATA,
|
||||
speed_base=0.5, speed_beat=3.0):
|
||||
"""Vectorized matrix rain. state dict persists column positions."""
|
||||
if "ry" not in state or len(state["ry"]) != g.cols:
|
||||
state["ry"] = np.random.uniform(-g.rows, g.rows, g.cols).astype(np.float32)
|
||||
state["rsp"] = np.random.uniform(0.3, 2.0, g.cols).astype(np.float32)
|
||||
state["rln"] = np.random.randint(8, 40, g.cols)
|
||||
state["rch"] = np.random.randint(0, len(pal), (g.rows, g.cols)) # pre-assign chars
|
||||
|
||||
speed_mult = speed_base + f.get("bass", 0.3)*speed_beat + f.get("sub_r", 0.3)*3
|
||||
if f.get("beat", 0) > 0: speed_mult *= 2.5
|
||||
state["ry"] += state["rsp"] * speed_mult
|
||||
|
||||
# Reset columns that fall past bottom
|
||||
rst = (state["ry"] - state["rln"]) > g.rows
|
||||
state["ry"][rst] = np.random.uniform(-25, -2, rst.sum())
|
||||
|
||||
# Vectorized draw using fancy indexing
|
||||
ch = np.full((g.rows, g.cols), " ", dtype="U1")
|
||||
co = np.zeros((g.rows, g.cols, 3), dtype=np.uint8)
|
||||
heads = state["ry"].astype(int)
|
||||
for c in range(g.cols):
|
||||
head = heads[c]
|
||||
trail_len = state["rln"][c]
|
||||
for i in range(trail_len):
|
||||
row = head - i
|
||||
if 0 <= row < g.rows:
|
||||
fade = 1.0 - i / trail_len
|
||||
ci = state["rch"][row, c] % len(pal)
|
||||
ch[row, c] = pal[ci]
|
||||
v = fade * bri * 255
|
||||
if i == 0: # head is bright white-ish
|
||||
co[row, c] = (int(v*0.9), int(min(255, v*1.1)), int(v*0.9))
|
||||
else:
|
||||
R, G, B = hsv2rgb_single(hue, 0.7, fade * bri)
|
||||
co[row, c] = (R, G, B)
|
||||
return ch, co, state
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Glitch / Data Effects
|
||||
|
||||
### Horizontal Band Displacement
|
||||
```python
|
||||
def eff_glitch_displace(ch, co, f, intensity=1.0):
|
||||
n_bands = int(8 + f.get("flux", 0.3)*25 + f.get("bdecay", 0)*15) * intensity
|
||||
for _ in range(int(n_bands)):
|
||||
y = random.randint(0, ch.shape[0]-1)
|
||||
h = random.randint(1, int(3 + f.get("sub", 0.3)*8))
|
||||
shift = int((random.random()-0.5) * f.get("rms", 0.3)*40 + f.get("bdecay", 0)*20*(random.random()-0.5))
|
||||
if shift != 0:
|
||||
for row in range(h):
|
||||
rr = y + row
|
||||
if 0 <= rr < ch.shape[0]:
|
||||
ch[rr] = np.roll(ch[rr], shift)
|
||||
co[rr] = np.roll(co[rr], shift, axis=0)
|
||||
return ch, co
|
||||
```
|
||||
|
||||
### Block Corruption
|
||||
```python
|
||||
def eff_block_corrupt(ch, co, f, char_pool=None, count_base=20):
|
||||
if char_pool is None:
|
||||
char_pool = list(PAL_BLOCKS[4:] + PAL_KATA[2:8])
|
||||
for _ in range(int(count_base + f.get("flux", 0.3)*60 + f.get("bdecay", 0)*40)):
|
||||
bx = random.randint(0, max(1, ch.shape[1]-6))
|
||||
by = random.randint(0, max(1, ch.shape[0]-4))
|
||||
bw, bh = random.randint(2,6), random.randint(1,4)
|
||||
block_char = random.choice(char_pool)
|
||||
# Fill rectangle with single char and random color
|
||||
for r in range(bh):
|
||||
for c in range(bw):
|
||||
rr, cc = by+r, bx+c
|
||||
if 0 <= rr < ch.shape[0] and 0 <= cc < ch.shape[1]:
|
||||
ch[rr, cc] = block_char
|
||||
co[rr, cc] = (random.randint(100,255), random.randint(0,100), random.randint(0,80))
|
||||
return ch, co
|
||||
```
|
||||
|
||||
### Scan Bars (Vertical)
|
||||
```python
|
||||
def eff_scanbars(ch, co, f, t, n_base=4, chars="|\u2551|!1l"):
|
||||
for bi in range(int(n_base + f.get("himid_r", 0.3)*12)):
|
||||
sx = int((t*50*(1+bi*0.3) + bi*37) % ch.shape[1])
|
||||
for rr in range(ch.shape[0]):
|
||||
if random.random() < 0.7:
|
||||
ch[rr, sx] = random.choice(chars)
|
||||
return ch, co
|
||||
```
|
||||
|
||||
### Error Messages
|
||||
```python
|
||||
# Parameterize the error vocabulary per project:
|
||||
ERRORS_TECH = ["SEGFAULT","0xDEADBEEF","BUFFER_OVERRUN","PANIC!","NULL_PTR",
|
||||
"CORRUPT","SIGSEGV","ERR_OVERFLOW","STACK_SMASH","BAD_ALLOC"]
|
||||
ERRORS_COSMIC = ["VOID_BREACH","ENTROPY_MAX","SINGULARITY","DIMENSION_FAULT",
|
||||
"REALITY_ERR","TIME_PARADOX","DARK_MATTER_LEAK","QUANTUM_DECOHERE"]
|
||||
ERRORS_ORGANIC = ["CELL_DIVISION_ERR","DNA_MISMATCH","MUTATION_OVERFLOW",
|
||||
"NEURAL_DEADLOCK","SYNAPSE_TIMEOUT","MEMBRANE_BREACH"]
|
||||
```
|
||||
|
||||
### Hex Data Stream
|
||||
```python
|
||||
hex_str = "".join(random.choice("0123456789ABCDEF") for _ in range(random.randint(8,20)))
|
||||
stamp(ch, co, hex_str, rand_row, rand_col, (0, 160, 80))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Spectrum / Visualization
|
||||
|
||||
### Mirrored Spectrum Bars
|
||||
```python
|
||||
def eff_spectrum(g, f, t, n_bars=64, pal=PAL_BLOCKS, mirror=True):
|
||||
bar_w = max(1, g.cols // n_bars); mid = g.rows // 2
|
||||
band_vals = np.array([f.get("sub",0.3), f.get("bass",0.3), f.get("lomid",0.3),
|
||||
f.get("mid",0.3), f.get("himid",0.3), f.get("hi",0.3)])
|
||||
ch = np.full((g.rows, g.cols), " ", dtype="U1")
|
||||
co = np.zeros((g.rows, g.cols, 3), dtype=np.uint8)
|
||||
for b in range(n_bars):
|
||||
frac = b / n_bars
|
||||
fi = frac * 5; lo_i = int(fi); hi_i = min(lo_i+1, 5)
|
||||
bval = min(1, (band_vals[lo_i]*(1-fi%1) + band_vals[hi_i]*(fi%1)) * 1.8)
|
||||
height = int(bval * (g.rows//2 - 2))
|
||||
for dy in range(height):
|
||||
hue = (f.get("cent",0.5)*0.3 + frac*0.3 + dy/max(height,1)*0.15) % 1.0
|
||||
ci = pal[min(int(dy/max(height,1)*len(pal)*0.7+len(pal)*0.2), len(pal)-1)]
|
||||
for dc in range(bar_w - (1 if bar_w > 2 else 0)):
|
||||
cc = b*bar_w + dc
|
||||
if 0 <= cc < g.cols:
|
||||
rows_to_draw = [mid - dy, mid + dy] if mirror else [g.rows - 1 - dy]
|
||||
for row in rows_to_draw:
|
||||
if 0 <= row < g.rows:
|
||||
ch[row, cc] = ci
|
||||
co[row, cc] = hsv_to_rgb_single(hue, 0.85, 0.5+dy/max(height,1)*0.5)
|
||||
return ch, co
|
||||
```
|
||||
|
||||
### Waveform
|
||||
```python
|
||||
def eff_waveform(g, f, t, row_offset=-5, hue=0.1):
|
||||
ch = np.full((g.rows, g.cols), " ", dtype="U1")
|
||||
co = np.zeros((g.rows, g.cols, 3), dtype=np.uint8)
|
||||
for c in range(g.cols):
|
||||
wv = (math.sin(c*0.15+t*5)*f.get("bass",0.3)*0.5
|
||||
+ math.sin(c*0.3+t*8)*f.get("mid",0.3)*0.3
|
||||
+ math.sin(c*0.6+t*12)*f.get("hi",0.3)*0.15)
|
||||
wr = g.rows + row_offset + int(wv * 4)
|
||||
if 0 <= wr < g.rows:
|
||||
ch[wr, c] = "~"
|
||||
v = int(120 + f.get("rms",0.3)*135)
|
||||
co[wr, c] = [v, int(v*0.7), int(v*0.4)]
|
||||
return ch, co
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Fire / Lava
|
||||
|
||||
### Fire Columns
|
||||
```python
|
||||
def eff_fire(g, f, t, n_base=20, hue_base=0.02, hue_range=0.12, pal=PAL_BLOCKS):
|
||||
n_cols = int(n_base + f.get("bass",0.3)*30 + f.get("sub_r",0.3)*20)
|
||||
ch = np.full((g.rows, g.cols), " ", dtype="U1")
|
||||
co = np.zeros((g.rows, g.cols, 3), dtype=np.uint8)
|
||||
for fi in range(n_cols):
|
||||
fx_c = int((fi*g.cols/n_cols + np.sin(t*2+fi*0.7)*3) % g.cols)
|
||||
height = int((f.get("bass",0.3)*0.4 + f.get("sub_r",0.3)*0.3 + f.get("rms",0.3)*0.3) * g.rows * 0.7)
|
||||
for dy in range(min(height, g.rows)):
|
||||
fr = g.rows - 1 - dy
|
||||
frac = dy / max(height, 1)
|
||||
bri = max(0.1, (1 - frac*0.6) * (0.5 + f.get("rms",0.3)*0.5))
|
||||
hue = hue_base + frac * hue_range
|
||||
ci = "\u2588" if frac<0.2 else ("\u2593" if frac<0.4 else ("\u2592" if frac<0.6 else "\u2591"))
|
||||
ch[fr, fx_c] = ci
|
||||
R, G, B = hsv2rgb_single(hue, 0.9, bri)
|
||||
co[fr, fx_c] = (R, G, B)
|
||||
return ch, co
|
||||
```
|
||||
|
||||
### Ice / Cold Fire (same structure, different hue range)
|
||||
```python
|
||||
# hue_base=0.55, hue_range=0.15 -- blue to cyan
|
||||
# Lower intensity, slower movement
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Text Overlays
|
||||
|
||||
### Scrolling Ticker
|
||||
```python
|
||||
def eff_ticker(ch, co, t, text, row, speed=15, color=(80, 100, 140)):
|
||||
off = int(t * speed) % max(len(text), 1)
|
||||
doubled = text + " " + text
|
||||
stamp(ch, co, doubled[off:off+ch.shape[1]], row, 0, color)
|
||||
```
|
||||
|
||||
### Beat-Triggered Words
|
||||
```python
|
||||
def eff_beat_words(ch, co, f, words, row_center=None, color=(255,240,220)):
|
||||
if f.get("beat", 0) > 0:
|
||||
w = random.choice(words)
|
||||
r = (row_center or ch.shape[0]//2) + random.randint(-5,5)
|
||||
stamp(ch, co, w, r, (ch.shape[1]-len(w))//2, color)
|
||||
```
|
||||
|
||||
### Fading Message Sequence
|
||||
```python
|
||||
def eff_fading_messages(ch, co, t, elapsed, messages, period=4.0, color_base=(220,220,220)):
|
||||
msg_idx = int(elapsed / period) % len(messages)
|
||||
phase = elapsed % period
|
||||
fade = max(0, min(1.0, phase) * min(1.0, period - phase))
|
||||
if fade > 0.05:
|
||||
v = fade
|
||||
msg = messages[msg_idx]
|
||||
cr, cg, cb = [int(c * v) for c in color_base]
|
||||
stamp(ch, co, msg, ch.shape[0]//2, (ch.shape[1]-len(msg))//2, (cr, cg, cb))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Screen Shake
|
||||
Shift entire char/color arrays on beat:
|
||||
```python
|
||||
def eff_shake(ch, co, f, x_amp=6, y_amp=3):
|
||||
shake_x = int(f.get("sub",0.3)*x_amp*(random.random()-0.5)*2 + f.get("bdecay",0)*4*(random.random()-0.5)*2)
|
||||
shake_y = int(f.get("bass",0.3)*y_amp*(random.random()-0.5)*2)
|
||||
if abs(shake_x) > 0:
|
||||
ch = np.roll(ch, shake_x, axis=1)
|
||||
co = np.roll(co, shake_x, axis=1)
|
||||
if abs(shake_y) > 0:
|
||||
ch = np.roll(ch, shake_y, axis=0)
|
||||
co = np.roll(co, shake_y, axis=0)
|
||||
return ch, co
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Composable Effect System
|
||||
|
||||
The real creative power comes from **composition**. There are three levels:
|
||||
|
||||
### Level 1: Character-Level Layering
|
||||
|
||||
Stack multiple effects as `(chars, colors)` layers:
|
||||
|
||||
```python
|
||||
class LayerStack(EffectNode):
|
||||
"""Render effects bottom-to-top with character-level compositing."""
|
||||
def add(self, effect, alpha=1.0):
|
||||
"""alpha < 1.0 = probabilistic override (sparse overlay)."""
|
||||
self.layers.append((effect, alpha))
|
||||
|
||||
# Usage:
|
||||
stack = LayerStack()
|
||||
stack.add(bg_effect) # base — fills screen
|
||||
stack.add(main_effect) # overlay on top (space chars = transparent)
|
||||
stack.add(particle_effect) # sparse overlay on top of that
|
||||
ch, co = stack.render(g, f, t, S)
|
||||
```
|
||||
|
||||
### Level 2: Pixel-Level Blending
|
||||
|
||||
After rendering to canvases, blend with Photoshop-style modes:
|
||||
|
||||
```python
|
||||
class PixelBlendStack:
|
||||
"""Stack canvases with blend modes for complex compositing."""
|
||||
def add(self, canvas, mode="normal", opacity=1.0)
|
||||
def composite(self) -> canvas
|
||||
|
||||
# Usage:
|
||||
pbs = PixelBlendStack()
|
||||
pbs.add(canvas_a) # base
|
||||
pbs.add(canvas_b, "screen", 0.7) # additive glow
|
||||
pbs.add(canvas_c, "difference", 0.5) # psychedelic interference
|
||||
result = pbs.composite()
|
||||
```
|
||||
|
||||
### Level 3: Temporal Feedback
|
||||
|
||||
Feed previous frame back into current frame for recursive effects:
|
||||
|
||||
```python
|
||||
fb = FeedbackBuffer()
|
||||
for each frame:
|
||||
canvas = render_current()
|
||||
canvas = fb.apply(canvas, decay=0.8, blend="screen",
|
||||
transform="zoom", transform_amt=0.015, hue_shift=0.02)
|
||||
```
|
||||
|
||||
### Effect Nodes — Uniform Interface
|
||||
|
||||
In the v2 protocol, effect nodes are used **inside** scene functions. The scene function itself returns a canvas. Effect nodes produce intermediate `(chars, colors)` that are rendered to canvas via the grid's `.render()` method or `_render_vf()`.
|
||||
|
||||
```python
|
||||
class EffectNode:
|
||||
def render(self, g, f, t, S) -> (chars, colors)
|
||||
|
||||
# Concrete implementations:
|
||||
class ValueFieldEffect(EffectNode):
|
||||
"""Wraps a value field function + hue field function + palette."""
|
||||
def __init__(self, val_fn, hue_fn, pal=PAL_DEFAULT, sat=0.7)
|
||||
|
||||
class LambdaEffect(EffectNode):
|
||||
"""Wrap any (g,f,t,S) -> (ch,co) function."""
|
||||
def __init__(self, fn)
|
||||
|
||||
class ConditionalEffect(EffectNode):
|
||||
"""Switch effects based on audio features."""
|
||||
def __init__(self, condition, if_true, if_false=None)
|
||||
```
|
||||
|
||||
### Value Field Generators (Atomic Building Blocks)
|
||||
|
||||
These produce float32 arrays `(rows, cols)` in range [0,1]. They are the raw visual patterns. All have signature `(g, f, t, S, **params) -> float32 array`.
|
||||
|
||||
```python
|
||||
def vf_sinefield(g, f, t, S, bri=0.5,
|
||||
freq=(0.13, 0.17, 0.07, 0.09), speed=(0.5, -0.4, -0.3, 0.2)):
|
||||
"""Layered sine field. General purpose background/texture."""
|
||||
v1 = np.sin(g.cc*freq[0] + t*speed[0]) * np.sin(g.rr*freq[1] - t*speed[1]) * 0.5 + 0.5
|
||||
v2 = np.sin(g.cc*freq[2] - t*speed[2] + g.rr*freq[3]) * 0.4 + 0.5
|
||||
v3 = np.sin(g.dist_n*5 + t*0.2) * 0.3 + 0.4
|
||||
return np.clip((v1*0.35 + v2*0.35 + v3*0.3) * bri * (0.6 + f.get("rms",0.3)*0.6), 0, 1)
|
||||
|
||||
def vf_smooth_noise(g, f, t, S, octaves=3, bri=0.5):
|
||||
"""Multi-octave sine approximation of Perlin noise."""
|
||||
val = np.zeros((g.rows, g.cols), dtype=np.float32)
|
||||
for i in range(octaves):
|
||||
freq = 0.05 * (2 ** i); amp = 0.5 / (i + 1)
|
||||
phase = t * (0.3 + i * 0.2)
|
||||
val = val + np.sin(g.cc*freq + phase) * np.cos(g.rr*freq*0.7 - phase*0.5) * amp
|
||||
return np.clip(val * 0.5 + 0.5, 0, 1) * bri
|
||||
|
||||
def vf_rings(g, f, t, S, n_base=6, spacing_base=4):
|
||||
"""Concentric rings, bass-driven count and wobble."""
|
||||
n = int(n_base + f.get("sub_r",0.3)*25 + f.get("bass",0.3)*10)
|
||||
sp = spacing_base + f.get("bass_r",0.3)*7 + f.get("rms",0.3)*3
|
||||
val = np.zeros((g.rows, g.cols), dtype=np.float32)
|
||||
for ri in range(n):
|
||||
rad = (ri+1)*sp + f.get("bdecay",0)*15
|
||||
wobble = f.get("mid_r",0.3)*5*np.sin(g.angle*3+t*4)
|
||||
rd = np.abs(g.dist - rad - wobble)
|
||||
th = 1 + f.get("sub",0.3)*3
|
||||
val = np.maximum(val, np.clip((1 - rd/th) * (0.4 + f.get("bass",0.3)*0.8), 0, 1))
|
||||
return val
|
||||
|
||||
def vf_spiral(g, f, t, S, n_arms=3, tightness=2.5):
|
||||
"""Logarithmic spiral arms."""
|
||||
val = np.zeros((g.rows, g.cols), dtype=np.float32)
|
||||
for ai in range(n_arms):
|
||||
offset = ai * 2*np.pi / n_arms
|
||||
log_r = np.log(g.dist + 1) * tightness
|
||||
arm_phase = g.angle + offset - log_r + t * 0.8
|
||||
arm_val = np.clip(np.cos(arm_phase * n_arms) * 0.6 + 0.2, 0, 1)
|
||||
arm_val *= (0.4 + f.get("rms",0.3)*0.6) * np.clip(1 - g.dist_n*0.5, 0.2, 1)
|
||||
val = np.maximum(val, arm_val)
|
||||
return val
|
||||
|
||||
def vf_tunnel(g, f, t, S, speed=3.0, complexity=6):
|
||||
"""Tunnel depth effect — infinite zoom feeling."""
|
||||
tunnel_d = 1.0 / (g.dist_n + 0.1)
|
||||
v1 = np.sin(tunnel_d*2 - t*speed) * 0.45 + 0.55
|
||||
v2 = np.sin(g.angle*complexity + tunnel_d*1.5 - t*2) * 0.35 + 0.55
|
||||
return np.clip(v1*0.5 + v2*0.5, 0, 1)
|
||||
|
||||
def vf_vortex(g, f, t, S, twist=3.0):
|
||||
"""Twisting radial pattern — distance modulates angle."""
|
||||
twisted = g.angle + g.dist_n * twist * np.sin(t * 0.5)
|
||||
val = np.sin(twisted * 4 - t * 2) * 0.5 + 0.5
|
||||
return np.clip(val * (0.5 + f.get("bass",0.3)*0.8), 0, 1)
|
||||
|
||||
def vf_interference(g, f, t, S, n_waves=6):
|
||||
"""Overlapping sine waves creating moire patterns."""
|
||||
drivers = ["mid_r", "himid_r", "bass_r", "lomid_r", "hi_r", "sub_r"]
|
||||
vals = np.zeros((g.rows, g.cols), dtype=np.float32)
|
||||
for i in range(min(n_waves, len(drivers))):
|
||||
angle = i * np.pi / n_waves
|
||||
freq = 0.06 + i * 0.03; sp = 0.5 + i * 0.3
|
||||
proj = g.cc * np.cos(angle) + g.rr * np.sin(angle)
|
||||
vals = vals + np.sin(proj*freq + t*sp) * f.get(drivers[i], 0.3) * 2.5
|
||||
return np.clip(vals * 0.12 + 0.45, 0.1, 1)
|
||||
|
||||
def vf_aurora(g, f, t, S, n_bands=3):
|
||||
"""Horizontal aurora bands."""
|
||||
val = np.zeros((g.rows, g.cols), dtype=np.float32)
|
||||
for i in range(n_bands):
|
||||
fr = 0.08 + i*0.04; fc = 0.012 + i*0.008
|
||||
sr = 0.7 + i*0.3; sc = 0.18 + i*0.12
|
||||
val = val + np.sin(g.rr*fr + t*sr) * np.sin(g.cc*fc + t*sc) * (0.6/n_bands)
|
||||
return np.clip(val * (f.get("lomid_r",0.3)*3 + 0.2), 0, 0.7)
|
||||
|
||||
def vf_ripple(g, f, t, S, sources=None, freq=0.3, damping=0.02):
|
||||
"""Concentric ripples from point sources."""
|
||||
if sources is None: sources = [(0.5, 0.5)]
|
||||
val = np.zeros((g.rows, g.cols), dtype=np.float32)
|
||||
for ry, rx in sources:
|
||||
dy = g.rr - g.rows*ry; dx = g.cc - g.cols*rx
|
||||
d = np.sqrt(dy**2 + dx**2)
|
||||
val = val + np.sin(d*freq - t*4) * np.exp(-d*damping) * 0.5
|
||||
return np.clip(val + 0.5, 0, 1)
|
||||
|
||||
def vf_plasma(g, f, t, S):
|
||||
"""Classic plasma: sum of sines at different orientations and speeds."""
|
||||
v = np.sin(g.cc * 0.03 + t * 0.7) * 0.5
|
||||
v = v + np.sin(g.rr * 0.04 - t * 0.5) * 0.4
|
||||
v = v + np.sin((g.cc * 0.02 + g.rr * 0.03) + t * 0.3) * 0.3
|
||||
v = v + np.sin(g.dist_n * 4 - t * 0.8) * 0.3
|
||||
return np.clip(v * 0.5 + 0.5, 0, 1)
|
||||
|
||||
def vf_diamond(g, f, t, S, freq=0.15):
|
||||
"""Diamond/checkerboard pattern."""
|
||||
val = np.abs(np.sin(g.cc * freq + t * 0.5)) * np.abs(np.sin(g.rr * freq * 1.2 - t * 0.3))
|
||||
return np.clip(val * (0.6 + f.get("rms",0.3)*0.8), 0, 1)
|
||||
|
||||
def vf_noise_static(g, f, t, S, density=0.4):
|
||||
"""Random noise — different each frame. Non-deterministic."""
|
||||
return np.random.random((g.rows, g.cols)).astype(np.float32) * density * (0.5 + f.get("rms",0.3)*0.5)
|
||||
```
|
||||
|
||||
### Hue Field Generators (Color Mapping)
|
||||
|
||||
These produce float32 hue arrays [0,1]. Independently combinable with any value field. Each is a factory returning a closure with signature `(g, f, t, S) -> float32 array`. Can also be a plain float for fixed hue.
|
||||
|
||||
```python
|
||||
def hf_fixed(hue):
|
||||
"""Single hue everywhere."""
|
||||
def fn(g, f, t, S):
|
||||
return np.full((g.rows, g.cols), hue, dtype=np.float32)
|
||||
return fn
|
||||
|
||||
def hf_angle(offset=0.0):
|
||||
"""Hue mapped to angle from center — rainbow wheel."""
|
||||
def fn(g, f, t, S):
|
||||
return (g.angle / (2 * np.pi) + offset + t * 0.05) % 1.0
|
||||
return fn
|
||||
|
||||
def hf_distance(base=0.5, scale=0.02):
|
||||
"""Hue mapped to distance from center."""
|
||||
def fn(g, f, t, S):
|
||||
return (base + g.dist * scale + t * 0.03) % 1.0
|
||||
return fn
|
||||
|
||||
def hf_time_cycle(speed=0.1):
|
||||
"""Hue cycles uniformly over time."""
|
||||
def fn(g, f, t, S):
|
||||
return np.full((g.rows, g.cols), (t * speed) % 1.0, dtype=np.float32)
|
||||
return fn
|
||||
|
||||
def hf_audio_cent():
|
||||
"""Hue follows spectral centroid — timbral color shifting."""
|
||||
def fn(g, f, t, S):
|
||||
return np.full((g.rows, g.cols), f.get("cent", 0.5) * 0.3, dtype=np.float32)
|
||||
return fn
|
||||
|
||||
def hf_gradient_h(start=0.0, end=1.0):
|
||||
"""Left-to-right hue gradient."""
|
||||
def fn(g, f, t, S):
|
||||
h = np.broadcast_to(
|
||||
start + (g.cc / g.cols) * (end - start),
|
||||
(g.rows, g.cols)
|
||||
).copy() # .copy() is CRITICAL — see troubleshooting.md
|
||||
return h % 1.0
|
||||
return fn
|
||||
|
||||
def hf_gradient_v(start=0.0, end=1.0):
|
||||
"""Top-to-bottom hue gradient."""
|
||||
def fn(g, f, t, S):
|
||||
h = np.broadcast_to(
|
||||
start + (g.rr / g.rows) * (end - start),
|
||||
(g.rows, g.cols)
|
||||
).copy()
|
||||
return h % 1.0
|
||||
return fn
|
||||
|
||||
def hf_plasma(speed=0.3):
|
||||
"""Plasma-style hue field — organic color variation."""
|
||||
def fn(g, f, t, S):
|
||||
return (np.sin(g.cc*0.02 + t*speed)*0.5 + np.sin(g.rr*0.015 + t*speed*0.7)*0.5) % 1.0
|
||||
return fn
|
||||
```
|
||||
|
||||
### Combining Value Fields
|
||||
|
||||
The combinatorial explosion comes from mixing value fields with math:
|
||||
|
||||
```python
|
||||
# Multiplication = intersection (only shows where both have brightness)
|
||||
combined = vf_plasma(g,f,t,S) * vf_vortex(g,f,t,S)
|
||||
|
||||
# Addition = union (shows both, clips at 1.0)
|
||||
combined = np.clip(vf_rings(g,f,t,S) + vf_spiral(g,f,t,S), 0, 1)
|
||||
|
||||
# Interference = beat pattern (shows XOR-like patterns)
|
||||
combined = np.abs(vf_plasma(g,f,t,S) - vf_tunnel(g,f,t,S))
|
||||
|
||||
# Modulation = one effect shapes the other
|
||||
combined = vf_rings(g,f,t,S) * (0.3 + 0.7 * vf_plasma(g,f,t,S))
|
||||
|
||||
# Maximum = shows the brightest of two effects
|
||||
combined = np.maximum(vf_spiral(g,f,t,S), vf_aurora(g,f,t,S))
|
||||
```
|
||||
|
||||
### Full Scene Example (v2 — Canvas Return)
|
||||
|
||||
A v2 scene function composes effects internally and returns a pixel canvas:
|
||||
|
||||
```python
|
||||
def scene_complex(r, f, t, S):
|
||||
"""v2 scene function: returns canvas (uint8 H,W,3).
|
||||
r = Renderer, f = audio features, t = time, S = persistent state dict."""
|
||||
g = r.grids["md"]
|
||||
rows, cols = g.rows, g.cols
|
||||
|
||||
# 1. Value field composition
|
||||
plasma = vf_plasma(g, f, t, S)
|
||||
vortex = vf_vortex(g, f, t, S, twist=4.0)
|
||||
combined = np.clip(plasma * 0.6 + vortex * 0.5 + plasma * vortex * 0.4, 0, 1)
|
||||
|
||||
# 2. Color from hue field
|
||||
h = (hf_angle(0.3)(g,f,t,S) * 0.5 + hf_time_cycle(0.08)(g,f,t,S) * 0.5) % 1.0
|
||||
|
||||
# 3. Render to canvas via _render_vf helper
|
||||
canvas = _render_vf(g, combined, h, sat=0.75, pal=PAL_DENSE)
|
||||
|
||||
# 4. Optional: blend a second layer
|
||||
overlay = _render_vf(r.grids["sm"], vf_rings(r.grids["sm"],f,t,S),
|
||||
hf_fixed(0.6)(r.grids["sm"],f,t,S), pal=PAL_BLOCK)
|
||||
canvas = blend_canvas(canvas, overlay, "screen", 0.4)
|
||||
|
||||
return canvas
|
||||
|
||||
# In the render_clip() loop (handled by the framework):
|
||||
# canvas = scene_fn(r, f, t, S)
|
||||
# canvas = tonemap(canvas, gamma=scene_gamma)
|
||||
# canvas = feedback.apply(canvas, ...)
|
||||
# canvas = shader_chain.apply(canvas, f=f, t=t)
|
||||
# pipe.stdin.write(canvas.tobytes())
|
||||
```
|
||||
|
||||
Vary the **value field combo**, **hue field**, **palette**, **blend modes**, **feedback config**, and **shader chain** per section for maximum visual variety. With 12 value fields × 8 hue fields × 14 palettes × 20 blend modes × 7 feedback transforms × 38 shaders, the combinations are effectively infinite.
|
||||
407
skills/creative/ascii-video/references/inputs.md
Normal file
407
skills/creative/ascii-video/references/inputs.md
Normal file
@@ -0,0 +1,407 @@
|
||||
# Input Sources
|
||||
|
||||
## Audio Analysis
|
||||
|
||||
### Loading
|
||||
|
||||
```python
|
||||
tmp = tempfile.mktemp(suffix=".wav")
|
||||
subprocess.run(["ffmpeg", "-y", "-i", input_path, "-ac", "1", "-ar", "22050",
|
||||
"-sample_fmt", "s16", tmp], capture_output=True, check=True)
|
||||
with wave.open(tmp) as wf:
|
||||
sr = wf.getframerate()
|
||||
raw = wf.readframes(wf.getnframes())
|
||||
samples = np.frombuffer(raw, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
```
|
||||
|
||||
### Per-Frame FFT
|
||||
|
||||
```python
|
||||
hop = sr // fps # samples per frame
|
||||
win = hop * 2 # analysis window (2x hop for overlap)
|
||||
window = np.hanning(win)
|
||||
freqs = rfftfreq(win, 1.0 / sr)
|
||||
|
||||
bands = {
|
||||
"sub": (freqs >= 20) & (freqs < 80),
|
||||
"bass": (freqs >= 80) & (freqs < 250),
|
||||
"lomid": (freqs >= 250) & (freqs < 500),
|
||||
"mid": (freqs >= 500) & (freqs < 2000),
|
||||
"himid": (freqs >= 2000)& (freqs < 6000),
|
||||
"hi": (freqs >= 6000),
|
||||
}
|
||||
```
|
||||
|
||||
For each frame: extract chunk, apply window, FFT, compute band energies.
|
||||
|
||||
### Feature Set
|
||||
|
||||
| Feature | Formula | Controls |
|
||||
|---------|---------|----------|
|
||||
| `rms` | `sqrt(mean(chunk²))` | Overall loudness/energy |
|
||||
| `sub`..`hi` | `sqrt(mean(band_magnitudes²))` | Per-band energy |
|
||||
| `centroid` | `sum(freq*mag) / sum(mag)` | Brightness/timbre |
|
||||
| `flatness` | `geomean(mag) / mean(mag)` | Noise vs tone |
|
||||
| `flux` | `sum(max(0, mag - prev_mag))` | Transient strength |
|
||||
| `sub_r`..`hi_r` | `band / sum(all_bands)` | Spectral shape (volume-independent) |
|
||||
| `cent_d` | `abs(gradient(centroid))` | Timbral change rate |
|
||||
| `beat` | Flux peak detection | Binary beat onset |
|
||||
| `bdecay` | Exponential decay from beats | Smooth beat pulse (0→1→0) |
|
||||
|
||||
**Band ratios are critical** — they decouple spectral shape from volume, so a quiet bass section and a loud bass section both read as "bassy" rather than just "loud" vs "quiet".
|
||||
|
||||
### Smoothing
|
||||
|
||||
EMA prevents visual jitter:
|
||||
|
||||
```python
|
||||
def ema(arr, alpha):
|
||||
out = np.empty_like(arr); out[0] = arr[0]
|
||||
for i in range(1, len(arr)):
|
||||
out[i] = alpha * arr[i] + (1 - alpha) * out[i-1]
|
||||
return out
|
||||
|
||||
# Slow-moving features (alpha=0.12): centroid, flatness, band ratios, cent_d
|
||||
# Fast-moving features (alpha=0.3): rms, flux, raw bands
|
||||
```
|
||||
|
||||
### Beat Detection
|
||||
|
||||
```python
|
||||
flux_smooth = np.convolve(flux, np.ones(5)/5, mode="same")
|
||||
peaks, _ = signal.find_peaks(flux_smooth, height=0.15, distance=fps//5, prominence=0.05)
|
||||
|
||||
beat = np.zeros(n_frames)
|
||||
bdecay = np.zeros(n_frames, dtype=np.float32)
|
||||
for p in peaks:
|
||||
beat[p] = 1.0
|
||||
for d in range(fps // 2):
|
||||
if p + d < n_frames:
|
||||
bdecay[p + d] = max(bdecay[p + d], math.exp(-d * 2.5 / (fps // 2)))
|
||||
```
|
||||
|
||||
`bdecay` gives smooth 0→1→0 pulse per beat, decaying over ~0.5s. Use for flash/glitch/mirror triggers.
|
||||
|
||||
### Normalization
|
||||
|
||||
After computing all frames, normalize each feature to 0-1:
|
||||
|
||||
```python
|
||||
for k in features:
|
||||
a = features[k]
|
||||
lo, hi = a.min(), a.max()
|
||||
features[k] = (a - lo) / (hi - lo + 1e-10)
|
||||
```
|
||||
|
||||
## Video Sampling
|
||||
|
||||
### Frame Extraction
|
||||
|
||||
```python
|
||||
# Method 1: ffmpeg pipe (memory efficient)
|
||||
cmd = ["ffmpeg", "-i", input_video, "-f", "rawvideo", "-pix_fmt", "rgb24",
|
||||
"-s", f"{target_w}x{target_h}", "-r", str(fps), "-"]
|
||||
pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
|
||||
frame_size = target_w * target_h * 3
|
||||
for fi in range(n_frames):
|
||||
raw = pipe.stdout.read(frame_size)
|
||||
if len(raw) < frame_size: break
|
||||
frame = np.frombuffer(raw, dtype=np.uint8).reshape(target_h, target_w, 3)
|
||||
# process frame...
|
||||
|
||||
# Method 2: OpenCV (if available)
|
||||
cap = cv2.VideoCapture(input_video)
|
||||
```
|
||||
|
||||
### Luminance-to-Character Mapping
|
||||
|
||||
Convert video pixels to ASCII characters based on brightness:
|
||||
|
||||
```python
|
||||
def frame_to_ascii(frame_rgb, grid, pal=PAL_DEFAULT):
|
||||
"""Convert video frame to character + color arrays."""
|
||||
rows, cols = grid.rows, grid.cols
|
||||
# Resize frame to grid dimensions
|
||||
small = np.array(Image.fromarray(frame_rgb).resize((cols, rows), Image.LANCZOS))
|
||||
# Luminance
|
||||
lum = (0.299 * small[:,:,0] + 0.587 * small[:,:,1] + 0.114 * small[:,:,2]) / 255.0
|
||||
# Map to chars
|
||||
chars = val2char(lum, lum > 0.02, pal)
|
||||
# Colors: use source pixel colors, scaled by luminance for visibility
|
||||
colors = np.clip(small * np.clip(lum[:,:,None] * 1.5 + 0.3, 0.3, 1), 0, 255).astype(np.uint8)
|
||||
return chars, colors
|
||||
```
|
||||
|
||||
### Edge-Weighted Character Mapping
|
||||
|
||||
Use edge detection for more detail in contour regions:
|
||||
|
||||
```python
|
||||
def frame_to_ascii_edges(frame_rgb, grid, pal=PAL_DEFAULT, edge_pal=PAL_BOX):
|
||||
gray = np.mean(frame_rgb, axis=2)
|
||||
small_gray = resize(gray, (grid.rows, grid.cols))
|
||||
lum = small_gray / 255.0
|
||||
|
||||
# Sobel edge detection
|
||||
gx = np.abs(small_gray[:, 2:] - small_gray[:, :-2])
|
||||
gy = np.abs(small_gray[2:, :] - small_gray[:-2, :])
|
||||
edge = np.zeros_like(small_gray)
|
||||
edge[:, 1:-1] += gx; edge[1:-1, :] += gy
|
||||
edge = np.clip(edge / edge.max(), 0, 1)
|
||||
|
||||
# Edge regions get box drawing chars, flat regions get brightness chars
|
||||
is_edge = edge > 0.15
|
||||
chars = val2char(lum, lum > 0.02, pal)
|
||||
edge_chars = val2char(edge, is_edge, edge_pal)
|
||||
chars[is_edge] = edge_chars[is_edge]
|
||||
|
||||
return chars, colors
|
||||
```
|
||||
|
||||
### Motion Detection
|
||||
|
||||
Detect pixel changes between frames for motion-reactive effects:
|
||||
|
||||
```python
|
||||
prev_frame = None
|
||||
def compute_motion(frame):
|
||||
global prev_frame
|
||||
if prev_frame is None:
|
||||
prev_frame = frame.astype(np.float32)
|
||||
return np.zeros(frame.shape[:2])
|
||||
diff = np.abs(frame.astype(np.float32) - prev_frame).mean(axis=2)
|
||||
prev_frame = frame.astype(np.float32) * 0.7 + prev_frame * 0.3 # smoothed
|
||||
return np.clip(diff / 30.0, 0, 1) # normalized motion map
|
||||
```
|
||||
|
||||
Use motion map to drive particle emission, glitch intensity, or character density.
|
||||
|
||||
### Video Feature Extraction
|
||||
|
||||
Per-frame features analogous to audio features, for driving effects:
|
||||
|
||||
```python
|
||||
def analyze_video_frame(frame_rgb):
|
||||
gray = np.mean(frame_rgb, axis=2)
|
||||
return {
|
||||
"brightness": gray.mean() / 255.0,
|
||||
"contrast": gray.std() / 128.0,
|
||||
"edge_density": compute_edge_density(gray),
|
||||
"motion": compute_motion(frame_rgb).mean(),
|
||||
"dominant_hue": compute_dominant_hue(frame_rgb),
|
||||
"color_variance": compute_color_variance(frame_rgb),
|
||||
}
|
||||
```
|
||||
|
||||
## Image Sequence
|
||||
|
||||
### Static Image to ASCII
|
||||
|
||||
Same as single video frame conversion. For animated sequences:
|
||||
|
||||
```python
|
||||
import glob
|
||||
frames = sorted(glob.glob("frames/*.png"))
|
||||
for fi, path in enumerate(frames):
|
||||
img = np.array(Image.open(path).resize((VW, VH)))
|
||||
chars, colors = frame_to_ascii(img, grid, pal)
|
||||
```
|
||||
|
||||
### Image as Texture Source
|
||||
|
||||
Use an image as a background texture that effects modulate:
|
||||
|
||||
```python
|
||||
def load_texture(path, grid):
|
||||
img = np.array(Image.open(path).resize((grid.cols, grid.rows)))
|
||||
lum = np.mean(img, axis=2) / 255.0
|
||||
return lum, img # luminance for char mapping, RGB for colors
|
||||
```
|
||||
|
||||
## Text / Lyrics
|
||||
|
||||
### SRT Parsing
|
||||
|
||||
```python
|
||||
import re
|
||||
def parse_srt(path):
|
||||
"""Returns [(start_sec, end_sec, text), ...]"""
|
||||
entries = []
|
||||
with open(path) as f:
|
||||
content = f.read()
|
||||
blocks = content.strip().split("\n\n")
|
||||
for block in blocks:
|
||||
lines = block.strip().split("\n")
|
||||
if len(lines) >= 3:
|
||||
times = lines[1]
|
||||
m = re.match(r"(\d+):(\d+):(\d+),(\d+) --> (\d+):(\d+):(\d+),(\d+)", times)
|
||||
if m:
|
||||
g = [int(x) for x in m.groups()]
|
||||
start = g[0]*3600 + g[1]*60 + g[2] + g[3]/1000
|
||||
end = g[4]*3600 + g[5]*60 + g[6] + g[7]/1000
|
||||
text = " ".join(lines[2:])
|
||||
entries.append((start, end, text))
|
||||
return entries
|
||||
```
|
||||
|
||||
### Lyrics Display Modes
|
||||
|
||||
- **Typewriter**: characters appear left-to-right over the time window
|
||||
- **Fade-in**: whole line fades from dark to bright
|
||||
- **Flash**: appear instantly on beat, fade out
|
||||
- **Scatter**: characters start at random positions, converge to final position
|
||||
- **Wave**: text follows a sine wave path
|
||||
|
||||
```python
|
||||
def lyrics_typewriter(ch, co, text, row, col, t, t_start, t_end, color):
|
||||
"""Reveal characters progressively over time window."""
|
||||
progress = np.clip((t - t_start) / (t_end - t_start), 0, 1)
|
||||
n_visible = int(len(text) * progress)
|
||||
stamp(ch, co, text[:n_visible], row, col, color)
|
||||
```
|
||||
|
||||
## Generative (No Input)
|
||||
|
||||
For pure generative ASCII art, the "features" dict is synthesized from time:
|
||||
|
||||
```python
|
||||
def synthetic_features(t, bpm=120):
|
||||
"""Generate audio-like features from time alone."""
|
||||
beat_period = 60.0 / bpm
|
||||
beat_phase = (t % beat_period) / beat_period
|
||||
return {
|
||||
"rms": 0.5 + 0.3 * math.sin(t * 0.5),
|
||||
"bass": 0.5 + 0.4 * math.sin(t * 2 * math.pi / beat_period),
|
||||
"sub": 0.3 + 0.3 * math.sin(t * 0.8),
|
||||
"mid": 0.4 + 0.3 * math.sin(t * 1.3),
|
||||
"hi": 0.3 + 0.2 * math.sin(t * 2.1),
|
||||
"cent": 0.5 + 0.2 * math.sin(t * 0.3),
|
||||
"flat": 0.4,
|
||||
"flux": 0.3 + 0.2 * math.sin(t * 3),
|
||||
"beat": 1.0 if beat_phase < 0.05 else 0.0,
|
||||
"bdecay": max(0, 1.0 - beat_phase * 4),
|
||||
# ratios
|
||||
"sub_r": 0.2, "bass_r": 0.25, "lomid_r": 0.15,
|
||||
"mid_r": 0.2, "himid_r": 0.12, "hi_r": 0.08,
|
||||
"cent_d": 0.1,
|
||||
}
|
||||
```
|
||||
|
||||
## TTS Integration
|
||||
|
||||
For narrated videos (testimonials, quotes, storytelling), generate speech audio per segment and mix with background music.
|
||||
|
||||
### ElevenLabs Voice Generation
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
def generate_tts(text, voice_id, api_key, output_path, model="eleven_multilingual_v2"):
|
||||
"""Generate TTS audio via ElevenLabs API."""
|
||||
url = f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}"
|
||||
headers = {"xi-api-key": api_key, "Content-Type": "application/json"}
|
||||
data = {"text": text, "model_id": model,
|
||||
"voice_settings": {"stability": 0.5, "similarity_boost": 0.75}}
|
||||
resp = requests.post(url, json=data, headers=headers, timeout=30)
|
||||
resp.raise_for_status()
|
||||
with open(output_path, "wb") as f:
|
||||
f.write(resp.content)
|
||||
```
|
||||
|
||||
### Voice Assignment
|
||||
|
||||
Use multiple voices for variety. Shuffle deterministically so re-runs are consistent:
|
||||
|
||||
```python
|
||||
import random as _rng
|
||||
|
||||
def assign_voices(n_quotes, voice_pool, seed=42):
|
||||
"""Assign a different voice to each quote, cycling if needed."""
|
||||
r = _rng.Random(seed)
|
||||
shuffled = list(voice_pool)
|
||||
r.shuffle(shuffled)
|
||||
return [shuffled[i % len(shuffled)] for i in range(n_quotes)]
|
||||
```
|
||||
|
||||
### Pronunciation Control
|
||||
|
||||
TTS text should be separate from display text. Common fixes:
|
||||
- Brand names: spell phonetically ("Nous" -> "Noose", "nginx" -> "engine-x")
|
||||
- Abbreviations: expand ("API" -> "A P I", "CLI" -> "C L I")
|
||||
- Technical terms: add phonetic hints
|
||||
|
||||
```python
|
||||
QUOTES = [("Display text here", "Author")]
|
||||
QUOTES_TTS = ["TTS text with phonetic spelling here"]
|
||||
# Keep both arrays in sync -- same indices
|
||||
```
|
||||
|
||||
### Audio Pipeline
|
||||
|
||||
1. Generate individual TTS clips (MP3/WAV per quote)
|
||||
2. Get duration of each clip
|
||||
3. Calculate timing: speech start/end per quote with gaps
|
||||
4. Concatenate into single TTS track with silence padding
|
||||
5. Mix with background music
|
||||
|
||||
```python
|
||||
def build_tts_track(tts_clips, target_duration, gap_seconds=2.0):
|
||||
"""Concatenate TTS clips with gaps, pad to target duration."""
|
||||
# Get durations
|
||||
durations = []
|
||||
for clip in tts_clips:
|
||||
result = subprocess.run(
|
||||
["ffprobe", "-v", "error", "-show_entries", "format=duration",
|
||||
"-of", "csv=p=0", clip],
|
||||
capture_output=True, text=True)
|
||||
durations.append(float(result.stdout.strip()))
|
||||
|
||||
# Calculate timing
|
||||
total_speech = sum(durations)
|
||||
total_gaps = target_duration - total_speech
|
||||
gap = max(0.5, total_gaps / (len(tts_clips) + 1))
|
||||
|
||||
timing = [] # (start, end, quote_index)
|
||||
t = gap # start after initial gap
|
||||
for i, dur in enumerate(durations):
|
||||
timing.append((t, t + dur, i))
|
||||
t += dur + gap
|
||||
|
||||
# Concatenate with ffmpeg
|
||||
# ... silence padding + concat filter
|
||||
return timing
|
||||
```
|
||||
|
||||
### Audio Mixing
|
||||
|
||||
Mix TTS (center) with background music (wide stereo, low volume):
|
||||
|
||||
```python
|
||||
def mix_audio(tts_path, bgm_path, output_path, bgm_volume=0.15):
|
||||
"""Mix TTS centered with BGM panned wide stereo."""
|
||||
cmd = [
|
||||
"ffmpeg", "-y",
|
||||
"-i", tts_path, # mono TTS
|
||||
"-i", bgm_path, # stereo BGM
|
||||
"-filter_complex",
|
||||
f"[0:a]aformat=sample_fmts=fltp:sample_rates=44100:channel_layouts=mono,"
|
||||
f"pan=stereo|c0=c0|c1=c0[tts];" # TTS center
|
||||
f"[1:a]loudnorm=I=-16:TP=-1.5:LRA=11,"
|
||||
f"volume={bgm_volume},"
|
||||
f"extrastereo=2.5[bgm];" # BGM wide stereo
|
||||
f"[tts][bgm]amix=inputs=2:duration=longest[out]",
|
||||
"-map", "[out]", "-c:a", "pcm_s16le", output_path
|
||||
]
|
||||
subprocess.run(cmd, capture_output=True, check=True)
|
||||
```
|
||||
|
||||
### Feature Analysis on Mixed Audio
|
||||
|
||||
Run the standard audio analysis (FFT, beat detection) on the final mixed track so visual effects react to both TTS and music:
|
||||
|
||||
```python
|
||||
# Analyze mixed_final.wav (not individual tracks)
|
||||
features = analyze_audio("mixed_final.wav", fps=24)
|
||||
```
|
||||
|
||||
This means visuals will pulse with both the music beats and the speech energy -- creating natural synchronization.
|
||||
435
skills/creative/ascii-video/references/optimization.md
Normal file
435
skills/creative/ascii-video/references/optimization.md
Normal file
@@ -0,0 +1,435 @@
|
||||
# Optimization Reference
|
||||
|
||||
## Hardware Detection
|
||||
|
||||
Detect the user's hardware at script startup and adapt rendering parameters automatically. Never hardcode worker counts or resolution.
|
||||
|
||||
### CPU and Memory Detection
|
||||
|
||||
```python
|
||||
import multiprocessing
|
||||
import platform
|
||||
import shutil
|
||||
import os
|
||||
|
||||
def detect_hardware():
|
||||
"""Detect hardware capabilities and return render config."""
|
||||
cpu_count = multiprocessing.cpu_count()
|
||||
|
||||
# Leave 1-2 cores free for OS + ffmpeg encoding
|
||||
if cpu_count >= 16:
|
||||
workers = cpu_count - 2
|
||||
elif cpu_count >= 8:
|
||||
workers = cpu_count - 1
|
||||
elif cpu_count >= 4:
|
||||
workers = cpu_count - 1
|
||||
else:
|
||||
workers = max(1, cpu_count)
|
||||
|
||||
# Memory detection (platform-specific)
|
||||
try:
|
||||
if platform.system() == "Darwin":
|
||||
import subprocess
|
||||
mem_bytes = int(subprocess.check_output(["sysctl", "-n", "hw.memsize"]).strip())
|
||||
elif platform.system() == "Linux":
|
||||
with open("/proc/meminfo") as f:
|
||||
for line in f:
|
||||
if line.startswith("MemTotal"):
|
||||
mem_bytes = int(line.split()[1]) * 1024
|
||||
break
|
||||
else:
|
||||
mem_bytes = 8 * 1024**3 # assume 8GB on unknown
|
||||
except Exception:
|
||||
mem_bytes = 8 * 1024**3
|
||||
|
||||
mem_gb = mem_bytes / (1024**3)
|
||||
|
||||
# Each worker uses ~50-150MB depending on grid sizes
|
||||
# Cap workers if memory is tight
|
||||
mem_per_worker_mb = 150
|
||||
max_workers_by_mem = int(mem_gb * 1024 * 0.6 / mem_per_worker_mb) # use 60% of RAM
|
||||
workers = min(workers, max_workers_by_mem)
|
||||
|
||||
# ffmpeg availability and codec support
|
||||
has_ffmpeg = shutil.which("ffmpeg") is not None
|
||||
|
||||
return {
|
||||
"cpu_count": cpu_count,
|
||||
"workers": workers,
|
||||
"mem_gb": mem_gb,
|
||||
"platform": platform.system(),
|
||||
"arch": platform.machine(),
|
||||
"has_ffmpeg": has_ffmpeg,
|
||||
}
|
||||
```
|
||||
|
||||
### Adaptive Quality Profiles
|
||||
|
||||
Scale resolution, FPS, CRF, and grid density based on hardware:
|
||||
|
||||
```python
|
||||
def quality_profile(hw, target_duration_s, user_preference="auto"):
|
||||
"""
|
||||
Returns render settings adapted to hardware.
|
||||
user_preference: "auto", "draft", "preview", "production", "max"
|
||||
"""
|
||||
if user_preference == "draft":
|
||||
return {"vw": 960, "vh": 540, "fps": 12, "crf": 28, "workers": min(4, hw["workers"]),
|
||||
"grid_scale": 0.5, "shaders": "minimal", "particles_max": 200}
|
||||
|
||||
if user_preference == "preview":
|
||||
return {"vw": 1280, "vh": 720, "fps": 15, "crf": 25, "workers": hw["workers"],
|
||||
"grid_scale": 0.75, "shaders": "standard", "particles_max": 500}
|
||||
|
||||
if user_preference == "max":
|
||||
return {"vw": 3840, "vh": 2160, "fps": 30, "crf": 15, "workers": hw["workers"],
|
||||
"grid_scale": 2.0, "shaders": "full", "particles_max": 3000}
|
||||
|
||||
# "production" or "auto"
|
||||
# Auto-detect: estimate render time, downgrade if it would take too long
|
||||
n_frames = int(target_duration_s * 24)
|
||||
est_seconds_per_frame = 0.18 # ~180ms at 1080p
|
||||
est_total_s = n_frames * est_seconds_per_frame / max(1, hw["workers"])
|
||||
|
||||
if hw["mem_gb"] < 4 or hw["cpu_count"] <= 2:
|
||||
# Low-end: 720p, 15fps
|
||||
return {"vw": 1280, "vh": 720, "fps": 15, "crf": 23, "workers": hw["workers"],
|
||||
"grid_scale": 0.75, "shaders": "standard", "particles_max": 500}
|
||||
|
||||
if est_total_s > 3600: # would take over an hour
|
||||
# Downgrade to 720p to speed up
|
||||
return {"vw": 1280, "vh": 720, "fps": 24, "crf": 20, "workers": hw["workers"],
|
||||
"grid_scale": 0.75, "shaders": "standard", "particles_max": 800}
|
||||
|
||||
# Standard production: 1080p 24fps
|
||||
return {"vw": 1920, "vh": 1080, "fps": 24, "crf": 20, "workers": hw["workers"],
|
||||
"grid_scale": 1.0, "shaders": "full", "particles_max": 1200}
|
||||
|
||||
|
||||
def apply_quality_profile(profile):
|
||||
"""Set globals from quality profile."""
|
||||
global VW, VH, FPS, N_WORKERS
|
||||
VW = profile["vw"]
|
||||
VH = profile["vh"]
|
||||
FPS = profile["fps"]
|
||||
N_WORKERS = profile["workers"]
|
||||
# Grid sizes scale with resolution
|
||||
# CRF passed to ffmpeg encoder
|
||||
# Shader set determines which post-processing is active
|
||||
```
|
||||
|
||||
### CLI Integration
|
||||
|
||||
```python
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--quality", choices=["draft", "preview", "production", "max", "auto"],
|
||||
default="auto", help="Render quality preset")
|
||||
parser.add_argument("--workers", type=int, default=0, help="Override worker count (0=auto)")
|
||||
parser.add_argument("--resolution", type=str, default="", help="Override resolution e.g. 1280x720")
|
||||
args = parser.parse_args()
|
||||
|
||||
hw = detect_hardware()
|
||||
if args.workers > 0:
|
||||
hw["workers"] = args.workers
|
||||
profile = quality_profile(hw, target_duration, args.quality)
|
||||
if args.resolution:
|
||||
w, h = args.resolution.split("x")
|
||||
profile["vw"], profile["vh"] = int(w), int(h)
|
||||
apply_quality_profile(profile)
|
||||
|
||||
log(f"Hardware: {hw['cpu_count']} cores, {hw['mem_gb']:.1f}GB RAM, {hw['platform']}")
|
||||
log(f"Render: {profile['vw']}x{profile['vh']} @{profile['fps']}fps, "
|
||||
f"CRF {profile['crf']}, {profile['workers']} workers")
|
||||
```
|
||||
|
||||
## Performance Budget
|
||||
|
||||
Target: 100-200ms per frame (5-10 fps single-threaded, 40-80 fps across 8 workers).
|
||||
|
||||
| Component | Time | Notes |
|
||||
|-----------|------|-------|
|
||||
| Feature extraction | 1-5ms | Pre-computed for all frames before render |
|
||||
| Effect function | 2-15ms | Vectorized numpy, avoid Python loops |
|
||||
| Character render | 80-150ms | **Bottleneck** -- per-cell Python loop |
|
||||
| Shader pipeline | 5-25ms | Depends on active shaders |
|
||||
| ffmpeg encode | ~5ms | Amortized by pipe buffering |
|
||||
|
||||
## Bitmap Pre-Rasterization
|
||||
|
||||
Rasterize every character at init, not per-frame:
|
||||
|
||||
```python
|
||||
# At init time -- done once
|
||||
for c in all_characters:
|
||||
img = Image.new("L", (cell_w, cell_h), 0)
|
||||
ImageDraw.Draw(img).text((0, 0), c, fill=255, font=font)
|
||||
bitmaps[c] = np.array(img, dtype=np.float32) / 255.0 # float32 for fast multiply
|
||||
|
||||
# At render time -- fast lookup
|
||||
bitmap = bitmaps[char]
|
||||
canvas[y:y+ch, x:x+cw] = np.maximum(canvas[y:y+ch, x:x+cw],
|
||||
(bitmap[:,:,None] * color).astype(np.uint8))
|
||||
```
|
||||
|
||||
Collect all characters from all palettes + overlay text into the init set. Lazy-init for any missed characters.
|
||||
|
||||
## Coordinate Array Caching
|
||||
|
||||
Pre-compute all grid-relative coordinate arrays at init, not per-frame:
|
||||
|
||||
```python
|
||||
# These are O(rows*cols) and used in every effect
|
||||
self.rr = np.arange(rows)[:, None] # row indices
|
||||
self.cc = np.arange(cols)[None, :] # col indices
|
||||
self.dist = np.sqrt(dx**2 + dy**2) # distance from center
|
||||
self.angle = np.arctan2(dy, dx) # angle from center
|
||||
self.dist_n = ... # normalized distance
|
||||
```
|
||||
|
||||
## Vectorized Effect Patterns
|
||||
|
||||
### Avoid Per-Cell Python Loops in Effects
|
||||
|
||||
The render loop (compositing bitmaps) is unavoidably per-cell. But effect functions must be fully vectorized numpy -- never iterate over rows/cols in Python.
|
||||
|
||||
Bad (O(rows*cols) Python loop):
|
||||
```python
|
||||
for r in range(rows):
|
||||
for c in range(cols):
|
||||
val[r, c] = math.sin(c * 0.1 + t) * math.cos(r * 0.1 - t)
|
||||
```
|
||||
|
||||
Good (vectorized):
|
||||
```python
|
||||
val = np.sin(g.cc * 0.1 + t) * np.cos(g.rr * 0.1 - t)
|
||||
```
|
||||
|
||||
### Vectorized Matrix Rain
|
||||
|
||||
The naive per-column per-trail-pixel loop is the second biggest bottleneck after the render loop. Use numpy fancy indexing:
|
||||
|
||||
```python
|
||||
# Instead of nested Python loops over columns and trail pixels:
|
||||
# Build row index arrays for all active trail pixels at once
|
||||
all_rows = []
|
||||
all_cols = []
|
||||
all_fades = []
|
||||
for c in range(cols):
|
||||
head = int(state["ry"][c])
|
||||
trail_len = state["rln"][c]
|
||||
for i in range(trail_len):
|
||||
row = head - i
|
||||
if 0 <= row < rows:
|
||||
all_rows.append(row)
|
||||
all_cols.append(c)
|
||||
all_fades.append(1.0 - i / trail_len)
|
||||
|
||||
# Vectorized assignment
|
||||
ar = np.array(all_rows)
|
||||
ac = np.array(all_cols)
|
||||
af = np.array(all_fades, dtype=np.float32)
|
||||
# Assign chars and colors in bulk using fancy indexing
|
||||
ch[ar, ac] = ... # vectorized char assignment
|
||||
co[ar, ac, 1] = (af * bri * 255).astype(np.uint8) # green channel
|
||||
```
|
||||
|
||||
### Vectorized Fire Columns
|
||||
|
||||
Same pattern -- accumulate index arrays, assign in bulk:
|
||||
|
||||
```python
|
||||
fire_val = np.zeros((rows, cols), dtype=np.float32)
|
||||
for fi in range(n_cols):
|
||||
fx_c = int((fi * cols / n_cols + np.sin(t * 2 + fi * 0.7) * 3) % cols)
|
||||
height = int(energy * rows * 0.7)
|
||||
dy = np.arange(min(height, rows))
|
||||
fr = rows - 1 - dy
|
||||
frac = dy / max(height, 1)
|
||||
# Width spread: base columns wider at bottom
|
||||
for dx in range(-1, 2): # 3-wide columns
|
||||
c = fx_c + dx
|
||||
if 0 <= c < cols:
|
||||
fire_val[fr, c] = np.maximum(fire_val[fr, c],
|
||||
(1 - frac * 0.6) * (0.5 + rms * 0.5))
|
||||
# Now map fire_val to chars and colors in one vectorized pass
|
||||
```
|
||||
|
||||
## Bloom Optimization
|
||||
|
||||
**Do NOT use `scipy.ndimage.uniform_filter`** -- measured at 424ms/frame.
|
||||
|
||||
Use 4x downsample + manual box blur instead -- 84ms/frame (5x faster):
|
||||
|
||||
```python
|
||||
sm = canvas[::4, ::4].astype(np.float32) # 4x downsample
|
||||
br = np.where(sm > threshold, sm, 0)
|
||||
for _ in range(3): # 3-pass manual box blur
|
||||
p = np.pad(br, ((1,1),(1,1),(0,0)), mode='edge')
|
||||
br = (p[:-2,:-2] + p[:-2,1:-1] + p[:-2,2:] +
|
||||
p[1:-1,:-2] + p[1:-1,1:-1] + p[1:-1,2:] +
|
||||
p[2:,:-2] + p[2:,1:-1] + p[2:,2:]) / 9.0
|
||||
bl = np.repeat(np.repeat(br, 4, axis=0), 4, axis=1)[:H, :W]
|
||||
```
|
||||
|
||||
## Vignette Caching
|
||||
|
||||
Distance field is resolution- and strength-dependent, never changes per frame:
|
||||
|
||||
```python
|
||||
_vig_cache = {}
|
||||
def sh_vignette(canvas, strength):
|
||||
key = (canvas.shape[0], canvas.shape[1], round(strength, 2))
|
||||
if key not in _vig_cache:
|
||||
Y = np.linspace(-1, 1, H)[:, None]
|
||||
X = np.linspace(-1, 1, W)[None, :]
|
||||
_vig_cache[key] = np.clip(1.0 - np.sqrt(X**2+Y**2) * strength, 0.15, 1).astype(np.float32)
|
||||
return np.clip(canvas * _vig_cache[key][:,:,None], 0, 255).astype(np.uint8)
|
||||
```
|
||||
|
||||
Same pattern for CRT barrel distortion (cache remap coordinates).
|
||||
|
||||
## Film Grain Optimization
|
||||
|
||||
Generate noise at half resolution, tile up:
|
||||
|
||||
```python
|
||||
noise = np.random.randint(-amt, amt+1, (H//2, W//2, 1), dtype=np.int16)
|
||||
noise = np.repeat(np.repeat(noise, 2, axis=0), 2, axis=1)[:H, :W]
|
||||
```
|
||||
|
||||
2x blocky grain looks like film grain and costs 1/4 the random generation.
|
||||
|
||||
## Parallel Rendering
|
||||
|
||||
### Worker Architecture
|
||||
|
||||
```python
|
||||
hw = detect_hardware()
|
||||
N_WORKERS = hw["workers"]
|
||||
|
||||
# Batch splitting (for non-clip architectures)
|
||||
batch_size = (n_frames + N_WORKERS - 1) // N_WORKERS
|
||||
batches = [(i, i*batch_size, min((i+1)*batch_size, n_frames), features, seg_path) ...]
|
||||
|
||||
with multiprocessing.Pool(N_WORKERS) as pool:
|
||||
segments = pool.starmap(render_batch, batches)
|
||||
```
|
||||
|
||||
### Per-Clip Parallelism (Preferred for Segmented Videos)
|
||||
|
||||
```python
|
||||
from concurrent.futures import ProcessPoolExecutor, as_completed
|
||||
|
||||
with ProcessPoolExecutor(max_workers=N_WORKERS) as pool:
|
||||
futures = {pool.submit(render_clip, seg, features, path): seg["id"]
|
||||
for seg, path in clip_args}
|
||||
for fut in as_completed(futures):
|
||||
clip_id = futures[fut]
|
||||
try:
|
||||
fut.result()
|
||||
log(f" {clip_id} done")
|
||||
except Exception as e:
|
||||
log(f" {clip_id} FAILED: {e}")
|
||||
```
|
||||
|
||||
### Worker Isolation
|
||||
|
||||
Each worker:
|
||||
- Creates its own `Renderer` instance (with full grid + bitmap init)
|
||||
- Opens its own ffmpeg subprocess
|
||||
- Has independent random seed (`random.seed(batch_id * 10000)`)
|
||||
- Writes to its own segment file and stderr log
|
||||
|
||||
### ffmpeg Pipe Safety
|
||||
|
||||
**CRITICAL**: Never `stderr=subprocess.PIPE` with long-running ffmpeg. The stderr buffer fills at ~64KB and deadlocks:
|
||||
|
||||
```python
|
||||
# WRONG -- will deadlock
|
||||
pipe = subprocess.Popen(cmd, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
|
||||
# RIGHT -- stderr to file
|
||||
stderr_fh = open(err_path, "w")
|
||||
pipe = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL, stderr=stderr_fh)
|
||||
# ... write all frames ...
|
||||
pipe.stdin.close()
|
||||
pipe.wait()
|
||||
stderr_fh.close()
|
||||
```
|
||||
|
||||
### Concatenation
|
||||
|
||||
```python
|
||||
with open(concat_file, "w") as cf:
|
||||
for seg in segments:
|
||||
cf.write(f"file '{seg}'\n")
|
||||
|
||||
cmd = ["ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", concat_file]
|
||||
if audio_path:
|
||||
cmd += ["-i", audio_path, "-c:v", "copy", "-c:a", "aac", "-b:a", "192k", "-shortest"]
|
||||
else:
|
||||
cmd += ["-c:v", "copy"]
|
||||
cmd.append(output_path)
|
||||
subprocess.run(cmd, capture_output=True, check=True)
|
||||
```
|
||||
|
||||
## Particle System Performance
|
||||
|
||||
Cap particle counts based on quality profile:
|
||||
|
||||
| System | Low | Standard | High |
|
||||
|--------|-----|----------|------|
|
||||
| Explosion | 300 | 1000 | 2500 |
|
||||
| Embers | 500 | 1500 | 3000 |
|
||||
| Starfield | 300 | 800 | 1500 |
|
||||
| Dissolve | 200 | 600 | 1200 |
|
||||
|
||||
Cull by truncating lists:
|
||||
```python
|
||||
MAX_PARTICLES = profile.get("particles_max", 1200)
|
||||
if len(S["px"]) > MAX_PARTICLES:
|
||||
for k in ("px", "py", "vx", "vy", "life", "char"):
|
||||
S[k] = S[k][-MAX_PARTICLES:] # keep newest
|
||||
```
|
||||
|
||||
## Memory Management
|
||||
|
||||
- Feature arrays: pre-computed for all frames, shared across workers via fork semantics (COW)
|
||||
- Canvas: allocated once per worker, reused (`np.zeros(...)`)
|
||||
- Character arrays: allocated per frame (cheap -- rows*cols U1 strings)
|
||||
- Bitmap cache: ~500KB per grid size, initialized once per worker
|
||||
|
||||
Total memory per worker: ~50-150MB. Total: ~400-800MB for 8 workers.
|
||||
|
||||
For low-memory systems (< 4GB), reduce worker count and use smaller grids.
|
||||
|
||||
## Brightness Verification
|
||||
|
||||
After render, spot-check brightness at sample timestamps:
|
||||
|
||||
```python
|
||||
for t in [2, 30, 60, 120, 180]:
|
||||
cmd = ["ffmpeg", "-ss", str(t), "-i", output_path,
|
||||
"-frames:v", "1", "-f", "rawvideo", "-pix_fmt", "rgb24", "-"]
|
||||
r = subprocess.run(cmd, capture_output=True)
|
||||
arr = np.frombuffer(r.stdout, dtype=np.uint8)
|
||||
print(f"t={t}s mean={arr.mean():.1f} max={arr.max()}")
|
||||
```
|
||||
|
||||
Target: mean > 5 for quiet sections, mean > 15 for active sections. If consistently below, increase brightness floor in effects and/or global boost multiplier.
|
||||
|
||||
## Render Time Estimates
|
||||
|
||||
Scale with hardware. Baseline: 1080p, 24fps, ~180ms/frame/worker.
|
||||
|
||||
| Duration | Frames | 4 workers | 8 workers | 16 workers |
|
||||
|----------|--------|-----------|-----------|------------|
|
||||
| 30s | 720 | ~3 min | ~2 min | ~1 min |
|
||||
| 2 min | 2,880 | ~13 min | ~7 min | ~4 min |
|
||||
| 3.5 min | 5,040 | ~23 min | ~12 min | ~6 min |
|
||||
| 5 min | 7,200 | ~33 min | ~17 min | ~9 min |
|
||||
| 10 min | 14,400 | ~65 min | ~33 min | ~17 min |
|
||||
|
||||
At 720p: multiply times by ~0.5. At 4K: multiply by ~4.
|
||||
|
||||
Heavier effects (many particles, dense grids, extra shader passes) add ~20-50%.
|
||||
382
skills/creative/ascii-video/references/scenes.md
Normal file
382
skills/creative/ascii-video/references/scenes.md
Normal file
@@ -0,0 +1,382 @@
|
||||
# Scene System Reference
|
||||
|
||||
Scenes are the top-level creative unit. Each scene is a time-bounded segment with its own effect function, shader chain, feedback configuration, and tone-mapping gamma.
|
||||
|
||||
## Scene Protocol (v2)
|
||||
|
||||
### Function Signature
|
||||
|
||||
```python
|
||||
def fx_scene_name(r, f, t, S) -> canvas:
|
||||
"""
|
||||
Args:
|
||||
r: Renderer instance — access multiple grids via r.get_grid("sm")
|
||||
f: dict of audio/video features, all values normalized to [0, 1]
|
||||
t: time in seconds (global, not local to scene)
|
||||
S: dict for persistent state (particles, rain columns, etc.)
|
||||
|
||||
Returns:
|
||||
canvas: numpy uint8 array, shape (VH, VW, 3) — full pixel frame
|
||||
"""
|
||||
```
|
||||
|
||||
This replaces the v1 protocol where scenes returned `(chars, colors)` tuples. The v2 protocol gives scenes full control over multi-grid rendering and pixel-level composition internally.
|
||||
|
||||
### The Renderer Class
|
||||
|
||||
```python
|
||||
class Renderer:
|
||||
def __init__(self):
|
||||
self.grids = {} # lazy-initialized grid cache
|
||||
self.g = None # "active" grid (for backward compat)
|
||||
self.S = {} # persistent state dict
|
||||
|
||||
def get_grid(self, key):
|
||||
"""Get or create a GridLayer by size key."""
|
||||
if key not in self.grids:
|
||||
sizes = {"xs": 8, "sm": 10, "md": 16, "lg": 20, "xl": 24, "xxl": 40}
|
||||
self.grids[key] = GridLayer(FONT_PATH, sizes[key])
|
||||
return self.grids[key]
|
||||
|
||||
def set_grid(self, key):
|
||||
"""Set active grid (legacy). Prefer get_grid() for multi-grid scenes."""
|
||||
self.g = self.get_grid(key)
|
||||
return self.g
|
||||
```
|
||||
|
||||
**Key difference from v1**: scenes call `r.get_grid("sm")`, `r.get_grid("lg")`, etc. to access multiple grids. Each grid is lazy-initialized and cached. The `set_grid()` method still works for single-grid scenes.
|
||||
|
||||
### Minimal Scene (Single Grid)
|
||||
|
||||
```python
|
||||
def fx_simple_rings(r, f, t, S):
|
||||
"""Single-grid scene: rings with distance-mapped hue."""
|
||||
canvas = _render_vf(r, "md",
|
||||
lambda g, f, t, S: vf_rings(g, f, t, S, n_base=8, spacing_base=3),
|
||||
hf_distance(0.3, 0.02), PAL_STARS, f, t, S, sat=0.85)
|
||||
return canvas
|
||||
```
|
||||
|
||||
### Standard Scene (Two Grids + Blend)
|
||||
|
||||
```python
|
||||
def fx_tunnel_ripple(r, f, t, S):
|
||||
"""Two-grid scene: tunnel depth exclusion-blended with ripple."""
|
||||
canvas_a = _render_vf(r, "md",
|
||||
lambda g, f, t, S: vf_tunnel(g, f, t, S, speed=5.0, complexity=10) * 1.3,
|
||||
hf_distance(0.55, 0.02), PAL_GREEK, f, t, S, sat=0.7)
|
||||
|
||||
canvas_b = _render_vf(r, "sm",
|
||||
lambda g, f, t, S: vf_ripple(g, f, t, S,
|
||||
sources=[(0.3,0.3), (0.7,0.7), (0.5,0.2)], freq=0.5, damping=0.012) * 1.4,
|
||||
hf_angle(0.1), PAL_STARS, f, t, S, sat=0.8)
|
||||
|
||||
return blend_canvas(canvas_a, canvas_b, "exclusion", 0.8)
|
||||
```
|
||||
|
||||
### Complex Scene (Three Grids + Conditional + Custom Rendering)
|
||||
|
||||
```python
|
||||
def fx_rings_explosion(r, f, t, S):
|
||||
"""Three-grid scene with particles and conditional kaleidoscope."""
|
||||
# Layer 1: rings
|
||||
canvas_a = _render_vf(r, "sm",
|
||||
lambda g, f, t, S: vf_rings(g, f, t, S, n_base=10, spacing_base=2) * 1.4,
|
||||
lambda g, f, t, S: (g.angle / (2*np.pi) + t * 0.15) % 1.0,
|
||||
PAL_STARS, f, t, S, sat=0.9)
|
||||
|
||||
# Layer 2: vortex on different grid
|
||||
canvas_b = _render_vf(r, "md",
|
||||
lambda g, f, t, S: vf_vortex(g, f, t, S, twist=6.0) * 1.2,
|
||||
hf_time_cycle(0.15), PAL_BLOCKS, f, t, S, sat=0.8)
|
||||
|
||||
result = blend_canvas(canvas_b, canvas_a, "screen", 0.7)
|
||||
|
||||
# Layer 3: particles (custom rendering, not _render_vf)
|
||||
g = r.get_grid("sm")
|
||||
if "px" not in S:
|
||||
S["px"], S["py"], S["vx"], S["vy"], S["life"], S["pch"] = (
|
||||
[], [], [], [], [], [])
|
||||
if f.get("beat", 0) > 0.5:
|
||||
chars = list("\u2605\u2736\u2733\u2738\u2726\u2728*+")
|
||||
for _ in range(int(80 + f.get("rms", 0.3) * 120)):
|
||||
ang = random.uniform(0, 2 * math.pi)
|
||||
sp = random.uniform(1, 10) * (0.5 + f.get("sub_r", 0.3) * 2)
|
||||
S["px"].append(float(g.cols // 2))
|
||||
S["py"].append(float(g.rows // 2))
|
||||
S["vx"].append(math.cos(ang) * sp * 2.5)
|
||||
S["vy"].append(math.sin(ang) * sp)
|
||||
S["life"].append(1.0)
|
||||
S["pch"].append(random.choice(chars))
|
||||
|
||||
# Update + draw particles
|
||||
ch_p = np.full((g.rows, g.cols), " ", dtype="U1")
|
||||
co_p = np.zeros((g.rows, g.cols, 3), dtype=np.uint8)
|
||||
i = 0
|
||||
while i < len(S["px"]):
|
||||
S["px"][i] += S["vx"][i]; S["py"][i] += S["vy"][i]
|
||||
S["vy"][i] += 0.03; S["life"][i] -= 0.02
|
||||
if S["life"][i] <= 0:
|
||||
for k in ("px","py","vx","vy","life","pch"): S[k].pop(i)
|
||||
else:
|
||||
pr, pc = int(S["py"][i]), int(S["px"][i])
|
||||
if 0 <= pr < g.rows and 0 <= pc < g.cols:
|
||||
ch_p[pr, pc] = S["pch"][i]
|
||||
co_p[pr, pc] = hsv2rgb_scalar(
|
||||
0.08 + (1-S["life"][i])*0.15, 0.95, S["life"][i])
|
||||
i += 1
|
||||
|
||||
canvas_p = g.render(ch_p, co_p)
|
||||
result = blend_canvas(result, canvas_p, "add", 0.8)
|
||||
|
||||
# Conditional kaleidoscope on strong beats
|
||||
if f.get("bdecay", 0) > 0.4:
|
||||
result = sh_kaleidoscope(result.copy(), folds=6)
|
||||
|
||||
return result
|
||||
```
|
||||
|
||||
### Scene with Custom Character Rendering (Matrix Rain)
|
||||
|
||||
When you need per-cell control beyond what `_render_vf()` provides:
|
||||
|
||||
```python
|
||||
def fx_matrix_layered(r, f, t, S):
|
||||
"""Matrix rain blended with tunnel — two grids, screen blend."""
|
||||
# Layer 1: Matrix rain (custom per-column rendering)
|
||||
g = r.get_grid("md")
|
||||
rows, cols = g.rows, g.cols
|
||||
pal = PAL_KATA
|
||||
|
||||
if "ry" not in S or len(S["ry"]) != cols:
|
||||
S["ry"] = np.random.uniform(-rows, rows, cols).astype(np.float32)
|
||||
S["rsp"] = np.random.uniform(0.3, 2.0, cols).astype(np.float32)
|
||||
S["rln"] = np.random.randint(8, 35, cols)
|
||||
S["rch"] = np.random.randint(1, len(pal), (rows, cols))
|
||||
|
||||
speed = 0.6 + f.get("bass", 0.3) * 3
|
||||
if f.get("beat", 0) > 0.5: speed *= 2.5
|
||||
S["ry"] += S["rsp"] * speed
|
||||
|
||||
ch = np.full((rows, cols), " ", dtype="U1")
|
||||
co = np.zeros((rows, cols, 3), dtype=np.uint8)
|
||||
heads = S["ry"].astype(int)
|
||||
for c in range(cols):
|
||||
head = heads[c]
|
||||
for i in range(S["rln"][c]):
|
||||
row = head - i
|
||||
if 0 <= row < rows:
|
||||
fade = 1.0 - i / S["rln"][c]
|
||||
ch[row, c] = pal[S["rch"][row, c] % len(pal)]
|
||||
if i == 0:
|
||||
v = int(min(255, fade * 300))
|
||||
co[row, c] = (int(v*0.9), v, int(v*0.9))
|
||||
else:
|
||||
v = int(fade * 240)
|
||||
co[row, c] = (int(v*0.1), v, int(v*0.4))
|
||||
canvas_a = g.render(ch, co)
|
||||
|
||||
# Layer 2: Tunnel on sm grid for depth texture
|
||||
canvas_b = _render_vf(r, "sm",
|
||||
lambda g, f, t, S: vf_tunnel(g, f, t, S, speed=5.0, complexity=10),
|
||||
hf_distance(0.3, 0.02), PAL_BLOCKS, f, t, S, sat=0.6)
|
||||
|
||||
return blend_canvas(canvas_a, canvas_b, "screen", 0.5)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Scene Table
|
||||
|
||||
The scene table defines the timeline: which scene plays when, with what configuration.
|
||||
|
||||
### Structure
|
||||
|
||||
```python
|
||||
SCENES = [
|
||||
{
|
||||
"start": 0.0, # start time in seconds
|
||||
"end": 3.96, # end time in seconds
|
||||
"name": "starfield", # identifier (used for clip filenames)
|
||||
"grid": "sm", # default grid (for render_clip setup)
|
||||
"fx": fx_starfield, # scene function reference (must be module-level)
|
||||
"gamma": 0.75, # tonemap gamma override (default 0.75)
|
||||
"shaders": [ # shader chain (applied after tonemap + feedback)
|
||||
("bloom", {"thr": 120}),
|
||||
("vignette", {"s": 0.2}),
|
||||
("grain", {"amt": 8}),
|
||||
],
|
||||
"feedback": None, # feedback buffer config (None = disabled)
|
||||
# "feedback": {"decay": 0.8, "blend": "screen", "opacity": 0.3,
|
||||
# "transform": "zoom", "transform_amt": 0.02, "hue_shift": 0.02},
|
||||
},
|
||||
{
|
||||
"start": 3.96,
|
||||
"end": 6.58,
|
||||
"name": "matrix_layered",
|
||||
"grid": "md",
|
||||
"fx": fx_matrix_layered,
|
||||
"shaders": [
|
||||
("crt", {"strength": 0.05}),
|
||||
("scanlines", {"intensity": 0.12}),
|
||||
("color_grade", {"tint": (0.7, 1.2, 0.7)}),
|
||||
("bloom", {"thr": 100}),
|
||||
],
|
||||
"feedback": {"decay": 0.5, "blend": "add", "opacity": 0.2},
|
||||
},
|
||||
# ... more scenes ...
|
||||
]
|
||||
```
|
||||
|
||||
### Beat-Synced Scene Cutting
|
||||
|
||||
Derive cut points from audio analysis:
|
||||
|
||||
```python
|
||||
# Get beat timestamps
|
||||
beats = [fi / FPS for fi in range(N_FRAMES) if features["beat"][fi] > 0.5]
|
||||
|
||||
# Group beats into phrase boundaries (every 4-8 beats)
|
||||
cuts = [0.0]
|
||||
for i in range(0, len(beats), 4): # cut every 4 beats
|
||||
cuts.append(beats[i])
|
||||
cuts.append(DURATION)
|
||||
|
||||
# Or use the music's structure: silence gaps, energy changes
|
||||
energy = features["rms"]
|
||||
# Find timestamps where energy drops significantly -> natural break points
|
||||
```
|
||||
|
||||
### `render_clip()` — The Render Loop
|
||||
|
||||
This function renders one scene to a clip file:
|
||||
|
||||
```python
|
||||
def render_clip(seg, features, clip_path):
|
||||
r = Renderer()
|
||||
r.set_grid(seg["grid"])
|
||||
S = r.S
|
||||
random.seed(hash(seg["id"]) + 42) # deterministic per scene
|
||||
|
||||
# Build shader chain from config
|
||||
chain = ShaderChain()
|
||||
for shader_name, kwargs in seg.get("shaders", []):
|
||||
chain.add(shader_name, **kwargs)
|
||||
|
||||
# Setup feedback buffer
|
||||
fb = None
|
||||
fb_cfg = seg.get("feedback", None)
|
||||
if fb_cfg:
|
||||
fb = FeedbackBuffer()
|
||||
|
||||
fx_fn = seg["fx"]
|
||||
|
||||
# Open ffmpeg pipe
|
||||
cmd = ["ffmpeg", "-y", "-f", "rawvideo", "-pix_fmt", "rgb24",
|
||||
"-s", f"{VW}x{VH}", "-r", str(FPS), "-i", "pipe:0",
|
||||
"-c:v", "libx264", "-preset", "fast", "-crf", "20",
|
||||
"-pix_fmt", "yuv420p", clip_path]
|
||||
stderr_fh = open(clip_path.replace(".mp4", ".log"), "w")
|
||||
pipe = subprocess.Popen(cmd, stdin=subprocess.PIPE,
|
||||
stdout=subprocess.DEVNULL, stderr=stderr_fh)
|
||||
|
||||
for fi in range(seg["frame_start"], seg["frame_end"]):
|
||||
t = fi / FPS
|
||||
feat = {k: float(features[k][fi]) for k in features}
|
||||
|
||||
# 1. Scene renders canvas
|
||||
canvas = fx_fn(r, feat, t, S)
|
||||
|
||||
# 2. Tonemap normalizes brightness
|
||||
canvas = tonemap(canvas, gamma=seg.get("gamma", 0.75))
|
||||
|
||||
# 3. Feedback adds temporal recursion
|
||||
if fb and fb_cfg:
|
||||
canvas = fb.apply(canvas, **{k: fb_cfg[k] for k in fb_cfg})
|
||||
|
||||
# 4. Shader chain adds post-processing
|
||||
canvas = chain.apply(canvas, f=feat, t=t)
|
||||
|
||||
pipe.stdin.write(canvas.tobytes())
|
||||
|
||||
pipe.stdin.close(); pipe.wait(); stderr_fh.close()
|
||||
```
|
||||
|
||||
### Building Segments from Scene Table
|
||||
|
||||
```python
|
||||
segments = []
|
||||
for i, scene in enumerate(SCENES):
|
||||
segments.append({
|
||||
"id": f"s{i:02d}_{scene['name']}",
|
||||
"name": scene["name"],
|
||||
"grid": scene["grid"],
|
||||
"fx": scene["fx"],
|
||||
"shaders": scene.get("shaders", []),
|
||||
"feedback": scene.get("feedback", None),
|
||||
"gamma": scene.get("gamma", 0.75),
|
||||
"frame_start": int(scene["start"] * FPS),
|
||||
"frame_end": int(scene["end"] * FPS),
|
||||
})
|
||||
```
|
||||
|
||||
### Parallel Rendering
|
||||
|
||||
Scenes are independent units dispatched to a process pool:
|
||||
|
||||
```python
|
||||
from concurrent.futures import ProcessPoolExecutor, as_completed
|
||||
|
||||
with ProcessPoolExecutor(max_workers=N_WORKERS) as pool:
|
||||
futures = {
|
||||
pool.submit(render_clip, seg, features, clip_path): seg["id"]
|
||||
for seg, clip_path in zip(segments, clip_paths)
|
||||
}
|
||||
for fut in as_completed(futures):
|
||||
try:
|
||||
fut.result()
|
||||
except Exception as e:
|
||||
log(f"ERROR {futures[fut]}: {e}")
|
||||
```
|
||||
|
||||
**Pickling constraint**: `ProcessPoolExecutor` serializes arguments via pickle. Module-level functions can be pickled; lambdas and closures cannot. All `fx_*` scene functions MUST be defined at module level, not as closures or class methods.
|
||||
|
||||
### Test-Frame Mode
|
||||
|
||||
Render a single frame at a specific timestamp to verify visuals without a full render:
|
||||
|
||||
```python
|
||||
if args.test_frame >= 0:
|
||||
fi = min(int(args.test_frame * FPS), N_FRAMES - 1)
|
||||
t = fi / FPS
|
||||
feat = {k: float(features[k][fi]) for k in features}
|
||||
scene = next(sc for sc in reversed(SCENES) if t >= sc["start"])
|
||||
r = Renderer()
|
||||
r.set_grid(scene["grid"])
|
||||
canvas = scene["fx"](r, feat, t, r.S)
|
||||
canvas = tonemap(canvas, gamma=scene.get("gamma", 0.75))
|
||||
chain = ShaderChain()
|
||||
for sn, kw in scene.get("shaders", []):
|
||||
chain.add(sn, **kw)
|
||||
canvas = chain.apply(canvas, f=feat, t=t)
|
||||
Image.fromarray(canvas).save(f"test_{args.test_frame:.1f}s.png")
|
||||
print(f"Mean brightness: {canvas.astype(float).mean():.1f}")
|
||||
```
|
||||
|
||||
CLI: `python reel.py --test-frame 10.0`
|
||||
|
||||
---
|
||||
|
||||
## Scene Design Checklist
|
||||
|
||||
For each scene:
|
||||
|
||||
1. **Choose 2-3 grid sizes** — different scales create interference
|
||||
2. **Choose different value fields** per layer — don't use the same effect on every grid
|
||||
3. **Choose different hue fields** per layer — or at minimum different hue offsets
|
||||
4. **Choose different palettes** per layer — mixing PAL_RUNE with PAL_BLOCKS looks different from PAL_RUNE with PAL_DENSE
|
||||
5. **Choose a blend mode** that matches the energy — screen for bright, difference for psychedelic, exclusion for subtle
|
||||
6. **Add conditional effects** on beat — kaleidoscope, mirror, glitch
|
||||
7. **Configure feedback** for trailing/recursive looks — or None for clean cuts
|
||||
8. **Set gamma** if using destructive shaders (solarize, posterize)
|
||||
9. **Test with --test-frame** at the scene's midpoint before full render
|
||||
1027
skills/creative/ascii-video/references/shaders.md
Normal file
1027
skills/creative/ascii-video/references/shaders.md
Normal file
File diff suppressed because it is too large
Load Diff
331
skills/creative/ascii-video/references/troubleshooting.md
Normal file
331
skills/creative/ascii-video/references/troubleshooting.md
Normal file
@@ -0,0 +1,331 @@
|
||||
# Troubleshooting Reference
|
||||
|
||||
Common bugs, gotchas, and platform-specific issues encountered during ASCII video development.
|
||||
|
||||
## NumPy Broadcasting
|
||||
|
||||
### The `broadcast_to().copy()` Trap
|
||||
|
||||
Hue field generators often return arrays that are broadcast views — they have shape `(1, cols)` or `(rows, 1)` that numpy broadcasts to `(rows, cols)`. These views are **read-only**. If any downstream code tries to modify them in-place (e.g., `h %= 1.0`), numpy raises:
|
||||
|
||||
```
|
||||
ValueError: output array is read-only
|
||||
```
|
||||
|
||||
**Fix**: Always `.copy()` after `broadcast_to()`:
|
||||
|
||||
```python
|
||||
h = np.broadcast_to(h, (g.rows, g.cols)).copy()
|
||||
```
|
||||
|
||||
This is especially important in `_render_vf()` where hue arrays flow through `hsv2rgb()`.
|
||||
|
||||
### The `+=` vs `+` Trap
|
||||
|
||||
Broadcasting also fails with in-place operators when operand shapes don't match exactly:
|
||||
|
||||
```python
|
||||
# FAILS if result is (rows,1) and operand is (rows, cols)
|
||||
val += np.sin(g.cc * 0.02 + t * 0.3) * 0.5
|
||||
|
||||
# WORKS — creates a new array
|
||||
val = val + np.sin(g.cc * 0.02 + t * 0.3) * 0.5
|
||||
```
|
||||
|
||||
The `vf_plasma()` function had this bug. Use `+` instead of `+=` when mixing different-shaped arrays.
|
||||
|
||||
### Shape Mismatch in `hsv2rgb()`
|
||||
|
||||
`hsv2rgb(h, s, v)` requires all three arrays to have identical shapes. If `h` is `(1, cols)` and `s` is `(rows, cols)`, the function crashes or produces wrong output.
|
||||
|
||||
**Fix**: Ensure all inputs are broadcast and copied to `(rows, cols)` before calling.
|
||||
|
||||
---
|
||||
|
||||
## Blend Mode Pitfalls
|
||||
|
||||
### Overlay Crushes Dark Inputs
|
||||
|
||||
`overlay(a, b) = 2*a*b` when `a < 0.5`. Two values of 0.12 produce `2 * 0.12 * 0.12 = 0.03`. The result is darker than either input.
|
||||
|
||||
**Impact**: If both layers are dark (which ASCII art usually is), overlay produces near-black output.
|
||||
|
||||
**Fix**: Use `screen` for dark source material. Screen always brightens: `1 - (1-a)*(1-b)`.
|
||||
|
||||
### Colordodge Division by Zero
|
||||
|
||||
`colordodge(a, b) = a / (1 - b)`. When `b = 1.0` (pure white pixels), this divides by zero.
|
||||
|
||||
**Fix**: Add epsilon: `a / (1 - b + 1e-6)`. The implementation in `BLEND_MODES` should include this.
|
||||
|
||||
### Colorburn Division by Zero
|
||||
|
||||
`colorburn(a, b) = 1 - (1-a) / b`. When `b = 0` (pure black pixels), this divides by zero.
|
||||
|
||||
**Fix**: Add epsilon: `1 - (1-a) / (b + 1e-6)`.
|
||||
|
||||
### Multiply Always Darkens
|
||||
|
||||
`multiply(a, b) = a * b`. Since both operands are [0,1], the result is always <= min(a,b). Never use multiply as a feedback blend mode — the frame goes black within a few frames.
|
||||
|
||||
**Fix**: Use `screen` for feedback, or `add` with low opacity.
|
||||
|
||||
---
|
||||
|
||||
## Multiprocessing
|
||||
|
||||
### Pickling Constraints
|
||||
|
||||
`ProcessPoolExecutor` serializes function arguments via pickle. This constrains what you can pass to workers:
|
||||
|
||||
| Can Pickle | Cannot Pickle |
|
||||
|-----------|---------------|
|
||||
| Module-level functions (`def fx_foo():`) | Lambdas (`lambda x: x + 1`) |
|
||||
| Dicts, lists, numpy arrays | Closures (functions defined inside functions) |
|
||||
| Class instances (with `__reduce__`) | Instance methods |
|
||||
| Strings, numbers | File handles, sockets |
|
||||
|
||||
**Impact**: All scene functions referenced in the SCENES table must be defined at module level with `def`. If you use a lambda or closure, you get:
|
||||
|
||||
```
|
||||
_pickle.PicklingError: Can't pickle <function <lambda> at 0x...>
|
||||
```
|
||||
|
||||
**Fix**: Define all scene functions at module top level. Lambdas used inside `_render_vf()` as val_fn/hue_fn are fine because they execute within the worker process — they're not pickled across process boundaries.
|
||||
|
||||
### macOS spawn vs Linux fork
|
||||
|
||||
On macOS, `multiprocessing` defaults to `spawn` (full serialization). On Linux, it defaults to `fork` (copy-on-write). This means:
|
||||
|
||||
- **macOS**: Feature arrays are serialized per worker (~57KB for 30s video, but scales with duration). Each worker re-imports the entire module.
|
||||
- **Linux**: Feature arrays are shared via COW. Workers inherit the parent's memory.
|
||||
|
||||
**Impact**: On macOS, module-level code (like `detect_hardware()`) runs in every worker process. If it has side effects (e.g., subprocess calls), those happen N+1 times.
|
||||
|
||||
### Per-Worker State Isolation
|
||||
|
||||
Each worker creates its own:
|
||||
- `Renderer` instance (with fresh grid cache)
|
||||
- `FeedbackBuffer` (feedback doesn't cross scene boundaries)
|
||||
- Random seed (`random.seed(hash(seg_id) + 42)`)
|
||||
|
||||
This means:
|
||||
- Particle state doesn't carry between scenes (expected)
|
||||
- Feedback trails reset at scene cuts (expected)
|
||||
- `np.random` state is NOT seeded by `random.seed()` — they use separate RNGs
|
||||
|
||||
**Fix for deterministic noise**: Use `np.random.RandomState(seed)` explicitly:
|
||||
|
||||
```python
|
||||
rng = np.random.RandomState(hash(seg_id) + 42)
|
||||
noise = rng.random((rows, cols))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Brightness Issues
|
||||
|
||||
### Dark Scenes After Tonemap
|
||||
|
||||
If a scene is still dark after tonemap, check:
|
||||
|
||||
1. **Gamma too high**: Lower gamma (0.5-0.6) for scenes with destructive post-processing
|
||||
2. **Shader destroying brightness**: Solarize, posterize, or contrast adjustments in the shader chain can undo tonemap's work. Move destructive shaders earlier in the chain, or increase gamma to compensate.
|
||||
3. **Feedback with multiply**: Multiply feedback darkens every frame. Switch to screen or add.
|
||||
4. **Overlay blend in scene**: If the scene function uses `blend_canvas(..., "overlay", ...)` with dark layers, switch to screen.
|
||||
|
||||
### Diagnostic: Test-Frame Brightness
|
||||
|
||||
```bash
|
||||
python reel.py --test-frame 10.0
|
||||
# Output: Mean brightness: 44.3, max: 255
|
||||
```
|
||||
|
||||
If mean < 20, the scene needs attention. Common fixes:
|
||||
- Lower gamma in the SCENES entry
|
||||
- Change internal blend modes from overlay/multiply to screen/add
|
||||
- Increase value field multipliers (e.g., `vf_plasma(...) * 1.5`)
|
||||
- Check that the shader chain doesn't have an aggressive solarize or threshold
|
||||
|
||||
### v1 Brightness Pattern (Deprecated)
|
||||
|
||||
The old pattern used a linear multiplier:
|
||||
|
||||
```python
|
||||
# OLD — don't use
|
||||
canvas = np.clip(canvas.astype(np.float32) * 2.0, 0, 255).astype(np.uint8)
|
||||
```
|
||||
|
||||
This fails because:
|
||||
- Dark scenes (mean 8): `8 * 2.0 = 16` — still dark
|
||||
- Bright scenes (mean 130): `130 * 2.0 = 255` — clipped, lost detail
|
||||
|
||||
Use `tonemap()` instead. See `composition.md` § Adaptive Tone Mapping.
|
||||
|
||||
---
|
||||
|
||||
## ffmpeg Issues
|
||||
|
||||
### Pipe Deadlock
|
||||
|
||||
The #1 production bug. If you use `stderr=subprocess.PIPE`:
|
||||
|
||||
```python
|
||||
# DEADLOCK — stderr buffer fills at 64KB, blocks ffmpeg, blocks your writes
|
||||
pipe = subprocess.Popen(cmd, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
```
|
||||
|
||||
**Fix**: Always redirect stderr to a file:
|
||||
|
||||
```python
|
||||
stderr_fh = open(err_path, "w")
|
||||
pipe = subprocess.Popen(cmd, stdin=subprocess.PIPE,
|
||||
stdout=subprocess.DEVNULL, stderr=stderr_fh)
|
||||
```
|
||||
|
||||
### Frame Count Mismatch
|
||||
|
||||
If the number of frames written to the pipe doesn't match what ffmpeg expects (based on `-r` and duration), the output may have:
|
||||
- Missing frames at the end
|
||||
- Incorrect duration
|
||||
- Audio-video desync
|
||||
|
||||
**Fix**: Calculate frame count explicitly: `n_frames = int(duration * FPS)`. Don't use `range(int(start*FPS), int(end*FPS))` without verifying the total matches.
|
||||
|
||||
### Concat Fails with "unsafe file name"
|
||||
|
||||
```
|
||||
[concat @ ...] Unsafe file name
|
||||
```
|
||||
|
||||
**Fix**: Always use `-safe 0`:
|
||||
```python
|
||||
["ffmpeg", "-f", "concat", "-safe", "0", "-i", concat_path, ...]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Font Issues
|
||||
|
||||
### Cell Height (macOS Pillow)
|
||||
|
||||
`textbbox()` and `getbbox()` return incorrect heights on some macOS Pillow versions. Use `getmetrics()`:
|
||||
|
||||
```python
|
||||
ascent, descent = font.getmetrics()
|
||||
cell_height = ascent + descent # correct
|
||||
# NOT: font.getbbox("M")[3] # wrong on some versions
|
||||
```
|
||||
|
||||
### Missing Unicode Glyphs
|
||||
|
||||
Not all fonts render all Unicode characters. If a palette character isn't in the font, the glyph renders as a blank or tofu box, appearing as a dark hole in the output.
|
||||
|
||||
**Fix**: Validate at init:
|
||||
|
||||
```python
|
||||
all_chars = set()
|
||||
for pal in [PAL_DEFAULT, PAL_DENSE, PAL_RUNE, ...]:
|
||||
all_chars.update(pal)
|
||||
|
||||
valid_chars = set()
|
||||
for c in all_chars:
|
||||
if c == " ":
|
||||
valid_chars.add(c)
|
||||
continue
|
||||
img = Image.new("L", (20, 20), 0)
|
||||
ImageDraw.Draw(img).text((0, 0), c, fill=255, font=font)
|
||||
if np.array(img).max() > 0:
|
||||
valid_chars.add(c)
|
||||
else:
|
||||
log(f"WARNING: '{c}' (U+{ord(c):04X}) missing from font")
|
||||
```
|
||||
|
||||
### Platform Font Paths
|
||||
|
||||
| Platform | Common Paths |
|
||||
|----------|-------------|
|
||||
| macOS | `/System/Library/Fonts/Menlo.ttc`, `/System/Library/Fonts/Monaco.ttf` |
|
||||
| Linux | `/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf` |
|
||||
| Windows | `C:\Windows\Fonts\consola.ttf` (Consolas) |
|
||||
|
||||
Always probe multiple paths and fall back gracefully. See `architecture.md` § Font Selection.
|
||||
|
||||
---
|
||||
|
||||
## Performance
|
||||
|
||||
### Slow Shaders
|
||||
|
||||
Some shaders use Python loops and are very slow at 1080p:
|
||||
|
||||
| Shader | Issue | Fix |
|
||||
|--------|-------|-----|
|
||||
| `wave_distort` | Per-row Python loop | Use vectorized fancy indexing |
|
||||
| `halftone` | Triple-nested loop | Vectorize with block reduction |
|
||||
| `matrix rain` | Per-column per-trail loop | Accumulate index arrays, bulk assign |
|
||||
|
||||
### Render Time Scaling
|
||||
|
||||
If render is taking much longer than expected:
|
||||
1. Check grid count — each extra grid adds ~100-150ms/frame for init
|
||||
2. Check particle count — cap at quality-appropriate limits
|
||||
3. Check shader count — each shader adds 2-25ms
|
||||
4. Check for accidental Python loops in effects (should be numpy only)
|
||||
|
||||
---
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
### Using `r.S` vs the `S` Parameter
|
||||
|
||||
The v2 scene protocol passes `S` (the state dict) as an explicit parameter. But `S` IS `r.S` — they're the same object. Both work:
|
||||
|
||||
```python
|
||||
def fx_scene(r, f, t, S):
|
||||
S["counter"] = S.get("counter", 0) + 1 # via parameter (preferred)
|
||||
r.S["counter"] = r.S.get("counter", 0) + 1 # via renderer (also works)
|
||||
```
|
||||
|
||||
Use the `S` parameter for clarity. The explicit parameter makes it obvious that the function has persistent state.
|
||||
|
||||
### Forgetting to Handle Empty Feature Values
|
||||
|
||||
Audio features default to 0.0 if the audio is silent. Use `.get()` with sensible defaults:
|
||||
|
||||
```python
|
||||
energy = f.get("bass", 0.3) # default to 0.3, not 0
|
||||
```
|
||||
|
||||
If you default to 0, effects go blank during silence.
|
||||
|
||||
### Writing New Files Instead of Editing Existing State
|
||||
|
||||
A common bug in particle systems: creating new arrays every frame instead of updating persistent state.
|
||||
|
||||
```python
|
||||
# WRONG — particles reset every frame
|
||||
S["px"] = []
|
||||
for _ in range(100):
|
||||
S["px"].append(random.random())
|
||||
|
||||
# RIGHT — only initialize once, update each frame
|
||||
if "px" not in S:
|
||||
S["px"] = []
|
||||
# ... emit new particles based on beats
|
||||
# ... update existing particles
|
||||
```
|
||||
|
||||
### Not Clipping Value Fields
|
||||
|
||||
Value fields should be [0, 1]. If they exceed this range, `val2char()` produces index errors:
|
||||
|
||||
```python
|
||||
# WRONG — vf_plasma() * 1.5 can exceed 1.0
|
||||
val = vf_plasma(g, f, t, S) * 1.5
|
||||
|
||||
# RIGHT — clip after scaling
|
||||
val = np.clip(vf_plasma(g, f, t, S) * 1.5, 0, 1)
|
||||
```
|
||||
|
||||
The `_render_vf()` helper clips automatically, but if you're building custom scenes, clip explicitly.
|
||||
Reference in New Issue
Block a user