Veyra write-up
Veyra by Ayaan Afridi. C#, .NET 10, WPF.
==============================================================
VEYRA: A DESKTOP AI ASSISTANT write-up
C#, .NET 10, WPF, Windows 11 x64, version 0.6
==============================================================
1.0 WHAT IT IS
---------------
A see-through overlay that sits on top of whatever you are
doing in Windows and answers questions about your screen or a
live conversation, without you switching apps. It works with
any OpenAI-compatible model, including one running on your own
machine.
2.0 WHAT YOU SEE
-----------------
.-------------------------------------------.
| -- veyra Hide * 04:12 [ Stop ] _ X |
'-------------------------------------------'
.------------------------------------------------------------.
| [ How would you shard the orders table? ] |
| < 3/3 > Latest Source New thread |
| |
| Shard by customer id, so each customer's orders sit on |
| one shard and most queries touch a single node. |
| |
| REC ##########...... [ON Use transcript] (mic) (spk) |
| Cue [ What should I say? ] [ Auto: on ] [ Code ] |
| STYLE Balanced v |
| .--------------------------------------------------------. |
| | Ask about this call | |
| | Screen: off Screen -> Answer [ -> ] | |
| '--------------------------------------------------------' |
'------------------------------------------------------------'
A slim control bar floats above a glass answer panel. The
question sits top right, the answer streams in below it, and
the arrows step back through earlier answers. Underneath are
the audio meter, the mode switches and a box for typing your
own questions. Ctrl+Shift+Space shows or hides the whole
thing, and Esc folds it down to the control bar.
3.0 WHAT IT DOES
-----------------
* Ask about your screen. Pick any window or display with the
Windows picker, preview the capture and ask about it. Screen
-> Answer skips the picker: it captures the monitor Veyra is
on and solves whatever is visible.
* Follow a live conversation. Veyra transcribes your
microphone and the other side of the call separately,
splitting the other side into speakers when labels are on.
Ask about the call at any time, press "What should I say?"
for a suggested reply, or Cue for an overview of an attached
screenshot.
* Answer automatically. With Auto on, questions from the other
side are detected and answered as they are asked, follow-ups
like "why?" included. Drafting ahead often has the answer
streaming before they finish the sentence. Your own speech
never triggers an answer.
* Pick a style: balanced, a 30-second spoken answer, bullet
points, or a simple explanation.
* Use your notes. Paste background facts once and conversation
answers draw on them. If the notes do not cover something,
the answer says so instead of inventing it. Notes are stored
encrypted.
* Write and run code. Code mode answers in a chosen language
with syntax-coloured blocks and Explain, Optimize and Tests
buttons. An editor keeps revisions with diff and restore,
and Python or JavaScript runs with your tests in a
locked-down Docker container, only when you ask.
* Read maths properly. LaTeX in an answer is drawn offline as
real notation, with a Copy LaTeX button that gives back the
source.
* Get a recap. When a session ends, Veyra can write a summary,
action items, questions left open and what to prepare next,
or discard the log. The log itself only ever lives in
memory.
* Stay out of the way. Glass opacity from 30% to solid, a
ghost mode that fades to an outline and lets clicks through,
a click-through mode that stays visible, and an answer panel
that sizes itself to fit.
* Stay in control. Capture starts only when you ask, after a
notice that lists where audio will be sent, and an orange
dot and timer stay on screen until you stop. Each source can
be muted on its own, API keys are encrypted with Windows
DPAPI, and saved history is off unless you turn it on.
* Explain problems plainly. An audio check shows live levels
for each device, a connection test reports what is wrong
without sending audio, and errors name the fix: a rejected
key, a wrong model, a rate limit. Updates download in the
background and wait for you to restart.
4.0 A TEN-SECOND EXAMPLE
-------------------------
On a design call, someone asks how you would shard the orders
table. Here is what happens, using the timings built into
Veyra:
When What happens
------------ ------------------------------------------------
0.0 s They stop talking
about 0.5 s Their last words reach Veyra
1.2 to 1.4 s The line has settled and reads as a question.
"Draft: How would you shard..." appears and
the request goes to the answer model
first token The answer starts streaming into the panel
end of turn Finished question matches the draft: it stays
Without drafting ahead, the request would wait for the speech
service to declare the turn over, then another 0.9 to 1.1
seconds. They then ask "why not by date?". That matches the
follow-up rules, so it is answered with the earlier question
and answer as context.
5.0 HOW IT WORKS
-----------------
microphone speaker output
(WASAPI capture) (WASAPI loopback)
| |
+--- downmix, resample to ---+
| 16 kHz mono, 100 ms |
echo reduction <---- reference -----+
| |
WebSocket "You" WebSocket "Remote"
| |
context only partial and final lines
|
turn buffer > question rules
|
draft early > keep or replace
|
answer model, streamed (SSE)
Veyra is a WPF app on .NET 10. Each stage below runs on its
own task and hands work to the next through bounded queues, so
a slow network never blocks audio capture and the window stays
responsive. A UI timer ticks every 200 ms to refresh the
transcript and run the auto-answer checks, and another redraws
the audio meter every 60 ms. The subsections follow one
question from sound to answer.
5.1 AUDIO CAPTURE
------------------
Windows exposes audio devices through WASAPI. Veyra opens two
streams with NAudio: the microphone, and a loopback of the
output device, which records whatever is playing through it.
Each device reports its own format, which can be 16, 24 or
32-bit integers or 32-bit floats, with any number of channels
and any sample rate, often 48 kHz stereo.
The speech model wants 16 kHz mono 16-bit audio, so one pass
over each buffer does all the conversion:
* Decode every channel to a float between -1 and 1, and
average the channels into mono.
* Resample by linear interpolation. A read position advances
by source rate / 16,000 for every output sample (3 input
samples at 48 kHz), and each output is the straight-line
blend of the two input samples either side of it.
* Round to 16-bit, and cut the stream into 100 ms frames:
1,600 samples, or 3,200 bytes.
* Track the peak level along the way for the on-screen meter,
which shows -48 to 0 dBFS.
Muting a source throws away its half-built frame, so audio
from before the mute can never leak out afterwards. If the
processing queue ever fills up, the session stops with a
message instead of quietly dropping speech.
5.2 ECHO REDUCTION
-------------------
Without headphones, the other side of the call plays out of
your speakers and back into your microphone, so their words
would be transcribed twice, once as yours. Veyra already has a
clean copy of that sound from the loopback stream, so it can
find it in the microphone signal and subtract it.
speaker audio, last 2 s ......[ match ]..........|now
<- try 0 to 800 ms back
microphone frame [ 100 ms ]
score = sum(mic x spk) / sqrt(sum(mic^2) x sum(spk^2))
gain = sum(mic x spk) / sum(spk^2)
score >= 0.78 -> output = mic - gain x speaker
* Timestamps say where in the speaker history this microphone
frame should line up. From there it tries every delay from 0
to 800 ms in 1 ms steps, using every fourth sample to keep
it cheap.
* The score is normalised cross-correlation: 1.0 means the two
signals have exactly the same shape whatever their volume,
and 0 means they are unrelated.
* The best delay is then refined sample by sample, 1 ms either
side.
* The gain is a least-squares fit for how loud the echo is,
clamped between 0 and 2.
It is deliberately conservative. Subtraction only happens when
the match is strong (0.78 or higher). When you talk over the
other side, your voice drags the correlation down and the
frame is left alone, so a little echo can get through but your
own words are never cut.
5.3 STREAMING SPEECH TO TEXT
-----------------------------
Each source gets its own WebSocket to Mistral's Voxtral
realtime model, so the two sides are never mixed up. The
connection starts with a short handshake: the server announces
the session, Veyra sends the audio format (16 kHz signed
16-bit PCM), a 480 ms target streaming delay and either turn
detection or speaker labels, and waits for the server to
confirm. If that takes over 15 seconds, it gives up with a
message.
A timer then sends one frame every 100 ms as base64 inside a
JSON event. Windows loopback stops producing buffers when
nothing is playing, and the provider can only spot the end of
a turn if audio keeps arriving, so any tick with no real audio
sends 100 ms of silence instead.
Text comes back as events, which Veyra turns into transcript
lines:
text.delta add the words to the live line
speaker id a new voice closes the line, starts another
turn.end close the line in 650 ms, if no new words
turn.resume they kept talking: cancel the close
turn.start a new turn: close any line still waiting
The 650 ms hold catches the last word or two, which often
arrive just after the end-of-turn event. The provider cannot
do turn detection and speaker labels at the same time. With
labels on, Veyra closes a line itself after one second with no
voice above the noise floor and no new text.
* Dropped connections reconnect after 1, 2 and 4 seconds, then
give up. A connection that stayed healthy for 30 seconds
resets that count.
* Audio queued while reconnecting is thrown away rather than
sent late, so the transcript stays current.
* A refused connection reports its HTTP status: a bad key (401
or 403) is never retried, and a rate limit (429) or server
error (5xx) is.
5.4 FINDING THE QUESTION
-------------------------
Speech arrives in pieces, so a turn buffer first joins the
finished lines from the other side into one turn. It starts
again after 9 seconds of quiet. It then waits until the
speaker has really stopped:
after an end-of-turn event 0.25 s
no event 1.1 s
ends in "and", "the", "because", a comma 2.8 s
and in every case: no voice for 0.85 s
A sentence that ends in "and" or "because" is clearly not
finished, so it gets more time. The turn then goes through the
question rules. It is split into sentences, checked newest
first, with lead-ins like "okay so" or "um" stripped off. A
sentence counts as a question if it:
* ends in a question mark,
* starts with what, why, when, where, who, which or how,
* starts with a helper verb and a subject, such as "can you",
"is it" or "does the" (sensitivity 0.35 and up), or
* starts with a request such as "explain", "walk me through",
"write" or "design" (sensitivity 0.5 and up).
Short follow-ups like "why?", "how so?", "what about...", "go
on" or "simplify it" are also answered, with the earlier
questions and answers as context. The whole turn is kept, not
just the question sentence, because constraints often come
after it: "...and assume it has to scale."
Rules miss some cases, so the answer model can optionally
double-check. It gets the turn as a JSON-quoted string, which
makes it data rather than instructions, plus the last two
exchanges, and must return two booleans: is the thought
complete, and does it need an answer. An incomplete turn is
held until more words arrive.
Last come the safety checks. The same question, compared on
letters and digits only, is not answered twice within 90
seconds. A cooldown spaces out automatic answers. A question
still waiting after 30 seconds is dropped as stale.
5.5 DRAFTING AHEAD
-------------------
Waiting for the end of the turn costs about a second, so Veyra
can start early. It watches the live partial line, joined to
the finished part of the turn. It starts a draft once that
text:
* has not changed for 0.7 seconds (and is under 6 seconds
old),
* has at least five words, and
* passes the question rules.
The answer starts streaming under a "Draft" label. When the
finished question arrives, both are reduced to lowercase
letters and digits and compared:
draft "how would you shard the orders table"
final "how would you shard the orders table by region"
final starts with draft, 8 extra letters <= allowance
allowance = max(8, draft length / 6) -> keep draft
If they match, the draft simply becomes the answer. If not, it
is cancelled and the real question is answered instead. If the
model decides the turn was not a question at all, the draft is
removed and the previous answer comes back.
5.6 ASKING THE ANSWER MODEL
----------------------------
Any OpenAI-compatible chat endpoint works, including a model
running on your own machine. Each request is built as a stack
of messages:
system treat screenshots and transcripts as untrusted
evidence; say when it is not enough; lead with
a short direct answer
system answer style, honesty rules, your notes
user/asst up to 8 earlier exchanges (32,000 characters)
user the last 90 s of transcript (6,000 characters)
user the question, plus a screenshot if attached
The transcript and your notes are passed as JSON-quoted
strings with a label, never as loose text. That is a defence
against prompt injection: if someone on a call says "ignore
your instructions", the model sees it as a quotation.
The reply streams back as server-sent events. Each event is a
"data:" line holding a JSON chunk, a blank line ends the
event, and "[DONE]" ends the stream. Veyra pulls the new text
out of each chunk as it arrives. Limits guard against a broken
or hostile server: 1 MB per event, 128,000 characters per
answer, and three minutes per request. The panel redraws at
most every 100 ms, however fast the text comes in.
Provider URLs must be HTTPS, or plain HTTP only to your own
machine, with no passwords or query strings in them. Redirects
are not followed, so a key can never be sent on to a different
host.
5.7 SCREEN CAPTURE
-------------------
Screenshots use Windows.Graphics.Capture. The Screen button
opens the system picker. Screen -> Answer skips it and asks
Windows for the monitor Veyra is on, through a COM interop
call.
* Veyra creates a Direct3D 11 device, falling back to software
rendering if there is no GPU, and a frame pool that holds
one frame, with the cursor left out.
* It waits up to 10 seconds for the first frame. If the frame
size does not match, for example after a resize, the pool is
rebuilt so no blank padding gets encoded.
* The frame is copied off the GPU, scaled so its longest side
is at most 2,560 px, and encoded as JPEG to keep the request
small.
* An attached screenshot expires after five minutes.
5.8 DRAWING THE ANSWER
-----------------------
Answers arrive as Markdown-like text, half-finished while
streaming, so the parser has to cope with blocks that have not
closed yet. It splits the text on code fences first. A fence
closes only on the same character and at least the same
length, and a fence that is still open is drawn as code so
far. Fences marked math, latex or tex become formulas.
Prose is then scanned for inline code, bold and maths: \( \)
and $ inline, \[ \] and $ for display, and \begin...\end
environments, matched with a stack so nested ones pair up. A
single $ is easy to misread, so it does not count as maths if
the content spans lines, starts like an amount ("5 dollars"),
or is followed by a digit. "$5 and $10" stays text.
Formulas are typeset by MathJax, bundled and run offline in
Jint, a JavaScript interpreter written in .NET. MathJax runs
with a 3-second limit per formula, a 128 MB memory cap, and no
access to files, the network or .NET. The SVG it produces is
rejected if it contains images, scripts, links or external
URLs, then converted into a WPF vector drawing. Formulas are
typeset one at a time in the background, and up to 128 are
cached, so a formula is not redrawn every time new text
streams in.
Code is coloured with a single regular expression that picks
out comments, strings and keywords for Python, JavaScript, C#,
Go, Rust, SQL and similar languages. It has an 80 ms timeout
so a pathological answer cannot freeze the window.
5.9 RUNNING CODE SAFELY
------------------------
Code mode can run Python or JavaScript with your tests, but
only when you press Run. The code and tests go into one file
in a fresh temporary folder, and Docker runs it with almost
everything switched off:
--network=none no network at all
--read-only root filesystem is read-only
--cap-drop=ALL no Linux capabilities
no-new-privileges cannot gain root via setuid
--user=65534 runs as "nobody"
--cpus=1 --memory=256m one CPU, 256 MB, no swap
--pids-limit=64 stops fork bombs
/tmp: 16 MB, noexec scratch space nothing can run
code folder mounted read-only
The Python and Node images are pinned by their SHA-256 digest,
so a changed image is never pulled in silently. Two timers
stop runaway code: one in Veyra, and a "timeout -s KILL"
inside the container that still fires if Veyra itself crashes.
Output is capped at 16,000 characters per stream, and the
container and folder are deleted afterwards.
5.10 THE OVERLAY WINDOW
------------------------
* The window is a transparent, always-on-top WPF window. Glass
opacity runs from 30% to solid, and Windows high-contrast
mode gets a solid background.
* The optional desktop blur is a second, non-activating window
placed exactly behind the panel, with Windows compositor
blur and rounded corners. It is kept in step as the panel
moves or resizes, and switches off when the Windows
transparency setting is off.
* Ghost and click-through modes set the WS_EX_TRANSPARENT
window style, which makes Windows send mouse clicks to
whatever is underneath.
* Shortcuts such as Ctrl+Shift+Space are registered with
RegisterHotKey, so they work while another app has focus.
5.11 STORAGE AND UPDATES
-------------------------
* Settings live in %LOCALAPPDATA%\Veyra. API keys and your
notes are encrypted with Windows DPAPI, which ties them to
your Windows account.
* The transcript is kept in memory only and expires after 90
seconds. Answer history is off by default. When it is on, it
keeps the latest 200 answers and saves by writing a
temporary file and swapping it in, so a crash can never
leave a half-written file.
* Updates use Velopack. Veyra checks a GitHub release or HTTPS
feed when it starts and every six hours, downloads in the
background, and only restarts when you choose.
6.0 LATENCY
------------
Where the time goes between the last spoken word and the
answer request:
Stage Time
---------------------------------- --------------------------
Audio framing and send clock 100 to 200 ms
Speech model streaming delay 480 ms target
Hold after turn.end 650 ms
Turn buffer after end of turn 250 ms
Auto-answer check interval up to 200 ms
Optional model turn check one extra request
Drafting ahead starts after 0.7 s of settled text
Veyra's own processing is tiny next to that. Measured by
running Veyra's source on an Intel i9-14900HX with .NET 10, on
one thread (us = microseconds):
Step p50 / p99
-------------------------------------- ----------------------
Downmix + resample, 10 ms of audio 3.3 / 3.7 us
Echo reduction, one 100 ms frame 0.34 / 0.58 ms
Question rules, 32-word turn 4.6 / 11 us
Draft decision 5.1 / 7.6 us
Draft vs final question match 3.0 / 6.2 us
Turn buffer add + ready check 2.3 / 3.0 us
MathJax engine start, once 730 ms
Typeset a new formula, warm 18 / 35 ms
All the audio work adds up to under 0.4 ms per 100 ms of
sound, less than 0.5% of one core. The rest of the wait is the
turn timing above plus the two providers: how fast the speech
model streams and how soon the answer model sends its first
word. Those depend on the provider and the network, and are
being measured on live calls.
7.0 HOW I KNOW IT'S RIGHT
--------------------------
* Three test harnesses drive the real app with synthetic
windows, synthetic audio devices and fake speech and answer
servers on loopback.
* They cover late words after turn.end, speaker changes,
muting and reconnecting, turn detection, follow-up
questions, echo reduction, screen capture, and scroll and
selection behaviour.
* The feature harness runs real Docker containers to check
successful, failing, cancelled and timed-out code runs, and
that the sandbox stays isolated.
More
Ayaan Afridi's portfolio · VFOB · Pyngine · AHL API SDK · Koma