/today, /yesterday, /week were queuing behind the entire periodic-check
backlog on the same rate-limited LICHESS_APP_TOKEN (observed ~2min wait on
prod for a single command). Add a two-tier priority queue to SharedTokenGate:
interactive on-demand requests jump ahead of background periodic checks,
which still drain normally when nothing interactive is waiting. Verified
locally end-to-end (isolated queue unit tests, live request with priority
correctly reaching Lichess, and a real successful accuracy fetch rendered
through StatsFormatter).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A bare mutex around games/user and user/activity wasn't enough: Lichess kept
rejecting the shared LICHESS_APP_TOKEN with 429 "Please only run 1
request(s) at a time" even when calls were strictly sequential with only
~100-300ms between one finishing and the next starting (confirmed live on
prod: near every request was getting 429'd right after the previous mutex
fix went out). Track the last release time and force a minimum 6s gap
before the next call is allowed to start.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
games/user and user/activity share one LICHESS_APP_TOKEN process-wide, and
Lichess allows only 1 concurrent request per token there (confirmed live:
429 "Please only run 1 request(s) at a time" under concurrent traffic).
The existing rate limiter only paced request starts 0.2s apart without
holding the lock through the request itself, so concurrent games/period
calls (periodic checks vs. on-demand /today etc.) could still collide and
get 429'd, silently dropping the accuracy shown in /today, /yesterday,
/week (that fetch is best-effort and swallows errors). Hold a dedicated
lock for the full request/response cycle on these two calls; puzzle
requests use per-user tokens and don't need it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Lichess started rejecting anonymous requests to games/user and
user/activity with a 404 (confirmed live: the same request with a valid
bearer token got a normal 429 instead), which our code silently read as
"user has no games" for every single tracked player, making periodic
checks report zero activity across the board. Add LICHESS_APP_TOKEN and
send it as a Bearer token on these two calls; it just needs to be any
valid token; it does not need to belong to the tracked player.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Forbidden from send_message was falling into the generic error handler,
which resets consecutive_errors to 0 right before the send attempt — so
backoff never grew past 60s and the pair retried forever, wasting request
queue capacity needed by other users' real periodic checks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A gamer's periodic check would get permanently stuck if their stored token
was revoked/expired: our stats API collapsed both "Lichess rejected the
token" (401/403, permanent) and genuine transient errors into the same 502
response, so the bot treated an invalid token exactly like a network blip —
retrying the same window forever at a capped 300s backoff, never advancing
the checkpoint (observed in prod: Dor1zz stuck for 100+ consecutive errors
over 8+ hours, admin alerts firing every 25 failures).
Preserve the distinction that already existed one layer down (lichess_client.py
already tells 401/403 apart from other failures) instead of collapsing it in
stats_service.py: add PuzzleOfPeriodResponse.auth_failed, have main.py return
401 specifically for that case, and have the bot raise a distinct
InvalidTokenError instead of returning None. On InvalidTokenError, the bot now
clears the token for that pair, notifies the user to reconnect via /addtoken,
and continues tracking games normally instead of stalling forever.
The bot's request_queue.py 4s FIFO gate wasn't protecting against Lichess's
rate limiter — that's already handled downstream in LichessWebServices/
rate_limiter.py (0.2s, shared across all callers of our stats service). The
bot-side gate only paced calls to our own local service, and since it awaited
each request to full completion before dequeuing the next, real dispatch gaps
were max(4s, previous request's duration) — with 454 tracked gamer/user pairs,
any burst (e.g. after a restart) piled into the queue and took 10-20+ minutes
to drain.
Replace it with a paced-dispatch + bounded-concurrency design: a hard 2s floor
between dispatches (still never lets 2+ requests through in that window),
decoupled from completion time, with up to 10 requests actually in flight at
once via a semaphore. Doesn't touch the real Lichess-facing rate limit at all.
Also add deterministic per-(user,gamer) checkpoint jitter: previously every
pair sharing the same period_minutes re-locked onto the same wall-clock phase
on every restart (backlog collapse snaps period_end_approx to `now` for
everyone overdue at once), recreating the pileup each time. Jitter is stable
across restarts (crc32-based, not Python's salted hash()) and capped well
under the 2h stale-backlog threshold. Small startup stagger added too, purely
cosmetic smoothing on top of the jitter fix.
The backlog-collapse fix (previous commit) correctly stops the checkpoint
from falling further behind, but it was still sending the full per-game
notification for whatever it caught up on — meaning a gamer whose checkpoint
had drifted weeks behind now dumps a multi-week, game-by-game report on the
user in one message. That's not a "periodic update" anymore, just spam.
When the collapsed window exceeds 2 hours, catch the checkpoint up silently
(still fixes the drift) but skip sending the notification for it — only
report activity that's actually recent going forward.
Persist user_data (PicklePersistence) so an in-flight /addgamer username
prompt survives a bot restart instead of being silently swallowed by
handle_username when the in-memory awaiting flag is gone.
Collapse periodic-check backlog into a single request spanning the whole
missed gap instead of replaying it one period_minutes window at a time —
with enough tracked gamers sharing one RequestQueue, per-window replay
could never catch up and the checkpoint fell further behind indefinitely.
Notification period label now reflects the actual queried span (minutes/
hours/days) instead of the configured interval, so a weeks-old catch-up
no longer gets mislabeled as "for 15 minutes".
Column-align per-mode rows (games/rating change/rating/W-L-D/accuracy)
same as the other stats commands, instead of loose free-text lines.
Strip the invisible variation selector from bullet/classical emoji so
it doesn't throw off column padding.
- /getgamers: column-aligned monospace table (username, ratings, period).
- Per-mode stats (rating/wins/losses/draws): column-aligned block, dropped
the meaningless multi-game average accuracy row.
- format_stats_response/format_period_notification now wrap the whole
message in a single markdown code block instead of alternating
plain-text headers and separately-fenced tables.
- Rating row gets an emoji prefix to match the other three rows, so
Telegram's monospace rendering keeps all values in the same column.
Set awaiting_addgamer_username as soon as the menu is shown so typing a
username directly (without tapping the button) is handled instead of
being silently ignored. Also stop reporting a valid username as "not
found" when Lichess returns a non-404 error (rate limit/timeout), and
surface an error message if the addgamer menu itself fails to send.
Add a small circle (⚪/⚫) glued right after the tracked player's name
in the per-game table to show which color they played, since the
white/black split was no longer visible once the opponent's name was
dropped. Replace the '1-0'/'0-1'/'1/2-1/2' result text with a plain
'-' separator — the leading outcome circle already conveys win/loss/draw.
Reworked the per-game row layout to be relative to the tracked player
instead of white/black: the tracked player's name+accuracy+rating are
always shown first, the opponent's rating+accuracy (no name) always
second, regardless of which color each side played in a given game -
this keeps columns aligned across rows even as the tracked player
switches sides. Dropped the rating-change-per-game column added
earlier; it pushed the line past mobile width limits.
Games-of-period responses can now include a per-game breakdown
(include_games) with Lichess post-analysis accuracy, plus which side
the tracked player was on and their rating change for that game.
- lichess_client.py requests accuracy=true from Lichess; stats_service
computes per-mode average accuracy and builds GameRow entries
(tracked_is_white, tracked_rating_diff) for blitz/rapid/classical
- /games/{username}/period gained an include_games query param
- formatters.py renders a column-aligned monospace table per game:
outcome circle (win/loss/draw relative to the tracked player),
rating change, accuracy, names/ratings, result — for today/yesterday
and periodic notifications; week keeps an aggregate accuracy line
- usernames are Markdown-escaped before formatting since messages are
now sent with parse_mode='Markdown'
Errors from Lichess (timeouts/5xx/invalid tokens) were being disguised
as "no activity" (HTTP 200, games_count/puzzles_in_period=0), causing
the bot to silently advance its checkpoint past real, undetected
activity. Puzzle-fetch failures weren't counted as errors at all, and
the periodic task died permanently after 5 consecutive errors with no
way to recover short of a manual restart. /setperiod also unconditionally
reset the checkpoint, dropping the window between the last check and
the command.
- API now returns success=false/502 on real errors instead of masking
them as zero activity (models.py, stats_service.py, main.py)
- Puzzle-fetch errors are now treated the same as game-fetch errors:
retry the same window instead of reporting "no puzzles"
- Notification delivery failures no longer silently advance the
checkpoint
- Replaced the hard 5-error kill switch with capped backoff that keeps
retrying indefinitely, plus an admin-bot notification if a player's
monitoring has been failing for a prolonged period (~2h+)
- /setperiod only clears the checkpoint when disabling notifications,
preserving continuity when a period is just changed
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Unused component that also carried several security issues (default
admin credentials, hardcoded Flask secret key, debug mode, plaintext
passwords). Cleaned up compose files, CI/CD pipeline, and docs
accordingly; also dropped the stale IS_PROD config docs in favor of
the current .env-based setup.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Remove IS_PROD flag and hardcoded tokens from config.py
- Read TELEGRAM_BOT_TOKEN and ADMINPANEL_TELEGRAM_BOT_TOKEN from env
- Add env_file: .env to lichess-bot and admin-bot in docker-compose.yml
- Add .env to .gitignore (no longer tracked)
- Add .env.example with test tokens
