Channel features: subscribers, new-videos badges, activity-driven sync

- Store subscriber count from YouTube statistics and show it on channel page
- Sync 50 videos per channel with playlistItems pagination support
- Show per-channel and per-category new-videos counters (2-day window)
- Replace hourly videos sync with activity trigger (2h idle) and
  incremental backfill (hard cap 200 per channel)
- Clicking the sidebar new-videos count filters the category feed to
  recent videos only (new_only)
- Update agent-team docs: deploy after green checks
This commit is contained in:
vrubelroman 2026-09-17 21:07:33 +00:00
parent fde9a439df
commit e10df8dcbd
27 changed files with 1420 additions and 107 deletions

View file

@ -26,7 +26,16 @@ METUBE_CONTAINER_DOWNLOAD_DIR=/downloads
METUBE_REQUEST_TIMEOUT_SECONDS=30
SUBSCRIPTIONS_SYNC_INTERVAL_HOURS=6
VIDEOS_SYNC_INTERVAL_MINUTES=60
VIDEOS_PER_CHANNEL_SYNC=10
# Необязательные настройки
# Авто-синк видео запускается по активности (запросы к API) после простоя
# в VIDEOS_SYNC_IDLE_HOURS с последнего завершённого синка.
VIDEOS_SYNC_IDLE_HOURS=2
# Максимум элементов плейлиста, рассматриваемых на канал за один синк.
VIDEOS_BACKFILL_CAP=200
# Остановить пагинацию плейлиста канала после стольких уже известных
# видео подряд (дальше — уже синхронизированная история).
VIDEOS_KNOWN_STOP_THRESHOLD=50
NEW_VIDEOS_WINDOW_DAYS=2
LOG_LEVEL=INFO

View file

@ -15,11 +15,12 @@ You are the only agent who normally speaks with the user. Coordinate the opencod
4. Submit the task to Coder with the Task tool (subagent_type `coder`). Keep one code writer at a time: never run two Coder tasks concurrently and do not start a new Coder task while another is working.
5. Read Coder's report (ends with `CODER_DONE`) and inspect the diff yourself. Then run Reviewer and Tester on the result. Reviewer must not edit; Tester must not fix. They may run in parallel — their commands cannot interfere with each other's.
6. Collect Critical, Major, and Minor findings with evidence. Send actionable findings back to Coder as a follow-up task. Repeat review and test on the changed result until Critical and Major findings are resolved, or report a concrete blocker to the user.
7. Give the user a concise final report: changes, affected files, review findings resolved or remaining, tests run and results, known limits, and worktree/branch. Do not claim visual or integration checks that were not performed.
6.5. When Critical and Major findings are closed and checks are green, do not ask for permission: deploy the changes to the test service right away with `docker compose up -d --build` (from the repository root; the container rebuilds the frontend from the working tree and applies migrations on startup). Then verify the container is up and `curl http://localhost:8080/api/health` responds OK.
7. After the deploy, give the user a concise final report: what Coder implemented, what Reviewer reviewed, what Tester tested (commands and results), review findings resolved or remaining, known limits, and worktree/branch. Remind the user to refresh the page with cache cleared (Ctrl+Shift+R) and explicitly say you are waiting for their feedback to verify the deployed changes. Do not claim visual or integration checks that were not performed.
## Working rules
- Do not merge, deploy, publish, or commit unless the user requested it or existing authorization covers it.
- Do not merge, publish, or commit unless the user requested it or existing authorization covers it. Deploying to the test service is authorized by this workflow (step 6.5); deploying elsewhere still requires the user's go-ahead.
- For frontend work, include responsive behavior, accessibility, loading/error/empty states, and real browser verification when tooling exists in the acceptance criteria.
- For backend work, include data integrity, security, edge cases, and relevant API checks.

View file

@ -9,7 +9,8 @@
- декомпозирует задачу и отправляет её Coder'у через Task tool;
- после Coder'а запускает Reviewer (только чтение) и Tester (проверка без правок) параллельно;
- возвращает Coder'у actionable findings и повторяет цикл, пока Critical/Major не закрыты;
- отдаёт финальный отчёт с изменениями, проверками и оставшимися рисками.
- когда Critical/Major закрыты и проверки зелёные — без вопроса деплоит на тестовый сервис (`docker compose up -d --build`, затем `curl http://localhost:8080/api/health`);
- отдаёт финальный отчёт уже после деплоя: что написано, просмотрено и протестировано, оставшиеся риски и просьба проверить в браузере с очисткой кэша (Ctrl+Shift+R).
Вручную писать субагентам не нужно — их вызывает только Orchestrator.

View file

@ -1,14 +1,19 @@
from datetime import datetime, timedelta, timezone
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, Field
from sqlalchemy import func
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app.config import settings
from app.core.auth_dependency import require_session
from app.core.slugify import unique_slugify
from app.db import get_db
from app.models.category import Category
from app.models.channel import Channel
from app.models.channel_category import channel_categories
from app.models.video import Video
router = APIRouter(dependencies=[Depends(require_session)])
@ -40,13 +45,15 @@ def _name_taken(db: Session, name: str, exclude_id: int | None = None) -> bool:
return any(row[0].casefold() == target for row in query.all())
def _serialize(db: Session, category: Category, counts: dict[int, int]) -> dict:
def _serialize(db: Session, category: Category, counts: dict[int, int], new_videos: dict[int, int] | None = None) -> dict:
new_videos = new_videos or {}
return {
"id": category.id,
"name": category.name,
"slug": category.slug,
"sort_order": category.sort_order,
"channel_count": counts.get(category.id, 0),
"new_videos_count": new_videos.get(category.id, 0),
}
@ -59,7 +66,17 @@ def list_categories(db: Session = Depends(get_db)) -> list[dict]:
.all()
)
counts = dict(count_rows)
return [_serialize(db, c, counts) for c in categories]
since = datetime.now(timezone.utc) - timedelta(days=settings.new_videos_window_days)
new_videos_rows = (
db.query(channel_categories.c.category_id, func.count(Video.id))
.join(Channel, Channel.id == channel_categories.c.channel_id)
.join(Video, Video.channel_id == Channel.id)
.filter(Channel.subscribed.is_(True), Video.published_at >= since)
.group_by(channel_categories.c.category_id)
.all()
)
new_videos = dict(new_videos_rows)
return [_serialize(db, c, counts, new_videos) for c in categories]
@router.post("/categories", status_code=201)

View file

@ -1,15 +1,18 @@
import logging
from datetime import datetime, timedelta, timezone
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from sqlalchemy import select
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app.config import settings
from app.core.auth_dependency import require_session
from app.db import get_db
from app.models.category import Category
from app.models.channel import Channel
from app.models.channel_category import channel_categories
from app.models.video import Video
from app.services import sync
from app.services.google_oauth import OAuthNotConnected
from app.services.youtube_client import YouTubeAPIError, YouTubeInsufficientScope, YouTubeQuotaExceeded
@ -37,7 +40,22 @@ def _category_ids_by_channel(db: Session, channel_ids: list[int]) -> dict[int, l
return result
def _serialize(channel: Channel, category_ids: list[int]) -> dict:
def _new_videos_counts(db: Session, channel_ids: list[int]) -> dict[int, int]:
"""Videos published within the new-videos window, per channel, in one
aggregate query."""
if not channel_ids:
return {}
since = datetime.now(timezone.utc) - timedelta(days=settings.new_videos_window_days)
rows = (
db.query(Video.channel_id, func.count(Video.id))
.filter(Video.channel_id.in_(channel_ids), Video.published_at >= since)
.group_by(Video.channel_id)
.all()
)
return {channel_id: count for channel_id, count in rows}
def _serialize(channel: Channel, category_ids: list[int], new_videos_count: int = 0) -> dict:
return {
"id": channel.id,
"youtube_channel_id": channel.youtube_channel_id,
@ -45,9 +63,11 @@ def _serialize(channel: Channel, category_ids: list[int]) -> dict:
"description": channel.description,
"thumbnail_url": channel.thumbnail_url,
"uploads_playlist_id": channel.uploads_playlist_id,
"subscriber_count": channel.subscriber_count,
"subscribed": channel.subscribed,
"last_synced_at": channel.last_synced_at,
"category_ids": category_ids,
"new_videos_count": new_videos_count,
}
@ -75,8 +95,10 @@ def list_channels(
query = query.filter(Channel.id.in_(channel_ids_in_category))
channels = query.order_by(Channel.title.asc()).all()
category_map = _category_ids_by_channel(db, [c.id for c in channels])
return [_serialize(c, category_map.get(c.id, [])) for c in channels]
channel_ids = [c.id for c in channels]
category_map = _category_ids_by_channel(db, channel_ids)
new_videos_map = _new_videos_counts(db, channel_ids)
return [_serialize(c, category_map.get(c.id, []), new_videos_map.get(c.id, 0)) for c in channels]
@router.get("/channels/{channel_id}")
@ -85,7 +107,8 @@ def get_channel(channel_id: int, db: Session = Depends(get_db)) -> dict:
if channel is None:
raise HTTPException(status_code=404, detail="Channel not found")
category_map = _category_ids_by_channel(db, [channel_id])
return _serialize(channel, category_map.get(channel_id, []))
new_videos_map = _new_videos_counts(db, [channel_id])
return _serialize(channel, category_map.get(channel_id, []), new_videos_map.get(channel_id, 0))
@router.put("/channels/{channel_id}/categories")
@ -110,7 +133,8 @@ def set_channel_categories(channel_id: int, payload: ChannelCategoriesUpdate, db
)
db.commit()
return _serialize(channel, sorted(unique_ids))
new_videos_map = _new_videos_counts(db, [channel_id])
return _serialize(channel, sorted(unique_ids), new_videos_map.get(channel_id, 0))
@router.post("/channels/{channel_id}/unsubscribe")
@ -139,4 +163,5 @@ def unsubscribe_channel(channel_id: int, db: Session = Depends(get_db)) -> dict:
raise HTTPException(status_code=502, detail="YouTube is unavailable")
category_map = _category_ids_by_channel(db, [channel_id])
return _serialize(channel, category_map.get(channel_id, []))
new_videos_map = _new_videos_counts(db, [channel_id])
return _serialize(channel, category_map.get(channel_id, []), new_videos_map.get(channel_id, 0))

View file

@ -1,10 +1,11 @@
import base64
from datetime import datetime, timezone
from datetime import datetime, timedelta, timezone
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy import func, or_, select
from sqlalchemy.orm import Session
from app.config import settings
from app.core.auth_dependency import require_session
from app.db import get_db
from app.models.channel import Channel
@ -80,6 +81,7 @@ def get_feed(
channel_id: int | None = None,
downloaded: bool = False,
search: str | None = Query(None, max_length=200),
new_only: bool = False,
limit: int = Query(DEFAULT_LIMIT, ge=1, le=MAX_LIMIT),
cursor: str | None = None,
db: Session = Depends(get_db),
@ -97,6 +99,15 @@ def get_feed(
)
query = query.filter(Video.channel_id.in_(channel_ids_in_category))
if new_only:
since = datetime.now(timezone.utc) - timedelta(days=settings.new_videos_window_days)
query = query.filter(Video.published_at >= since)
# Align with the sidebar badge count (categories.py), which only
# counts subscribed channels: after unsubscribing, a channel's
# fresh videos must disappear from "new only" too.
subscribed_channel_ids = select(Channel.id).where(Channel.subscribed.is_(True))
query = query.filter(Video.channel_id.in_(subscribed_channel_ids))
if downloaded:
query = query.filter(Video.id.in_(_completed_video_ids()))

View file

@ -32,8 +32,19 @@ class Settings(BaseSettings):
metube_request_timeout_seconds: int = 30
subscriptions_sync_interval_hours: int = 6
videos_sync_interval_minutes: int = 60
videos_per_channel_sync: int = 10
# Videos sync is no longer scheduled: it is triggered by user activity
# (any authenticated request) after this much idle time since the last
# completed sync.
videos_sync_idle_hours: int = 2
# Hard history-depth limit per channel per videos sync: at most the
# newest N playlist items are considered; anything older is never
# backfilled by design (full channel history is out of scope).
videos_backfill_cap: int = 200
# Stop playlistItems pagination for a channel once this many already
# known video ids are seen in a row (the rest of the playlist is history
# we have already synced).
videos_known_stop_threshold: int = 50
new_videos_window_days: int = 2
log_level: str = "INFO"

View file

@ -1,6 +1,11 @@
from fastapi import HTTPException, Request
from app.services.sync_trigger import maybe_trigger_videos_sync
def require_session(request: Request) -> None:
if not request.session.get("authenticated"):
raise HTTPException(status_code=401, detail="Not authenticated")
# Every authenticated request may trigger a background videos sync when it
# is due (cheap check: lock + one app_settings row read).
maybe_trigger_videos_sync()

View file

@ -1,6 +1,6 @@
from datetime import datetime
from sqlalchemy import Boolean, DateTime, String, Text, func
from sqlalchemy import BigInteger, Boolean, DateTime, String, Text, func
from sqlalchemy.orm import Mapped, mapped_column
from app.db import Base
@ -16,6 +16,7 @@ class Channel(Base):
description: Mapped[str | None] = mapped_column(Text, nullable=True)
thumbnail_url: Mapped[str | None] = mapped_column(String, nullable=True)
uploads_playlist_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
subscriber_count: Mapped[int | None] = mapped_column(BigInteger, nullable=True)
subscribed: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
last_synced_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now(), nullable=False)

View file

@ -21,19 +21,9 @@ def _run_subscriptions_sync() -> None:
db.close()
def _run_videos_sync() -> None:
db = SessionLocal()
try:
sync.sync_videos(db)
except sync.SyncInProgress:
logger.info("Scheduled videos sync skipped: already running")
except Exception:
logger.exception("Scheduled videos sync failed")
finally:
db.close()
def create_scheduler() -> BackgroundScheduler:
# Videos sync is not scheduled anymore: it runs on user activity via
# services.sync_trigger.maybe_trigger_videos_sync (see require_session).
scheduler = BackgroundScheduler(timezone="UTC")
scheduler.add_job(
_run_subscriptions_sync,
@ -41,10 +31,4 @@ def create_scheduler() -> BackgroundScheduler:
hours=settings.subscriptions_sync_interval_hours,
id="subscriptions_sync",
)
scheduler.add_job(
_run_videos_sync,
"interval",
minutes=settings.videos_sync_interval_minutes,
id="videos_sync",
)
return scheduler

View file

@ -52,6 +52,10 @@ def get_videos_sync_status(db: Session) -> dict:
return _get_status(db, VIDEOS_SYNC_STATUS_KEY, _videos_lock)
def is_videos_sync_running() -> bool:
return _videos_lock.locked()
def sync_subscriptions(db: Session) -> dict:
if not _subscriptions_lock.acquire(blocking=False):
raise SyncInProgress("Subscriptions sync already in progress")
@ -108,10 +112,13 @@ def sync_subscriptions(db: Session) -> dict:
subscribed_ids = [c.youtube_channel_id for c in existing.values() if c.subscribed]
try:
uploads = youtube_client.fetch_uploads_playlists(credentials, subscribed_ids)
for channel_id, uploads_playlist_id in uploads.items():
for channel_id, data in uploads.items():
channel = existing.get(channel_id)
if channel is not None:
channel.uploads_playlist_id = uploads_playlist_id
if data["uploads_playlist_id"] is not None:
channel.uploads_playlist_id = data["uploads_playlist_id"]
if data["subscriber_count"] is not None:
channel.subscriber_count = data["subscriber_count"]
db.commit()
except Exception:
logger.exception("Failed to fetch uploads playlists during subscriptions sync")
@ -166,21 +173,38 @@ def sync_videos(db: Session) -> dict:
)
channel_by_youtube_id = {c.youtube_channel_id: c for c in channels}
# One query for all known video ids (grouped by channel) so there is
# no N+1 at the DB level; the YouTube API is still queried per channel.
known_ids_by_channel: dict[int, set[str]] = {}
for youtube_video_id, channel_id in db.query(Video.youtube_video_id, Video.channel_id).all():
known_ids_by_channel.setdefault(channel_id, set()).add(youtube_video_id)
candidate_video_ids: set[str] = set()
for channel in channels:
try:
video_ids = youtube_client.fetch_playlist_video_ids(
credentials, channel.uploads_playlist_id, settings.videos_per_channel_sync
# videos_backfill_cap is a hard history-depth limit per
# channel: only the newest N playlist items are ever
# considered; anything older is not backfilled by design.
# The early stop on consecutive known ids saves playlistItems
# pages on repeated syncs inside that window.
new_ids = youtube_client.fetch_playlist_video_ids_incremental(
credentials,
channel.uploads_playlist_id,
known_ids_by_channel.get(channel.id, set()),
settings.videos_backfill_cap,
settings.videos_known_stop_threshold,
)
candidate_video_ids.update(video_ids)
candidate_video_ids.update(new_ids)
except Exception:
logger.exception("Failed to fetch playlist items for channel %s", channel.youtube_channel_id)
# Details are fetched only for ids that are not in the DB yet (the
# incremental fetch above already filters out known ids), so every
# returned item is a new video to insert. Existing videos' metadata is
# deliberately not refreshed by the videos sync.
details = youtube_client.fetch_videos_details(credentials, list(candidate_video_ids))
existing = {v.youtube_video_id: v for v in db.query(Video).all()}
added = 0
updated = 0
skipped = 0
for item in details:
@ -197,9 +221,8 @@ def sync_videos(db: Session) -> dict:
duration_seconds = parse_iso8601_duration(item["duration_iso8601"])
youtube_url = settings.youtube_watch_url_template.format(video_id=item["youtube_video_id"])
video = existing.get(item["youtube_video_id"])
if video is None:
video = Video(
db.add(
Video(
youtube_video_id=item["youtube_video_id"],
channel_id=channel.id,
title=item["title"],
@ -209,18 +232,8 @@ def sync_videos(db: Session) -> dict:
duration_seconds=duration_seconds,
youtube_url=youtube_url,
)
db.add(video)
existing[item["youtube_video_id"]] = video
)
added += 1
else:
video.channel_id = channel.id
video.title = item["title"]
video.description = item["description"]
video.thumbnail_url = item["thumbnail_url"]
video.published_at = published_at
video.duration_seconds = duration_seconds
video.youtube_url = youtube_url
updated += 1
db.commit()
@ -230,12 +243,14 @@ def sync_videos(db: Session) -> dict:
"finished_at": _now_iso(),
"error": None,
"videos_added": added,
"videos_updated": updated,
# Kept for API compatibility: always 0, because the videos sync
# never refreshes metadata of existing videos.
"videos_updated": 0,
"videos_skipped": skipped,
"channels_checked": len(channels),
}
_save_status(db, VIDEOS_SYNC_STATUS_KEY, result)
logger.info("Videos sync completed: added=%d updated=%d skipped=%d", added, updated, skipped)
logger.info("Videos sync completed: added=%d skipped=%d", added, skipped)
return result
except Exception as exc:

View file

@ -0,0 +1,91 @@
"""Activity-triggered videos sync.
The hourly scheduled videos sync is gone; instead, every authenticated request
(require_session) calls maybe_trigger_videos_sync(). The check must stay cheap:
if the sync is not running and enough idle time has passed since the last
completed sync, a background thread is started which calls sync.sync_videos
with its own DB session.
"""
import json
import logging
import threading
from datetime import datetime, timedelta, timezone
from app.config import settings
from app.db import SessionLocal
from app.services import sync
from app.services.state import get_setting
logger = logging.getLogger(__name__)
def is_videos_sync_due(finished_at_iso: str | None, now: datetime, idle_hours: int) -> bool:
"""Pure predicate: should an automatic videos sync start now?
True when the last completed sync's finished_at is missing/unparseable or
older than `idle_hours`."""
if not finished_at_iso:
return True
if not isinstance(finished_at_iso, str):
return True
try:
finished_at = datetime.fromisoformat(finished_at_iso.replace("Z", "+00:00"))
except (ValueError, TypeError):
return True
if finished_at.tzinfo is None:
finished_at = finished_at.replace(tzinfo=timezone.utc)
if now.tzinfo is None:
now = now.replace(tzinfo=timezone.utc)
return now - finished_at >= timedelta(hours=idle_hours)
def _extract_finished_at(raw: str | None) -> str | None:
if not raw:
return None
try:
payload = json.loads(raw)
except (ValueError, TypeError):
return None
if not isinstance(payload, dict):
return None
return payload.get("finished_at")
def _run_videos_sync() -> None:
db = SessionLocal()
try:
sync.sync_videos(db)
except sync.SyncInProgress:
# Lost the race against a manual sync or another trigger: fine, the
# other run will record the status.
logger.info("Triggered videos sync skipped: already running")
except Exception:
logger.exception("Triggered videos sync failed")
finally:
db.close()
def maybe_trigger_videos_sync() -> None:
"""Cheap per-request check; never raises. Starts a background videos sync
when due, or does nothing when a sync is already running."""
try:
if sync.is_videos_sync_running():
return
db = SessionLocal()
try:
raw = get_setting(db, sync.VIDEOS_SYNC_STATUS_KEY)
finished_at = _extract_finished_at(raw)
if not is_videos_sync_due(finished_at, datetime.now(timezone.utc), settings.videos_sync_idle_hours):
return
finally:
db.close()
logger.info(
"Videos sync due (last finished: %s), starting in background thread", finished_at
)
threading.Thread(target=_run_videos_sync, daemon=True).start()
except Exception:
# Never break the request because of the trigger itself.
logger.exception("Failed to evaluate videos sync trigger")

View file

@ -9,6 +9,12 @@ logger = logging.getLogger(__name__)
BATCH_SIZE = 50
# Hard safety cap for playlistItems pagination: at most this many pages
# (BATCH_SIZE items each, i.e. 500 ids) per playlist, so the pageToken loop
# can never run away. Raise both the cap and the caller's max_results
# together if more is ever needed.
MAX_PLAYLIST_PAGES = 10
class YouTubeQuotaExceeded(Exception):
pass
@ -102,26 +108,108 @@ def fetch_subscriptions(credentials: Credentials) -> list[dict]:
def fetch_playlist_video_ids(credentials: Credentials, playlist_id: str, max_results: int) -> list[str]:
"""Low-level primitive: fetch up to `max_results` playlist item ids,
newest first, with no knowledge of what is already synced. The videos sync
uses fetch_playlist_video_ids_incremental instead; this stays as the plain
paginated helper (kept for tests and any future non-incremental callers)."""
video_ids: list[str] = []
page_token: str | None = None
pages_fetched = 0
with httpx.Client(timeout=settings.metube_request_timeout_seconds) as client:
while pages_fetched < MAX_PLAYLIST_PAGES and len(video_ids) < max_results:
params = {
"part": "contentDetails",
"playlistId": playlist_id,
"maxResults": min(max_results, 50),
"maxResults": min(max_results - len(video_ids), BATCH_SIZE),
}
if page_token:
params["pageToken"] = page_token
response = client.get(f"{settings.youtube_api_base_url}/playlistItems", params=params, headers=_headers(credentials))
if response.status_code == 404:
return []
_raise_for_status(response)
data = response.json()
video_ids = []
for item in data.get("items", []):
video_id = item.get("contentDetails", {}).get("videoId")
if video_id:
video_ids.append(video_id)
pages_fetched += 1
page_token = data.get("nextPageToken")
if not page_token:
break
return video_ids
def fetch_playlist_video_ids_incremental(
credentials: Credentials,
playlist_id: str,
known_ids: set[str],
max_results: int,
stop_threshold: int,
) -> list[str]:
"""Fetch up to `max_results` previously-unknown video ids from a playlist,
stopping pagination early once `stop_threshold` consecutive ids that are
already in `known_ids` are encountered. Uploads playlists are ordered
newest-first, so a long run of known ids means we reached history that was
already synced and there is nothing new further down.
`max_results` is a hard history-depth limit: at most the newest
`max_results` videos of the channel are ever considered, and everything
older than that is intentionally not backfilled (we do not mirror full
channel history). The early stop on known ids only saves pages on repeated
syncs within that window.
Returns only the unknown ids, in playlist order."""
new_ids: list[str] = []
seen_new: set[str] = set()
consecutive_known = 0
page_token: str | None = None
pages_fetched = 0
with httpx.Client(timeout=settings.metube_request_timeout_seconds) as client:
while pages_fetched < MAX_PLAYLIST_PAGES and len(new_ids) < max_results:
params = {
"part": "contentDetails",
"playlistId": playlist_id,
"maxResults": BATCH_SIZE,
}
if page_token:
params["pageToken"] = page_token
response = client.get(f"{settings.youtube_api_base_url}/playlistItems", params=params, headers=_headers(credentials))
if response.status_code == 404:
return new_ids
_raise_for_status(response)
data = response.json()
for item in data.get("items", []):
video_id = item.get("contentDetails", {}).get("videoId")
if not video_id:
continue
if video_id in known_ids or video_id in seen_new:
consecutive_known += 1
if consecutive_known >= stop_threshold:
return new_ids
else:
seen_new.add(video_id)
new_ids.append(video_id)
consecutive_known = 0
if len(new_ids) >= max_results:
return new_ids
pages_fetched += 1
page_token = data.get("nextPageToken")
if not page_token:
break
return new_ids
def fetch_videos_details(credentials: Credentials, video_ids: list[str]) -> list[dict]:
results: list[dict] = []
@ -171,14 +259,24 @@ def unsubscribe(credentials: Credentials, youtube_subscription_id: str) -> None:
_raise_for_status(response)
def fetch_uploads_playlists(credentials: Credentials, channel_ids: list[str]) -> dict[str, str]:
result: dict[str, str] = {}
def _parse_subscriber_count(statistics: dict) -> int | None:
raw = statistics.get("subscriberCount")
if raw is None:
return None
try:
return int(raw)
except (TypeError, ValueError):
return None
def fetch_uploads_playlists(credentials: Credentials, channel_ids: list[str]) -> dict[str, dict]:
result: dict[str, dict] = {}
with httpx.Client(timeout=settings.metube_request_timeout_seconds) as client:
for i in range(0, len(channel_ids), BATCH_SIZE):
batch = channel_ids[i : i + BATCH_SIZE]
params = {
"part": "snippet,contentDetails",
"part": "snippet,contentDetails,statistics",
"id": ",".join(batch),
"maxResults": BATCH_SIZE,
}
@ -191,7 +289,10 @@ def fetch_uploads_playlists(credentials: Credentials, channel_ids: list[str]) ->
uploads = (
item.get("contentDetails", {}).get("relatedPlaylists", {}).get("uploads")
)
if channel_id and uploads:
result[channel_id] = uploads
if channel_id:
result[channel_id] = {
"uploads_playlist_id": uploads or None,
"subscriber_count": _parse_subscriber_count(item.get("statistics", {})),
}
return result

View file

@ -33,7 +33,12 @@
.category-dot { display: inline-block; width: 8px; height: 8px; border-radius: 50%; flex: none; background: var(--accent); box-shadow: 0 0 0 3px var(--accent-soft); }
.category-link { gap: 15px; padding-left: 18px; }
.category-link.active .category-dot { background: var(--accent); }
.sidebar-count { margin-left: auto; font-size: 11px; color: var(--subtle); }
.sidebar-category-row { display: flex; align-items: center; gap: 4px; }
.sidebar-category-row .sidebar-link { flex: 1; min-width: 0; }
.sidebar-badge { margin-left: auto; flex: none; padding: 1px 8px; border-radius: 999px; background: rgba(255,107,104,.13); color: var(--error); font-size: 11px; font-weight: 650; line-height: 1.5; white-space: nowrap; }
.sidebar-badge-link { cursor: pointer; text-decoration: none; transition: background .16s; }
.sidebar-badge-link:hover { background: rgba(255,107,104,.26); }
.sidebar-badge-link:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
.sidebar-add { color: var(--accent); font-size: 13px; margin-top: 4px; }
.sidebar-hint { margin: 0; padding: 5px 13px 9px; color: var(--subtle); font-size: 12px; }
.sidebar-bottom { margin-top: auto; padding-bottom: 0; }
@ -46,6 +51,8 @@ h1, h2, h3, p { margin-top: 0; }
h1 { color: var(--text); font-size: clamp(25px, 2.25vw, 34px); line-height: 1.18; letter-spacing: -.035em; font-weight: 700; margin-bottom: 8px; }
h2 { color: var(--text); font-size: 18px; line-height: 1.3; font-weight: 650; }
.page-subtitle { margin: 0; color: var(--muted); font-size: 14px; line-height: 1.5; }
.page-subtitle a { color: var(--accent); text-decoration: none; font-weight: 650; }
.page-subtitle a:hover { text-decoration: underline; }
button, .button-primary, .button-secondary, .button-quiet, .button-danger, .button-link { transition: background .16s, color .16s, border-color .16s; }
.button-primary, .button-secondary, .button-danger, .button-link { min-height: 42px; display: inline-flex; align-items: center; justify-content: center; gap: 8px; border-radius: 8px; padding: 0 15px; text-decoration: none; font-size: 13px; font-weight: 650; white-space: nowrap; }
.button-primary { border: 1px solid var(--accent); background: var(--accent); color: #081521; }
@ -125,6 +132,7 @@ button, .button-primary, .button-secondary, .button-quiet, .button-danger, .butt
.channel-title { font-size: 15px; font-weight: 650; text-decoration: none; }
.channel-title:hover { color: var(--accent); }
.badge { font-size: 11px; color: var(--subtle); }
.badge-new { padding: 2px 8px; border-radius: 999px; background: rgba(255,107,104,.13); color: var(--error); font-weight: 650; white-space: nowrap; }
.unsubscribe-button { margin-left: auto; min-height: 28px; border: 0; border-radius: 6px; padding: 0 8px; color: var(--subtle); background: transparent; font-size: 11px; }
.unsubscribe-button:hover { color: var(--error); background: rgba(255,107,104,.1); }
.channel-categories { display: flex; align-items: center; flex-wrap: wrap; gap: 7px; margin-top: 10px; }
@ -271,7 +279,8 @@ button, .button-primary, .button-secondary, .button-quiet, .button-danger, .butt
}
@media (min-width: 621px) and (max-width: 699px) { .video-grid { grid-template-columns: 1fr; } }
@media (pointer: coarse) {
.video-actions .button-link, .video-actions .button-secondary, .download-badge, .category-nav button, .local-filter-nav a, .mobile-category-nav a, .chip, .chip-edit, .category-checkbox, .unsubscribe-button { min-height: 44px; }
.video-actions .button-link, .video-actions .button-secondary, .download-badge, .category-nav button, .local-filter-nav a, .mobile-category-nav a, .chip, .chip-edit, .category-checkbox, .unsubscribe-button, .sidebar-badge { min-height: 44px; }
.sidebar-badge { display: inline-flex; align-items: center; }
.reorder-buttons .icon-button { width: 40px; height: 40px; }
.popover-create input, .popover-create button { height: 44px; }
.popover-create button { width: 44px; }

View file

@ -58,9 +58,11 @@ export interface ChannelDto {
description: string | null
thumbnail_url: string | null
uploads_playlist_id: string | null
subscriber_count: number | null
subscribed: boolean
last_synced_at: string | null
category_ids: number[]
new_videos_count: number
}
export function getChannel(channelId: number) {
@ -96,6 +98,7 @@ export interface CategoryDto {
slug: string
sort_order: number
channel_count: number
new_videos_count: number
}
export function listCategories() {
@ -211,6 +214,8 @@ export function getFeed(
downloaded?: boolean
search?: string
cursor?: string
limit?: number
newOnly?: boolean
} = {},
) {
const qs = new URLSearchParams()
@ -219,7 +224,9 @@ export function getFeed(
if (params.channelId != null) qs.set('channel_id', String(params.channelId))
if (params.downloaded) qs.set('downloaded', 'true')
if (params.search) qs.set('search', params.search)
if (params.newOnly) qs.set('new_only', 'true')
if (params.cursor) qs.set('cursor', params.cursor)
if (params.limit != null) qs.set('limit', String(params.limit))
const suffix = qs.toString() ? `?${qs.toString()}` : ''
return request<{ items: FeedVideoDto[]; next_cursor: string | null }>(`/api/feed${suffix}`)
}

View file

@ -49,7 +49,10 @@ function Sidebar({ close }: { close: () => void }) {
</div>
<div className="sidebar-group sidebar-categories">
<div className="sidebar-heading">Категории</div>
{categories.map((category) => <NavLink key={category.id} to={`/category/${category.id}`} onClick={close} className={({ isActive }) => `sidebar-link category-link ${isActive ? 'active' : ''}`}><span className="category-dot" /><span className="truncate">{category.name}</span><span className="sidebar-count">{category.channel_count}</span></NavLink>)}
{categories.map((category) => <div key={category.id} className="sidebar-category-row">
<NavLink to={`/category/${category.id}`} onClick={close} className={({ isActive }) => `sidebar-link category-link ${isActive ? 'active' : ''}`}><span className="category-dot" /><span className="truncate">{category.name}</span></NavLink>
{category.new_videos_count > 0 && <Link to={`/category/${category.id}?new=1`} onClick={close} className="sidebar-badge sidebar-badge-link" aria-label={`${category.new_videos_count} новых видео в категории «${category.name}»`}>{category.new_videos_count}</Link>}
</div>)}
{categories.length === 0 && !categoriesQuery.isLoading && <p className="sidebar-hint">Пока нет категорий</p>}
<Link to="/settings/categories" onClick={close} className="sidebar-link sidebar-add"><Icon name="plus" size={18} /><span>Новая категория</span></Link>
</div>

View file

@ -82,6 +82,7 @@ function ChannelCard({ channel, categories }: Props) {
{channel.thumbnail_url ? <img src={channel.thumbnail_url} alt="" loading="lazy" /> : <span className="channel-avatar-placeholder">{channel.title.charAt(0).toUpperCase()}</span>}
<div className="channel-info">
<div className="channel-head"><Link to={`/channels/${channel.id}/videos`} className="channel-title">{channel.title}</Link>
{channel.new_videos_count > 0 && <span className="badge badge-new" title="Недавно вышедшие видео">{channel.new_videos_count} новых</span>}
{!channel.subscribed && <span className="badge">отписан</span>}
{channel.subscribed && (
<button

View file

@ -3,17 +3,19 @@ import { Link, useParams } from 'react-router-dom'
import { getChannel, getFeed } from '../api/client'
import Icon from '../components/Icon'
import VideoCard from '../components/VideoCard'
import { formatSubscriberCount } from '../utils/format'
function ChannelVideos() {
const { channelId } = useParams<{ channelId: string }>()
const id = Number(channelId)
const valid = Number.isInteger(id) && id > 0
const channelQuery = useQuery({ queryKey: ['channel', id], queryFn: () => getChannel(id), enabled: valid })
const feedQuery = useInfiniteQuery({ queryKey: ['feed', 'channel', id], queryFn: ({ pageParam }) => getFeed({ channelId: id, cursor: pageParam }), initialPageParam: undefined as string | undefined, getNextPageParam: (lastPage) => lastPage.next_cursor ?? undefined, enabled: valid })
const feedQuery = useInfiniteQuery({ queryKey: ['feed', 'channel', id], queryFn: ({ pageParam }) => getFeed({ channelId: id, limit: 20, cursor: pageParam }), initialPageParam: undefined as string | undefined, getNextPageParam: (lastPage) => lastPage.next_cursor ?? undefined, enabled: valid })
const items = feedQuery.data?.pages.flatMap((page) => page.items) ?? []
const subscriberLabel = channelQuery.data ? formatSubscriberCount(channelQuery.data.subscriber_count) : null
return <section className="page">
<Link to="/channels" className="back-link"><Icon name="arrow" size={17} /> Каналы</Link>
<div className="page-heading"><div><p className="eyebrow">Видео канала</p><h1>{channelQuery.data?.title ?? 'Канал'}</h1><p className="page-subtitle">Последние ролики из твоих подписок.</p></div></div>
<div className="page-heading"><div><p className="eyebrow">Видео канала</p><h1>{channelQuery.data?.title ?? 'Канал'}</h1><p className="page-subtitle">{subscriberLabel ?? 'Последние ролики из твоих подписок.'}</p></div></div>
{(channelQuery.isError || feedQuery.isError || !valid) && <div className="empty-state" role="alert"><h2>Не удалось открыть канал</h2><Link to="/channels" className="button-primary">К списку каналов</Link></div>}
{feedQuery.isLoading && <div className="video-grid">{Array.from({ length: 6 }, (_, index) => <div className="video-card" key={index}><div className="skeleton-thumbnail" /><div className="video-info"><div className="skeleton-line wide" /><div className="skeleton-line" /></div></div>)}</div>}
{!feedQuery.isLoading && !feedQuery.isError && items.length > 0 && <ul className="video-grid">{items.map((video) => <VideoCard key={video.youtube_video_id} video={video} />)}</ul>}

View file

@ -17,6 +17,7 @@ function Feed() {
const categoryNumber = categoryId && /^\d+$/.test(categoryId) ? Number(categoryId) : undefined
const isLocal = location.pathname === '/local'
const isUncategorized = location.pathname === '/uncategorized'
const newOnly = categoryNumber !== undefined && searchParams.get('new') === '1'
const localCategoryRaw = isLocal ? searchParams.get('category') : null
const localCategory = localCategoryRaw && /^\d+$/.test(localCategoryRaw) ? Number(localCategoryRaw) : undefined
const localUncategorized = isLocal && searchParams.get('uncategorized') === 'true'
@ -30,12 +31,13 @@ function Feed() {
enabled: isUncategorized || (!categoryNumber && !isLocal && !search),
})
const feedQuery = useInfiniteQuery({
queryKey: ['feed', location.pathname, search, localCategory, localUncategorized],
queryKey: ['feed', location.pathname, search, localCategory, localUncategorized, newOnly],
queryFn: ({ pageParam }) => getFeed({
categoryId: isLocal ? localCategory : categoryNumber,
uncategorized: isUncategorized || localUncategorized,
downloaded: isLocal,
search,
newOnly,
cursor: pageParam,
}),
initialPageParam: undefined as string | undefined,
@ -63,7 +65,7 @@ function Feed() {
onSuccess: () => { queryClient.invalidateQueries({ queryKey: ['sync-status'] }); queryClient.invalidateQueries({ queryKey: ['feed'] }) },
})
const title = search ? `Поиск: ${search}` : isLocal ? 'На сервере' : isUncategorized ? 'Без категории' : categoryId ? category?.name ?? 'Категория' : 'Все видео'
const subtitle = search ? 'Результаты в вашей ленте' : isLocal ? 'Видео, которые вы сохранили для просмотра' : isUncategorized ? `${countQuery.data?.length ?? '…'} каналов пока без категории` : category ? `${category.channel_count} каналов в категории` : `${countQuery.data?.length ?? '…'} подписок в вашей ленте`
const subtitle = newOnly ? 'Только новые видео' : search ? 'Результаты в вашей ленте' : isLocal ? 'Видео, которые вы сохранили для просмотра' : isUncategorized ? `${countQuery.data?.length ?? '…'} каналов пока без категории` : category ? `${category.channel_count} каналов в категории` : `${countQuery.data?.length ?? '…'} подписок в вашей ленте`
const items = feedQuery.data?.pages.flatMap((page) => page.items) ?? []
useEffect(() => {
if (feedQuery.isLoading) return
@ -78,7 +80,7 @@ function Feed() {
return <section className="page">
<div className="page-heading">
<div><p className="eyebrow">Моя лента</p><h1>{title}</h1><p className="page-subtitle">{subtitle}</p></div>
<div><p className="eyebrow">Моя лента</p><h1>{title}</h1><p className="page-subtitle">{subtitle}{newOnly && <> · <Link to={`/category/${categoryNumber}`}>Показать все</Link></>}</p></div>
{!isLocal && !search && <button type="button" className="button-secondary refresh-button" onClick={() => syncMutation.mutate()} disabled={syncMutation.isPending || syncStatusQuery.data?.videos.running}><Icon name="refresh" size={17} className={syncStatusQuery.data?.videos.running ? 'spin' : ''} />{syncStatusQuery.data?.videos.running ? 'Обновляется' : 'Обновить'}</button>}
</div>
<nav className="mobile-category-nav" aria-label="Категории">

View file

@ -9,6 +9,13 @@ export function formatDuration(seconds: number | null): string | null {
return `${m}:${String(s).padStart(2, '0')}`
}
const SUBSCRIBERS_FMT = new Intl.NumberFormat('ru-RU', { notation: 'compact', maximumFractionDigits: 1 })
export function formatSubscriberCount(value: number | null | undefined): string | null {
if (value == null) return null
return `${SUBSCRIBERS_FMT.format(value)} подписчиков`
}
const RTF = new Intl.RelativeTimeFormat('ru', { numeric: 'auto' })
export function formatRelativeTime(iso: string): string {

View file

@ -0,0 +1,26 @@
"""channels.subscriber_count
Revision ID: 0008_channel_subscriber_count
Revises: 0007_drop_access_expiry
Create Date: 2026-09-17
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "0008_channel_subscriber_count"
down_revision: Union[str, None] = "0007_drop_access_expiry"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# Subscriber count comes from YouTube channels.list (statistics part);
# hidden counts stay NULL.
op.add_column("channels", sa.Column("subscriber_count", sa.BigInteger(), nullable=True))
def downgrade() -> None:
op.drop_column("channels", "subscriber_count")

View file

@ -1,10 +1,13 @@
import pytest
from datetime import datetime, timedelta, timezone
from fastapi.testclient import TestClient
from app.core.auth_dependency import require_session
from app.db import get_db
from app.main import app
from app.models.channel import Channel
from app.models.video import Video
@pytest.fixture
@ -29,6 +32,19 @@ def _create_channel(db_session, youtube_channel_id="chanA", title="Channel A"):
return channel
def _seed_video(db_session, channel, video_id, published_at):
video = Video(
youtube_video_id=video_id,
channel_id=channel.id,
title=f"Video {video_id}",
published_at=published_at,
youtube_url=f"https://www.youtube.com/watch?v={video_id}",
)
db_session.add(video)
db_session.commit()
return video
def test_category_crud(client):
created = client.post("/api/categories", json={"name": "Linux"}).json()
assert created["name"] == "Linux"
@ -118,3 +134,64 @@ def test_reorder_rejects_mismatched_ids(client):
client.post("/api/categories", json={"name": "A"})
resp = client.post("/api/categories/reorder", json={"category_ids": [9999]})
assert resp.status_code == 400
def test_category_new_videos_count_sums_recent_videos_of_subscribed_channels(client, db_session):
now = datetime.now(timezone.utc)
category = client.post("/api/categories", json={"name": "Linux"}).json()
assert category["new_videos_count"] == 0
channel_a = _create_channel(db_session, "chanA", "Channel A")
channel_b = _create_channel(db_session, "chanB", "Channel B")
channel_c = _create_channel(db_session, "chanC", "Channel C")
channel_c.subscribed = False
db_session.commit()
client.put(f"/api/channels/{channel_a.id}/categories", json={"category_ids": [category["id"]]})
client.put(f"/api/channels/{channel_b.id}/categories", json={"category_ids": [category["id"]]})
client.put(f"/api/channels/{channel_c.id}/categories", json={"category_ids": [category["id"]]})
_seed_video(db_session, channel_a, "vidARecent1", now - timedelta(days=1))
_seed_video(db_session, channel_a, "vidARecent2", now - timedelta(hours=2))
_seed_video(db_session, channel_a, "vidAOld", now - timedelta(days=30))
_seed_video(db_session, channel_b, "vidBRecent", now - timedelta(days=1))
# Unsubscribed channel: its videos must not count towards the category.
_seed_video(db_session, channel_c, "vidCUnsub", now - timedelta(days=1))
listed = client.get("/api/categories").json()
assert len(listed) == 1
assert listed[0]["channel_count"] == 3
assert listed[0]["new_videos_count"] == 3
def test_category_new_videos_count_excludes_other_categories(client, db_session):
now = datetime.now(timezone.utc)
cat1 = client.post("/api/categories", json={"name": "Linux"}).json()
cat2 = client.post("/api/categories", json={"name": "IT"}).json()
channel_a = _create_channel(db_session, "chanA", "Channel A")
channel_b = _create_channel(db_session, "chanB", "Channel B")
client.put(f"/api/channels/{channel_a.id}/categories", json={"category_ids": [cat1["id"]]})
client.put(f"/api/channels/{channel_b.id}/categories", json={"category_ids": [cat2["id"]]})
# Only an old video in cat1, a recent one in cat2: counts must not leak.
_seed_video(db_session, channel_a, "vidAOld", now - timedelta(days=30))
_seed_video(db_session, channel_b, "vidBRecent", now - timedelta(days=1))
listed = {c["id"]: c for c in client.get("/api/categories").json()}
assert listed[cat1["id"]]["new_videos_count"] == 0
assert listed[cat2["id"]]["new_videos_count"] == 1
def test_category_new_videos_count_zero_without_recent_videos(client, db_session):
now = datetime.now(timezone.utc)
category = client.post("/api/categories", json={"name": "Linux"}).json()
channel = _create_channel(db_session)
client.put(f"/api/channels/{channel.id}/categories", json={"category_ids": [category["id"]]})
_seed_video(db_session, channel, "vidOld", now - timedelta(days=30))
listed = client.get("/api/categories").json()
assert len(listed) == 1
assert listed[0]["channel_count"] == 1
assert listed[0]["new_videos_count"] == 0

View file

@ -1,3 +1,5 @@
from datetime import datetime, timedelta, timezone
import pytest
from fastapi.testclient import TestClient
@ -5,6 +7,7 @@ from app.core.auth_dependency import require_session
from app.db import get_db
from app.main import app
from app.models.channel import Channel
from app.models.video import Video
from app.services import sync
from app.services.youtube_client import YouTubeInsufficientScope
@ -86,3 +89,54 @@ def test_unsubscribe_insufficient_scope_returns_403(client, db_session, monkeypa
def test_unsubscribe_channel_not_found(client):
resp = client.post("/api/channels/9999/unsubscribe")
assert resp.status_code == 404
def _seed_video(db_session, channel, video_id, published_at):
video = Video(
youtube_video_id=video_id,
channel_id=channel.id,
title=f"Video {video_id}",
published_at=published_at,
youtube_url=f"https://www.youtube.com/watch?v={video_id}",
)
db_session.add(video)
db_session.commit()
return video
def test_channel_responses_include_subscriber_and_new_videos_counts(client, db_session):
channel = _seed_channel(db_session)
channel.subscriber_count = 1200000
db_session.commit()
now = datetime.now(timezone.utc)
_seed_video(db_session, channel, "vidRecent", now - timedelta(days=1))
_seed_video(db_session, channel, "vidOld", now - timedelta(days=30))
resp = client.get("/api/channels")
assert resp.status_code == 200
payload = resp.json()
assert len(payload) == 1
assert payload[0]["subscriber_count"] == 1200000
# Only the video from 1 day ago is inside the 7-day window.
assert payload[0]["new_videos_count"] == 1
resp_single = client.get(f"/api/channels/{channel.id}")
assert resp_single.status_code == 200
single = resp_single.json()
assert single["subscriber_count"] == 1200000
assert single["new_videos_count"] == 1
def test_channel_new_videos_count_zero_without_recent_videos(client, db_session):
channel = _seed_channel(db_session)
now = datetime.now(timezone.utc)
_seed_video(db_session, channel, "vidOld", now - timedelta(days=30))
resp = client.get("/api/channels")
assert resp.status_code == 200
payload = resp.json()
assert len(payload) == 1
assert payload[0]["subscriber_count"] is None
assert payload[0]["new_videos_count"] == 0

View file

@ -124,6 +124,133 @@ def test_feed_filters_uncategorized(client, db_session):
assert ids == {"vid1", "vid3"}
def _seed_new_only(db_session):
"""Two categories: channel A in a category (recent + old video),
channel B uncategorized (recent + old video). Published dates are
relative to now so the new_videos_window filter is exercised."""
now = datetime.now(timezone.utc)
channel_a = Channel(youtube_channel_id="chanNewA", title="Channel New A", subscribed=True)
channel_b = Channel(youtube_channel_id="chanNewB", title="Channel New B", subscribed=True)
db_session.add_all([channel_a, channel_b])
db_session.commit()
category = Category(name="Recent", slug="recent", sort_order=0)
db_session.add(category)
db_session.commit()
db_session.execute(channel_categories.insert().values(channel_id=channel_a.id, category_id=category.id))
db_session.commit()
videos = [
Video(
youtube_video_id="vidRecentA",
channel_id=channel_a.id,
title="Recent A",
published_at=now - timedelta(days=1),
youtube_url="https://www.youtube.com/watch?v=vidRecentA",
),
Video(
youtube_video_id="vidOldA",
channel_id=channel_a.id,
title="Old A",
published_at=now - timedelta(days=30),
youtube_url="https://www.youtube.com/watch?v=vidOldA",
),
Video(
youtube_video_id="vidRecentB",
channel_id=channel_b.id,
title="Recent B",
published_at=now - timedelta(hours=2),
youtube_url="https://www.youtube.com/watch?v=vidRecentB",
),
Video(
youtube_video_id="vidOldB",
channel_id=channel_b.id,
title="Old B",
published_at=now - timedelta(days=30),
youtube_url="https://www.youtube.com/watch?v=vidOldB",
),
]
db_session.add_all(videos)
db_session.commit()
return channel_a, channel_b, category, videos
def test_feed_new_only_includes_recent_and_excludes_old(client, db_session):
_seed_new_only(db_session)
resp = client.get("/api/feed?new_only=true").json()
ids = {i["youtube_video_id"] for i in resp["items"]}
assert ids == {"vidRecentA", "vidRecentB"}
# Without new_only nothing changes: old videos appear as usual.
all_ids = {i["youtube_video_id"] for i in client.get("/api/feed").json()["items"]}
assert all_ids == {"vidRecentA", "vidOldA", "vidRecentB", "vidOldB"}
def test_feed_new_only_combines_with_category(client, db_session):
_, _, category, _ = _seed_new_only(db_session)
resp = client.get(f"/api/feed?category_id={category.id}&new_only=true").json()
ids = {i["youtube_video_id"] for i in resp["items"]}
assert ids == {"vidRecentA"}
def test_feed_new_only_excludes_unsubscribed_channels(client, db_session):
now = datetime.now(timezone.utc)
channel = Channel(youtube_channel_id="chanUnsub", title="Channel Unsub", subscribed=False)
db_session.add(channel)
db_session.commit()
db_session.add(
Video(
youtube_video_id="vidFreshUnsub",
channel_id=channel.id,
title="Fresh Unsub",
published_at=now - timedelta(hours=2),
youtube_url="https://www.youtube.com/watch?v=vidFreshUnsub",
)
)
db_session.commit()
resp = client.get("/api/feed?new_only=true").json()
assert resp["items"] == []
# Regular feed is not filtered by subscription.
all_ids = {i["youtube_video_id"] for i in client.get("/api/feed").json()["items"]}
assert all_ids == {"vidFreshUnsub"}
def test_feed_new_only_pagination_cursor_works_in_filtered_set(client, db_session):
now = datetime.now(timezone.utc)
channel = Channel(youtube_channel_id="chanPage", title="Channel Page", subscribed=True)
db_session.add(channel)
db_session.commit()
db_session.add_all(
[
Video(
youtube_video_id=f"vidNew{i}",
channel_id=channel.id,
title=f"Video New {i}",
published_at=now - timedelta(hours=i),
youtube_url=f"https://www.youtube.com/watch?v=vidNew{i}",
)
for i in range(3)
]
)
db_session.commit()
page1 = client.get("/api/feed?new_only=true&limit=2").json()
assert [i["youtube_video_id"] for i in page1["items"]] == ["vidNew0", "vidNew1"]
assert page1["next_cursor"] is not None
page2 = client.get(f"/api/feed?new_only=true&limit=2&cursor={page1['next_cursor']}").json()
assert [i["youtube_video_id"] for i in page2["items"]] == ["vidNew2"]
assert page2["next_cursor"] is None
def test_feed_filters_downloaded(client, db_session):
_, _, _, videos = _seed(db_session)

View file

@ -1,3 +1,6 @@
from datetime import datetime, timezone
from app.config import settings
from app.models.channel import Channel
from app.models.video import Video
from app.services import sync
@ -32,7 +35,7 @@ def test_sync_subscriptions_idempotent_and_unsubscribes(monkeypatch, db_session)
monkeypatch.setattr(
sync.youtube_client,
"fetch_uploads_playlists",
lambda creds, ids: {cid: f"UU{cid}" for cid in ids},
lambda creds, ids: {cid: {"uploads_playlist_id": f"UU{cid}", "subscriber_count": 12345} for cid in ids},
)
result = sync.sync_subscriptions(db_session)
@ -45,6 +48,7 @@ def test_sync_subscriptions_idempotent_and_unsubscribes(monkeypatch, db_session)
assert [c.youtube_channel_id for c in channels] == ["chanA", "chanB"]
assert all(c.subscribed for c in channels)
assert channels[0].uploads_playlist_id == "UUchanA"
assert channels[0].subscriber_count == 12345
# Second sync: chanA disappears from subscriptions, chanC appears.
monkeypatch.setattr(
@ -81,6 +85,46 @@ def test_sync_subscriptions_idempotent_and_unsubscribes(monkeypatch, db_session)
assert channels["chanC"].subscribed is True
def test_sync_subscriptions_skips_none_subscriber_count(monkeypatch, db_session):
channel = Channel(
youtube_channel_id="chanA",
youtube_subscription_id="subA",
title="Channel A",
subscribed=True,
subscriber_count=12345,
)
db_session.add(channel)
db_session.commit()
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: _fake_credentials())
monkeypatch.setattr(
sync.youtube_client,
"fetch_subscriptions",
lambda creds: [
{
"youtube_channel_id": "chanA",
"youtube_subscription_id": "subA",
"title": "Channel A",
"description": "d",
"thumbnail_url": "t",
},
],
)
monkeypatch.setattr(
sync.youtube_client,
"fetch_uploads_playlists",
lambda creds, ids: {cid: {"uploads_playlist_id": f"UU{cid}", "subscriber_count": None} for cid in ids},
)
result = sync.sync_subscriptions(db_session)
assert result["status"] == "completed"
db_session.refresh(channel)
assert channel.uploads_playlist_id == "UUchanA"
# None in the API response must not overwrite the previously stored count.
assert channel.subscriber_count == 12345
def test_sync_in_progress_raises(monkeypatch, db_session):
sync._subscriptions_lock.acquire()
try:
@ -106,12 +150,14 @@ def _seed_channel(db_session, youtube_channel_id="chanA", uploads_playlist_id="U
return channel
def test_sync_videos_adds_and_updates(monkeypatch, db_session):
def test_sync_videos_adds_new_videos(monkeypatch, db_session):
channel = _seed_channel(db_session)
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: object())
monkeypatch.setattr(
sync.youtube_client, "fetch_playlist_video_ids", lambda creds, playlist_id, max_results: ["vid1"]
sync.youtube_client,
"fetch_playlist_video_ids_incremental",
lambda creds, playlist_id, known_ids, max_results, stop_threshold: ["vid1"],
)
monkeypatch.setattr(
sync.youtube_client,
@ -133,12 +179,23 @@ def test_sync_videos_adds_and_updates(monkeypatch, db_session):
assert result["status"] == "completed"
assert result["videos_added"] == 1
assert result["channels_checked"] == 1
video = db_session.query(Video).filter_by(youtube_video_id="vid1").one()
assert video.title == "Video One"
assert video.duration_seconds == 300
assert video.youtube_url == "https://www.youtube.com/watch?v=vid1"
def test_sync_videos_second_run_is_idempotent_and_fetches_no_details(monkeypatch, db_session):
channel = _seed_channel(db_session)
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: object())
monkeypatch.setattr(
sync.youtube_client,
"fetch_playlist_video_ids_incremental",
lambda creds, playlist_id, known_ids, max_results, stop_threshold: ["vid1"],
)
monkeypatch.setattr(
sync.youtube_client,
"fetch_videos_details",
@ -146,29 +203,142 @@ def test_sync_videos_adds_and_updates(monkeypatch, db_session):
{
"youtube_video_id": "vid1",
"youtube_channel_id": channel.youtube_channel_id,
"title": "Video One Updated",
"description": "d2",
"thumbnail_url": "t2",
"title": "Video One",
"description": "d",
"thumbnail_url": "t",
"published_at": "2026-09-10T12:00:00Z",
"duration_iso8601": "PT6M",
"duration_iso8601": "PT5M",
}
],
)
result2 = sync.sync_videos(db_session)
assert result2["videos_added"] == 0
assert result2["videos_updated"] == 1
first = sync.sync_videos(db_session)
assert first["videos_added"] == 1
videos = db_session.query(Video).all()
assert len(videos) == 1
assert videos[0].title == "Video One Updated"
# Second run: vid1 is now known, so the incremental fetch reports nothing
# new and no details are requested at all (existing videos are not
# metadata-refreshed by design).
details_calls = []
monkeypatch.setattr(
sync.youtube_client,
"fetch_playlist_video_ids_incremental",
lambda creds, playlist_id, known_ids, max_results, stop_threshold: [],
)
monkeypatch.setattr(
sync.youtube_client,
"fetch_videos_details",
lambda creds, ids: details_calls.append(list(ids)) or [],
)
second = sync.sync_videos(db_session)
assert second["videos_added"] == 0
assert second["videos_updated"] == 0
assert details_calls == [[]]
assert db_session.query(Video).count() == 1
def test_sync_videos_passes_known_ids_and_backfill_settings(monkeypatch, db_session):
channel = _seed_channel(db_session)
db_session.add_all(
[
Video(
youtube_video_id="known1",
channel_id=channel.id,
title="Known 1",
published_at=datetime(2026, 9, 1, tzinfo=timezone.utc),
youtube_url="https://www.youtube.com/watch?v=known1",
),
Video(
youtube_video_id="known2",
channel_id=channel.id,
title="Known 2",
published_at=datetime(2026, 9, 1, tzinfo=timezone.utc),
youtube_url="https://www.youtube.com/watch?v=known2",
),
]
)
db_session.commit()
captured = {}
def fake_incremental(creds, playlist_id, known_ids, max_results, stop_threshold):
captured.update(known_ids=known_ids, max_results=max_results, stop_threshold=stop_threshold)
return ["vid1"]
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: object())
monkeypatch.setattr(sync.youtube_client, "fetch_playlist_video_ids_incremental", fake_incremental)
monkeypatch.setattr(
sync.youtube_client,
"fetch_videos_details",
lambda creds, ids: [
{
"youtube_video_id": "vid1",
"youtube_channel_id": channel.youtube_channel_id,
"title": "Video One",
"description": "d",
"thumbnail_url": "t",
"published_at": "2026-09-10T12:00:00Z",
"duration_iso8601": "PT5M",
}
],
)
result = sync.sync_videos(db_session)
assert result["videos_added"] == 1
assert captured["known_ids"] == {"known1", "known2"}
assert captured["max_results"] == settings.videos_backfill_cap
assert captured["stop_threshold"] == settings.videos_known_stop_threshold
def test_sync_videos_backfill_cap_ingests_all_candidates(monkeypatch, db_session):
channel = _seed_channel(db_session)
cap = settings.videos_backfill_cap
new_ids = [f"vid{i}" for i in range(cap)]
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: object())
monkeypatch.setattr(
sync.youtube_client,
"fetch_playlist_video_ids_incremental",
lambda creds, playlist_id, known_ids, max_results, stop_threshold: new_ids,
)
details_calls = []
monkeypatch.setattr(
sync.youtube_client,
"fetch_videos_details",
lambda creds, ids: details_calls.append(list(ids))
or [
{
"youtube_video_id": vid,
"youtube_channel_id": channel.youtube_channel_id,
"title": f"Title {vid}",
"description": "d",
"thumbnail_url": "t",
"published_at": "2026-09-10T12:00:00Z",
"duration_iso8601": None,
}
for vid in ids
],
)
result = sync.sync_videos(db_session)
assert result["videos_added"] == cap
assert result["videos_updated"] == 0
assert set(details_calls[0]) == set(new_ids)
assert db_session.query(Video).count() == cap
def test_sync_videos_skips_unknown_channel(monkeypatch, db_session):
_seed_channel(db_session, "chanA")
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: object())
monkeypatch.setattr(sync.youtube_client, "fetch_playlist_video_ids", lambda creds, playlist_id, max_results: [])
monkeypatch.setattr(
sync.youtube_client,
"fetch_playlist_video_ids_incremental",
lambda creds, playlist_id, known_ids, max_results, stop_threshold: [],
)
monkeypatch.setattr(
sync.youtube_client,
"fetch_videos_details",
@ -189,3 +359,211 @@ def test_sync_videos_skips_unknown_channel(monkeypatch, db_session):
assert result["videos_skipped"] == 1
assert db_session.query(Video).count() == 0
class _FakeResponse:
def __init__(self, payload, status_code=200):
self.status_code = status_code
self._payload = payload
self.text = str(payload)
def json(self):
return self._payload
class _FakeCredentials:
token = "fake-token"
class _FakeClient:
"""Serves canned playlistItems pages through the real incremental fetcher."""
def __init__(self, pages):
self._pages = list(pages)
self.requests = []
def __enter__(self):
return self
def __exit__(self, *args):
return False
def get(self, url, params=None, headers=None):
self.requests.append({"url": url, "params": params})
return self._pages.pop(0)
def _page(items, next_token=None):
return _FakeResponse(
{
"items": [{"contentDetails": {"videoId": vid}} for vid in items],
**({"nextPageToken": next_token} if next_token else {}),
}
)
def test_sync_videos_stops_pagination_on_known_run(monkeypatch, db_session):
"""A channel with many known videos in a row after the new ones: the
incremental fetch must stop paginating once the known-run threshold is hit
and only the new ids must get details."""
channel = _seed_channel(db_session)
known = [f"known{i}" for i in range(60)]
db_session.add_all(
[
Video(
youtube_video_id=vid,
channel_id=channel.id,
title=f"Known {vid}",
published_at=datetime(2026, 9, 1, tzinfo=timezone.utc),
youtube_url=f"https://www.youtube.com/watch?v={vid}",
)
for vid in known
]
)
db_session.commit()
# Page 1: two new videos then 48 known; page 2 continues with known ids,
# so the run crosses 50 on the second page and pagination must stop there
# (a third page exists and must never be requested).
client = _FakeClient(
[
_page(["new1", "new2", *known[:48]], next_token="page2"),
_page(known[48:58], next_token="page3"),
_page(["never-seen"]),
]
)
details_calls = []
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: _FakeCredentials())
monkeypatch.setattr(sync.youtube_client.httpx, "Client", lambda timeout: client)
monkeypatch.setattr(
sync.youtube_client,
"fetch_videos_details",
lambda creds, ids: details_calls.append(list(ids))
or [
{
"youtube_video_id": vid,
"youtube_channel_id": channel.youtube_channel_id,
"title": f"Title {vid}",
"description": "",
"thumbnail_url": None,
"published_at": "2026-09-10T12:00:00Z",
"duration_iso8601": None,
}
for vid in ids
],
)
result = sync.sync_videos(db_session)
assert result["videos_added"] == 2
assert len(client.requests) == 2 # stopped on page 2, page 3 never fetched
assert set(details_calls[0]) == {"new1", "new2"}
assert db_session.query(Video).count() == 62
def test_sync_videos_stops_at_backfill_cap(monkeypatch, db_session):
"""More new videos than the cap: pagination stops once the cap is reached
and details are fetched for exactly the capped candidates."""
channel = _seed_channel(db_session)
cap = settings.videos_backfill_cap
pages = [_page([f"vid{page * 50 + i}" for i in range(50)], next_token=f"p{page + 2}") for page in range(6)]
client = _FakeClient(pages)
details_calls = []
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: _FakeCredentials())
monkeypatch.setattr(sync.youtube_client.httpx, "Client", lambda timeout: client)
monkeypatch.setattr(
sync.youtube_client,
"fetch_videos_details",
lambda creds, ids: details_calls.append(list(ids))
or [
{
"youtube_video_id": vid,
"youtube_channel_id": channel.youtube_channel_id,
"title": f"Title {vid}",
"description": "",
"thumbnail_url": None,
"published_at": "2026-09-10T12:00:00Z",
"duration_iso8601": None,
}
for vid in ids
],
)
result = sync.sync_videos(db_session)
assert len(client.requests) == 4 # 4 pages x 50 = cap reached
assert result["videos_added"] == cap
assert len(details_calls[0]) == cap
assert db_session.query(Video).count() == cap
def test_sync_videos_cap_is_hard_history_depth_limit(monkeypatch, db_session):
"""videos_backfill_cap is an intentional history-depth limit, not a
per-sync batch: a 250-video channel ingests only the newest 200 videos
ever; a later sync without new videos stops after one page of known ids
and adds nothing; a new video on top of the playlist is picked up while
the beyond-the-cap history never surfaces."""
channel = _seed_channel(db_session)
cap = settings.videos_backfill_cap
def _details(creds, ids):
return [
{
"youtube_video_id": vid,
"youtube_channel_id": channel.youtube_channel_id,
"title": f"Title {vid}",
"description": "",
"thumbnail_url": None,
"published_at": "2026-09-10T12:00:00Z",
"duration_iso8601": None,
}
for vid in ids
]
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: _FakeCredentials())
monkeypatch.setattr(sync.youtube_client, "fetch_videos_details", _details)
# Uploads playlist, newest first: vid249 .. vid0.
playlist = [f"vid{i}" for i in range(249, -1, -1)]
def _run(ids):
client = _FakeClient(
[_page(ids[i : i + 50], next_token=f"page{i // 50}") for i in range(0, len(ids), 50)]
)
monkeypatch.setattr(sync.youtube_client.httpx, "Client", lambda timeout: client)
return sync.sync_videos(db_session), client
# First sync: 250 videos, only the newest 200 fit under the cap.
result1, client1 = _run(playlist)
assert result1["videos_added"] == cap
assert db_session.query(Video).count() == cap
assert len(client1.requests) == 4 # 4 pages x 50 = cap, older pages untouched
synced = {v.youtube_video_id for v in db_session.query(Video).all()}
assert synced == {f"vid{i}" for i in range(50, 250)}
assert "vid49" not in synced # older than the cap: never backfilled by design
# Second sync, nothing new: one page of 50 known ids in a row stops it.
result2, client2 = _run(playlist)
assert result2["videos_added"] == 0
assert len(client2.requests) == 1
assert db_session.query(Video).count() == cap
# Third sync, one new video on top: only that one is added, the
# beyond-the-cap history still does not surface.
result3, client3 = _run(["vid250", *playlist])
assert result3["videos_added"] == 1
assert len(client3.requests) == 2 # page 2 needed to confirm 50 known in a row
synced_after = {v.youtube_video_id for v in db_session.query(Video).all()}
assert synced_after == {f"vid{i}" for i in range(50, 251)}
assert "vid49" not in synced_after
def test_is_videos_sync_running_reflects_lock():
sync._videos_lock.acquire()
try:
assert sync.is_videos_sync_running() is True
finally:
sync._videos_lock.release()
assert sync.is_videos_sync_running() is False

159
tests/test_sync_trigger.py Normal file
View file

@ -0,0 +1,159 @@
import json
from datetime import datetime, timedelta, timezone
from types import SimpleNamespace
from app.services import sync_trigger
def _now():
return datetime.now(timezone.utc)
def test_is_videos_sync_due_without_finished_at():
assert sync_trigger.is_videos_sync_due(None, _now(), 2) is True
def test_is_videos_sync_due_after_idle():
finished = (_now() - timedelta(hours=3)).isoformat()
assert sync_trigger.is_videos_sync_due(finished, _now(), 2) is True
def test_is_videos_sync_due_at_exact_threshold():
finished = (_now() - timedelta(hours=2)).isoformat()
assert sync_trigger.is_videos_sync_due(finished, _now(), 2) is True
def test_is_videos_sync_due_before_idle():
finished = (_now() - timedelta(hours=1)).isoformat()
assert sync_trigger.is_videos_sync_due(finished, _now(), 2) is False
def test_is_videos_sync_due_unparseable_finished_at():
assert sync_trigger.is_videos_sync_due("not-a-date", _now(), 2) is True
assert sync_trigger.is_videos_sync_due(12345, _now(), 2) is True # non-str garbage
def test_is_videos_sync_due_naive_finished_at():
naive = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(hours=3)
assert sync_trigger.is_videos_sync_due(naive.isoformat(), _now(), 2) is True
def test_extract_finished_at():
assert sync_trigger._extract_finished_at(None) is None
assert sync_trigger._extract_finished_at("not json") is None
assert sync_trigger._extract_finished_at(json.dumps(["a"])) is None
assert sync_trigger._extract_finished_at(json.dumps({"status": "running", "finished_at": None})) is None
assert (
sync_trigger._extract_finished_at(json.dumps({"status": "completed", "finished_at": "2026-09-10T12:00:00+00:00"}))
== "2026-09-10T12:00:00+00:00"
)
class _FakeDB:
def __init__(self):
self.closed = False
def close(self):
self.closed = True
def _patch_threads(monkeypatch, started):
monkeypatch.setattr(
sync_trigger,
"threading",
SimpleNamespace(Thread=lambda target=None, daemon=None: started.append({"target": target, "daemon": daemon})),
)
def test_maybe_trigger_skips_when_sync_running(monkeypatch):
started = []
monkeypatch.setattr(sync_trigger.sync, "is_videos_sync_running", lambda: True)
monkeypatch.setattr(
sync_trigger,
"SessionLocal",
lambda: (_ for _ in ()).throw(AssertionError("DB must not be touched while running")),
)
_patch_threads(monkeypatch, started)
sync_trigger.maybe_trigger_videos_sync()
assert started == []
def test_maybe_trigger_skips_when_not_due(monkeypatch):
started = []
db = _FakeDB()
raw = json.dumps({"status": "completed", "finished_at": (_now() - timedelta(hours=1)).isoformat()})
monkeypatch.setattr(sync_trigger.sync, "is_videos_sync_running", lambda: False)
monkeypatch.setattr(sync_trigger, "SessionLocal", lambda: db)
monkeypatch.setattr(sync_trigger, "get_setting", lambda session, key: raw)
_patch_threads(monkeypatch, started)
sync_trigger.maybe_trigger_videos_sync()
assert started == []
assert db.closed is True
def test_maybe_trigger_starts_thread_when_due_never_synced(monkeypatch):
started = []
db = _FakeDB()
monkeypatch.setattr(sync_trigger.sync, "is_videos_sync_running", lambda: False)
monkeypatch.setattr(sync_trigger, "SessionLocal", lambda: db)
monkeypatch.setattr(sync_trigger, "get_setting", lambda session, key: None)
_patch_threads(monkeypatch, started)
sync_trigger.maybe_trigger_videos_sync()
assert len(started) == 1
assert started[0]["target"] == sync_trigger._run_videos_sync
assert started[0]["daemon"] is True
assert db.closed is True
def test_maybe_trigger_starts_thread_when_due_after_idle(monkeypatch):
started = []
db = _FakeDB()
raw = json.dumps({"status": "completed", "finished_at": (_now() - timedelta(hours=5)).isoformat()})
monkeypatch.setattr(sync_trigger.sync, "is_videos_sync_running", lambda: False)
monkeypatch.setattr(sync_trigger, "SessionLocal", lambda: db)
monkeypatch.setattr(sync_trigger, "get_setting", lambda session, key: raw)
_patch_threads(monkeypatch, started)
sync_trigger.maybe_trigger_videos_sync()
assert len(started) == 1
assert started[0]["daemon"] is True
def test_maybe_trigger_never_raises_when_db_unavailable(monkeypatch):
started = []
monkeypatch.setattr(sync_trigger.sync, "is_videos_sync_running", lambda: False)
monkeypatch.setattr(
sync_trigger,
"SessionLocal",
lambda: (_ for _ in ()).throw(RuntimeError("db down")),
)
_patch_threads(monkeypatch, started)
sync_trigger.maybe_trigger_videos_sync() # must not raise
assert started == []
def test_run_videos_sync_swallows_sync_in_progress(monkeypatch):
closed = []
monkeypatch.setattr(
sync_trigger,
"SessionLocal",
lambda: SimpleNamespace(close=lambda: closed.append(True)),
)
monkeypatch.setattr(
sync_trigger.sync,
"sync_videos",
lambda db: (_ for _ in ()).throw(sync_trigger.sync.SyncInProgress("busy")),
)
sync_trigger._run_videos_sync() # must not raise
assert closed == [True]

View file

@ -115,6 +115,160 @@ def test_fetch_playlist_video_ids_returns_empty_on_404(monkeypatch):
assert result == []
def test_fetch_playlist_video_ids_paginates(monkeypatch):
page1 = FakeResponse(
200,
{
"items": [{"contentDetails": {"videoId": f"vid{i}"}} for i in range(50)],
"nextPageToken": "page2",
},
)
page2 = FakeResponse(
200,
{"items": [{"contentDetails": {"videoId": f"vid{i}"}} for i in range(50, 80)]},
)
seen = []
class RecordingClient(FakeClient):
def get(self, url, params=None, headers=None):
seen.append({"url": url, "params": params})
return self._responses.pop(0)
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: RecordingClient([page1, page2]))
result = youtube_client.fetch_playlist_video_ids(_credentials(), "UUplaylist", 80)
assert result == [f"vid{i}" for i in range(80)]
assert len(seen) == 2
assert seen[0]["params"]["maxResults"] == 50
assert "pageToken" not in seen[0]["params"]
assert seen[1]["params"]["maxResults"] == 30
assert seen[1]["params"]["pageToken"] == "page2"
assert seen[0]["params"]["part"] == "contentDetails"
assert seen[1]["params"]["part"] == "contentDetails"
def test_fetch_playlist_video_ids_stops_at_safety_cap(monkeypatch):
page = FakeResponse(
200,
{"items": [{"contentDetails": {"videoId": "vid"}}], "nextPageToken": "next"},
)
seen = []
class RecordingClient(FakeClient):
def get(self, url, params=None, headers=None):
seen.append(params)
return self._responses.pop(0)
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: RecordingClient([page] * 100))
result = youtube_client.fetch_playlist_video_ids(_credentials(), "UUplaylist", 10000)
assert len(result) == youtube_client.MAX_PLAYLIST_PAGES
assert len(seen) == youtube_client.MAX_PLAYLIST_PAGES
assert "pageToken" not in seen[0]
assert seen[1]["pageToken"] == "next"
def _page_items(video_ids, next_token=None):
payload = {"items": [{"contentDetails": {"videoId": vid}} for vid in video_ids]}
if next_token:
payload["nextPageToken"] = next_token
return FakeResponse(200, payload)
def test_fetch_playlist_video_ids_incremental_returns_only_unknown(monkeypatch):
page = _page_items(["new1", "known1", "new2", "known2"])
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: FakeClient([page]))
result = youtube_client.fetch_playlist_video_ids_incremental(
_credentials(), "UUplaylist", known_ids={"known1", "known2"}, max_results=10, stop_threshold=50
)
assert result == ["new1", "new2"]
def test_fetch_playlist_video_ids_incremental_stops_on_known_run(monkeypatch):
page1 = _page_items([f"known{i}" for i in range(50)], next_token="page2")
page2 = _page_items(["new1", "new2"], next_token="page3")
seen = []
class RecordingClient(FakeClient):
def get(self, url, params=None, headers=None):
seen.append(params)
return self._responses.pop(0)
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: RecordingClient([page1, page2]))
result = youtube_client.fetch_playlist_video_ids_incremental(
_credentials(), "UUplaylist", known_ids={f"known{i}" for i in range(50)}, max_results=10, stop_threshold=50
)
assert result == []
assert len(seen) == 1 # page2 never requested
def test_fetch_playlist_video_ids_incremental_known_run_crosses_pages(monkeypatch):
page1 = _page_items(["new1"] + [f"known{i}" for i in range(49)], next_token="page2")
page2 = _page_items([f"known{i}" for i in range(49, 60)], next_token="page3")
seen = []
class RecordingClient(FakeClient):
def get(self, url, params=None, headers=None):
seen.append(params)
return self._responses.pop(0)
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: RecordingClient([page1, page2]))
result = youtube_client.fetch_playlist_video_ids_incremental(
_credentials(), "UUplaylist", known_ids={f"known{i}" for i in range(60)}, max_results=10, stop_threshold=50
)
assert result == ["new1"]
assert len(seen) == 2 # stopped mid-page-2, page3 never requested
def test_fetch_playlist_video_ids_incremental_stops_at_cap(monkeypatch):
pages = [_page_items([f"vid{page * 50 + i}" for i in range(50)], next_token=f"p{page + 2}") for page in range(6)]
seen = []
class RecordingClient(FakeClient):
def get(self, url, params=None, headers=None):
seen.append(params)
return self._responses.pop(0)
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: RecordingClient(pages))
result = youtube_client.fetch_playlist_video_ids_incremental(
_credentials(), "UUplaylist", known_ids=set(), max_results=200, stop_threshold=50
)
assert len(result) == 200
assert len(seen) == 4
def test_fetch_playlist_video_ids_incremental_empty_on_404(monkeypatch):
response = FakeResponse(404, {"error": {"message": "playlist not found"}})
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: FakeClient([response]))
result = youtube_client.fetch_playlist_video_ids_incremental(
_credentials(), "UUplaylist", known_ids=set(), max_results=10, stop_threshold=50
)
assert result == []
def test_fetch_playlist_video_ids_incremental_dedupes_new_ids(monkeypatch):
page = _page_items(["new1", "new1", "known1"])
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: FakeClient([page]))
result = youtube_client.fetch_playlist_video_ids_incremental(
_credentials(), "UUplaylist", known_ids={"known1"}, max_results=10, stop_threshold=50
)
assert result == ["new1"]
def test_fetch_videos_details(monkeypatch):
response = FakeResponse(
200,
@ -194,13 +348,48 @@ def test_fetch_uploads_playlists_batches(monkeypatch):
200,
{
"items": [
{"id": "chanA", "contentDetails": {"relatedPlaylists": {"uploads": "UUchanA"}}},
{"id": "chanB", "contentDetails": {"relatedPlaylists": {"uploads": "UUchanB"}}},
{
"id": "chanA",
"contentDetails": {"relatedPlaylists": {"uploads": "UUchanA"}},
"statistics": {"subscriberCount": "12345"},
},
{
"id": "chanB",
"contentDetails": {"relatedPlaylists": {"uploads": "UUchanB"}},
"statistics": {"hiddenSubscriberCount": True},
},
{"id": "chanC", "contentDetails": {"relatedPlaylists": {"uploads": "UUchanC"}}},
{"id": "chanD", "contentDetails": {"relatedPlaylists": {}}, "statistics": {"subscriberCount": "42"}},
]
},
)
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: FakeClient([response]))
result = youtube_client.fetch_uploads_playlists(_credentials(), ["chanA", "chanB"])
result = youtube_client.fetch_uploads_playlists(_credentials(), ["chanA", "chanB", "chanC", "chanD"])
assert result == {"chanA": "UUchanA", "chanB": "UUchanB"}
assert result == {
"chanA": {"uploads_playlist_id": "UUchanA", "subscriber_count": 12345},
"chanB": {"uploads_playlist_id": "UUchanB", "subscriber_count": None},
"chanC": {"uploads_playlist_id": "UUchanC", "subscriber_count": None},
"chanD": {"uploads_playlist_id": None, "subscriber_count": 42},
}
def test_fetch_uploads_playlists_subscriber_count_not_an_int_is_none(monkeypatch):
response = FakeResponse(
200,
{
"items": [
{
"id": "chanA",
"contentDetails": {"relatedPlaylists": {"uploads": "UUchanA"}},
"statistics": {"subscriberCount": "not-a-number"},
}
]
},
)
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: FakeClient([response]))
result = youtube_client.fetch_uploads_playlists(_credentials(), ["chanA"])
assert result == {"chanA": {"uploads_playlist_id": "UUchanA", "subscriber_count": None}}