Channel features: subscribers, new-videos badges, activity-driven sync
- Store subscriber count from YouTube statistics and show it on channel page - Sync 50 videos per channel with playlistItems pagination support - Show per-channel and per-category new-videos counters (2-day window) - Replace hourly videos sync with activity trigger (2h idle) and incremental backfill (hard cap 200 per channel) - Clicking the sidebar new-videos count filters the category feed to recent videos only (new_only) - Update agent-team docs: deploy after green checks
This commit is contained in:
parent
fde9a439df
commit
e10df8dcbd
27 changed files with 1420 additions and 107 deletions
13
.env.example
13
.env.example
|
|
@ -26,7 +26,16 @@ METUBE_CONTAINER_DOWNLOAD_DIR=/downloads
|
|||
METUBE_REQUEST_TIMEOUT_SECONDS=30
|
||||
|
||||
SUBSCRIPTIONS_SYNC_INTERVAL_HOURS=6
|
||||
VIDEOS_SYNC_INTERVAL_MINUTES=60
|
||||
VIDEOS_PER_CHANNEL_SYNC=10
|
||||
|
||||
# Необязательные настройки
|
||||
# Авто-синк видео запускается по активности (запросы к API) после простоя
|
||||
# в VIDEOS_SYNC_IDLE_HOURS с последнего завершённого синка.
|
||||
VIDEOS_SYNC_IDLE_HOURS=2
|
||||
# Максимум элементов плейлиста, рассматриваемых на канал за один синк.
|
||||
VIDEOS_BACKFILL_CAP=200
|
||||
# Остановить пагинацию плейлиста канала после стольких уже известных
|
||||
# видео подряд (дальше — уже синхронизированная история).
|
||||
VIDEOS_KNOWN_STOP_THRESHOLD=50
|
||||
NEW_VIDEOS_WINDOW_DAYS=2
|
||||
|
||||
LOG_LEVEL=INFO
|
||||
|
|
|
|||
|
|
@ -15,11 +15,12 @@ You are the only agent who normally speaks with the user. Coordinate the opencod
|
|||
4. Submit the task to Coder with the Task tool (subagent_type `coder`). Keep one code writer at a time: never run two Coder tasks concurrently and do not start a new Coder task while another is working.
|
||||
5. Read Coder's report (ends with `CODER_DONE`) and inspect the diff yourself. Then run Reviewer and Tester on the result. Reviewer must not edit; Tester must not fix. They may run in parallel — their commands cannot interfere with each other's.
|
||||
6. Collect Critical, Major, and Minor findings with evidence. Send actionable findings back to Coder as a follow-up task. Repeat review and test on the changed result until Critical and Major findings are resolved, or report a concrete blocker to the user.
|
||||
7. Give the user a concise final report: changes, affected files, review findings resolved or remaining, tests run and results, known limits, and worktree/branch. Do not claim visual or integration checks that were not performed.
|
||||
6.5. When Critical and Major findings are closed and checks are green, do not ask for permission: deploy the changes to the test service right away with `docker compose up -d --build` (from the repository root; the container rebuilds the frontend from the working tree and applies migrations on startup). Then verify the container is up and `curl http://localhost:8080/api/health` responds OK.
|
||||
7. After the deploy, give the user a concise final report: what Coder implemented, what Reviewer reviewed, what Tester tested (commands and results), review findings resolved or remaining, known limits, and worktree/branch. Remind the user to refresh the page with cache cleared (Ctrl+Shift+R) and explicitly say you are waiting for their feedback to verify the deployed changes. Do not claim visual or integration checks that were not performed.
|
||||
|
||||
## Working rules
|
||||
|
||||
- Do not merge, deploy, publish, or commit unless the user requested it or existing authorization covers it.
|
||||
- Do not merge, publish, or commit unless the user requested it or existing authorization covers it. Deploying to the test service is authorized by this workflow (step 6.5); deploying elsewhere still requires the user's go-ahead.
|
||||
- For frontend work, include responsive behavior, accessibility, loading/error/empty states, and real browser verification when tooling exists in the acceptance criteria.
|
||||
- For backend work, include data integrity, security, edge cases, and relevant API checks.
|
||||
|
||||
|
|
|
|||
|
|
@ -9,7 +9,8 @@
|
|||
- декомпозирует задачу и отправляет её Coder'у через Task tool;
|
||||
- после Coder'а запускает Reviewer (только чтение) и Tester (проверка без правок) параллельно;
|
||||
- возвращает Coder'у actionable findings и повторяет цикл, пока Critical/Major не закрыты;
|
||||
- отдаёт финальный отчёт с изменениями, проверками и оставшимися рисками.
|
||||
- когда Critical/Major закрыты и проверки зелёные — без вопроса деплоит на тестовый сервис (`docker compose up -d --build`, затем `curl http://localhost:8080/api/health`);
|
||||
- отдаёт финальный отчёт уже после деплоя: что написано, просмотрено и протестировано, оставшиеся риски и просьба проверить в браузере с очисткой кэша (Ctrl+Shift+R).
|
||||
|
||||
Вручную писать субагентам не нужно — их вызывает только Orchestrator.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,14 +1,19 @@
|
|||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy import func
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import settings
|
||||
from app.core.auth_dependency import require_session
|
||||
from app.core.slugify import unique_slugify
|
||||
from app.db import get_db
|
||||
from app.models.category import Category
|
||||
from app.models.channel import Channel
|
||||
from app.models.channel_category import channel_categories
|
||||
from app.models.video import Video
|
||||
|
||||
router = APIRouter(dependencies=[Depends(require_session)])
|
||||
|
||||
|
|
@ -40,13 +45,15 @@ def _name_taken(db: Session, name: str, exclude_id: int | None = None) -> bool:
|
|||
return any(row[0].casefold() == target for row in query.all())
|
||||
|
||||
|
||||
def _serialize(db: Session, category: Category, counts: dict[int, int]) -> dict:
|
||||
def _serialize(db: Session, category: Category, counts: dict[int, int], new_videos: dict[int, int] | None = None) -> dict:
|
||||
new_videos = new_videos or {}
|
||||
return {
|
||||
"id": category.id,
|
||||
"name": category.name,
|
||||
"slug": category.slug,
|
||||
"sort_order": category.sort_order,
|
||||
"channel_count": counts.get(category.id, 0),
|
||||
"new_videos_count": new_videos.get(category.id, 0),
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -59,7 +66,17 @@ def list_categories(db: Session = Depends(get_db)) -> list[dict]:
|
|||
.all()
|
||||
)
|
||||
counts = dict(count_rows)
|
||||
return [_serialize(db, c, counts) for c in categories]
|
||||
since = datetime.now(timezone.utc) - timedelta(days=settings.new_videos_window_days)
|
||||
new_videos_rows = (
|
||||
db.query(channel_categories.c.category_id, func.count(Video.id))
|
||||
.join(Channel, Channel.id == channel_categories.c.channel_id)
|
||||
.join(Video, Video.channel_id == Channel.id)
|
||||
.filter(Channel.subscribed.is_(True), Video.published_at >= since)
|
||||
.group_by(channel_categories.c.category_id)
|
||||
.all()
|
||||
)
|
||||
new_videos = dict(new_videos_rows)
|
||||
return [_serialize(db, c, counts, new_videos) for c in categories]
|
||||
|
||||
|
||||
@router.post("/categories", status_code=201)
|
||||
|
|
|
|||
|
|
@ -1,15 +1,18 @@
|
|||
import logging
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import settings
|
||||
from app.core.auth_dependency import require_session
|
||||
from app.db import get_db
|
||||
from app.models.category import Category
|
||||
from app.models.channel import Channel
|
||||
from app.models.channel_category import channel_categories
|
||||
from app.models.video import Video
|
||||
from app.services import sync
|
||||
from app.services.google_oauth import OAuthNotConnected
|
||||
from app.services.youtube_client import YouTubeAPIError, YouTubeInsufficientScope, YouTubeQuotaExceeded
|
||||
|
|
@ -37,7 +40,22 @@ def _category_ids_by_channel(db: Session, channel_ids: list[int]) -> dict[int, l
|
|||
return result
|
||||
|
||||
|
||||
def _serialize(channel: Channel, category_ids: list[int]) -> dict:
|
||||
def _new_videos_counts(db: Session, channel_ids: list[int]) -> dict[int, int]:
|
||||
"""Videos published within the new-videos window, per channel, in one
|
||||
aggregate query."""
|
||||
if not channel_ids:
|
||||
return {}
|
||||
since = datetime.now(timezone.utc) - timedelta(days=settings.new_videos_window_days)
|
||||
rows = (
|
||||
db.query(Video.channel_id, func.count(Video.id))
|
||||
.filter(Video.channel_id.in_(channel_ids), Video.published_at >= since)
|
||||
.group_by(Video.channel_id)
|
||||
.all()
|
||||
)
|
||||
return {channel_id: count for channel_id, count in rows}
|
||||
|
||||
|
||||
def _serialize(channel: Channel, category_ids: list[int], new_videos_count: int = 0) -> dict:
|
||||
return {
|
||||
"id": channel.id,
|
||||
"youtube_channel_id": channel.youtube_channel_id,
|
||||
|
|
@ -45,9 +63,11 @@ def _serialize(channel: Channel, category_ids: list[int]) -> dict:
|
|||
"description": channel.description,
|
||||
"thumbnail_url": channel.thumbnail_url,
|
||||
"uploads_playlist_id": channel.uploads_playlist_id,
|
||||
"subscriber_count": channel.subscriber_count,
|
||||
"subscribed": channel.subscribed,
|
||||
"last_synced_at": channel.last_synced_at,
|
||||
"category_ids": category_ids,
|
||||
"new_videos_count": new_videos_count,
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -75,8 +95,10 @@ def list_channels(
|
|||
query = query.filter(Channel.id.in_(channel_ids_in_category))
|
||||
|
||||
channels = query.order_by(Channel.title.asc()).all()
|
||||
category_map = _category_ids_by_channel(db, [c.id for c in channels])
|
||||
return [_serialize(c, category_map.get(c.id, [])) for c in channels]
|
||||
channel_ids = [c.id for c in channels]
|
||||
category_map = _category_ids_by_channel(db, channel_ids)
|
||||
new_videos_map = _new_videos_counts(db, channel_ids)
|
||||
return [_serialize(c, category_map.get(c.id, []), new_videos_map.get(c.id, 0)) for c in channels]
|
||||
|
||||
|
||||
@router.get("/channels/{channel_id}")
|
||||
|
|
@ -85,7 +107,8 @@ def get_channel(channel_id: int, db: Session = Depends(get_db)) -> dict:
|
|||
if channel is None:
|
||||
raise HTTPException(status_code=404, detail="Channel not found")
|
||||
category_map = _category_ids_by_channel(db, [channel_id])
|
||||
return _serialize(channel, category_map.get(channel_id, []))
|
||||
new_videos_map = _new_videos_counts(db, [channel_id])
|
||||
return _serialize(channel, category_map.get(channel_id, []), new_videos_map.get(channel_id, 0))
|
||||
|
||||
|
||||
@router.put("/channels/{channel_id}/categories")
|
||||
|
|
@ -110,7 +133,8 @@ def set_channel_categories(channel_id: int, payload: ChannelCategoriesUpdate, db
|
|||
)
|
||||
db.commit()
|
||||
|
||||
return _serialize(channel, sorted(unique_ids))
|
||||
new_videos_map = _new_videos_counts(db, [channel_id])
|
||||
return _serialize(channel, sorted(unique_ids), new_videos_map.get(channel_id, 0))
|
||||
|
||||
|
||||
@router.post("/channels/{channel_id}/unsubscribe")
|
||||
|
|
@ -139,4 +163,5 @@ def unsubscribe_channel(channel_id: int, db: Session = Depends(get_db)) -> dict:
|
|||
raise HTTPException(status_code=502, detail="YouTube is unavailable")
|
||||
|
||||
category_map = _category_ids_by_channel(db, [channel_id])
|
||||
return _serialize(channel, category_map.get(channel_id, []))
|
||||
new_videos_map = _new_videos_counts(db, [channel_id])
|
||||
return _serialize(channel, category_map.get(channel_id, []), new_videos_map.get(channel_id, 0))
|
||||
|
|
|
|||
|
|
@ -1,10 +1,11 @@
|
|||
import base64
|
||||
from datetime import datetime, timezone
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy import func, or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import settings
|
||||
from app.core.auth_dependency import require_session
|
||||
from app.db import get_db
|
||||
from app.models.channel import Channel
|
||||
|
|
@ -80,6 +81,7 @@ def get_feed(
|
|||
channel_id: int | None = None,
|
||||
downloaded: bool = False,
|
||||
search: str | None = Query(None, max_length=200),
|
||||
new_only: bool = False,
|
||||
limit: int = Query(DEFAULT_LIMIT, ge=1, le=MAX_LIMIT),
|
||||
cursor: str | None = None,
|
||||
db: Session = Depends(get_db),
|
||||
|
|
@ -97,6 +99,15 @@ def get_feed(
|
|||
)
|
||||
query = query.filter(Video.channel_id.in_(channel_ids_in_category))
|
||||
|
||||
if new_only:
|
||||
since = datetime.now(timezone.utc) - timedelta(days=settings.new_videos_window_days)
|
||||
query = query.filter(Video.published_at >= since)
|
||||
# Align with the sidebar badge count (categories.py), which only
|
||||
# counts subscribed channels: after unsubscribing, a channel's
|
||||
# fresh videos must disappear from "new only" too.
|
||||
subscribed_channel_ids = select(Channel.id).where(Channel.subscribed.is_(True))
|
||||
query = query.filter(Video.channel_id.in_(subscribed_channel_ids))
|
||||
|
||||
if downloaded:
|
||||
query = query.filter(Video.id.in_(_completed_video_ids()))
|
||||
|
||||
|
|
|
|||
|
|
@ -32,8 +32,19 @@ class Settings(BaseSettings):
|
|||
metube_request_timeout_seconds: int = 30
|
||||
|
||||
subscriptions_sync_interval_hours: int = 6
|
||||
videos_sync_interval_minutes: int = 60
|
||||
videos_per_channel_sync: int = 10
|
||||
# Videos sync is no longer scheduled: it is triggered by user activity
|
||||
# (any authenticated request) after this much idle time since the last
|
||||
# completed sync.
|
||||
videos_sync_idle_hours: int = 2
|
||||
# Hard history-depth limit per channel per videos sync: at most the
|
||||
# newest N playlist items are considered; anything older is never
|
||||
# backfilled by design (full channel history is out of scope).
|
||||
videos_backfill_cap: int = 200
|
||||
# Stop playlistItems pagination for a channel once this many already
|
||||
# known video ids are seen in a row (the rest of the playlist is history
|
||||
# we have already synced).
|
||||
videos_known_stop_threshold: int = 50
|
||||
new_videos_window_days: int = 2
|
||||
|
||||
log_level: str = "INFO"
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,11 @@
|
|||
from fastapi import HTTPException, Request
|
||||
|
||||
from app.services.sync_trigger import maybe_trigger_videos_sync
|
||||
|
||||
|
||||
def require_session(request: Request) -> None:
|
||||
if not request.session.get("authenticated"):
|
||||
raise HTTPException(status_code=401, detail="Not authenticated")
|
||||
# Every authenticated request may trigger a background videos sync when it
|
||||
# is due (cheap check: lock + one app_settings row read).
|
||||
maybe_trigger_videos_sync()
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import Boolean, DateTime, String, Text, func
|
||||
from sqlalchemy import BigInteger, Boolean, DateTime, String, Text, func
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from app.db import Base
|
||||
|
|
@ -16,6 +16,7 @@ class Channel(Base):
|
|||
description: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
thumbnail_url: Mapped[str | None] = mapped_column(String, nullable=True)
|
||||
uploads_playlist_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
subscriber_count: Mapped[int | None] = mapped_column(BigInteger, nullable=True)
|
||||
subscribed: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
|
||||
last_synced_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now(), nullable=False)
|
||||
|
|
|
|||
|
|
@ -21,19 +21,9 @@ def _run_subscriptions_sync() -> None:
|
|||
db.close()
|
||||
|
||||
|
||||
def _run_videos_sync() -> None:
|
||||
db = SessionLocal()
|
||||
try:
|
||||
sync.sync_videos(db)
|
||||
except sync.SyncInProgress:
|
||||
logger.info("Scheduled videos sync skipped: already running")
|
||||
except Exception:
|
||||
logger.exception("Scheduled videos sync failed")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def create_scheduler() -> BackgroundScheduler:
|
||||
# Videos sync is not scheduled anymore: it runs on user activity via
|
||||
# services.sync_trigger.maybe_trigger_videos_sync (see require_session).
|
||||
scheduler = BackgroundScheduler(timezone="UTC")
|
||||
scheduler.add_job(
|
||||
_run_subscriptions_sync,
|
||||
|
|
@ -41,10 +31,4 @@ def create_scheduler() -> BackgroundScheduler:
|
|||
hours=settings.subscriptions_sync_interval_hours,
|
||||
id="subscriptions_sync",
|
||||
)
|
||||
scheduler.add_job(
|
||||
_run_videos_sync,
|
||||
"interval",
|
||||
minutes=settings.videos_sync_interval_minutes,
|
||||
id="videos_sync",
|
||||
)
|
||||
return scheduler
|
||||
|
|
|
|||
|
|
@ -52,6 +52,10 @@ def get_videos_sync_status(db: Session) -> dict:
|
|||
return _get_status(db, VIDEOS_SYNC_STATUS_KEY, _videos_lock)
|
||||
|
||||
|
||||
def is_videos_sync_running() -> bool:
|
||||
return _videos_lock.locked()
|
||||
|
||||
|
||||
def sync_subscriptions(db: Session) -> dict:
|
||||
if not _subscriptions_lock.acquire(blocking=False):
|
||||
raise SyncInProgress("Subscriptions sync already in progress")
|
||||
|
|
@ -108,10 +112,13 @@ def sync_subscriptions(db: Session) -> dict:
|
|||
subscribed_ids = [c.youtube_channel_id for c in existing.values() if c.subscribed]
|
||||
try:
|
||||
uploads = youtube_client.fetch_uploads_playlists(credentials, subscribed_ids)
|
||||
for channel_id, uploads_playlist_id in uploads.items():
|
||||
for channel_id, data in uploads.items():
|
||||
channel = existing.get(channel_id)
|
||||
if channel is not None:
|
||||
channel.uploads_playlist_id = uploads_playlist_id
|
||||
if data["uploads_playlist_id"] is not None:
|
||||
channel.uploads_playlist_id = data["uploads_playlist_id"]
|
||||
if data["subscriber_count"] is not None:
|
||||
channel.subscriber_count = data["subscriber_count"]
|
||||
db.commit()
|
||||
except Exception:
|
||||
logger.exception("Failed to fetch uploads playlists during subscriptions sync")
|
||||
|
|
@ -166,21 +173,38 @@ def sync_videos(db: Session) -> dict:
|
|||
)
|
||||
channel_by_youtube_id = {c.youtube_channel_id: c for c in channels}
|
||||
|
||||
# One query for all known video ids (grouped by channel) so there is
|
||||
# no N+1 at the DB level; the YouTube API is still queried per channel.
|
||||
known_ids_by_channel: dict[int, set[str]] = {}
|
||||
for youtube_video_id, channel_id in db.query(Video.youtube_video_id, Video.channel_id).all():
|
||||
known_ids_by_channel.setdefault(channel_id, set()).add(youtube_video_id)
|
||||
|
||||
candidate_video_ids: set[str] = set()
|
||||
for channel in channels:
|
||||
try:
|
||||
video_ids = youtube_client.fetch_playlist_video_ids(
|
||||
credentials, channel.uploads_playlist_id, settings.videos_per_channel_sync
|
||||
# videos_backfill_cap is a hard history-depth limit per
|
||||
# channel: only the newest N playlist items are ever
|
||||
# considered; anything older is not backfilled by design.
|
||||
# The early stop on consecutive known ids saves playlistItems
|
||||
# pages on repeated syncs inside that window.
|
||||
new_ids = youtube_client.fetch_playlist_video_ids_incremental(
|
||||
credentials,
|
||||
channel.uploads_playlist_id,
|
||||
known_ids_by_channel.get(channel.id, set()),
|
||||
settings.videos_backfill_cap,
|
||||
settings.videos_known_stop_threshold,
|
||||
)
|
||||
candidate_video_ids.update(video_ids)
|
||||
candidate_video_ids.update(new_ids)
|
||||
except Exception:
|
||||
logger.exception("Failed to fetch playlist items for channel %s", channel.youtube_channel_id)
|
||||
|
||||
# Details are fetched only for ids that are not in the DB yet (the
|
||||
# incremental fetch above already filters out known ids), so every
|
||||
# returned item is a new video to insert. Existing videos' metadata is
|
||||
# deliberately not refreshed by the videos sync.
|
||||
details = youtube_client.fetch_videos_details(credentials, list(candidate_video_ids))
|
||||
|
||||
existing = {v.youtube_video_id: v for v in db.query(Video).all()}
|
||||
added = 0
|
||||
updated = 0
|
||||
skipped = 0
|
||||
|
||||
for item in details:
|
||||
|
|
@ -197,9 +221,8 @@ def sync_videos(db: Session) -> dict:
|
|||
duration_seconds = parse_iso8601_duration(item["duration_iso8601"])
|
||||
youtube_url = settings.youtube_watch_url_template.format(video_id=item["youtube_video_id"])
|
||||
|
||||
video = existing.get(item["youtube_video_id"])
|
||||
if video is None:
|
||||
video = Video(
|
||||
db.add(
|
||||
Video(
|
||||
youtube_video_id=item["youtube_video_id"],
|
||||
channel_id=channel.id,
|
||||
title=item["title"],
|
||||
|
|
@ -209,18 +232,8 @@ def sync_videos(db: Session) -> dict:
|
|||
duration_seconds=duration_seconds,
|
||||
youtube_url=youtube_url,
|
||||
)
|
||||
db.add(video)
|
||||
existing[item["youtube_video_id"]] = video
|
||||
added += 1
|
||||
else:
|
||||
video.channel_id = channel.id
|
||||
video.title = item["title"]
|
||||
video.description = item["description"]
|
||||
video.thumbnail_url = item["thumbnail_url"]
|
||||
video.published_at = published_at
|
||||
video.duration_seconds = duration_seconds
|
||||
video.youtube_url = youtube_url
|
||||
updated += 1
|
||||
)
|
||||
added += 1
|
||||
|
||||
db.commit()
|
||||
|
||||
|
|
@ -230,12 +243,14 @@ def sync_videos(db: Session) -> dict:
|
|||
"finished_at": _now_iso(),
|
||||
"error": None,
|
||||
"videos_added": added,
|
||||
"videos_updated": updated,
|
||||
# Kept for API compatibility: always 0, because the videos sync
|
||||
# never refreshes metadata of existing videos.
|
||||
"videos_updated": 0,
|
||||
"videos_skipped": skipped,
|
||||
"channels_checked": len(channels),
|
||||
}
|
||||
_save_status(db, VIDEOS_SYNC_STATUS_KEY, result)
|
||||
logger.info("Videos sync completed: added=%d updated=%d skipped=%d", added, updated, skipped)
|
||||
logger.info("Videos sync completed: added=%d skipped=%d", added, skipped)
|
||||
return result
|
||||
|
||||
except Exception as exc:
|
||||
|
|
|
|||
91
backend/app/services/sync_trigger.py
Normal file
91
backend/app/services/sync_trigger.py
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
"""Activity-triggered videos sync.
|
||||
|
||||
The hourly scheduled videos sync is gone; instead, every authenticated request
|
||||
(require_session) calls maybe_trigger_videos_sync(). The check must stay cheap:
|
||||
if the sync is not running and enough idle time has passed since the last
|
||||
completed sync, a background thread is started which calls sync.sync_videos
|
||||
with its own DB session.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import threading
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from app.config import settings
|
||||
from app.db import SessionLocal
|
||||
from app.services import sync
|
||||
from app.services.state import get_setting
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def is_videos_sync_due(finished_at_iso: str | None, now: datetime, idle_hours: int) -> bool:
|
||||
"""Pure predicate: should an automatic videos sync start now?
|
||||
|
||||
True when the last completed sync's finished_at is missing/unparseable or
|
||||
older than `idle_hours`."""
|
||||
if not finished_at_iso:
|
||||
return True
|
||||
if not isinstance(finished_at_iso, str):
|
||||
return True
|
||||
try:
|
||||
finished_at = datetime.fromisoformat(finished_at_iso.replace("Z", "+00:00"))
|
||||
except (ValueError, TypeError):
|
||||
return True
|
||||
if finished_at.tzinfo is None:
|
||||
finished_at = finished_at.replace(tzinfo=timezone.utc)
|
||||
if now.tzinfo is None:
|
||||
now = now.replace(tzinfo=timezone.utc)
|
||||
return now - finished_at >= timedelta(hours=idle_hours)
|
||||
|
||||
|
||||
def _extract_finished_at(raw: str | None) -> str | None:
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
payload = json.loads(raw)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
if not isinstance(payload, dict):
|
||||
return None
|
||||
return payload.get("finished_at")
|
||||
|
||||
|
||||
def _run_videos_sync() -> None:
|
||||
db = SessionLocal()
|
||||
try:
|
||||
sync.sync_videos(db)
|
||||
except sync.SyncInProgress:
|
||||
# Lost the race against a manual sync or another trigger: fine, the
|
||||
# other run will record the status.
|
||||
logger.info("Triggered videos sync skipped: already running")
|
||||
except Exception:
|
||||
logger.exception("Triggered videos sync failed")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def maybe_trigger_videos_sync() -> None:
|
||||
"""Cheap per-request check; never raises. Starts a background videos sync
|
||||
when due, or does nothing when a sync is already running."""
|
||||
try:
|
||||
if sync.is_videos_sync_running():
|
||||
return
|
||||
|
||||
db = SessionLocal()
|
||||
try:
|
||||
raw = get_setting(db, sync.VIDEOS_SYNC_STATUS_KEY)
|
||||
finished_at = _extract_finished_at(raw)
|
||||
if not is_videos_sync_due(finished_at, datetime.now(timezone.utc), settings.videos_sync_idle_hours):
|
||||
return
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
logger.info(
|
||||
"Videos sync due (last finished: %s), starting in background thread", finished_at
|
||||
)
|
||||
threading.Thread(target=_run_videos_sync, daemon=True).start()
|
||||
except Exception:
|
||||
# Never break the request because of the trigger itself.
|
||||
logger.exception("Failed to evaluate videos sync trigger")
|
||||
|
|
@ -9,6 +9,12 @@ logger = logging.getLogger(__name__)
|
|||
|
||||
BATCH_SIZE = 50
|
||||
|
||||
# Hard safety cap for playlistItems pagination: at most this many pages
|
||||
# (BATCH_SIZE items each, i.e. 500 ids) per playlist, so the pageToken loop
|
||||
# can never run away. Raise both the cap and the caller's max_results
|
||||
# together if more is ever needed.
|
||||
MAX_PLAYLIST_PAGES = 10
|
||||
|
||||
|
||||
class YouTubeQuotaExceeded(Exception):
|
||||
pass
|
||||
|
|
@ -102,26 +108,108 @@ def fetch_subscriptions(credentials: Credentials) -> list[dict]:
|
|||
|
||||
|
||||
def fetch_playlist_video_ids(credentials: Credentials, playlist_id: str, max_results: int) -> list[str]:
|
||||
with httpx.Client(timeout=settings.metube_request_timeout_seconds) as client:
|
||||
params = {
|
||||
"part": "contentDetails",
|
||||
"playlistId": playlist_id,
|
||||
"maxResults": min(max_results, 50),
|
||||
}
|
||||
response = client.get(f"{settings.youtube_api_base_url}/playlistItems", params=params, headers=_headers(credentials))
|
||||
if response.status_code == 404:
|
||||
return []
|
||||
_raise_for_status(response)
|
||||
data = response.json()
|
||||
"""Low-level primitive: fetch up to `max_results` playlist item ids,
|
||||
newest first, with no knowledge of what is already synced. The videos sync
|
||||
uses fetch_playlist_video_ids_incremental instead; this stays as the plain
|
||||
paginated helper (kept for tests and any future non-incremental callers)."""
|
||||
video_ids: list[str] = []
|
||||
page_token: str | None = None
|
||||
pages_fetched = 0
|
||||
|
||||
with httpx.Client(timeout=settings.metube_request_timeout_seconds) as client:
|
||||
while pages_fetched < MAX_PLAYLIST_PAGES and len(video_ids) < max_results:
|
||||
params = {
|
||||
"part": "contentDetails",
|
||||
"playlistId": playlist_id,
|
||||
"maxResults": min(max_results - len(video_ids), BATCH_SIZE),
|
||||
}
|
||||
if page_token:
|
||||
params["pageToken"] = page_token
|
||||
|
||||
response = client.get(f"{settings.youtube_api_base_url}/playlistItems", params=params, headers=_headers(credentials))
|
||||
if response.status_code == 404:
|
||||
return []
|
||||
_raise_for_status(response)
|
||||
data = response.json()
|
||||
|
||||
for item in data.get("items", []):
|
||||
video_id = item.get("contentDetails", {}).get("videoId")
|
||||
if video_id:
|
||||
video_ids.append(video_id)
|
||||
|
||||
pages_fetched += 1
|
||||
page_token = data.get("nextPageToken")
|
||||
if not page_token:
|
||||
break
|
||||
|
||||
video_ids = []
|
||||
for item in data.get("items", []):
|
||||
video_id = item.get("contentDetails", {}).get("videoId")
|
||||
if video_id:
|
||||
video_ids.append(video_id)
|
||||
return video_ids
|
||||
|
||||
|
||||
def fetch_playlist_video_ids_incremental(
|
||||
credentials: Credentials,
|
||||
playlist_id: str,
|
||||
known_ids: set[str],
|
||||
max_results: int,
|
||||
stop_threshold: int,
|
||||
) -> list[str]:
|
||||
"""Fetch up to `max_results` previously-unknown video ids from a playlist,
|
||||
stopping pagination early once `stop_threshold` consecutive ids that are
|
||||
already in `known_ids` are encountered. Uploads playlists are ordered
|
||||
newest-first, so a long run of known ids means we reached history that was
|
||||
already synced and there is nothing new further down.
|
||||
|
||||
`max_results` is a hard history-depth limit: at most the newest
|
||||
`max_results` videos of the channel are ever considered, and everything
|
||||
older than that is intentionally not backfilled (we do not mirror full
|
||||
channel history). The early stop on known ids only saves pages on repeated
|
||||
syncs within that window.
|
||||
|
||||
Returns only the unknown ids, in playlist order."""
|
||||
new_ids: list[str] = []
|
||||
seen_new: set[str] = set()
|
||||
consecutive_known = 0
|
||||
page_token: str | None = None
|
||||
pages_fetched = 0
|
||||
|
||||
with httpx.Client(timeout=settings.metube_request_timeout_seconds) as client:
|
||||
while pages_fetched < MAX_PLAYLIST_PAGES and len(new_ids) < max_results:
|
||||
params = {
|
||||
"part": "contentDetails",
|
||||
"playlistId": playlist_id,
|
||||
"maxResults": BATCH_SIZE,
|
||||
}
|
||||
if page_token:
|
||||
params["pageToken"] = page_token
|
||||
|
||||
response = client.get(f"{settings.youtube_api_base_url}/playlistItems", params=params, headers=_headers(credentials))
|
||||
if response.status_code == 404:
|
||||
return new_ids
|
||||
_raise_for_status(response)
|
||||
data = response.json()
|
||||
|
||||
for item in data.get("items", []):
|
||||
video_id = item.get("contentDetails", {}).get("videoId")
|
||||
if not video_id:
|
||||
continue
|
||||
if video_id in known_ids or video_id in seen_new:
|
||||
consecutive_known += 1
|
||||
if consecutive_known >= stop_threshold:
|
||||
return new_ids
|
||||
else:
|
||||
seen_new.add(video_id)
|
||||
new_ids.append(video_id)
|
||||
consecutive_known = 0
|
||||
if len(new_ids) >= max_results:
|
||||
return new_ids
|
||||
|
||||
pages_fetched += 1
|
||||
page_token = data.get("nextPageToken")
|
||||
if not page_token:
|
||||
break
|
||||
|
||||
return new_ids
|
||||
|
||||
|
||||
def fetch_videos_details(credentials: Credentials, video_ids: list[str]) -> list[dict]:
|
||||
results: list[dict] = []
|
||||
|
||||
|
|
@ -171,14 +259,24 @@ def unsubscribe(credentials: Credentials, youtube_subscription_id: str) -> None:
|
|||
_raise_for_status(response)
|
||||
|
||||
|
||||
def fetch_uploads_playlists(credentials: Credentials, channel_ids: list[str]) -> dict[str, str]:
|
||||
result: dict[str, str] = {}
|
||||
def _parse_subscriber_count(statistics: dict) -> int | None:
|
||||
raw = statistics.get("subscriberCount")
|
||||
if raw is None:
|
||||
return None
|
||||
try:
|
||||
return int(raw)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def fetch_uploads_playlists(credentials: Credentials, channel_ids: list[str]) -> dict[str, dict]:
|
||||
result: dict[str, dict] = {}
|
||||
|
||||
with httpx.Client(timeout=settings.metube_request_timeout_seconds) as client:
|
||||
for i in range(0, len(channel_ids), BATCH_SIZE):
|
||||
batch = channel_ids[i : i + BATCH_SIZE]
|
||||
params = {
|
||||
"part": "snippet,contentDetails",
|
||||
"part": "snippet,contentDetails,statistics",
|
||||
"id": ",".join(batch),
|
||||
"maxResults": BATCH_SIZE,
|
||||
}
|
||||
|
|
@ -191,7 +289,10 @@ def fetch_uploads_playlists(credentials: Credentials, channel_ids: list[str]) ->
|
|||
uploads = (
|
||||
item.get("contentDetails", {}).get("relatedPlaylists", {}).get("uploads")
|
||||
)
|
||||
if channel_id and uploads:
|
||||
result[channel_id] = uploads
|
||||
if channel_id:
|
||||
result[channel_id] = {
|
||||
"uploads_playlist_id": uploads or None,
|
||||
"subscriber_count": _parse_subscriber_count(item.get("statistics", {})),
|
||||
}
|
||||
|
||||
return result
|
||||
|
|
|
|||
|
|
@ -33,7 +33,12 @@
|
|||
.category-dot { display: inline-block; width: 8px; height: 8px; border-radius: 50%; flex: none; background: var(--accent); box-shadow: 0 0 0 3px var(--accent-soft); }
|
||||
.category-link { gap: 15px; padding-left: 18px; }
|
||||
.category-link.active .category-dot { background: var(--accent); }
|
||||
.sidebar-count { margin-left: auto; font-size: 11px; color: var(--subtle); }
|
||||
.sidebar-category-row { display: flex; align-items: center; gap: 4px; }
|
||||
.sidebar-category-row .sidebar-link { flex: 1; min-width: 0; }
|
||||
.sidebar-badge { margin-left: auto; flex: none; padding: 1px 8px; border-radius: 999px; background: rgba(255,107,104,.13); color: var(--error); font-size: 11px; font-weight: 650; line-height: 1.5; white-space: nowrap; }
|
||||
.sidebar-badge-link { cursor: pointer; text-decoration: none; transition: background .16s; }
|
||||
.sidebar-badge-link:hover { background: rgba(255,107,104,.26); }
|
||||
.sidebar-badge-link:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
|
||||
.sidebar-add { color: var(--accent); font-size: 13px; margin-top: 4px; }
|
||||
.sidebar-hint { margin: 0; padding: 5px 13px 9px; color: var(--subtle); font-size: 12px; }
|
||||
.sidebar-bottom { margin-top: auto; padding-bottom: 0; }
|
||||
|
|
@ -46,6 +51,8 @@ h1, h2, h3, p { margin-top: 0; }
|
|||
h1 { color: var(--text); font-size: clamp(25px, 2.25vw, 34px); line-height: 1.18; letter-spacing: -.035em; font-weight: 700; margin-bottom: 8px; }
|
||||
h2 { color: var(--text); font-size: 18px; line-height: 1.3; font-weight: 650; }
|
||||
.page-subtitle { margin: 0; color: var(--muted); font-size: 14px; line-height: 1.5; }
|
||||
.page-subtitle a { color: var(--accent); text-decoration: none; font-weight: 650; }
|
||||
.page-subtitle a:hover { text-decoration: underline; }
|
||||
button, .button-primary, .button-secondary, .button-quiet, .button-danger, .button-link { transition: background .16s, color .16s, border-color .16s; }
|
||||
.button-primary, .button-secondary, .button-danger, .button-link { min-height: 42px; display: inline-flex; align-items: center; justify-content: center; gap: 8px; border-radius: 8px; padding: 0 15px; text-decoration: none; font-size: 13px; font-weight: 650; white-space: nowrap; }
|
||||
.button-primary { border: 1px solid var(--accent); background: var(--accent); color: #081521; }
|
||||
|
|
@ -125,6 +132,7 @@ button, .button-primary, .button-secondary, .button-quiet, .button-danger, .butt
|
|||
.channel-title { font-size: 15px; font-weight: 650; text-decoration: none; }
|
||||
.channel-title:hover { color: var(--accent); }
|
||||
.badge { font-size: 11px; color: var(--subtle); }
|
||||
.badge-new { padding: 2px 8px; border-radius: 999px; background: rgba(255,107,104,.13); color: var(--error); font-weight: 650; white-space: nowrap; }
|
||||
.unsubscribe-button { margin-left: auto; min-height: 28px; border: 0; border-radius: 6px; padding: 0 8px; color: var(--subtle); background: transparent; font-size: 11px; }
|
||||
.unsubscribe-button:hover { color: var(--error); background: rgba(255,107,104,.1); }
|
||||
.channel-categories { display: flex; align-items: center; flex-wrap: wrap; gap: 7px; margin-top: 10px; }
|
||||
|
|
@ -271,7 +279,8 @@ button, .button-primary, .button-secondary, .button-quiet, .button-danger, .butt
|
|||
}
|
||||
@media (min-width: 621px) and (max-width: 699px) { .video-grid { grid-template-columns: 1fr; } }
|
||||
@media (pointer: coarse) {
|
||||
.video-actions .button-link, .video-actions .button-secondary, .download-badge, .category-nav button, .local-filter-nav a, .mobile-category-nav a, .chip, .chip-edit, .category-checkbox, .unsubscribe-button { min-height: 44px; }
|
||||
.video-actions .button-link, .video-actions .button-secondary, .download-badge, .category-nav button, .local-filter-nav a, .mobile-category-nav a, .chip, .chip-edit, .category-checkbox, .unsubscribe-button, .sidebar-badge { min-height: 44px; }
|
||||
.sidebar-badge { display: inline-flex; align-items: center; }
|
||||
.reorder-buttons .icon-button { width: 40px; height: 40px; }
|
||||
.popover-create input, .popover-create button { height: 44px; }
|
||||
.popover-create button { width: 44px; }
|
||||
|
|
|
|||
|
|
@ -58,9 +58,11 @@ export interface ChannelDto {
|
|||
description: string | null
|
||||
thumbnail_url: string | null
|
||||
uploads_playlist_id: string | null
|
||||
subscriber_count: number | null
|
||||
subscribed: boolean
|
||||
last_synced_at: string | null
|
||||
category_ids: number[]
|
||||
new_videos_count: number
|
||||
}
|
||||
|
||||
export function getChannel(channelId: number) {
|
||||
|
|
@ -96,6 +98,7 @@ export interface CategoryDto {
|
|||
slug: string
|
||||
sort_order: number
|
||||
channel_count: number
|
||||
new_videos_count: number
|
||||
}
|
||||
|
||||
export function listCategories() {
|
||||
|
|
@ -211,6 +214,8 @@ export function getFeed(
|
|||
downloaded?: boolean
|
||||
search?: string
|
||||
cursor?: string
|
||||
limit?: number
|
||||
newOnly?: boolean
|
||||
} = {},
|
||||
) {
|
||||
const qs = new URLSearchParams()
|
||||
|
|
@ -219,7 +224,9 @@ export function getFeed(
|
|||
if (params.channelId != null) qs.set('channel_id', String(params.channelId))
|
||||
if (params.downloaded) qs.set('downloaded', 'true')
|
||||
if (params.search) qs.set('search', params.search)
|
||||
if (params.newOnly) qs.set('new_only', 'true')
|
||||
if (params.cursor) qs.set('cursor', params.cursor)
|
||||
if (params.limit != null) qs.set('limit', String(params.limit))
|
||||
const suffix = qs.toString() ? `?${qs.toString()}` : ''
|
||||
return request<{ items: FeedVideoDto[]; next_cursor: string | null }>(`/api/feed${suffix}`)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -49,7 +49,10 @@ function Sidebar({ close }: { close: () => void }) {
|
|||
</div>
|
||||
<div className="sidebar-group sidebar-categories">
|
||||
<div className="sidebar-heading">Категории</div>
|
||||
{categories.map((category) => <NavLink key={category.id} to={`/category/${category.id}`} onClick={close} className={({ isActive }) => `sidebar-link category-link ${isActive ? 'active' : ''}`}><span className="category-dot" /><span className="truncate">{category.name}</span><span className="sidebar-count">{category.channel_count}</span></NavLink>)}
|
||||
{categories.map((category) => <div key={category.id} className="sidebar-category-row">
|
||||
<NavLink to={`/category/${category.id}`} onClick={close} className={({ isActive }) => `sidebar-link category-link ${isActive ? 'active' : ''}`}><span className="category-dot" /><span className="truncate">{category.name}</span></NavLink>
|
||||
{category.new_videos_count > 0 && <Link to={`/category/${category.id}?new=1`} onClick={close} className="sidebar-badge sidebar-badge-link" aria-label={`${category.new_videos_count} новых видео в категории «${category.name}»`}>{category.new_videos_count}</Link>}
|
||||
</div>)}
|
||||
{categories.length === 0 && !categoriesQuery.isLoading && <p className="sidebar-hint">Пока нет категорий</p>}
|
||||
<Link to="/settings/categories" onClick={close} className="sidebar-link sidebar-add"><Icon name="plus" size={18} /><span>Новая категория</span></Link>
|
||||
</div>
|
||||
|
|
|
|||
|
|
@ -82,6 +82,7 @@ function ChannelCard({ channel, categories }: Props) {
|
|||
{channel.thumbnail_url ? <img src={channel.thumbnail_url} alt="" loading="lazy" /> : <span className="channel-avatar-placeholder">{channel.title.charAt(0).toUpperCase()}</span>}
|
||||
<div className="channel-info">
|
||||
<div className="channel-head"><Link to={`/channels/${channel.id}/videos`} className="channel-title">{channel.title}</Link>
|
||||
{channel.new_videos_count > 0 && <span className="badge badge-new" title="Недавно вышедшие видео">{channel.new_videos_count} новых</span>}
|
||||
{!channel.subscribed && <span className="badge">отписан</span>}
|
||||
{channel.subscribed && (
|
||||
<button
|
||||
|
|
|
|||
|
|
@ -3,17 +3,19 @@ import { Link, useParams } from 'react-router-dom'
|
|||
import { getChannel, getFeed } from '../api/client'
|
||||
import Icon from '../components/Icon'
|
||||
import VideoCard from '../components/VideoCard'
|
||||
import { formatSubscriberCount } from '../utils/format'
|
||||
|
||||
function ChannelVideos() {
|
||||
const { channelId } = useParams<{ channelId: string }>()
|
||||
const id = Number(channelId)
|
||||
const valid = Number.isInteger(id) && id > 0
|
||||
const channelQuery = useQuery({ queryKey: ['channel', id], queryFn: () => getChannel(id), enabled: valid })
|
||||
const feedQuery = useInfiniteQuery({ queryKey: ['feed', 'channel', id], queryFn: ({ pageParam }) => getFeed({ channelId: id, cursor: pageParam }), initialPageParam: undefined as string | undefined, getNextPageParam: (lastPage) => lastPage.next_cursor ?? undefined, enabled: valid })
|
||||
const feedQuery = useInfiniteQuery({ queryKey: ['feed', 'channel', id], queryFn: ({ pageParam }) => getFeed({ channelId: id, limit: 20, cursor: pageParam }), initialPageParam: undefined as string | undefined, getNextPageParam: (lastPage) => lastPage.next_cursor ?? undefined, enabled: valid })
|
||||
const items = feedQuery.data?.pages.flatMap((page) => page.items) ?? []
|
||||
const subscriberLabel = channelQuery.data ? formatSubscriberCount(channelQuery.data.subscriber_count) : null
|
||||
return <section className="page">
|
||||
<Link to="/channels" className="back-link"><Icon name="arrow" size={17} /> Каналы</Link>
|
||||
<div className="page-heading"><div><p className="eyebrow">Видео канала</p><h1>{channelQuery.data?.title ?? 'Канал'}</h1><p className="page-subtitle">Последние ролики из твоих подписок.</p></div></div>
|
||||
<div className="page-heading"><div><p className="eyebrow">Видео канала</p><h1>{channelQuery.data?.title ?? 'Канал'}</h1><p className="page-subtitle">{subscriberLabel ?? 'Последние ролики из твоих подписок.'}</p></div></div>
|
||||
{(channelQuery.isError || feedQuery.isError || !valid) && <div className="empty-state" role="alert"><h2>Не удалось открыть канал</h2><Link to="/channels" className="button-primary">К списку каналов</Link></div>}
|
||||
{feedQuery.isLoading && <div className="video-grid">{Array.from({ length: 6 }, (_, index) => <div className="video-card" key={index}><div className="skeleton-thumbnail" /><div className="video-info"><div className="skeleton-line wide" /><div className="skeleton-line" /></div></div>)}</div>}
|
||||
{!feedQuery.isLoading && !feedQuery.isError && items.length > 0 && <ul className="video-grid">{items.map((video) => <VideoCard key={video.youtube_video_id} video={video} />)}</ul>}
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ function Feed() {
|
|||
const categoryNumber = categoryId && /^\d+$/.test(categoryId) ? Number(categoryId) : undefined
|
||||
const isLocal = location.pathname === '/local'
|
||||
const isUncategorized = location.pathname === '/uncategorized'
|
||||
const newOnly = categoryNumber !== undefined && searchParams.get('new') === '1'
|
||||
const localCategoryRaw = isLocal ? searchParams.get('category') : null
|
||||
const localCategory = localCategoryRaw && /^\d+$/.test(localCategoryRaw) ? Number(localCategoryRaw) : undefined
|
||||
const localUncategorized = isLocal && searchParams.get('uncategorized') === 'true'
|
||||
|
|
@ -30,12 +31,13 @@ function Feed() {
|
|||
enabled: isUncategorized || (!categoryNumber && !isLocal && !search),
|
||||
})
|
||||
const feedQuery = useInfiniteQuery({
|
||||
queryKey: ['feed', location.pathname, search, localCategory, localUncategorized],
|
||||
queryKey: ['feed', location.pathname, search, localCategory, localUncategorized, newOnly],
|
||||
queryFn: ({ pageParam }) => getFeed({
|
||||
categoryId: isLocal ? localCategory : categoryNumber,
|
||||
uncategorized: isUncategorized || localUncategorized,
|
||||
downloaded: isLocal,
|
||||
search,
|
||||
newOnly,
|
||||
cursor: pageParam,
|
||||
}),
|
||||
initialPageParam: undefined as string | undefined,
|
||||
|
|
@ -63,7 +65,7 @@ function Feed() {
|
|||
onSuccess: () => { queryClient.invalidateQueries({ queryKey: ['sync-status'] }); queryClient.invalidateQueries({ queryKey: ['feed'] }) },
|
||||
})
|
||||
const title = search ? `Поиск: ${search}` : isLocal ? 'На сервере' : isUncategorized ? 'Без категории' : categoryId ? category?.name ?? 'Категория' : 'Все видео'
|
||||
const subtitle = search ? 'Результаты в вашей ленте' : isLocal ? 'Видео, которые вы сохранили для просмотра' : isUncategorized ? `${countQuery.data?.length ?? '…'} каналов пока без категории` : category ? `${category.channel_count} каналов в категории` : `${countQuery.data?.length ?? '…'} подписок в вашей ленте`
|
||||
const subtitle = newOnly ? 'Только новые видео' : search ? 'Результаты в вашей ленте' : isLocal ? 'Видео, которые вы сохранили для просмотра' : isUncategorized ? `${countQuery.data?.length ?? '…'} каналов пока без категории` : category ? `${category.channel_count} каналов в категории` : `${countQuery.data?.length ?? '…'} подписок в вашей ленте`
|
||||
const items = feedQuery.data?.pages.flatMap((page) => page.items) ?? []
|
||||
useEffect(() => {
|
||||
if (feedQuery.isLoading) return
|
||||
|
|
@ -78,7 +80,7 @@ function Feed() {
|
|||
|
||||
return <section className="page">
|
||||
<div className="page-heading">
|
||||
<div><p className="eyebrow">Моя лента</p><h1>{title}</h1><p className="page-subtitle">{subtitle}</p></div>
|
||||
<div><p className="eyebrow">Моя лента</p><h1>{title}</h1><p className="page-subtitle">{subtitle}{newOnly && <> · <Link to={`/category/${categoryNumber}`}>Показать все</Link></>}</p></div>
|
||||
{!isLocal && !search && <button type="button" className="button-secondary refresh-button" onClick={() => syncMutation.mutate()} disabled={syncMutation.isPending || syncStatusQuery.data?.videos.running}><Icon name="refresh" size={17} className={syncStatusQuery.data?.videos.running ? 'spin' : ''} />{syncStatusQuery.data?.videos.running ? 'Обновляется' : 'Обновить'}</button>}
|
||||
</div>
|
||||
<nav className="mobile-category-nav" aria-label="Категории">
|
||||
|
|
|
|||
|
|
@ -9,6 +9,13 @@ export function formatDuration(seconds: number | null): string | null {
|
|||
return `${m}:${String(s).padStart(2, '0')}`
|
||||
}
|
||||
|
||||
const SUBSCRIBERS_FMT = new Intl.NumberFormat('ru-RU', { notation: 'compact', maximumFractionDigits: 1 })
|
||||
|
||||
export function formatSubscriberCount(value: number | null | undefined): string | null {
|
||||
if (value == null) return null
|
||||
return `${SUBSCRIBERS_FMT.format(value)} подписчиков`
|
||||
}
|
||||
|
||||
const RTF = new Intl.RelativeTimeFormat('ru', { numeric: 'auto' })
|
||||
|
||||
export function formatRelativeTime(iso: string): string {
|
||||
|
|
|
|||
26
migrations/versions/0008_channel_subscriber_count.py
Normal file
26
migrations/versions/0008_channel_subscriber_count.py
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
"""channels.subscriber_count
|
||||
|
||||
Revision ID: 0008_channel_subscriber_count
|
||||
Revises: 0007_drop_access_expiry
|
||||
Create Date: 2026-09-17
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision: str = "0008_channel_subscriber_count"
|
||||
down_revision: Union[str, None] = "0007_drop_access_expiry"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# Subscriber count comes from YouTube channels.list (statistics part);
|
||||
# hidden counts stay NULL.
|
||||
op.add_column("channels", sa.Column("subscriber_count", sa.BigInteger(), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("channels", "subscriber_count")
|
||||
|
|
@ -1,10 +1,13 @@
|
|||
import pytest
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.core.auth_dependency import require_session
|
||||
from app.db import get_db
|
||||
from app.main import app
|
||||
from app.models.channel import Channel
|
||||
from app.models.video import Video
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
|
|
@ -29,6 +32,19 @@ def _create_channel(db_session, youtube_channel_id="chanA", title="Channel A"):
|
|||
return channel
|
||||
|
||||
|
||||
def _seed_video(db_session, channel, video_id, published_at):
|
||||
video = Video(
|
||||
youtube_video_id=video_id,
|
||||
channel_id=channel.id,
|
||||
title=f"Video {video_id}",
|
||||
published_at=published_at,
|
||||
youtube_url=f"https://www.youtube.com/watch?v={video_id}",
|
||||
)
|
||||
db_session.add(video)
|
||||
db_session.commit()
|
||||
return video
|
||||
|
||||
|
||||
def test_category_crud(client):
|
||||
created = client.post("/api/categories", json={"name": "Linux"}).json()
|
||||
assert created["name"] == "Linux"
|
||||
|
|
@ -118,3 +134,64 @@ def test_reorder_rejects_mismatched_ids(client):
|
|||
client.post("/api/categories", json={"name": "A"})
|
||||
resp = client.post("/api/categories/reorder", json={"category_ids": [9999]})
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
def test_category_new_videos_count_sums_recent_videos_of_subscribed_channels(client, db_session):
|
||||
now = datetime.now(timezone.utc)
|
||||
category = client.post("/api/categories", json={"name": "Linux"}).json()
|
||||
assert category["new_videos_count"] == 0
|
||||
|
||||
channel_a = _create_channel(db_session, "chanA", "Channel A")
|
||||
channel_b = _create_channel(db_session, "chanB", "Channel B")
|
||||
channel_c = _create_channel(db_session, "chanC", "Channel C")
|
||||
channel_c.subscribed = False
|
||||
db_session.commit()
|
||||
|
||||
client.put(f"/api/channels/{channel_a.id}/categories", json={"category_ids": [category["id"]]})
|
||||
client.put(f"/api/channels/{channel_b.id}/categories", json={"category_ids": [category["id"]]})
|
||||
client.put(f"/api/channels/{channel_c.id}/categories", json={"category_ids": [category["id"]]})
|
||||
|
||||
_seed_video(db_session, channel_a, "vidARecent1", now - timedelta(days=1))
|
||||
_seed_video(db_session, channel_a, "vidARecent2", now - timedelta(hours=2))
|
||||
_seed_video(db_session, channel_a, "vidAOld", now - timedelta(days=30))
|
||||
_seed_video(db_session, channel_b, "vidBRecent", now - timedelta(days=1))
|
||||
# Unsubscribed channel: its videos must not count towards the category.
|
||||
_seed_video(db_session, channel_c, "vidCUnsub", now - timedelta(days=1))
|
||||
|
||||
listed = client.get("/api/categories").json()
|
||||
assert len(listed) == 1
|
||||
assert listed[0]["channel_count"] == 3
|
||||
assert listed[0]["new_videos_count"] == 3
|
||||
|
||||
|
||||
def test_category_new_videos_count_excludes_other_categories(client, db_session):
|
||||
now = datetime.now(timezone.utc)
|
||||
cat1 = client.post("/api/categories", json={"name": "Linux"}).json()
|
||||
cat2 = client.post("/api/categories", json={"name": "IT"}).json()
|
||||
|
||||
channel_a = _create_channel(db_session, "chanA", "Channel A")
|
||||
channel_b = _create_channel(db_session, "chanB", "Channel B")
|
||||
client.put(f"/api/channels/{channel_a.id}/categories", json={"category_ids": [cat1["id"]]})
|
||||
client.put(f"/api/channels/{channel_b.id}/categories", json={"category_ids": [cat2["id"]]})
|
||||
|
||||
# Only an old video in cat1, a recent one in cat2: counts must not leak.
|
||||
_seed_video(db_session, channel_a, "vidAOld", now - timedelta(days=30))
|
||||
_seed_video(db_session, channel_b, "vidBRecent", now - timedelta(days=1))
|
||||
|
||||
listed = {c["id"]: c for c in client.get("/api/categories").json()}
|
||||
assert listed[cat1["id"]]["new_videos_count"] == 0
|
||||
assert listed[cat2["id"]]["new_videos_count"] == 1
|
||||
|
||||
|
||||
def test_category_new_videos_count_zero_without_recent_videos(client, db_session):
|
||||
now = datetime.now(timezone.utc)
|
||||
category = client.post("/api/categories", json={"name": "Linux"}).json()
|
||||
|
||||
channel = _create_channel(db_session)
|
||||
client.put(f"/api/channels/{channel.id}/categories", json={"category_ids": [category["id"]]})
|
||||
_seed_video(db_session, channel, "vidOld", now - timedelta(days=30))
|
||||
|
||||
listed = client.get("/api/categories").json()
|
||||
assert len(listed) == 1
|
||||
assert listed[0]["channel_count"] == 1
|
||||
assert listed[0]["new_videos_count"] == 0
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
|
|
@ -5,6 +7,7 @@ from app.core.auth_dependency import require_session
|
|||
from app.db import get_db
|
||||
from app.main import app
|
||||
from app.models.channel import Channel
|
||||
from app.models.video import Video
|
||||
from app.services import sync
|
||||
from app.services.youtube_client import YouTubeInsufficientScope
|
||||
|
||||
|
|
@ -86,3 +89,54 @@ def test_unsubscribe_insufficient_scope_returns_403(client, db_session, monkeypa
|
|||
def test_unsubscribe_channel_not_found(client):
|
||||
resp = client.post("/api/channels/9999/unsubscribe")
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
def _seed_video(db_session, channel, video_id, published_at):
|
||||
video = Video(
|
||||
youtube_video_id=video_id,
|
||||
channel_id=channel.id,
|
||||
title=f"Video {video_id}",
|
||||
published_at=published_at,
|
||||
youtube_url=f"https://www.youtube.com/watch?v={video_id}",
|
||||
)
|
||||
db_session.add(video)
|
||||
db_session.commit()
|
||||
return video
|
||||
|
||||
|
||||
def test_channel_responses_include_subscriber_and_new_videos_counts(client, db_session):
|
||||
channel = _seed_channel(db_session)
|
||||
channel.subscriber_count = 1200000
|
||||
db_session.commit()
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
_seed_video(db_session, channel, "vidRecent", now - timedelta(days=1))
|
||||
_seed_video(db_session, channel, "vidOld", now - timedelta(days=30))
|
||||
|
||||
resp = client.get("/api/channels")
|
||||
assert resp.status_code == 200
|
||||
payload = resp.json()
|
||||
assert len(payload) == 1
|
||||
assert payload[0]["subscriber_count"] == 1200000
|
||||
# Only the video from 1 day ago is inside the 7-day window.
|
||||
assert payload[0]["new_videos_count"] == 1
|
||||
|
||||
resp_single = client.get(f"/api/channels/{channel.id}")
|
||||
assert resp_single.status_code == 200
|
||||
single = resp_single.json()
|
||||
assert single["subscriber_count"] == 1200000
|
||||
assert single["new_videos_count"] == 1
|
||||
|
||||
|
||||
def test_channel_new_videos_count_zero_without_recent_videos(client, db_session):
|
||||
channel = _seed_channel(db_session)
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
_seed_video(db_session, channel, "vidOld", now - timedelta(days=30))
|
||||
|
||||
resp = client.get("/api/channels")
|
||||
assert resp.status_code == 200
|
||||
payload = resp.json()
|
||||
assert len(payload) == 1
|
||||
assert payload[0]["subscriber_count"] is None
|
||||
assert payload[0]["new_videos_count"] == 0
|
||||
|
|
|
|||
|
|
@ -124,6 +124,133 @@ def test_feed_filters_uncategorized(client, db_session):
|
|||
assert ids == {"vid1", "vid3"}
|
||||
|
||||
|
||||
def _seed_new_only(db_session):
|
||||
"""Two categories: channel A in a category (recent + old video),
|
||||
channel B uncategorized (recent + old video). Published dates are
|
||||
relative to now so the new_videos_window filter is exercised."""
|
||||
now = datetime.now(timezone.utc)
|
||||
channel_a = Channel(youtube_channel_id="chanNewA", title="Channel New A", subscribed=True)
|
||||
channel_b = Channel(youtube_channel_id="chanNewB", title="Channel New B", subscribed=True)
|
||||
db_session.add_all([channel_a, channel_b])
|
||||
db_session.commit()
|
||||
|
||||
category = Category(name="Recent", slug="recent", sort_order=0)
|
||||
db_session.add(category)
|
||||
db_session.commit()
|
||||
|
||||
db_session.execute(channel_categories.insert().values(channel_id=channel_a.id, category_id=category.id))
|
||||
db_session.commit()
|
||||
|
||||
videos = [
|
||||
Video(
|
||||
youtube_video_id="vidRecentA",
|
||||
channel_id=channel_a.id,
|
||||
title="Recent A",
|
||||
published_at=now - timedelta(days=1),
|
||||
youtube_url="https://www.youtube.com/watch?v=vidRecentA",
|
||||
),
|
||||
Video(
|
||||
youtube_video_id="vidOldA",
|
||||
channel_id=channel_a.id,
|
||||
title="Old A",
|
||||
published_at=now - timedelta(days=30),
|
||||
youtube_url="https://www.youtube.com/watch?v=vidOldA",
|
||||
),
|
||||
Video(
|
||||
youtube_video_id="vidRecentB",
|
||||
channel_id=channel_b.id,
|
||||
title="Recent B",
|
||||
published_at=now - timedelta(hours=2),
|
||||
youtube_url="https://www.youtube.com/watch?v=vidRecentB",
|
||||
),
|
||||
Video(
|
||||
youtube_video_id="vidOldB",
|
||||
channel_id=channel_b.id,
|
||||
title="Old B",
|
||||
published_at=now - timedelta(days=30),
|
||||
youtube_url="https://www.youtube.com/watch?v=vidOldB",
|
||||
),
|
||||
]
|
||||
db_session.add_all(videos)
|
||||
db_session.commit()
|
||||
|
||||
return channel_a, channel_b, category, videos
|
||||
|
||||
|
||||
def test_feed_new_only_includes_recent_and_excludes_old(client, db_session):
|
||||
_seed_new_only(db_session)
|
||||
|
||||
resp = client.get("/api/feed?new_only=true").json()
|
||||
|
||||
ids = {i["youtube_video_id"] for i in resp["items"]}
|
||||
assert ids == {"vidRecentA", "vidRecentB"}
|
||||
|
||||
# Without new_only nothing changes: old videos appear as usual.
|
||||
all_ids = {i["youtube_video_id"] for i in client.get("/api/feed").json()["items"]}
|
||||
assert all_ids == {"vidRecentA", "vidOldA", "vidRecentB", "vidOldB"}
|
||||
|
||||
|
||||
def test_feed_new_only_combines_with_category(client, db_session):
|
||||
_, _, category, _ = _seed_new_only(db_session)
|
||||
|
||||
resp = client.get(f"/api/feed?category_id={category.id}&new_only=true").json()
|
||||
|
||||
ids = {i["youtube_video_id"] for i in resp["items"]}
|
||||
assert ids == {"vidRecentA"}
|
||||
|
||||
|
||||
def test_feed_new_only_excludes_unsubscribed_channels(client, db_session):
|
||||
now = datetime.now(timezone.utc)
|
||||
channel = Channel(youtube_channel_id="chanUnsub", title="Channel Unsub", subscribed=False)
|
||||
db_session.add(channel)
|
||||
db_session.commit()
|
||||
db_session.add(
|
||||
Video(
|
||||
youtube_video_id="vidFreshUnsub",
|
||||
channel_id=channel.id,
|
||||
title="Fresh Unsub",
|
||||
published_at=now - timedelta(hours=2),
|
||||
youtube_url="https://www.youtube.com/watch?v=vidFreshUnsub",
|
||||
)
|
||||
)
|
||||
db_session.commit()
|
||||
|
||||
resp = client.get("/api/feed?new_only=true").json()
|
||||
assert resp["items"] == []
|
||||
|
||||
# Regular feed is not filtered by subscription.
|
||||
all_ids = {i["youtube_video_id"] for i in client.get("/api/feed").json()["items"]}
|
||||
assert all_ids == {"vidFreshUnsub"}
|
||||
|
||||
|
||||
def test_feed_new_only_pagination_cursor_works_in_filtered_set(client, db_session):
|
||||
now = datetime.now(timezone.utc)
|
||||
channel = Channel(youtube_channel_id="chanPage", title="Channel Page", subscribed=True)
|
||||
db_session.add(channel)
|
||||
db_session.commit()
|
||||
db_session.add_all(
|
||||
[
|
||||
Video(
|
||||
youtube_video_id=f"vidNew{i}",
|
||||
channel_id=channel.id,
|
||||
title=f"Video New {i}",
|
||||
published_at=now - timedelta(hours=i),
|
||||
youtube_url=f"https://www.youtube.com/watch?v=vidNew{i}",
|
||||
)
|
||||
for i in range(3)
|
||||
]
|
||||
)
|
||||
db_session.commit()
|
||||
|
||||
page1 = client.get("/api/feed?new_only=true&limit=2").json()
|
||||
assert [i["youtube_video_id"] for i in page1["items"]] == ["vidNew0", "vidNew1"]
|
||||
assert page1["next_cursor"] is not None
|
||||
|
||||
page2 = client.get(f"/api/feed?new_only=true&limit=2&cursor={page1['next_cursor']}").json()
|
||||
assert [i["youtube_video_id"] for i in page2["items"]] == ["vidNew2"]
|
||||
assert page2["next_cursor"] is None
|
||||
|
||||
|
||||
def test_feed_filters_downloaded(client, db_session):
|
||||
_, _, _, videos = _seed(db_session)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,3 +1,6 @@
|
|||
from datetime import datetime, timezone
|
||||
|
||||
from app.config import settings
|
||||
from app.models.channel import Channel
|
||||
from app.models.video import Video
|
||||
from app.services import sync
|
||||
|
|
@ -32,7 +35,7 @@ def test_sync_subscriptions_idempotent_and_unsubscribes(monkeypatch, db_session)
|
|||
monkeypatch.setattr(
|
||||
sync.youtube_client,
|
||||
"fetch_uploads_playlists",
|
||||
lambda creds, ids: {cid: f"UU{cid}" for cid in ids},
|
||||
lambda creds, ids: {cid: {"uploads_playlist_id": f"UU{cid}", "subscriber_count": 12345} for cid in ids},
|
||||
)
|
||||
|
||||
result = sync.sync_subscriptions(db_session)
|
||||
|
|
@ -45,6 +48,7 @@ def test_sync_subscriptions_idempotent_and_unsubscribes(monkeypatch, db_session)
|
|||
assert [c.youtube_channel_id for c in channels] == ["chanA", "chanB"]
|
||||
assert all(c.subscribed for c in channels)
|
||||
assert channels[0].uploads_playlist_id == "UUchanA"
|
||||
assert channels[0].subscriber_count == 12345
|
||||
|
||||
# Second sync: chanA disappears from subscriptions, chanC appears.
|
||||
monkeypatch.setattr(
|
||||
|
|
@ -81,6 +85,46 @@ def test_sync_subscriptions_idempotent_and_unsubscribes(monkeypatch, db_session)
|
|||
assert channels["chanC"].subscribed is True
|
||||
|
||||
|
||||
def test_sync_subscriptions_skips_none_subscriber_count(monkeypatch, db_session):
|
||||
channel = Channel(
|
||||
youtube_channel_id="chanA",
|
||||
youtube_subscription_id="subA",
|
||||
title="Channel A",
|
||||
subscribed=True,
|
||||
subscriber_count=12345,
|
||||
)
|
||||
db_session.add(channel)
|
||||
db_session.commit()
|
||||
|
||||
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: _fake_credentials())
|
||||
monkeypatch.setattr(
|
||||
sync.youtube_client,
|
||||
"fetch_subscriptions",
|
||||
lambda creds: [
|
||||
{
|
||||
"youtube_channel_id": "chanA",
|
||||
"youtube_subscription_id": "subA",
|
||||
"title": "Channel A",
|
||||
"description": "d",
|
||||
"thumbnail_url": "t",
|
||||
},
|
||||
],
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
sync.youtube_client,
|
||||
"fetch_uploads_playlists",
|
||||
lambda creds, ids: {cid: {"uploads_playlist_id": f"UU{cid}", "subscriber_count": None} for cid in ids},
|
||||
)
|
||||
|
||||
result = sync.sync_subscriptions(db_session)
|
||||
|
||||
assert result["status"] == "completed"
|
||||
db_session.refresh(channel)
|
||||
assert channel.uploads_playlist_id == "UUchanA"
|
||||
# None in the API response must not overwrite the previously stored count.
|
||||
assert channel.subscriber_count == 12345
|
||||
|
||||
|
||||
def test_sync_in_progress_raises(monkeypatch, db_session):
|
||||
sync._subscriptions_lock.acquire()
|
||||
try:
|
||||
|
|
@ -106,12 +150,14 @@ def _seed_channel(db_session, youtube_channel_id="chanA", uploads_playlist_id="U
|
|||
return channel
|
||||
|
||||
|
||||
def test_sync_videos_adds_and_updates(monkeypatch, db_session):
|
||||
def test_sync_videos_adds_new_videos(monkeypatch, db_session):
|
||||
channel = _seed_channel(db_session)
|
||||
|
||||
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: object())
|
||||
monkeypatch.setattr(
|
||||
sync.youtube_client, "fetch_playlist_video_ids", lambda creds, playlist_id, max_results: ["vid1"]
|
||||
sync.youtube_client,
|
||||
"fetch_playlist_video_ids_incremental",
|
||||
lambda creds, playlist_id, known_ids, max_results, stop_threshold: ["vid1"],
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
sync.youtube_client,
|
||||
|
|
@ -133,12 +179,23 @@ def test_sync_videos_adds_and_updates(monkeypatch, db_session):
|
|||
|
||||
assert result["status"] == "completed"
|
||||
assert result["videos_added"] == 1
|
||||
assert result["channels_checked"] == 1
|
||||
|
||||
video = db_session.query(Video).filter_by(youtube_video_id="vid1").one()
|
||||
assert video.title == "Video One"
|
||||
assert video.duration_seconds == 300
|
||||
assert video.youtube_url == "https://www.youtube.com/watch?v=vid1"
|
||||
|
||||
|
||||
def test_sync_videos_second_run_is_idempotent_and_fetches_no_details(monkeypatch, db_session):
|
||||
channel = _seed_channel(db_session)
|
||||
|
||||
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: object())
|
||||
monkeypatch.setattr(
|
||||
sync.youtube_client,
|
||||
"fetch_playlist_video_ids_incremental",
|
||||
lambda creds, playlist_id, known_ids, max_results, stop_threshold: ["vid1"],
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
sync.youtube_client,
|
||||
"fetch_videos_details",
|
||||
|
|
@ -146,29 +203,142 @@ def test_sync_videos_adds_and_updates(monkeypatch, db_session):
|
|||
{
|
||||
"youtube_video_id": "vid1",
|
||||
"youtube_channel_id": channel.youtube_channel_id,
|
||||
"title": "Video One Updated",
|
||||
"description": "d2",
|
||||
"thumbnail_url": "t2",
|
||||
"title": "Video One",
|
||||
"description": "d",
|
||||
"thumbnail_url": "t",
|
||||
"published_at": "2026-09-10T12:00:00Z",
|
||||
"duration_iso8601": "PT6M",
|
||||
"duration_iso8601": "PT5M",
|
||||
}
|
||||
],
|
||||
)
|
||||
|
||||
result2 = sync.sync_videos(db_session)
|
||||
assert result2["videos_added"] == 0
|
||||
assert result2["videos_updated"] == 1
|
||||
first = sync.sync_videos(db_session)
|
||||
assert first["videos_added"] == 1
|
||||
|
||||
videos = db_session.query(Video).all()
|
||||
assert len(videos) == 1
|
||||
assert videos[0].title == "Video One Updated"
|
||||
# Second run: vid1 is now known, so the incremental fetch reports nothing
|
||||
# new and no details are requested at all (existing videos are not
|
||||
# metadata-refreshed by design).
|
||||
details_calls = []
|
||||
monkeypatch.setattr(
|
||||
sync.youtube_client,
|
||||
"fetch_playlist_video_ids_incremental",
|
||||
lambda creds, playlist_id, known_ids, max_results, stop_threshold: [],
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
sync.youtube_client,
|
||||
"fetch_videos_details",
|
||||
lambda creds, ids: details_calls.append(list(ids)) or [],
|
||||
)
|
||||
|
||||
second = sync.sync_videos(db_session)
|
||||
|
||||
assert second["videos_added"] == 0
|
||||
assert second["videos_updated"] == 0
|
||||
assert details_calls == [[]]
|
||||
assert db_session.query(Video).count() == 1
|
||||
|
||||
|
||||
def test_sync_videos_passes_known_ids_and_backfill_settings(monkeypatch, db_session):
|
||||
channel = _seed_channel(db_session)
|
||||
db_session.add_all(
|
||||
[
|
||||
Video(
|
||||
youtube_video_id="known1",
|
||||
channel_id=channel.id,
|
||||
title="Known 1",
|
||||
published_at=datetime(2026, 9, 1, tzinfo=timezone.utc),
|
||||
youtube_url="https://www.youtube.com/watch?v=known1",
|
||||
),
|
||||
Video(
|
||||
youtube_video_id="known2",
|
||||
channel_id=channel.id,
|
||||
title="Known 2",
|
||||
published_at=datetime(2026, 9, 1, tzinfo=timezone.utc),
|
||||
youtube_url="https://www.youtube.com/watch?v=known2",
|
||||
),
|
||||
]
|
||||
)
|
||||
db_session.commit()
|
||||
|
||||
captured = {}
|
||||
|
||||
def fake_incremental(creds, playlist_id, known_ids, max_results, stop_threshold):
|
||||
captured.update(known_ids=known_ids, max_results=max_results, stop_threshold=stop_threshold)
|
||||
return ["vid1"]
|
||||
|
||||
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: object())
|
||||
monkeypatch.setattr(sync.youtube_client, "fetch_playlist_video_ids_incremental", fake_incremental)
|
||||
monkeypatch.setattr(
|
||||
sync.youtube_client,
|
||||
"fetch_videos_details",
|
||||
lambda creds, ids: [
|
||||
{
|
||||
"youtube_video_id": "vid1",
|
||||
"youtube_channel_id": channel.youtube_channel_id,
|
||||
"title": "Video One",
|
||||
"description": "d",
|
||||
"thumbnail_url": "t",
|
||||
"published_at": "2026-09-10T12:00:00Z",
|
||||
"duration_iso8601": "PT5M",
|
||||
}
|
||||
],
|
||||
)
|
||||
|
||||
result = sync.sync_videos(db_session)
|
||||
|
||||
assert result["videos_added"] == 1
|
||||
assert captured["known_ids"] == {"known1", "known2"}
|
||||
assert captured["max_results"] == settings.videos_backfill_cap
|
||||
assert captured["stop_threshold"] == settings.videos_known_stop_threshold
|
||||
|
||||
|
||||
def test_sync_videos_backfill_cap_ingests_all_candidates(monkeypatch, db_session):
|
||||
channel = _seed_channel(db_session)
|
||||
cap = settings.videos_backfill_cap
|
||||
new_ids = [f"vid{i}" for i in range(cap)]
|
||||
|
||||
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: object())
|
||||
monkeypatch.setattr(
|
||||
sync.youtube_client,
|
||||
"fetch_playlist_video_ids_incremental",
|
||||
lambda creds, playlist_id, known_ids, max_results, stop_threshold: new_ids,
|
||||
)
|
||||
details_calls = []
|
||||
monkeypatch.setattr(
|
||||
sync.youtube_client,
|
||||
"fetch_videos_details",
|
||||
lambda creds, ids: details_calls.append(list(ids))
|
||||
or [
|
||||
{
|
||||
"youtube_video_id": vid,
|
||||
"youtube_channel_id": channel.youtube_channel_id,
|
||||
"title": f"Title {vid}",
|
||||
"description": "d",
|
||||
"thumbnail_url": "t",
|
||||
"published_at": "2026-09-10T12:00:00Z",
|
||||
"duration_iso8601": None,
|
||||
}
|
||||
for vid in ids
|
||||
],
|
||||
)
|
||||
|
||||
result = sync.sync_videos(db_session)
|
||||
|
||||
assert result["videos_added"] == cap
|
||||
assert result["videos_updated"] == 0
|
||||
assert set(details_calls[0]) == set(new_ids)
|
||||
assert db_session.query(Video).count() == cap
|
||||
|
||||
|
||||
def test_sync_videos_skips_unknown_channel(monkeypatch, db_session):
|
||||
_seed_channel(db_session, "chanA")
|
||||
|
||||
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: object())
|
||||
monkeypatch.setattr(sync.youtube_client, "fetch_playlist_video_ids", lambda creds, playlist_id, max_results: [])
|
||||
monkeypatch.setattr(
|
||||
sync.youtube_client,
|
||||
"fetch_playlist_video_ids_incremental",
|
||||
lambda creds, playlist_id, known_ids, max_results, stop_threshold: [],
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
sync.youtube_client,
|
||||
"fetch_videos_details",
|
||||
|
|
@ -189,3 +359,211 @@ def test_sync_videos_skips_unknown_channel(monkeypatch, db_session):
|
|||
|
||||
assert result["videos_skipped"] == 1
|
||||
assert db_session.query(Video).count() == 0
|
||||
|
||||
|
||||
class _FakeResponse:
|
||||
def __init__(self, payload, status_code=200):
|
||||
self.status_code = status_code
|
||||
self._payload = payload
|
||||
self.text = str(payload)
|
||||
|
||||
def json(self):
|
||||
return self._payload
|
||||
|
||||
|
||||
class _FakeCredentials:
|
||||
token = "fake-token"
|
||||
|
||||
|
||||
class _FakeClient:
|
||||
"""Serves canned playlistItems pages through the real incremental fetcher."""
|
||||
|
||||
def __init__(self, pages):
|
||||
self._pages = list(pages)
|
||||
self.requests = []
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *args):
|
||||
return False
|
||||
|
||||
def get(self, url, params=None, headers=None):
|
||||
self.requests.append({"url": url, "params": params})
|
||||
return self._pages.pop(0)
|
||||
|
||||
|
||||
def _page(items, next_token=None):
|
||||
return _FakeResponse(
|
||||
{
|
||||
"items": [{"contentDetails": {"videoId": vid}} for vid in items],
|
||||
**({"nextPageToken": next_token} if next_token else {}),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def test_sync_videos_stops_pagination_on_known_run(monkeypatch, db_session):
|
||||
"""A channel with many known videos in a row after the new ones: the
|
||||
incremental fetch must stop paginating once the known-run threshold is hit
|
||||
and only the new ids must get details."""
|
||||
channel = _seed_channel(db_session)
|
||||
known = [f"known{i}" for i in range(60)]
|
||||
db_session.add_all(
|
||||
[
|
||||
Video(
|
||||
youtube_video_id=vid,
|
||||
channel_id=channel.id,
|
||||
title=f"Known {vid}",
|
||||
published_at=datetime(2026, 9, 1, tzinfo=timezone.utc),
|
||||
youtube_url=f"https://www.youtube.com/watch?v={vid}",
|
||||
)
|
||||
for vid in known
|
||||
]
|
||||
)
|
||||
db_session.commit()
|
||||
|
||||
# Page 1: two new videos then 48 known; page 2 continues with known ids,
|
||||
# so the run crosses 50 on the second page and pagination must stop there
|
||||
# (a third page exists and must never be requested).
|
||||
client = _FakeClient(
|
||||
[
|
||||
_page(["new1", "new2", *known[:48]], next_token="page2"),
|
||||
_page(known[48:58], next_token="page3"),
|
||||
_page(["never-seen"]),
|
||||
]
|
||||
)
|
||||
details_calls = []
|
||||
|
||||
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: _FakeCredentials())
|
||||
monkeypatch.setattr(sync.youtube_client.httpx, "Client", lambda timeout: client)
|
||||
monkeypatch.setattr(
|
||||
sync.youtube_client,
|
||||
"fetch_videos_details",
|
||||
lambda creds, ids: details_calls.append(list(ids))
|
||||
or [
|
||||
{
|
||||
"youtube_video_id": vid,
|
||||
"youtube_channel_id": channel.youtube_channel_id,
|
||||
"title": f"Title {vid}",
|
||||
"description": "",
|
||||
"thumbnail_url": None,
|
||||
"published_at": "2026-09-10T12:00:00Z",
|
||||
"duration_iso8601": None,
|
||||
}
|
||||
for vid in ids
|
||||
],
|
||||
)
|
||||
|
||||
result = sync.sync_videos(db_session)
|
||||
|
||||
assert result["videos_added"] == 2
|
||||
assert len(client.requests) == 2 # stopped on page 2, page 3 never fetched
|
||||
assert set(details_calls[0]) == {"new1", "new2"}
|
||||
assert db_session.query(Video).count() == 62
|
||||
|
||||
|
||||
def test_sync_videos_stops_at_backfill_cap(monkeypatch, db_session):
|
||||
"""More new videos than the cap: pagination stops once the cap is reached
|
||||
and details are fetched for exactly the capped candidates."""
|
||||
channel = _seed_channel(db_session)
|
||||
cap = settings.videos_backfill_cap
|
||||
pages = [_page([f"vid{page * 50 + i}" for i in range(50)], next_token=f"p{page + 2}") for page in range(6)]
|
||||
client = _FakeClient(pages)
|
||||
details_calls = []
|
||||
|
||||
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: _FakeCredentials())
|
||||
monkeypatch.setattr(sync.youtube_client.httpx, "Client", lambda timeout: client)
|
||||
monkeypatch.setattr(
|
||||
sync.youtube_client,
|
||||
"fetch_videos_details",
|
||||
lambda creds, ids: details_calls.append(list(ids))
|
||||
or [
|
||||
{
|
||||
"youtube_video_id": vid,
|
||||
"youtube_channel_id": channel.youtube_channel_id,
|
||||
"title": f"Title {vid}",
|
||||
"description": "",
|
||||
"thumbnail_url": None,
|
||||
"published_at": "2026-09-10T12:00:00Z",
|
||||
"duration_iso8601": None,
|
||||
}
|
||||
for vid in ids
|
||||
],
|
||||
)
|
||||
|
||||
result = sync.sync_videos(db_session)
|
||||
|
||||
assert len(client.requests) == 4 # 4 pages x 50 = cap reached
|
||||
assert result["videos_added"] == cap
|
||||
assert len(details_calls[0]) == cap
|
||||
assert db_session.query(Video).count() == cap
|
||||
|
||||
|
||||
def test_sync_videos_cap_is_hard_history_depth_limit(monkeypatch, db_session):
|
||||
"""videos_backfill_cap is an intentional history-depth limit, not a
|
||||
per-sync batch: a 250-video channel ingests only the newest 200 videos
|
||||
ever; a later sync without new videos stops after one page of known ids
|
||||
and adds nothing; a new video on top of the playlist is picked up while
|
||||
the beyond-the-cap history never surfaces."""
|
||||
channel = _seed_channel(db_session)
|
||||
cap = settings.videos_backfill_cap
|
||||
|
||||
def _details(creds, ids):
|
||||
return [
|
||||
{
|
||||
"youtube_video_id": vid,
|
||||
"youtube_channel_id": channel.youtube_channel_id,
|
||||
"title": f"Title {vid}",
|
||||
"description": "",
|
||||
"thumbnail_url": None,
|
||||
"published_at": "2026-09-10T12:00:00Z",
|
||||
"duration_iso8601": None,
|
||||
}
|
||||
for vid in ids
|
||||
]
|
||||
|
||||
monkeypatch.setattr(sync.google_oauth, "get_credentials", lambda db: _FakeCredentials())
|
||||
monkeypatch.setattr(sync.youtube_client, "fetch_videos_details", _details)
|
||||
|
||||
# Uploads playlist, newest first: vid249 .. vid0.
|
||||
playlist = [f"vid{i}" for i in range(249, -1, -1)]
|
||||
|
||||
def _run(ids):
|
||||
client = _FakeClient(
|
||||
[_page(ids[i : i + 50], next_token=f"page{i // 50}") for i in range(0, len(ids), 50)]
|
||||
)
|
||||
monkeypatch.setattr(sync.youtube_client.httpx, "Client", lambda timeout: client)
|
||||
return sync.sync_videos(db_session), client
|
||||
|
||||
# First sync: 250 videos, only the newest 200 fit under the cap.
|
||||
result1, client1 = _run(playlist)
|
||||
assert result1["videos_added"] == cap
|
||||
assert db_session.query(Video).count() == cap
|
||||
assert len(client1.requests) == 4 # 4 pages x 50 = cap, older pages untouched
|
||||
synced = {v.youtube_video_id for v in db_session.query(Video).all()}
|
||||
assert synced == {f"vid{i}" for i in range(50, 250)}
|
||||
assert "vid49" not in synced # older than the cap: never backfilled by design
|
||||
|
||||
# Second sync, nothing new: one page of 50 known ids in a row stops it.
|
||||
result2, client2 = _run(playlist)
|
||||
assert result2["videos_added"] == 0
|
||||
assert len(client2.requests) == 1
|
||||
assert db_session.query(Video).count() == cap
|
||||
|
||||
# Third sync, one new video on top: only that one is added, the
|
||||
# beyond-the-cap history still does not surface.
|
||||
result3, client3 = _run(["vid250", *playlist])
|
||||
assert result3["videos_added"] == 1
|
||||
assert len(client3.requests) == 2 # page 2 needed to confirm 50 known in a row
|
||||
synced_after = {v.youtube_video_id for v in db_session.query(Video).all()}
|
||||
assert synced_after == {f"vid{i}" for i in range(50, 251)}
|
||||
assert "vid49" not in synced_after
|
||||
|
||||
|
||||
def test_is_videos_sync_running_reflects_lock():
|
||||
sync._videos_lock.acquire()
|
||||
try:
|
||||
assert sync.is_videos_sync_running() is True
|
||||
finally:
|
||||
sync._videos_lock.release()
|
||||
assert sync.is_videos_sync_running() is False
|
||||
|
|
|
|||
159
tests/test_sync_trigger.py
Normal file
159
tests/test_sync_trigger.py
Normal file
|
|
@ -0,0 +1,159 @@
|
|||
import json
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from types import SimpleNamespace
|
||||
|
||||
from app.services import sync_trigger
|
||||
|
||||
|
||||
def _now():
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def test_is_videos_sync_due_without_finished_at():
|
||||
assert sync_trigger.is_videos_sync_due(None, _now(), 2) is True
|
||||
|
||||
|
||||
def test_is_videos_sync_due_after_idle():
|
||||
finished = (_now() - timedelta(hours=3)).isoformat()
|
||||
assert sync_trigger.is_videos_sync_due(finished, _now(), 2) is True
|
||||
|
||||
|
||||
def test_is_videos_sync_due_at_exact_threshold():
|
||||
finished = (_now() - timedelta(hours=2)).isoformat()
|
||||
assert sync_trigger.is_videos_sync_due(finished, _now(), 2) is True
|
||||
|
||||
|
||||
def test_is_videos_sync_due_before_idle():
|
||||
finished = (_now() - timedelta(hours=1)).isoformat()
|
||||
assert sync_trigger.is_videos_sync_due(finished, _now(), 2) is False
|
||||
|
||||
|
||||
def test_is_videos_sync_due_unparseable_finished_at():
|
||||
assert sync_trigger.is_videos_sync_due("not-a-date", _now(), 2) is True
|
||||
assert sync_trigger.is_videos_sync_due(12345, _now(), 2) is True # non-str garbage
|
||||
|
||||
|
||||
def test_is_videos_sync_due_naive_finished_at():
|
||||
naive = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(hours=3)
|
||||
assert sync_trigger.is_videos_sync_due(naive.isoformat(), _now(), 2) is True
|
||||
|
||||
|
||||
def test_extract_finished_at():
|
||||
assert sync_trigger._extract_finished_at(None) is None
|
||||
assert sync_trigger._extract_finished_at("not json") is None
|
||||
assert sync_trigger._extract_finished_at(json.dumps(["a"])) is None
|
||||
assert sync_trigger._extract_finished_at(json.dumps({"status": "running", "finished_at": None})) is None
|
||||
assert (
|
||||
sync_trigger._extract_finished_at(json.dumps({"status": "completed", "finished_at": "2026-09-10T12:00:00+00:00"}))
|
||||
== "2026-09-10T12:00:00+00:00"
|
||||
)
|
||||
|
||||
|
||||
class _FakeDB:
|
||||
def __init__(self):
|
||||
self.closed = False
|
||||
|
||||
def close(self):
|
||||
self.closed = True
|
||||
|
||||
|
||||
def _patch_threads(monkeypatch, started):
|
||||
monkeypatch.setattr(
|
||||
sync_trigger,
|
||||
"threading",
|
||||
SimpleNamespace(Thread=lambda target=None, daemon=None: started.append({"target": target, "daemon": daemon})),
|
||||
)
|
||||
|
||||
|
||||
def test_maybe_trigger_skips_when_sync_running(monkeypatch):
|
||||
started = []
|
||||
monkeypatch.setattr(sync_trigger.sync, "is_videos_sync_running", lambda: True)
|
||||
monkeypatch.setattr(
|
||||
sync_trigger,
|
||||
"SessionLocal",
|
||||
lambda: (_ for _ in ()).throw(AssertionError("DB must not be touched while running")),
|
||||
)
|
||||
_patch_threads(monkeypatch, started)
|
||||
|
||||
sync_trigger.maybe_trigger_videos_sync()
|
||||
|
||||
assert started == []
|
||||
|
||||
|
||||
def test_maybe_trigger_skips_when_not_due(monkeypatch):
|
||||
started = []
|
||||
db = _FakeDB()
|
||||
raw = json.dumps({"status": "completed", "finished_at": (_now() - timedelta(hours=1)).isoformat()})
|
||||
monkeypatch.setattr(sync_trigger.sync, "is_videos_sync_running", lambda: False)
|
||||
monkeypatch.setattr(sync_trigger, "SessionLocal", lambda: db)
|
||||
monkeypatch.setattr(sync_trigger, "get_setting", lambda session, key: raw)
|
||||
_patch_threads(monkeypatch, started)
|
||||
|
||||
sync_trigger.maybe_trigger_videos_sync()
|
||||
|
||||
assert started == []
|
||||
assert db.closed is True
|
||||
|
||||
|
||||
def test_maybe_trigger_starts_thread_when_due_never_synced(monkeypatch):
|
||||
started = []
|
||||
db = _FakeDB()
|
||||
monkeypatch.setattr(sync_trigger.sync, "is_videos_sync_running", lambda: False)
|
||||
monkeypatch.setattr(sync_trigger, "SessionLocal", lambda: db)
|
||||
monkeypatch.setattr(sync_trigger, "get_setting", lambda session, key: None)
|
||||
_patch_threads(monkeypatch, started)
|
||||
|
||||
sync_trigger.maybe_trigger_videos_sync()
|
||||
|
||||
assert len(started) == 1
|
||||
assert started[0]["target"] == sync_trigger._run_videos_sync
|
||||
assert started[0]["daemon"] is True
|
||||
assert db.closed is True
|
||||
|
||||
|
||||
def test_maybe_trigger_starts_thread_when_due_after_idle(monkeypatch):
|
||||
started = []
|
||||
db = _FakeDB()
|
||||
raw = json.dumps({"status": "completed", "finished_at": (_now() - timedelta(hours=5)).isoformat()})
|
||||
monkeypatch.setattr(sync_trigger.sync, "is_videos_sync_running", lambda: False)
|
||||
monkeypatch.setattr(sync_trigger, "SessionLocal", lambda: db)
|
||||
monkeypatch.setattr(sync_trigger, "get_setting", lambda session, key: raw)
|
||||
_patch_threads(monkeypatch, started)
|
||||
|
||||
sync_trigger.maybe_trigger_videos_sync()
|
||||
|
||||
assert len(started) == 1
|
||||
assert started[0]["daemon"] is True
|
||||
|
||||
|
||||
def test_maybe_trigger_never_raises_when_db_unavailable(monkeypatch):
|
||||
started = []
|
||||
monkeypatch.setattr(sync_trigger.sync, "is_videos_sync_running", lambda: False)
|
||||
monkeypatch.setattr(
|
||||
sync_trigger,
|
||||
"SessionLocal",
|
||||
lambda: (_ for _ in ()).throw(RuntimeError("db down")),
|
||||
)
|
||||
_patch_threads(monkeypatch, started)
|
||||
|
||||
sync_trigger.maybe_trigger_videos_sync() # must not raise
|
||||
|
||||
assert started == []
|
||||
|
||||
|
||||
def test_run_videos_sync_swallows_sync_in_progress(monkeypatch):
|
||||
closed = []
|
||||
monkeypatch.setattr(
|
||||
sync_trigger,
|
||||
"SessionLocal",
|
||||
lambda: SimpleNamespace(close=lambda: closed.append(True)),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
sync_trigger.sync,
|
||||
"sync_videos",
|
||||
lambda db: (_ for _ in ()).throw(sync_trigger.sync.SyncInProgress("busy")),
|
||||
)
|
||||
|
||||
sync_trigger._run_videos_sync() # must not raise
|
||||
|
||||
assert closed == [True]
|
||||
|
|
@ -115,6 +115,160 @@ def test_fetch_playlist_video_ids_returns_empty_on_404(monkeypatch):
|
|||
assert result == []
|
||||
|
||||
|
||||
def test_fetch_playlist_video_ids_paginates(monkeypatch):
|
||||
page1 = FakeResponse(
|
||||
200,
|
||||
{
|
||||
"items": [{"contentDetails": {"videoId": f"vid{i}"}} for i in range(50)],
|
||||
"nextPageToken": "page2",
|
||||
},
|
||||
)
|
||||
page2 = FakeResponse(
|
||||
200,
|
||||
{"items": [{"contentDetails": {"videoId": f"vid{i}"}} for i in range(50, 80)]},
|
||||
)
|
||||
seen = []
|
||||
|
||||
class RecordingClient(FakeClient):
|
||||
def get(self, url, params=None, headers=None):
|
||||
seen.append({"url": url, "params": params})
|
||||
return self._responses.pop(0)
|
||||
|
||||
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: RecordingClient([page1, page2]))
|
||||
|
||||
result = youtube_client.fetch_playlist_video_ids(_credentials(), "UUplaylist", 80)
|
||||
|
||||
assert result == [f"vid{i}" for i in range(80)]
|
||||
assert len(seen) == 2
|
||||
assert seen[0]["params"]["maxResults"] == 50
|
||||
assert "pageToken" not in seen[0]["params"]
|
||||
assert seen[1]["params"]["maxResults"] == 30
|
||||
assert seen[1]["params"]["pageToken"] == "page2"
|
||||
assert seen[0]["params"]["part"] == "contentDetails"
|
||||
assert seen[1]["params"]["part"] == "contentDetails"
|
||||
|
||||
|
||||
def test_fetch_playlist_video_ids_stops_at_safety_cap(monkeypatch):
|
||||
page = FakeResponse(
|
||||
200,
|
||||
{"items": [{"contentDetails": {"videoId": "vid"}}], "nextPageToken": "next"},
|
||||
)
|
||||
seen = []
|
||||
|
||||
class RecordingClient(FakeClient):
|
||||
def get(self, url, params=None, headers=None):
|
||||
seen.append(params)
|
||||
return self._responses.pop(0)
|
||||
|
||||
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: RecordingClient([page] * 100))
|
||||
|
||||
result = youtube_client.fetch_playlist_video_ids(_credentials(), "UUplaylist", 10000)
|
||||
|
||||
assert len(result) == youtube_client.MAX_PLAYLIST_PAGES
|
||||
assert len(seen) == youtube_client.MAX_PLAYLIST_PAGES
|
||||
assert "pageToken" not in seen[0]
|
||||
assert seen[1]["pageToken"] == "next"
|
||||
|
||||
|
||||
def _page_items(video_ids, next_token=None):
|
||||
payload = {"items": [{"contentDetails": {"videoId": vid}} for vid in video_ids]}
|
||||
if next_token:
|
||||
payload["nextPageToken"] = next_token
|
||||
return FakeResponse(200, payload)
|
||||
|
||||
|
||||
def test_fetch_playlist_video_ids_incremental_returns_only_unknown(monkeypatch):
|
||||
page = _page_items(["new1", "known1", "new2", "known2"])
|
||||
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: FakeClient([page]))
|
||||
|
||||
result = youtube_client.fetch_playlist_video_ids_incremental(
|
||||
_credentials(), "UUplaylist", known_ids={"known1", "known2"}, max_results=10, stop_threshold=50
|
||||
)
|
||||
|
||||
assert result == ["new1", "new2"]
|
||||
|
||||
|
||||
def test_fetch_playlist_video_ids_incremental_stops_on_known_run(monkeypatch):
|
||||
page1 = _page_items([f"known{i}" for i in range(50)], next_token="page2")
|
||||
page2 = _page_items(["new1", "new2"], next_token="page3")
|
||||
seen = []
|
||||
|
||||
class RecordingClient(FakeClient):
|
||||
def get(self, url, params=None, headers=None):
|
||||
seen.append(params)
|
||||
return self._responses.pop(0)
|
||||
|
||||
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: RecordingClient([page1, page2]))
|
||||
|
||||
result = youtube_client.fetch_playlist_video_ids_incremental(
|
||||
_credentials(), "UUplaylist", known_ids={f"known{i}" for i in range(50)}, max_results=10, stop_threshold=50
|
||||
)
|
||||
|
||||
assert result == []
|
||||
assert len(seen) == 1 # page2 never requested
|
||||
|
||||
|
||||
def test_fetch_playlist_video_ids_incremental_known_run_crosses_pages(monkeypatch):
|
||||
page1 = _page_items(["new1"] + [f"known{i}" for i in range(49)], next_token="page2")
|
||||
page2 = _page_items([f"known{i}" for i in range(49, 60)], next_token="page3")
|
||||
seen = []
|
||||
|
||||
class RecordingClient(FakeClient):
|
||||
def get(self, url, params=None, headers=None):
|
||||
seen.append(params)
|
||||
return self._responses.pop(0)
|
||||
|
||||
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: RecordingClient([page1, page2]))
|
||||
|
||||
result = youtube_client.fetch_playlist_video_ids_incremental(
|
||||
_credentials(), "UUplaylist", known_ids={f"known{i}" for i in range(60)}, max_results=10, stop_threshold=50
|
||||
)
|
||||
|
||||
assert result == ["new1"]
|
||||
assert len(seen) == 2 # stopped mid-page-2, page3 never requested
|
||||
|
||||
|
||||
def test_fetch_playlist_video_ids_incremental_stops_at_cap(monkeypatch):
|
||||
pages = [_page_items([f"vid{page * 50 + i}" for i in range(50)], next_token=f"p{page + 2}") for page in range(6)]
|
||||
seen = []
|
||||
|
||||
class RecordingClient(FakeClient):
|
||||
def get(self, url, params=None, headers=None):
|
||||
seen.append(params)
|
||||
return self._responses.pop(0)
|
||||
|
||||
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: RecordingClient(pages))
|
||||
|
||||
result = youtube_client.fetch_playlist_video_ids_incremental(
|
||||
_credentials(), "UUplaylist", known_ids=set(), max_results=200, stop_threshold=50
|
||||
)
|
||||
|
||||
assert len(result) == 200
|
||||
assert len(seen) == 4
|
||||
|
||||
|
||||
def test_fetch_playlist_video_ids_incremental_empty_on_404(monkeypatch):
|
||||
response = FakeResponse(404, {"error": {"message": "playlist not found"}})
|
||||
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: FakeClient([response]))
|
||||
|
||||
result = youtube_client.fetch_playlist_video_ids_incremental(
|
||||
_credentials(), "UUplaylist", known_ids=set(), max_results=10, stop_threshold=50
|
||||
)
|
||||
|
||||
assert result == []
|
||||
|
||||
|
||||
def test_fetch_playlist_video_ids_incremental_dedupes_new_ids(monkeypatch):
|
||||
page = _page_items(["new1", "new1", "known1"])
|
||||
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: FakeClient([page]))
|
||||
|
||||
result = youtube_client.fetch_playlist_video_ids_incremental(
|
||||
_credentials(), "UUplaylist", known_ids={"known1"}, max_results=10, stop_threshold=50
|
||||
)
|
||||
|
||||
assert result == ["new1"]
|
||||
|
||||
|
||||
def test_fetch_videos_details(monkeypatch):
|
||||
response = FakeResponse(
|
||||
200,
|
||||
|
|
@ -194,13 +348,48 @@ def test_fetch_uploads_playlists_batches(monkeypatch):
|
|||
200,
|
||||
{
|
||||
"items": [
|
||||
{"id": "chanA", "contentDetails": {"relatedPlaylists": {"uploads": "UUchanA"}}},
|
||||
{"id": "chanB", "contentDetails": {"relatedPlaylists": {"uploads": "UUchanB"}}},
|
||||
{
|
||||
"id": "chanA",
|
||||
"contentDetails": {"relatedPlaylists": {"uploads": "UUchanA"}},
|
||||
"statistics": {"subscriberCount": "12345"},
|
||||
},
|
||||
{
|
||||
"id": "chanB",
|
||||
"contentDetails": {"relatedPlaylists": {"uploads": "UUchanB"}},
|
||||
"statistics": {"hiddenSubscriberCount": True},
|
||||
},
|
||||
{"id": "chanC", "contentDetails": {"relatedPlaylists": {"uploads": "UUchanC"}}},
|
||||
{"id": "chanD", "contentDetails": {"relatedPlaylists": {}}, "statistics": {"subscriberCount": "42"}},
|
||||
]
|
||||
},
|
||||
)
|
||||
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: FakeClient([response]))
|
||||
|
||||
result = youtube_client.fetch_uploads_playlists(_credentials(), ["chanA", "chanB"])
|
||||
result = youtube_client.fetch_uploads_playlists(_credentials(), ["chanA", "chanB", "chanC", "chanD"])
|
||||
|
||||
assert result == {"chanA": "UUchanA", "chanB": "UUchanB"}
|
||||
assert result == {
|
||||
"chanA": {"uploads_playlist_id": "UUchanA", "subscriber_count": 12345},
|
||||
"chanB": {"uploads_playlist_id": "UUchanB", "subscriber_count": None},
|
||||
"chanC": {"uploads_playlist_id": "UUchanC", "subscriber_count": None},
|
||||
"chanD": {"uploads_playlist_id": None, "subscriber_count": 42},
|
||||
}
|
||||
|
||||
|
||||
def test_fetch_uploads_playlists_subscriber_count_not_an_int_is_none(monkeypatch):
|
||||
response = FakeResponse(
|
||||
200,
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"id": "chanA",
|
||||
"contentDetails": {"relatedPlaylists": {"uploads": "UUchanA"}},
|
||||
"statistics": {"subscriberCount": "not-a-number"},
|
||||
}
|
||||
]
|
||||
},
|
||||
)
|
||||
monkeypatch.setattr(youtube_client.httpx, "Client", lambda timeout: FakeClient([response]))
|
||||
|
||||
result = youtube_client.fetch_uploads_playlists(_credentials(), ["chanA"])
|
||||
|
||||
assert result == {"chanA": {"uploads_playlist_id": "UUchanA", "subscriber_count": None}}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue