Skip to content

fix: avoid scheduler SQLite lockups#272

Merged
ModerRAS merged 2 commits intomasterfrom
fix/search-cache-cleanup-lockup
Apr 21, 2026
Merged

fix: avoid scheduler SQLite lockups#272
ModerRAS merged 2 commits intomasterfrom
fix/search-cache-cleanup-lockup

Conversation

@ModerRAS
Copy link
Copy Markdown
Owner

@ModerRAS ModerRAS commented Apr 21, 2026

Summary

  • keep the earlier scheduler lockup fix in place
  • move search pagination cache traffic into a dedicated SearchCache.sqlite database under Env.WorkDirn- initialize the isolated cache database on startup and route cache writes/cleanup through SearchCacheDbContextn- keep the primary Data.sqlite focused on durable bot data so cache churn no longer competes with message ingestion
  • cover the isolated cache path in tests

Why

SQLite serializes writers per database file. The original problem was not just multiple DbContext instances, but that cache cleanup and normal message persistence were contending on the same Data.sqlite file. By splitting disposable search-page cache data into its own SQLite file, cache writes and cleanup no longer block the main message-processing database.

Validation

  • dotnet build TelegramSearchBot.sln -c Release
  • dotnet test TelegramSearchBot.sln -c Release --no-build

Summary by CodeRabbit

  • Bug Fixes

    • Enabled error logging for failed controller executions to improve troubleshooting and monitoring.
  • Chores

    • Optimized SQLite database maintenance during search cache cleanup operations.
    • Improved application initialization to properly set up all database components at startup.

Replace SQLite VACUUM in scheduled cache cleanup with PRAGMA optimize, resolve DbContext instances per task execution, and log controller failures so blocked updates stop going silent.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 21, 2026

📝 Walkthrough

Walkthrough

Adds a new unit test for search-cache cleanup; introduces a dedicated SearchCacheDbContext; switches scheduled tasks and services from direct DataDbContext injection to IServiceProvider-scoped DbContext resolution; adjusts SQLite maintenance and controller error logging; updates SendService and related tests to use search-cache storage APIs.

Changes

Cohort / File(s) Summary
Tests — Scheduler
TelegramSearchBot.Test/Service/Scheduler/SearchPageCacheCleanupTaskTests.cs
New xUnit test that seeds expired and recent SearchPageCache rows, runs SearchPageCacheCleanupTask, asserts expired entries removed and heartbeat invoked. Implements shared in-memory SQLite connection and disposes resources.
Controller Logging
TelegramSearchBot/Executor/ControllerExecutor.cs
Re-enabled active error logging (Serilog) for exceptions thrown during controller execution, logging controller type and update id.
Scheduler Tasks (scoping)
TelegramSearchBot/Service/Scheduler/SearchPageCacheCleanupTask.cs, TelegramSearchBot/Service/Scheduler/WordCloudTask.cs
Replaced constructor-injected DataDbContext with IServiceProvider; create DI scope inside ExecuteAsync() and resolve scoped DbContext. SearchPageCacheCleanupTask now queries scoped SearchCacheDbContext; SQLite maintenance changed from unconditional VACUUM to conditional PRAGMA optimize. WordCloudTask now passes scoped DataDbContext into helper methods and batches user lookups into a single query/dictionary.
Search cache DB model & registration
TelegramSearchBot.Database/Model/SearchCacheDbContext.cs, TelegramSearchBot/Extension/ServiceCollectionExtension.cs, TelegramSearchBot/AppBootstrap/GeneralBootstrap.cs
Added SearchCacheDbContext with DbSet<SearchPageCache> and unique index on UUID. Service registration added second SQLite DB (SearchCache.sqlite) using shared-cache connection string helper. Startup ensures SearchCacheDbContext.Database.EnsureCreatedAsync() during initialization.
SendService & storage
TelegramSearchBot/Service/BotAPI/SendService.cs, TelegramSearchBot/Service/Search/SearchOptionStorageService.cs, TelegramSearchBot.Test/Service/BotAPI/SendServiceTests.cs
SendService now depends on SearchOptionStorageService (not DataDbContext). Keyboard generation delegates cache UUID creation/removal to SearchOptionStorageService. SearchOptionStorageService switched to use SearchCacheDbContext and uses ExecuteDeleteAsync for bulk cleanup. Tests updated to use real SearchCacheDbContext in-memory instance and assert cache entries where applicable.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🐰 I nibbled logs and scoped them neat,

databases warmed with shared-cache heat,
Old pages gone, new lookups run,
Heartbeats counted—cleanup done.
🥕✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 11.54% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the primary change: separating search cache into a dedicated SQLite database to prevent scheduler lockups caused by database contention.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/search-cache-cleanup-lockup

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 21, 2026

🔍 PR检查报告

📋 检查概览

🧪 测试结果

平台 状态 详情
Ubuntu 🔴 失败 测试结果不可用
Windows 🔴 失败 测试结果不可用

📊 代码质量

  • ✅ 代码格式化检查
  • ✅ 安全漏洞扫描
  • ✅ 依赖包分析
  • ✅ 代码覆盖率收集

📁 测试产物

  • 测试结果文件已上传为artifacts
  • 代码覆盖率已上传到Codecov

🔗 相关链接


此报告由GitHub Actions自动生成

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
TelegramSearchBot.Test/Service/Scheduler/SearchPageCacheCleanupTaskTests.cs (1)

24-38: Optional: register DataDbContext as Scoped and drop the inner using on scope-resolved contexts.

AddDbContext defaults to ServiceLifetime.Scoped for a reason — the scope owns the context's lifetime and disposes it when the scope ends. Registering as Transient plus the manual using var dbContext = scope.ServiceProvider.GetRequiredService<DataDbContext>(); pattern causes a second Dispose() call (harmless on DbContext, but unusual) and diverges from how the production code resolves it. Switching to Scoped better mirrors real runtime wiring and simplifies the test.

♻️ Optional refactor
-            services.AddDbContext<DataDbContext>(
-                options => options.UseSqlite(_connection),
-                ServiceLifetime.Transient);
+            services.AddDbContext<DataDbContext>(options => options.UseSqlite(_connection));
...
-            using var scope = _serviceProvider.CreateScope();
-            using var dbContext = scope.ServiceProvider.GetRequiredService<DataDbContext>();
-            dbContext.Database.EnsureCreated();
+            using var scope = _serviceProvider.CreateScope();
+            var dbContext = scope.ServiceProvider.GetRequiredService<DataDbContext>();
+            dbContext.Database.EnsureCreated();

Apply the same using var scope / var dbContext shape to the seed block (lines 37–52) and the verification block (lines 65–69).

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@TelegramSearchBot.Test/Service/Scheduler/SearchPageCacheCleanupTaskTests.cs`
around lines 24 - 38, Change the test DI registration to use the normal scoped
lifetime instead of transient by updating the AddDbContext call (remove
ServiceLifetime.Transient or replace with ServiceLifetime.Scoped) so
DataDbContext is registered as scoped; then, when resolving from
_serviceProvider.CreateScope(), stop wrapping the resolved DataDbContext in an
extra using (replace "using var dbContext =
scope.ServiceProvider.GetRequiredService<DataDbContext>()" with "var dbContext =
scope.ServiceProvider.GetRequiredService<DataDbContext>()") so the scope owns
disposal and the test mirrors production wiring (apply the same using-scope /
var dbContext pattern to the seed and verification blocks that use
DataDbContext).
TelegramSearchBot/Service/Scheduler/WordCloudTask.cs (1)

141-154: Nice improvement — N+1 user lookup replaced with a single batched query.

Correctness looks good: ToDictionaryAsync uses Func selectors so the name-formatting runs client-side on materialized results, no translation risk. One small nit: the $"{u.FirstName} {u.LastName}".Trim() expression is built twice per row; extracting it once is slightly cleaner and avoids a second allocation.

♻️ Optional refactor
-                    var userNames = await dbContext.UserData
-                        .Where(u => topUserIds.Contains(u.Id))
-                        .ToDictionaryAsync(
-                            u => u.Id,
-                            u => string.IsNullOrWhiteSpace($"{u.FirstName} {u.LastName}".Trim())
-                                ? $"用户{u.Id}"
-                                : $"{u.FirstName} {u.LastName}".Trim());
+                    var userNames = await dbContext.UserData
+                        .Where(u => topUserIds.Contains(u.Id))
+                        .ToDictionaryAsync(
+                            u => u.Id,
+                            u => {
+                                var fullName = $"{u.FirstName} {u.LastName}".Trim();
+                                return string.IsNullOrWhiteSpace(fullName) ? $"用户{u.Id}" : fullName;
+                            });
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@TelegramSearchBot/Service/Scheduler/WordCloudTask.cs` around lines 141 - 154,
The ToDictionaryAsync selector in WordCloudTask.cs currently builds
$"{u.FirstName} {u.LastName}".Trim() twice per row; change the selector passed
to ToDictionaryAsync (used to populate userNames) to compute the trimmed
fullName once into a local variable (e.g., fullName = $"{u.FirstName}
{u.LastName}".Trim()) and then use string.IsNullOrWhiteSpace(fullName) ?
$"用户{u.Id}" : fullName so the allocation and trimming happen only once; leave
the rest of the topUsers projection (which uses userNames and
topUserContributors) unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@TelegramSearchBot.Test/Service/Scheduler/SearchPageCacheCleanupTaskTests.cs`:
- Around line 24-38: Change the test DI registration to use the normal scoped
lifetime instead of transient by updating the AddDbContext call (remove
ServiceLifetime.Transient or replace with ServiceLifetime.Scoped) so
DataDbContext is registered as scoped; then, when resolving from
_serviceProvider.CreateScope(), stop wrapping the resolved DataDbContext in an
extra using (replace "using var dbContext =
scope.ServiceProvider.GetRequiredService<DataDbContext>()" with "var dbContext =
scope.ServiceProvider.GetRequiredService<DataDbContext>()") so the scope owns
disposal and the test mirrors production wiring (apply the same using-scope /
var dbContext pattern to the seed and verification blocks that use
DataDbContext).

In `@TelegramSearchBot/Service/Scheduler/WordCloudTask.cs`:
- Around line 141-154: The ToDictionaryAsync selector in WordCloudTask.cs
currently builds $"{u.FirstName} {u.LastName}".Trim() twice per row; change the
selector passed to ToDictionaryAsync (used to populate userNames) to compute the
trimmed fullName once into a local variable (e.g., fullName = $"{u.FirstName}
{u.LastName}".Trim()) and then use string.IsNullOrWhiteSpace(fullName) ?
$"用户{u.Id}" : fullName so the allocation and trimming happen only once; leave
the rest of the topUsers projection (which uses userNames and
topUserContributors) unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b09cb8d8-d177-41aa-b6d1-64d67c26dc66

📥 Commits

Reviewing files that changed from the base of the PR and between 328eb6b and 7a36d90.

📒 Files selected for processing (4)
  • TelegramSearchBot.Test/Service/Scheduler/SearchPageCacheCleanupTaskTests.cs
  • TelegramSearchBot/Executor/ControllerExecutor.cs
  • TelegramSearchBot/Service/Scheduler/SearchPageCacheCleanupTask.cs
  • TelegramSearchBot/Service/Scheduler/WordCloudTask.cs

Move search pagination cache traffic into a dedicated SearchCache.sqlite database, initialize it at startup, and route cache writes/cleanup through the isolated context so the primary SQLite file is no longer contended by cache maintenance.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
TelegramSearchBot/Service/Scheduler/SearchPageCacheCleanupTask.cs (1)

11-12: 🛠️ Refactor suggestion | 🟠 Major

Use the required singleton lifetime for scheduled tasks.

This task implements IScheduledTask, but the injectable lifetime is still Transient. The constructor now only captures IServiceProvider and ILogger, and the DbContext is resolved inside ExecuteAsync(), so switching to singleton should not capture scoped state.

Suggested change
-    [Injectable(Microsoft.Extensions.DependencyInjection.ServiceLifetime.Transient)]
+    [Injectable(Microsoft.Extensions.DependencyInjection.ServiceLifetime.Singleton)]
     public class SearchPageCacheCleanupTask : IScheduledTask {

As per coding guidelines, **/Service/Scheduler/**/*.cs: “All scheduled tasks must implement IScheduledTask and use [Injectable(ServiceLifetime.Singleton)] attribute”.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@TelegramSearchBot/Service/Scheduler/SearchPageCacheCleanupTask.cs` around
lines 11 - 12, The SearchPageCacheCleanupTask is registered as Transient but
must be a singleton per guidelines; update the Injectable attribute on the
SearchPageCacheCleanupTask class (which implements IScheduledTask) to use
Microsoft.Extensions.DependencyInjection.ServiceLifetime.Singleton instead of
Transient, keeping the constructor that accepts IServiceProvider and ILogger
as-is since DbContext is resolved inside ExecuteAsync().
🧹 Nitpick comments (1)
TelegramSearchBot.Database/Model/SearchCacheDbContext.cs (1)

17-19: Add an index for cleanup scans.

Cleanup deletes filter by CreatedTime, but this model only indexes UUID. Add a non-unique CreatedTime index so cache cleanup does not full-scan a high-churn table and hold the cache DB write lock longer than needed.

Suggested change
             modelBuilder.Entity<SearchPageCache>()
                 .HasIndex(cache => cache.UUID)
                 .IsUnique();
+
+            modelBuilder.Entity<SearchPageCache>()
+                .HasIndex(cache => cache.CreatedTime);

Based on learnings, search results caching should be implemented via SearchPageCacheCleanupTask.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@TelegramSearchBot.Database/Model/SearchCacheDbContext.cs` around lines 17 -
19, The SearchPageCache entity currently only has a unique index on UUID which
causes cleanup scans that filter by CreatedTime to full-scan the table; add a
non-unique index on CreatedTime in the EF model configuration (in the same block
where modelBuilder.Entity<SearchPageCache>() is configured) by adding a
HasIndex(cache => cache.CreatedTime) without IsUnique(), so the cleanup task
(SearchPageCacheCleanupTask) can efficiently delete by CreatedTime without long
DB write-lock full scans.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@TelegramSearchBot/Service/Scheduler/SearchPageCacheCleanupTask.cs`:
- Around line 11-12: The SearchPageCacheCleanupTask is registered as Transient
but must be a singleton per guidelines; update the Injectable attribute on the
SearchPageCacheCleanupTask class (which implements IScheduledTask) to use
Microsoft.Extensions.DependencyInjection.ServiceLifetime.Singleton instead of
Transient, keeping the constructor that accepts IServiceProvider and ILogger
as-is since DbContext is resolved inside ExecuteAsync().

---

Nitpick comments:
In `@TelegramSearchBot.Database/Model/SearchCacheDbContext.cs`:
- Around line 17-19: The SearchPageCache entity currently only has a unique
index on UUID which causes cleanup scans that filter by CreatedTime to full-scan
the table; add a non-unique index on CreatedTime in the EF model configuration
(in the same block where modelBuilder.Entity<SearchPageCache>() is configured)
by adding a HasIndex(cache => cache.CreatedTime) without IsUnique(), so the
cleanup task (SearchPageCacheCleanupTask) can efficiently delete by CreatedTime
without long DB write-lock full scans.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8604a159-c20a-48a2-905c-a6d39d6f66a0

📥 Commits

Reviewing files that changed from the base of the PR and between 7a36d90 and 27edc1c.

📒 Files selected for processing (8)
  • TelegramSearchBot.Database/Model/SearchCacheDbContext.cs
  • TelegramSearchBot.Test/Service/BotAPI/SendServiceTests.cs
  • TelegramSearchBot.Test/Service/Scheduler/SearchPageCacheCleanupTaskTests.cs
  • TelegramSearchBot/AppBootstrap/GeneralBootstrap.cs
  • TelegramSearchBot/Extension/ServiceCollectionExtension.cs
  • TelegramSearchBot/Service/BotAPI/SendService.cs
  • TelegramSearchBot/Service/Scheduler/SearchPageCacheCleanupTask.cs
  • TelegramSearchBot/Service/Search/SearchOptionStorageService.cs

@ModerRAS ModerRAS merged commit 72eecc5 into master Apr 21, 2026
6 checks passed
@ModerRAS ModerRAS deleted the fix/search-cache-cleanup-lockup branch April 21, 2026 11:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant