1. Completed: 路线一：/proc 进程句柄

   5 分钟，落空——进程没有长持被删文件的 fd

2. Completed: 路线二：ext4 日志恢复（ext4magic）

   30 分钟，落空——忙盘 3 小时，journal 已滚掉

3. 03

   Current: 路线三：photorec 空闲块雕刻

   20 分钟，成功——雕出 42 个候选，锚点鉴定出删除时刻的那份

4. 04

   Pending: 路线四：旧备份兜底 + 手工补差

   备而未用

📖 How It Happened

A self-hosted service was running on a Debian 12 VPS: a Docker container with a bind-mounted data volume, where the core data was an SQLite database. During a disk cleanup, a directory that “appeared to be a remnant of an old version” was completely rm -rf .

The problem was: that wasn’t an old version directory—it was the data volume of a running container.

To make matters worse, besides the database, this directory also contained two other items:

• A daily backup script (create a tar file → GPG AES256 symmetric encryption → upload to cloud storage)
• The GPG backup passphrase file—the only copy in the entire universe

The encrypted backups for the last 5 days sit quietly on the cloud storage, but not a single one can be decrypted:

gpg: decryption failed: Bad session key

The backups are all there, but the key is gone. The latest plaintext backup I have right now is from a month ago.

🧊 Golden Rule: Preserve the Scene First

Your first reaction upon discovering an accidental deletion should not be “fix it quickly,” but rather to stop any urge to write:

• Do not restart the container/service. Processes may still hold handles to the deleted files; restarting will result in their permanent loss. Even if the handles are gone, the process will continue to run using its in-memory configuration, buying you time to troubleshoot;
• Minimize writes to the target disk. The data blocks of the deleted files are still residing in free blocks, and every write to the disk could overwrite them;
• Always write the output of recovery operations to a different disk.

Next, we followed four approaches in order of increasing cost and lengthening time window.

❌ Approach 1: Retrieve from process handles (5 minutes, unsuccessful)

Under Linux, when a file is deleted but remains open by a process, the data is actually still present and can be copied back exactly as is from /proc :

# 找被删但仍被打开的文件
ls -l /proc/<pid>/fd/ | grep -i deleted

# 找到的话直接复制回来
cp /proc/<pid>/fd/<N> /安全位置/recovered.db

This is the approach with the lowest cost and highest success rate; always try this first.

Unfortunately, this attempt failed: the service’s SQLite connection is established on-demand, and the process does not hold the file handle for an extended period.

❌ Approach 2: ext4 Journal Recovery (30 minutes, unsuccessful)

The ext4 journal temporarily retains metadata for deleted inodes; ext4magic can be used to recover the complete directory tree:

apt-get install -y ext4magic

ext4magic /dev/vdb1 \
  -a "$(date -d '今天 03:00' +%s)" \
  -b "$(date -d '删除前一刻' +%s)" \
  -f "相对挂载点的路径" \
  -r -d /另一块盘/recover

I ran into three pitfalls, all worth noting:

1. -b must use a timestamp prior to deletion. Using a timestamp after deletion will result in Inode not found —the directory inode has already been released, so resolving the path based on the "current" time will naturally fail;
2. Beware of false leads.The Inode found ... is allocated in the output looks like “the directory was actually moved with mv,” but checking with find /挂载点 -inum <号> revealed that it was the parent directory’s inode—the tool only parsed up to the parent level;
3. -m ’s full-scan “magic” mode directly causes a segmentation fault (an old bug in version 0.3.2); don’t count on it as a last resort.

The final conclusion is No undeled inode in journal found : The deletion occurred 3 hours ago, and other services on the same disk are still writing to it—the journal has long since been flushed. On a busy disk, the time window for journal recovery may be only a few minutes—if you’re going down this route, you’ll have to act fast.

✅ Approach 3: photorec Free-Block Carving (20 minutes, successful)

File carving does not rely on any file system metadata; it directly extracts entire files from free blocks on the disk based on file header signatures. SQLite is an ideal target:

• Fixed file headers (SQLite format 3\0 ) with strong signatures;
• Small individual file sizes (tens to hundreds of KB), which are likely stored contiguously and are less prone to fragmentation;
• Databases are repeatedly flushed to disk, so free blocks often contain an entire time series of historical versions.

apt-get install -y testdisk sqlite3

photorec /log /d /另一块盘/carve /cmd /dev/vdb1 \
  partition_none,options,freespace,fileopt,everything,disable,sqlite,enable,search

Parameter Breakdown:

• options,freespace — Scans only free blocks, leaving in-use data untouched, which is much faster;
• fileopt,everything,disable,sqlite,enable — Disables all signatures first, then enables the SQLite signature separately to avoid extracting a bunch of irrelevant files;
• The output directory is on a separate disk.

A 107GB partition was scanned in a few minutes, yielding 42 SQLite files.

🔍 Candidate Analysis: Finding the one with the “deletion time” among the 42 residual files

What was extracted isn’t “a single answer,” but rather dozens of historical snapshots left behind as the database was repeatedly written to disk over the past few months. Which one is the most recent?

Step 1: Batch Checkup—first eliminate the corrupted ones:

for f in carve-hits/*.db; do
  ok=$(sqlite3 "$f" "pragma integrity_check;" | head -1)
  # 再抽几个业务表的行数和关键字段
done

Step 2: Identify monotonically increasing metrics in the business data. The files themselves lack reliable timestamps (the mtime of the extracted files is set to the extraction time), but the business data contains a natural clock:

• Cumulative traffic counters—monotonically increasing within the same billing cycle;
• Number of records—only increases for users/orders;
• Expiration date fields—renewals only push the date forward.

Step 3: Find external time anchors. A status file from a scheduled task on the server (recording which users received expiration reminders on a given day) escaped accidental deletion; the expiration dates it records match the fields in the candidate database exactly—this allows us to pin the candidate to the timeline.

The intersection of these three lines of evidence pinpointed a single candidate that contained both the “most recent renewal” and the “most recent new user.” integrity_check returned “ok,” confirming that all configuration tables were complete and that the key settings matched the production environment. This is the final version from the time of deletion (or extremely close to it).

🚀 Deployment and Validation

mkdir -p /数据卷路径/db
cp candidate-9.db /数据卷路径/db/app.db && chmod 600 /数据卷路径/db/app.db
docker restart <容器>

Don’t limit validation to just “the container is up”: ensure logs contain no database errors, the web service returns a 200 status code, external domain links are reachable, and walk through each core business function one by one. If everything passes, there is zero data loss at the configuration level; the worst-case scenario is merely a slight loss of statistical data from the very last segment before deletion.

🛡️ Cleanup: Don’t Let the Incident Have a Sequel

1. Rebuild the backup chain on the same day:flock for re-entry prevention → Stop containers to ensure SQLite consistency → tar → GPG AES256 → rclone upload → remote verification → local/remote retention policy, and manually test the entire workflow once—writing a backup script without testing it is pointless;
2. Saved a second copy of the new password in a password manager immediately after generating it—the most painful part of this incident wasn’t deleting the database; it was that “all the backups were there, but there was only one key, and it was locked in the same drawer as the database”;
3. Retain temporary files generated during the recovery process for a few days to monitor them; clean them up only after confirming everything is in order.

🧭 Quick Reference Table for Recovery Paths

Try them from top to bottom: the cost increases, and the time window decreases. May you never need to use option 3—but if you do, remember to freeze the scene first.