Skip to content
Notifications
Clear all

Help: The database cleanup job keeps failing - out of disk space

20 Posts
19 Users
0 Reactions
74 Views
(@crm_hopper_2025_new)
Honorable Member
Joined: 4 months ago
Posts: 365
 

You're right about it checking the specific filesystem, but that line is often buried. And even if you spot it, the percentage it's checking against is sometimes a global config, not the actual job requirement.

The moving target problem isn't just VACUUM FULL. It's that the job often does a preliminary sort or index rebuild *before* the delete, and that temporary working set needs its own space. So you clear the threshold, the job starts, and then immediately trips on the next step. I've seen it need double the initially logged requirement to actually finish.

What's the point of a preflight check that doesn't account for its own operational overhead?



   
ReplyQuote
(@cost_optimizer_99)
Prominent Member
Joined: 5 months ago
Posts: 632
 

Wrong logs. Everyone's focused on the database. The issue is the binary logs for Xray's internal queue (`$JFROG_HOME/xray/var/log/queue.log.*`). They bloat when cleanup fails and get purged *after* a successful run, creating a circular dependency.

You need to manually truncate those first. Find the mount point with `df -h $JFROG_HOME/xray/var/log` and clear 20% there, not just the database volume. That's usually enough headroom for the preflight check to pass and the real cleanup to start.


show the math


   
ReplyQuote
(@bench_beast)
Noble Member
Joined: 4 months ago
Posts: 723
Topic starter  

You've only checked the Postgres logs. The critical failure line is in the xray-server.log, not the database logs. Look for a line containing "requires" and "bytes free".

That's the exact threshold you need to hit. The config you posted is irrelevant.


Benchmarks don't lie.


   
ReplyQuote
(@docker_diver)
Honorable Member
Joined: 4 months ago
Posts: 496
 

Oh, that's a good catch. I was only checking the PG logs for errors.

Where does xray-server.log live? Is it usually in `/var/log/` under the Xray install directory? And is that "requires X bytes free" line pretty clear once you find it? I want to look for it but I'm not sure where to start.


Containers are magic, but I want to know how the magic works.


   
ReplyQuote
(@gregm)
Honorable Member
Joined: 3 months ago
Posts: 424
 

Checking Postgres logs for this is barking up the wrong tree. The generic "Cleanup process failed" in your Xray logs is the starting symptom, not the root cause.

The actual failure threshold is in `$JFROG_HOME/xray/var/log/xray-server.log`. Look for a line with "requires" and "bytes free". That number isn't a suggestion, it's a hard stop before the job even tries to run.

But hitting that number is just the first hoop. If your `events` table isn't partitioned, manually deleting rows to clear space won't immediately shrink the table file on disk. The cleanup job likely still needs to run a VACUUM to reclaim that space, which requires its own overhead. So you can clear enough to start the job, only to have it immediately fail again when it tries to process.


Trust but verify


   
ReplyQuote
Page 2 / 2