The failed executions index page loads all failures' stack traces, consuming large amounts of memory

[searls]

I'm currently debugging an issue that's causing HTTP requests to spiral out of control in infinite recursion, which is resulting in each associated failed job to persist a >1MB backtrace. Best I can tell, the failed jobs index controller (/jobs/applications/appname/failed/jobs) is querying Solid Queue's failed executions table and loading all of these into RAM, which is instantly killing my Heroku instance for vastly exceeding its memory quota (seeing >2GB of memory consumption by this route to load the index page).

I haven't seen anyone else mention this as an issue, but it seems like a reasonable approach might be to selectively select the columns needed to render the failed#index page so they're not all being fully loaded into memory.

[rosa]

😮 Wow. I'm really surprised about that huge memory consumption because in all pages that list jobs, the list is paginated, with a default page size of 10, so at most, 10 jobs should be being loaded there, so they shouldn't be using 2GB of memory by any means, unless your backtrace is much larger (in MySQL you're limited by the column size but I don't think there's a limit in PostgreSQL and SQLite).

[searls]

Yeah, I was surprised too because it is paginated, which is why I suspected the full collection is being loaded into memory even if it's not being rendered. Any one page of results would trigger the issue.

I just did a plaintext SQL dump of just this week's failures and it was about 189MB, FWIW. Of course, memory usage in Active Record is greater than the literal text being loaded, so a 1500MB (assuming a 500MB app baseline) memory spike after loading 189MB of results doesn't seem totally unbelievable?

[willtcarey]

I just got hit by this as well. I haven't profiled anything yet, but looking through the code, my guess is that it happens when we're loading the failed job class names.

mission_control-jobs/app/controllers/mission_control/jobs/jobs_controller.rb

Line 7 in 894a3ac

@job_class_names = jobs_with_status.job_class_names

Which loads the first 500 jobs in order to grab the failing class name

mission_control-jobs/lib/active_job/jobs_relation.rb

Lines 183 to 185 in 894a3ac

	def job_class_names(from_first: 500)
	first(from_first).collect(&:job_class_name).uniq
	end

That's the only obvious piece of code I see bypassing the page limit on that page