How LeadTables’ Sequential Job Processing Works
I have a vision for LeadTables that one day, you’ll be able to program complex “enrichment chains” where you pre-program a sequence of Data Modules and “start the chain” and it’ll just run one type of enrichment after another per your programmed sequence.
But alas, that’s not how it works right now 🙂
Right now, when you queue up enrichment jobs, they build their leads lists at the moment of queuing, so you can’t reliably queue up a bunch of enrichments in advance, if the subsequent ones will rely on the output of previous ones for filtering.
For example, suppose you wanted this…
- “AI Prompt” Data Module to produce a custom lead score
- Filter your table with all leads with a custom lead score of 7 or higher
- Different “AI Prompt” Data Module writes a personalization line
As of today, if you were to queue up that first one and second one at the same time, when the second one runs, none of the leads from the first one will be included in the result set, because at the moment of queueing, they didn’t have custom lead scores generated yet.
Same deal with uploads; if you queue an upload and then queue a formula while it’s still uploading, you may find that many/all of the leads that got added from the upload don’t get the formula run for them.
So for this stage of LeadTables, it’s much safer to…
- Queue a job
- and let it fully run
- before queuing another.
But that said, here’s how LeadTables works in terms of sequentialness…
The LeadTables Global Job Queue
My servers can only handle so much load, and additionally, many of the external data providers we let you use directly in the platform have rate limits that I have to operate within.
(For example, Snov’s rate limit only allows processing like 15 leads per minute across ALLLLLL LeadTables users 🤮)
((Little ole me’s just a small baby SaaS, so until I can profitably afford to be on multi-thousand-dollar-a-month enterprise plans with data providers, this kind of thing will be a challenge))
Because of this, whenever you queue up a job, it gets added to the queue shared across all users.
And additionally, you personally can only have one job at a time running across all LeadTables in your Teamspace.
So even though most jobs instantly process for now while there aren’t many of you in the platform, know that this is not actually technically how things work — you just got lucky because no one already had jobs running when you went to process yours 🙂
Currently, my database can handle ~1,000,000 records per hour across all users, so things like formulas should hopefully always be pretty much instant-ish for the foreseeable future.
TL/DR: When you queue up a job, if a lot of other jobs are ahead of you, you might need to wait a bit before it gets processed.
☝️ NB: Keep An Eye Out For “Stalled Jobs!”
This whole “jobs queue” part of LeadTables is one of the most complicated parts of the code, and managing stalled jobs is extra-challenging from a code perspective.
So if you ever run a job that starts processing but then inexplicably stops and doesn’t move for like 5+ minutes — even when you go to the LeadTable’s jobs page and refresh the browser tab — it’s probably “stalled.”
Eventually I’ll auto-kill stalled jobs but at the time of writing it’s a manual process.
In the next lesson I’ll show you how to cancel a stalled in-progress job.