Log in

No account? Create an account

Previous Entry | Next Entry

Gearman and TheSchwartz, explained!

Gearman and TheSchwartz are two job scheduling engines that LiveJournal and Dreamwidth and everything on the whole collected codebase use, for different tasks as appropriate.

xb95 that could work. and ofc, now I'm wondering if we can't just use theschwartz workesr instead of gearman workers.
xb95 then it's just one level of abstraction
durandal goes to look things up
goesboom bliiiiink.
goesboom (Jazz) Someone someday want to explain things to me as far as this goes?
durandal ok, stupidity setting in, but, considering how much I DONT know about em (cause I am uninformed like that), which one is actually better suited to running the xpost?
exor674 makes a bug for something at least to do in the meantime
xb95 damn kareila is gone
exor674 .. oh wait it IS evaled
exor674 hrm then
xb95 durandal: TheSchwartz, really. it's durable - which means that if it fails, it retries. or if a worker disappears, it retries.
xb95 goesboom: What it means, wrt Gearman/TheSchwartz?
goesboom xb95: Yes!
durandal mark: ok... that being the case, why are gearman in the mix?
xb95 Well, both have pros and cons, right. TheSchwartz is very stable, solid, but much like a rock. It's a lot of overhead (it's heavy) and a lot of setup and I have a lower impression of it overall.
xb95 But Gearman is very fast and basically lets you just say "hey Gearman, do something for me" and it does.
xb95 Of course, if it happens to crash while doing it, your thing goes away.
xb95 Amusingly, the Gearman team is adding persistent queue support which makes it basically equivalent to TheSchwartz...
exor674 xb95: there we go... that's the issue with it not... failing properly
exor674 as soon as Bugsy wakes up
xb95 And even more confusing, 'Gearman' was named because it's an anagram of 'Manager', as it's a job manager system. 'TheSchwartz' was named after Brad's manager, Mike Schwartz, the VP of Engineering at Six Apart.
Bugsy New Dreamwidth Development bug 1635 filed by anall@andreanall.com.
Bugsy Bug http://bugs.dwscoalition.org/show_bug.cgi?id=1635 enhancement, P-, -undecided-, nobody, NEW, crossposter: get_proxy call is not safe if xmlrpc call failed.
goesboom (Jazz) So, each does what?
JD oh it's not just a spaceballs reference?
goesboom (Jazz) I know absolutely nothing about any of this. But I would sorta like to.
durandal ok, so how hard is it to slave gear to a scwartz master?
xb95 JD: It is not a Spaceballs reference. :)
xb95 Or, well, it could be I suppose. But it was named after Mike.
durandal mark, with Brad, you never know
xb95 goesboom: Both of them are ways of doing asyncronous jobs. Which is like, if you call up your friend and say, "hey, write a paper for me" and then you go play video games while he does. Eventually, when the job is done, he gives you the paper and you can use it.
xb95 The difference between the two is, if your friend pops off to China and becomes a monk, in Gearman your paper never gets done. It's gone. But in TheSchwartz, you get a new friend who will make sure the paper gets done
goesboom nods.
xb95 We use it for things that can run asyncronously; we don't care if they get done *now*, we just want them to get done *sometime*. So we can tell the user "your entry is posted!" and then fire off a job to do the xpost.
exor674 also TheSchwartz jobs are more "do this and tell me later", while Gearman jobs are "do this somewhere else"
xb95 yeah
xb95 dre is more accurate
goesboom Ah, alright. That makes enough sense for our little layman's brain. :)
goesboom Thank you.
goesboom (Jazz) bounces.
xb95 durandal: very tru.e
xb95 durandal: It was proably double entendre!
xb95 durandal: and as for how hard, it's not hard to have a schwartz job fire off a gearman job, but I think the usefulness of doing that is low. MIght as well just run schwartz jobs in lots of places.
durandal mark: *nod*

And that concludes this episode of [info - personal] xb95 and [info - personal] exor674 Explain It All!
Gone away, gone ahead,
Echoes roll unanswered.
Empty, open, dusty, dead.
Why have all the Weyrfolk fled?

Where have dragons gone together
Leaving weyrs to wind and weather,
Setting herdbeasts free of tether;
Gone, our safeguards, gone, but whither?

Have they flown to some new weyr
Where cruel Threads some others fear?
Are they worlds away from here?
Why, oh why the empty weyr?

-- "The Question Song", Anne McCaffrey
Powered by LiveJournal.com
Designed by yoksel