The report was accurate: the worker had exited after deferring the task. It was also late. Before the scheduler processed it, the trigger had resumed the task and its continuation had reached QUEUED. The event and the database state described different moments in the same execution.

What bothered me was that neither side had to be lying. The scheduler had enough information to recognize the ordering, but its existing guard stopped one state too early.

Two timelines meet at the scheduler

A deferrable task releases its worker while waiting for a trigger. Resuming the task and processing the worker’s executor event can progress independently. That permits this ordering:

By the time the older SUCCESS arrives, QUEUED describes the continuation. Treating that pair as an ordinary mismatch can punish a task that has already made progress.

For an ML workflow waiting on an external job or an artifact, deferral is a useful way to avoid occupying a worker during the wait. The same lifecycle gives the control plane more than one event to reconcile. A success from one phase doesn’t tell the scheduler what the resumed phase should do next.

I find it easier to reason about this race with the events in order than with a list of state names. SUCCESS and QUEUED sound contradictory in isolation. Place the defer exit before the trigger, and the contradiction disappears.

The continuation marker narrows the exception

Airflow already handled the resumed task while it was SCHEDULED. The existing check also required next_method, which identifies the continuation, before ignoring the stale executor success. The missing case was a continuation that had advanced to QUEUED.

I extended the state check to include both. Written out, the resume-after-defer exception requires:

(
    ti.state in (TaskInstanceState.SCHEDULED, TaskInstanceState.QUEUED)
    and state == TaskInstanceState.SUCCESS
    and ti.next_method is not None
)

Each condition does work. The current state must belong to the resumed path. The arriving event must be a success. The task must still carry a continuation marker. A queued task without that marker can represent a real mismatch and should retain the existing handling.

Keeping those conditions together mattered more to me than making the check shorter. This patch recognizes a specific ordering; it doesn’t change the scheduler’s general treatment of state disagreements.

Remove the reason for the exception

The regression sets up a queued task with next_method = "execute_callback", then delivers the older success event. The task must remain queued. The scheduler must send no failure callback and record no unexpected metric.

That checks the race, but a guard that ignored every success for every queued task could pass it too. The second half removes next_method and delivers the same event again. This time the scheduler must record scheduler.tasks.killed_externally.

That is the assertion I would look for first in review. The test changes only the evidence that justified the exception, then asks the scheduler to resume its ordinary behavior. It makes an overly broad fix visible without needing a second, unrelated setup.

The implementation is a small extension to a state check. Its scope is easier to trust because the regression tests both the continuation and the case where the continuation marker is absent.

When I read orchestration code now, I ask which phase an event belongs to before deciding whether its state conflicts with the database. A pipeline can spend much longer waiting for external work than running Python. Correctly handing that work back to the scheduler is part of keeping the pipeline reliable.

My Apache Airflow PR #68741 merged on July 2, 2026. State guard and regression test.

Under gevent, overlapping access to the shared pub/sub object could raise ConcurrentObjectUseError. Following the second caller took me out of the polling loop and into AsyncResult.__del__, through result removal, and finally to unsubscribe().

I had been reading cleanup as code that ran after the interesting work. Here it was another active participant in the protocol, issuing I/O while the drainer was still using the connection. That changed where I put the lock, and which kind of lock the consumer needed.

Cleanup is a socket user

The drainer calls get_message() to collect task updates. Result destruction can reach remove_pending_result, then cancel_for, and eventually pubsub.unsubscribe(). Redis-py pub/sub objects don’t support concurrent use, so protecting only the polling call leaves the other caller free to enter the same object.

The patch gives ResultConsumer a shared RLock around pub/sub access, including subscription changes, polling, reconnection, and closing the object. The subscription set is updated under that lock too. The ownership rule needs to cover the consumer’s view of the subscriptions as well as the socket calls.

I find the destructor path useful to keep visible in a diagram. A reviewer looking only for public methods that explicitly subscribe or poll can miss a caller whose entry point is object lifetime.

The callback comes back through the same lock

While holding the lock, the drainer can process a message through on_state_change. If the task is ready, that callback can call cancel_for, which needs to unsubscribe. The consumer reaches the lock again before the outer call has released it.

A plain mutex would make the consumer wait on itself. The reentrant lock lets that nested call proceed under the same ownership. Reconnection also reaches nested consumer operations, so choosing RLock requires following the callbacks, not just counting concurrent callers.

That was the most revealing part of the review for me. The declaration fits on one line; the reason for it sits several calls away. Adding synchronization without tracing those calls would trade a socket race for a deadlock.

Two less visible boundaries matter too. When no pub/sub object exists, drain_events sleeps outside the lock. Otherwise, a caller trying to start consumption would have to wait through an idle sleep. After a fork, on_after_fork creates a fresh lock before cleanup: the inherited lock may have been held by a thread that doesn’t exist in the child.

Test ownership before testing the whole queue

The unit test uses a fake pub/sub object that records overlapping calls. A drainer polls while other threads subscribe and unsubscribe; the fake makes concurrent entry observable. That gives the regression a precise failure condition without relying on a real socket race to happen at the right moment.

Other unit checks cover lock ownership on the relevant paths, sleeping outside the lock when idle, and replacing the inherited lock after a fork. The PR also adds Redis integration tests for concurrent result collection and subscription churn. Those exercise the connection behavior the fake can’t reproduce.

I want both layers. The fake helps explain what broke when a test fails. The integration test checks that the ownership rule still makes sense against Redis, including callers entering through the normal result API.

Serialization has a latency cost

Polling holds the lock while waiting for a message. A new subscription or cancellation must wait for that poll to return. The PR discusses a drainer poll timeout of up to one second; that timeout describes the poll, not an end-to-end subscription latency guarantee.

For a queue feeding embedding jobs or offline model evaluations, I would measure result collection and subscription latency alongside worker execution time. Even after a fast worker finishes, the caller can still be waiting behind a poll in the result consumer.

A design in which the drainer owns the socket and accepts subscription commands through a queue could move that boundary. This patch keeps the existing consumer structure and serializes its callers. It fixes ownership at a smaller scope, with a waiting cost that a deployment can measure.

My Celery PR #10671 merged on September 21, 2026. Implementation and tests.

An embedding or storage failure should leave work for a retry. In Swiftide, filter_cached could write the cache entry before either stage had finished. The next run saw that entry and treated the document as handled. A temporary downstream failure had become a persistent reason to leave the document out of the index.

My first sketch was embarrassingly short: move the cache write to the end. Then I tried to define what “the end” meant for a document that became several chunks, lost an error to filter_errors, and passed through a merged pipeline. Most of the work in this patch lives in that definition.

The cache was claiming success too early

The original sequence was a cache miss, an immediate cache write, then downstream processing. If embedding or storage failed, the cache still remembered the early write. A retry faithfully skipped the source that needed another attempt.

For a retrieval pipeline, that failure is uncomfortable because the index can remain usable. Other documents are searchable, and a model can still produce an answer. The missing source shows up later as missing evidence. When debugging that kind of gap, I’d check whether the source reached the index before spending time on the embedding model or prompt.

The patch makes the filter record which sources need work. Cache writes happen in run(), after the stream finishes without an unhandled error. Nodes reaching the end provide completion candidates; the final decision also checks failures and which cache filters each source actually passed.

Chunking changes the unit of completion

A cache lookup starts with a source document. A chunker can turn it into several derived nodes. The storage backend sees those children, so the pipeline needs to carry the original source identity all the way through.

Derived nodes now preserve the first source ancestor in parent_id. Redis, redb, and DuckDB cache integrations use that identity, and completion candidates are deduplicated. Several successful children can lead to one cache write for their source.

One successful child still doesn’t prove the source finished. This is the regression I find most useful:

The chunker yields a child successfully, then returns an error. With filter_errors, the pipeline can store that child and return successfully. If completion relied only on the returned status and the stored child, the parent would be cached even though its fan-out failed partway through.

The patch records failed source IDs before the error can disappear from the stream. That history survives filter_errors, and the final marking phase excludes the source. The successful child stays stored; the source stays eligible for retry.

I like this test because it makes two plausible success signals insufficient on purpose. A stored node and an Ok result are both real. Neither one says that every part of the source completed.

A merged pipeline needs branch history

There is another way to overstate completion. Suppose a source passes through the right branch’s cache filter, then the branches merge. Marking it in the left branch’s cache would claim work that branch never performed. A later change to the split predicate could then skip the source on the left.

The completion bookkeeping therefore uses pairs of cache registration and source ID. A completed source only updates a cache whose filter it actually passed. Independently built pipelines also link their failure and cache-pass registries when they merge, so the final decision can see both sides’ history.

The rule is easier to review when I write out its conditions:

The distinction between a source ID and a (cache, source) pair was the part I had to be most careful with. Deduplicating sources solves repeated writes after chunking. It doesn’t establish which branch did the work.

Retrying can repeat successful writes

An unhandled stream error exits run() before cache marking. A source that completed earlier in the same run can therefore be processed again next time. Storage and cache updates aren’t one transaction, so downstream writes still need appropriate retry behavior, such as idempotent persistence where the application requires it.

I’m willing to accept that repeated work here. It has a path to recovery. A premature cache entry can keep the unfinished source out of every later run unless someone invalidates it. For an indexing job, that is a much harder failure to notice from the query side.

The final cache writes use bounded concurrency rather than a serial round trip for every source. The trait also gains a required NodeCache.set_by_id method, so custom cache implementations need to support the deferred write. This is an API cost worth making visible when adopting the change.

The merged regressions cover ordinary success and failure, parent deduplication after chunking, partial chunk failure, and branch-specific completion. What I want to see in an indexing review now is the source’s retry path: if any stage loses part of this document, what state makes the next run pick it up again?

My Swiftide PR #1164 merged on September 16, 2026. Changes and regression tests · Pipeline at the merge commit.

A passing test makes me hesitate in a way a failing one doesn’t. Before changing the implementation, I wanted an answer small enough to check without TensorFlow. Fix y, draw the function, and ask what happens just to either side of zero. The line goes straight through the origin. Its slope never disappears.

That left two things to change: the backward pass and the test that had been defending it.

A zero value can still carry a gradient

For positive y, xlogy(x, y) is x * log(y). Holding y = 3.1 makes it a straight line with slope log(3.1). Setting x to zero picks a point on that line; it doesn’t change its slope.

Move the point away from zero and the old gradient agrees with the correct one. Reset it, and the error appears at the origin. This particular bug is easy to miss if a gradient check samples ordinary nonzero inputs.

The numerical check I kept coming back to needs only two nearby evaluations:

from math import isclose, log

y = 3.1
h = 1e-6
slope = ((h * log(y)) - (-h * log(y))) / (2 * h)
assert isclose(slope, log(y), rel_tol=1e-12)

This matters in a training graph because the backward pass sends a sensitivity upstream. If the loss contributes an upstream gradient g, this operation should return g * log(y) with respect to x, before any broadcast reduction. Returning zero erases that contribution. A forward-value check can pass while the optimizer receives the wrong signal.

The forward shortcut leaked into autodiff

The old implementation was compact. It built a mask that was one wherever x was nonzero, then used xlogy(mask, y) as the partial derivative with respect to x.

I can see why that looked reasonable. At a nonzero x, the mask becomes one, and xlogy(1, y) gives log(y). It also reuses the forward operation’s handling of zero. Unfortunately, that is exactly where the derivative gets lost: xlogy(0, y) deliberately returns zero.

The forward operation has a special value convention. Reusing it inside the gradient accidentally turned that convention into a statement about local sensitivity. Those are separate decisions, and the derivative only becomes obvious once you write them down separately.

xlog1py used the same construction. Inside its logarithm domain, y > -1, the function is x * log1p(y), so the slope with respect to x is log1p(y) at the origin too.

The patch uses the logarithms directly:

The surrounding gradient machinery still multiplies by the upstream gradient, reduces broadcast dimensions, and reshapes the result to the input shape. I left the y partials alone. A correction to one derivative shouldn’t acquire unrelated changes to broadcasting or the other input’s gradient.

The regression needed an independent answer

Changing the code made the existing zero-x expectations wrong. I changed those expectations to the logarithms and kept the checks that the y gradients are zero for these inputs. The regression now distinguishes the two partials instead of letting a shared zero hide the difference.

The merged tests exercise float16, float32, and float64. They also cover y = 0 for xlogy and y = -1 for xlog1py, where TensorFlow’s implemented x gradient is negative infinity. These boundary cases have no finite classical derivative. The straight-line argument above applies inside the logarithm domain; the boundary tests pin down the framework’s chosen behavior.

I care about that distinction because a numerical check and an API boundary test answer different questions. At y = 3.1, I can derive a finite slope and compare it with nearby values. At the singularity, I need the test to state the intended behavior explicitly.

This patch changed only a few expressions, but it changed what a green test meant to me. When a gradient expectation looks suspicious, I now want a derivation or an independent numerical check beside it. Reproducing the implementation in the expected value can preserve the same mistake for years.

My TensorFlow PR #119869 merged on July 16, 2026. Implementation and test changes.

The diff shows what changed. I want the write-up to explain why that change is enough, which test would have caught the failure, and what tradeoffs remain.

Each technical post links to its merged PR so readers can check the implementation and tests.