Your ingestion job is green. Your job assistant is still quoting yesterday's requirements from a posting whose id hasn't changed.

Only-new monitoring is an identity feed; refreshing existing answers requires comparing their content.

Lab note 136: can a daily job feed keep role answers current? The catch comes first: the API provides no update or closed-job timestamp. We need repeated observations to notice a rewrite, and those observations can't tell us when the employer made it.

What did your daily refresh actually refresh?

The job-board Actor's onlyNewSinceLastRun option emits postings unseen by an earlier run with the same query shape. The first run returns everything.

That's useful for new-role alerts. Using it to refresh requirements in retrieval doesn't work: an existing posting can change without becoming a new posting.

We'd make the snapshot setting explicit, even though false is the default.

This fetches a bounded snapshot with description text available for comparison:

import hashlib import json import os from apify_client import ApifyClient run_input = { "mode": "getJobs", "companySlugs": ["wealthfront"], "boardRegion": "global", "maxItems": 500, "includeDescription": True, "includeHtml": False, "onlyNewSinceLastRun": False, } scope = (run_input["boardRegion"], run_input["companySlugs"][0]) client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor( "vexaloom-labs/lever-job-board-postings-by-company-slug" ).call(run_input=run_input) if run["status"] != "SUCCEEDED": raise RuntimeError(f"Snapshot unavailable: {run['status']}") items = list(client.dataset(run["defaultDatasetId"]).iterate_items()) 

This example pins a canonical company slug and a region. Those input values supply the namespace around each posting ID.

For multiple boards, preserve the actual company and region associated with each record. In particular, auto is a routing choice, so it shouldn't become the region in your document identity.

The distinction we're building around looks like this:

board region + company slug + posting ID | stable document | selected facts -> content hash | changed hash -> replace version 

A successful fetch gets observations into the pipeline. It doesn't update your index by itself. The green checkmark has a very narrow job description.

Keep observation time out of the version hash

Hashing the entire row looks convenient until scrapedAt changes.

That field records an observation. Including it in the content hash can turn an unchanged role into another embedding job, consume the ingestion token budget, and leave duplicate versions competing in retrieval.

We'd hash an explicit factual projection instead. The following function also retains timestamps separately for provenance:

FACT_FIELDS = ( "title", "categories", "employmentType", "isRemote", "descriptionPlain", "salaryRange", "salaryMax", "salaryCurrency", "salaryInterval", "salaryText", "salaryDescriptionPlain", "applyUrl", ) def project(row, board_scope): posting_id = row.get("id") if not isinstance(posting_id, str) or not posting_id: raise ValueError("Posting identity is missing") facts = {name: row[name] for name in FACT_FIELDS if name in row} content = json.dumps( facts, sort_keys=True, separators=(",", ":"), ensure_ascii=False ) key = json.dumps([*board_scope, posting_id], separators=(",", ":")) return { "key": key, "version": hashlib.sha256(content.encode("utf-8")).hexdigest(), "facts": facts, "provenance": { name: row[name] for name in ("createdAt", "scrapedAt") if name in row }, } 

Sorting object keys prevents serialization order from manufacturing revisions. It doesn't normalize away punctuation, salary text, or requirement wording. We'd rather inspect an extra change than erase a meaningful distinction.

This is a version detector for the selected fields. If your assistant starts answering from another field, add that field to the projection before trusting the detector to track it.

The same boundary helps chunking: choose a description representation for embedding rather than feeding combined text, overlapping sections, and HTML into the same document. Repeated wording can spend context tokens without adding evidence.

An upsert can keep pay that disappeared

Salary fields are absent when the employer hasn't published a value. That creates a particularly quiet refresh bug.

Suppose your storage layer merges incoming properties into an existing parent document. When a salary field disappears from the new record, the old value can survive the merge. Your assistant now has a pay claim that the latest observation doesn't support.

The filter must replace the factual document, including omissions. It must also retire the previous chunks.

This chooses a complete replacement when the content changes and a provenance update when it doesn't:

def instruction(previous, current): old = previous.get(current["key"]) if old is not None and old["version"] == current["version"]: return { "operation": "observe", "key": current["key"], "provenance": current["provenance"], } return { "operation": "replace_parent_and_chunks", "key": current["key"], "document": current, } 

These operation names belong to our ingestion adapter. They aren't Actor fields.

The adapter must interpret replacement literally: replace the parent facts, generate chunks from the new document, and make the old chunks ineligible for retrieval.

A practical implementation writes the new chunks under their content version, then switches an active-version pointer. Retrieval must enforce that pointer. Otherwise, even an accurate new chunk can sit beside an obsolete requirement and give the model contradictory evidence.

Keep the previous version active if replacement fails. Don't commit the new hash before its searchable chunks are ready, or a retry can mistake unfinished work for a completed refresh.

Silence can't tell you a role closed

The Actor reads currently published postings. A closed posting disappears, but an absent row in your dataset isn't sufficient evidence of closure.

The snapshot also has a global record cap, and filters can remove postings from view. Changing the query changes what absence means.

Deleting every unseen ID would turn a coverage boundary into an invented employment event. Leaving those IDs eligible for current-role answers would preserve stale claims.

We'd emit a separate uncertainty instruction:

def unobserved_instructions(previous, current): return [ {"operation": "mark_unobserved", "key": key} for key in previous.keys() - current.keys() ] 

The adapter should preserve those records for history while excluding them from answers that require a current observation. It should never translate mark_unobserved into “closed.”

That policy trades some answer coverage for a clearer evidence boundary. If a posting matters, perform a targeted follow-up before making a stronger claim.

Only compare presence within the same collection scope. Keep filters stable, track whether your cap could truncate coverage, and treat a changed collection policy as a new scope.

Combine the filters before touching the index

We want validation and comparison to finish before any storage mutation begins. Otherwise, a malformed record halfway through ingestion can leave a partially refreshed corpus.

This combines the projection, revision comparison, and absence handling into a plan:

def plan_snapshot(rows, previous, board_scope): current = {} for row in rows: document = project(row, board_scope) key = document["key"] if key in current: raise ValueError("Duplicate posting identity in snapshot") current[key] = document return [ instruction(previous, document) for document in current.values() ] + unobserved_instructions(previous, current) 

Pass the fetched items, the persisted documents for the matching scope, and scope into this function. An empty previous mapping is appropriate only when bootstrapping that scope.

Persist the complete projected document after a successful replacement. For an observe instruction, update provenance without rebuilding unchanged chunks.

Instruction Storage behavior Retrieval consequence
observe Update observation provenance Existing content remains eligible
replace_parent_and_chunks Replace facts and activate the new chunk version Superseded requirements stop competing
mark_unobserved Retain history and flag uncertain presence Current-role answers exclude unsupported freshness

The planner deliberately stops at instructions. Your database and vector index determine how to commit them safely, and that boundary deserves an explicit implementation rather than a hopeful comment beside an upsert.

Fresh observations still have a boundary

We'd ship repeated snapshots for factual refresh, with content hashes deciding which documents need replacement.

The worked example uses Lever Job Board Postings by Company Slug, but the failure belongs to any retrieval pipeline that treats discovery as synchronization.

There are limits to the repair. A rewrite between observations can go unseen. A revised posting's createdAt doesn't become an edit timestamp. A later scrapedAt establishes when the pipeline observed content, not when the employer changed it.

Your answer layer should preserve that distinction. “Observed with these requirements” is supportable. “The employer changed these requirements at this time” needs evidence this feed doesn't provide.

Reproduce a rewrite without editing a board

Use synthetic records to check the ingestion contract before connecting your storage adapter.

Example records below keep the posting identity stable while changing the observation time, requirement text, and presence of salary text:

before = { "id": "example-posting", "descriptionPlain": "Experience with Python.", "salaryText": "Example published pay", "scrapedAt": "2026-10-01T12:00:00Z", } seen_again = { **before, "scrapedAt": "2026-10-02T12:00:00Z", } edited = { **seen_again, "descriptionPlain": "Experience with Python and retrieval systems.", } edited.pop("salaryText") baseline = project(before, scope) previous = {baseline["key"]: baseline} observation_plan = plan_snapshot([seen_again], previous, scope) assert observation_plan[0]["operation"] == "observe" revision_plan = plan_snapshot([edited], previous, scope) assert revision_plan[0]["operation"] == "replace_parent_and_chunks" assert "salaryText" not in revision_plan[0]["document"]["facts"] absence_plan = plan_snapshot([], previous, scope) assert absence_plan[0]["operation"] == "mark_unobserved" 

To reproduce the integration:

  • Run the assertions to check the planner's behavior.
  • Implement the adapter operations, including active-version enforcement during retrieval.
  • Fetch a baseline and persist its projected documents after indexing succeeds.
  • Repeat the same snapshot input, compare against that persisted state, and apply the resulting instructions.

Keep the collection input identical between observations so the comparison measures content changes within the same scope:

{ "mode": "getJobs", "companySlugs": ["wealthfront"], "boardRegion": "global", "maxItems": 500, "includeDescription": true, "includeHtml": false, "onlyNewSinceLastRun": false } 

Drafted with AI assistance and reviewed for accuracy.