Your row limit counts records; your retrieval task counts matches.
Set maxItems low and it's tempting to embed each returned row. For tennis, that row can carry a tournament's nested matches. The context window gets the tournament, even when your question concerns a match.
Lab note 141: can we make a match independently retrievable without copying its siblings into the embedding? That's a schema-level bet. Better retrieval scores would still need evaluation against your questions.
Where the record cap stops
maxItems caps data records, not nested tennis matches or tokens.
Its default is 50, and its allowed range is 1 through 5000. Those bounds control acquisition. They don't establish an embedding budget.
The Actor's tennis scoreboard returns event records with a flat matches[] list alongside groupings[]. The event row is a useful transport container. Making it your retrieval unit is a separate decision.
This Example acquisition selects tennis explicitly and streams the resulting dataset:
import hashlib import json import os import tiktoken from apify_client import ApifyClient run_input = { "mode": "scoreboard", "league": "atp", "datePreset": "last7Days", "maxItems": 1, } client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor( "vexaloom-labs/espn-scores-standings-teams-and-news-by-league" ).call(run_input=run_input) if run["status"] != "SUCCEEDED": raise RuntimeError(f"Actor ended with status {run['status']}") rows = client.dataset(run["defaultDatasetId"]).iterate_items() A successful run still needs content inspection: an unsupported or empty request can produce a notice row. That belongs in your ingestion report, outside the sports corpus.
What does a match need to carry?
Our target task is narrow: retrieve the relevant tennis match, then give an assistant enough context to identify it and describe its available state.
That suggests a projection rather than an indiscriminate tournament dump.
| Field or structure | Destination | Reason |
|---|---|---|
Input mode, league, maxItems | Ingestion configuration; copy the selected league into parent metadata | Preserve acquisition intent without treating controls as match content |
recordType | Row routing | Separate events from notices and unexpected record types |
Event eventId, name, date | Parent manifest and every match chunk | A retrieved match must remain identifiable away from its original row |
Match matchId | Chunk identity and match content | Identify the child within its event |
Match round, court | Match content | Distinguish encounters within the tournament |
Match notes, status | Match content, intact | Preserve available qualifications and state |
Match competitors[] | Match content, intact | Keep the participants together |
matches[] | Flatten into child chunks | Align the retrieval unit with the question's subject |
groupings[] | Exclude from embedding text | Avoid indexing parallel representations of match material |
The catch is inside competitors and status: knowing those containers exist doesn't justify inventing their internal paths.
We preserve them intact. We'd rather carry an opaque object than confidently extract a participant name from a key we guessed.
The event's date also stays under the event namespace. Renaming it to “match start time” would add meaning the field mapping hasn't established.
Keep a projection, not both trees
The intended split looks like this:
event row ├─ event identity ───────────────────> parent manifest ├─ matches[] ──> match + event context ──> embedding candidate └─ groupings[] ──────────────────────> excluded from embedding text Flattening both tennis representations risks giving repeated material extra retrieval weight. More serialized text doesn't necessarily mean more evidence. Sometimes it means the same evidence brought a folding chair.
We'd keep the original event payload in a separate raw-data archive if reprocessing matters. The embedding projection has a narrower job.
Some repetition is useful: the tournament identity travels with each match. Repeating a small parent envelope buys independence; repeating sibling matches changes what the chunk is about.
The trade-off appears in broader questions. A tournament-wide recap may need several retrieved matches or an event-level aggregation. Match chunks are a deliberate fit for match retrieval, not a universal replacement for every sports query.
Count the text you actually embed
The serializer below defines the flat-list field explicitly, preserves the selected fields, and measures the exact string intended for embedding.
Its token budget and encoding are explicit application choices. Match the encoding to your embedding system before treating the count as its operational limit.
MATCH_LIST_FIELD = "matches" EVENT_FIELDS = ("eventId", "name", "date") MATCH_FIELDS = ( "matchId", "round", "court", "notes", "status", "competitors" ) encoding = tiktoken.get_encoding("cl100k_base") token_budget = 1200 def compact(value): return json.dumps( value, ensure_ascii=False, sort_keys=True, separators=(",", ":") ) def project(record, fields): return {field: record[field] for field in fields if field in record} def stable_key(*parts): return hashlib.sha256(compact(parts).encode("utf-8")).hexdigest() def serialize_match(parent, match): text = compact({ "event": parent, "match": project(match, MATCH_FIELDS), }) return { "chunk_id": stable_key( parent["league"], parent["eventId"], match["matchId"] ), "metadata": {**parent, "matchId": match["matchId"]}, "text": text, "token_count": len( encoding.encode(text, disallowed_special=()) ), } The chunk key combines the selected league, event identity, and match identity. It makes no assumption that matchId is globally unique.
That distinction matters during refreshes. A match's status can change without becoming a different match. Use the stable key for corpus updates; using a content hash as the only identity would turn each changed snapshot into another retrievable version.
The count covers text. Metadata, the JSONL envelope, and instructions added to a later generation prompt need their own accounting.
Write match chunks and an event manifest
This cell writes accepted chunks, parent records, and a review queue. Oversized candidates retain their complete content for a later structured split.
parents = {} chunks = {} with open("review.jsonl", "w", encoding="utf-8") as review_file: def route(reason, record): review_file.write( compact({"reason": reason, "record": record}) + "\n" ) for row in rows: record_type = row.get("recordType") if record_type == "notice": route("notice", row) continue if record_type != "event": route("unexpected_record_type", row) continue if row.get("eventId") in (None, ""): route("missing_event_id", row) continue parent = { "league": run_input["league"], **project(row, EVENT_FIELDS), } event_key = stable_key(parent["league"], parent["eventId"]) parents[event_key] = {"event_key": event_key, **parent} matches = row.get(MATCH_LIST_FIELD) if not isinstance(matches, list) or not matches: route("missing_matches", row) continue for match in matches: if ( not isinstance(match, dict) or match.get("matchId") in (None, "") ): route("missing_match_id", { "event": parent, "match": match }) continue chunk = serialize_match(parent, match) chunks[chunk["chunk_id"]] = chunk with open("matches.jsonl", "w", encoding="utf-8") as match_file: for chunk in chunks.values(): if chunk["token_count"] > token_budget: route("over_token_budget", chunk) continue match_file.write(compact(chunk) + "\n") with open("events.jsonl", "w", encoding="utf-8") as event_file: for parent_record in parents.values(): event_file.write(compact(parent_record) + "\n") The dictionaries retain the last encountered version of each key within this batch. That's an ordering policy, not a freshness guarantee. Across ingestions, store an observation timestamp and define which snapshot may replace an existing record.
This version also buffers projected chunks in memory. For a larger corpus, move that keyed staging step into persistent storage rather than accumulating every candidate in a process.
At embedding time, submit each accepted record's text and retain its chunk_id and metadata. Sending the entire JSONL envelope would repeat context the serializer already included.
What refuses to fit
Cutting serialized JSON at a token boundary doesn't work as a semantic policy. It can remove a competitor, detach a status qualification, or leave invalid JSON. A smaller string isn't automatically a smaller fact.
The review queue gives oversized matches an explicit destination. A follow-up splitter can repeat event identity, complete competitors, and status while separating independently meaningful notes.
If the mandatory context alone exceeds your budget, keep that match out of the embedding batch until you change the budget or design a smaller, verified projection.
Requesting gameDetail doesn't supply a richer tennis fallback either. These individual-sport scoreboards don't expose the same team-style summary; that request returns a notice.
Missing fields also remain missing. The Actor omits empty values, so the serializer can't turn absent status or participant detail into evidence. Your answer layer needs permission to say the record doesn't establish something.
The event transports; the match retrieves
For this task, we'd embed the match projection and keep the event as its parent.
The flat list gives us a suitable boundary. It doesn't establish that every match will fit, that every question is match-shaped, or that retrieval quality improves without evaluation.
ESPN Scores, Standings, Teams & News by League is the worked example. The reusable decision is to choose the embedding unit from the question's subject.
For a retrieval evaluation, check whether the relevant match appears, whether nearby matches crowd it out, and whether the assistant can distinguish missing evidence from a negative answer. A compact chunk can still be the wrong chunk.
Reproduce the boundary
- Install
apify-clientandtiktoken, and setAPIFY_TOKENin your environment. - Run the Python cells in order with the input below.
- Inspect
events.jsonl,matches.jsonl, andreview.jsonlseparately. Empty requests belong in the review path. - Feed accepted
textvalues into your embedding pipeline and retain their stable identities for updates.
Use this exact Example input:
{ "mode": "scoreboard", "league": "atp", "datePreset": "last7Days", "maxItems": 1 } Drafted with AI assistance and reviewed for accuracy.