In my last post I pulled Gulf jobs from employer job feeds. The feed gives you a title, a location and a description. Everything a job seeker actually filters on (visa sponsorship, remote or on-site, years of experience) is buried in free text.

So I wrote rules to pull those fields out. The index now holds about 12,000 to 13,700 open jobs from 742 employers, refreshed daily. Here is what broke along the way.

1. "We do not offer visa sponsorship" matches "visa sponsorship"

My first visa rule was a single regex looking for phrases like visa sponsorship or sponsor your visa. It flagged jobs as sponsoring visas when the posting said the exact opposite:

Please note we do not offer visa sponsorship for this role.

The fix was ordering. I check the negative patterns first and only look for the positive ones if no negative matched:

def flag(text, pos, neg, low, hints): if not any(h in low for h in hints): return None # the post never mentions it if neg.search(text): return False # explicitly says no if pos.search(text): return True # explicitly says yes return None # mentions it, but not clearly 

The negative list grew more than the positive one: "must already hold a valid visa", "own visa", "transferable visa", "without sponsorship", "unable to sponsor".

2. Three states, not two

The bigger lesson was in that last return None. My first version returned False whenever it found no positive match. That made the data claim thousands of employers refuse sponsorship, when really they just never mentioned it.

So every extracted field has three states:

Value Meaning
true The post says yes
false The post says no
null The post doesn't say

The honest result is humbling. Only about 50 of roughly 13,700 jobs state visa sponsorship in a way a rule can read. In the Gulf, sponsorship is usually assumed for expat hires, so employers rarely write it down. A filter that promises "visa sponsorship jobs" can only ever return those 50, and I now say so in the docs instead of padding the field.

3. "Hybrid & Hydrogen Engineer" is not a hybrid job

Work arrangement looked easy: if the title or description says "hybrid" or "remote", tag it. Then I read the matches:

  • Hybrid & Hydrogen Engineer, an engineering specialism, not a work arrangement
  • Remote Sensing roles, which are about satellite data, not working from home

Now a word in the title only counts when it looks like a tag: (Hybrid), - Remote, Remote working. Words that are part of the job name are ignored. Descriptions need a fuller phrase like "hybrid working model" or "work from home two days a week".

4. Some career sites tag every job the same way

Some job systems have a structured "workplace type" field, which should be better than any regex. But on a few employer sites, every job carries the same value. So drivers, chefs and technicians showed up as Remote or Hybrid.

I didn't want to throw the field away, because for office roles it's usually right. So there's one sanity rule on top: Remote or Hybrid is dropped back to null for hands-on titles (driver, chef, cashier, technician, nurse, physician and about 40 more). On-site is kept, because it's never wrong for those roles.

After the cleanup, about 2,270 of 13,700 jobs state a work arrangement: roughly 1,670 on-site, 410 hybrid and 185 remote. That's only 16%, but every one of them is something the employer actually said.

Bonus: watch each source, not the total

This one happened this week. My daily health check flagged that only 514 employers had open jobs, when it's usually about 718. The total job count looked almost normal (12,087 instead of about 13,700), so a total-only check would have missed it.

Splitting by source showed the problem immediately. One job system fell from 2,125 jobs to 380 overnight, while every other source was steady. A test run confirmed that system was answering every request with HTTP 429 (too many requests). It wasn't my code. The host was throttling shared cloud IPs.

The fix I'm building is boring but important: if an employer's site refuses us, keep its last good jobs for up to three days instead of silently dropping them. And the health check now compares per-source counts, not just the total.

What I'd tell myself on day one

  • Absence is not a "no". Use null and document what it means.
  • Check negatives before positives. The sentence that says no usually contains the words that say yes.
  • Read 50 matches before trusting a regex. Every bug above was obvious within the first page of real output.
  • Monitor by source. A healthy total can hide a dead source.

I built the Apify Actor that collects this data: Gulf Jobs Scraper (pay per result). If you've extracted fields from job posts in another region, I'd like to hear which ones gave you the most false positives.