A surprising amount of business still runs on xBase files. GoldMine CRM keeps its contacts in dBASE tables. So do older accounting and inventory systems, and Siemens STEP 7 projects. The programs that write them are decades old, and they still work, which is exactly why nobody has replaced them.

The trouble starts when something modern needs the same data. The Visual FoxPro OLE DB and Jet dBASE drivers are 32-bit and no longer developed, so a .NET app that talks to these files ends up pinned to x86 and Windows. Reading the files is the easy part. The hard part is writing to them while the old program is still running, without breaking its indexes or its locks.

I built Thunder.Dbf, an MIT-licensed C# library for .NET 10 that reads and writes dBASE, FoxPro, Visual FoxPro and Clipper tables, memos and indexes in plain managed code, on Windows, Linux and macOS. This post is about the part that took the longest: matching the old engines byte for byte.

There is no server, only files and manners

Old xBase programs don't use a database server. Every copy of the app opens the same files on a shared drive and coordinates through byte-range locks and the index files themselves. Anything else that writes to those files has to follow the same rules, or the old app finds a corrupt index the next morning.

So Thunder.Dbf takes the same record and header locks dBASE takes, updates every index tag on every write, and leaves the bytes of fields you didn't change exactly as they were. It opens read-only unless you ask otherwise:

using Thunder.Dbf; using var db = DbfDatabase.Open(@"\\server\goldmine\data"); Console.Write(db.Check()); // what's readable, writable, locked or damaged var contacts = db["CONTACT1"]; using var index = contacts.OpenProductionIndex()!; var acme = contacts.Seek(index["CONTCOMP"], "ACME"); // like dBASE's SEEK, through GoldMine's own index

Unit tests prove a library agrees with itself

That isn't the question that matters. The question is whether Visual FoxPro still trusts the files after we've written them. So a Windows test machine runs the real programs against the same tables: Visual FoxPro 9 SP2, its OLE DB driver, Jet 4.0 and Harbour (the open-source Clipper). After each round of writes, Visual FoxPro itself walks every index tag, looks up every record by every key, and compares each index with its own REINDEX.

That turned up the kind of detail no format document mentions.

Every writer has its own idea of a blank

  • Jet keys a blank date as the double 1E100, so in its .mdx indexes blank dates sort after every real date. A blank keyed as 0 instead looks fine to everything except Jet, whose IS NULL no longer finds the record, and whose own UPDATE then leaves a stale key behind.
  • Visual FoxPro puts a 0x80 marker byte in front of every nullable field's key, and keys .NULL. as all zeros. Character fields are the well-known case, but it does the same for currency, numeric, integer, date, datetime and double fields. Miss that, and seeks on those tags find nothing.
  • A blank date that isn't .NULL. reads back as null, but is keyed differently from .NULL.. Seeking null has to find both.

Numbers aren't just numbers

  • In a dBASE IV .mdx, a numeric key is 12-byte BCD, and byte 1 carries a digit count. dBASE counts significant digits, so 15940 counts 4; Jet counts 10 for every whole number. Jet steps from one match to the next by comparing raw bytes, so a key written in the other style hides a live record behind a deleted one.
  • Visual FoxPro rounds currency half away from zero (1.00005 is stored as 1.0001). .NET's decimal.Round rounds half to even (1.0000), so a value on an exact midpoint gets stored differently from what a FoxPro user would get by typing the same number.
  • A number whose decimals don't fit its field is stored by Visual FoxPro with the decimals that do: 12345678.9 goes into an N(10,2) field as 12345678.9, one decimal short. A library that throws instead can read that value but can't write it back.

None of these show up if you only test against files your own code wrote.

What the testing found

Over twenty rounds, Thunder.Dbf and Visual FoxPro took turns writing the same tables with random appends, updates, deletes and recalls: 616,640 writes, and 39.4 million index entries and records checked. Every one agreed.

In eleven code pages, from Western to Cyrillic, Greek, Turkish and Japanese, Thunder.Dbf's .cdx keys come out byte-for-byte identical to Visual FoxPro's, including case-insensitive UPPER() indexes.

Then the harsh one: we killed writers at random moments mid-change, and let Visual FoxPro judge the table with its own tools. Visual FoxPro and its OLE DB driver left a wrong index with no warning 51 times in 180. Thunder.Dbf did so 0 times in 120: when something was wrong, it said so and refused to write until a rebuild. On default settings it also lost no confirmed write across 50 hard power cuts to a virtual machine.

The full numbers, including speed against the OLE DB driver, are in the test write-up.

From reading the data to moving off it

Most teams don't stop at reading. The same packages cover LINQ over tables, an ADO.NET provider (DbfConnection) to replace the 32-bit drivers, migration to SQL Server, PostgreSQL or SQLite with a live two-way sync until cut-over, and a CLI:

dotnet add package Thunder.Dbf dotnet tool install -g thunder-dbf

If you maintain one of these systems, I'd like to hear what breaks. The source, docs and issues are at github.com/pthunder/Thunder.Dbf.