“Store the data” is only the beginning of the problem. The difficult questions usually come afterward: What structure does this data actually have? Which fields are missing or inconsistent? Which values are invalid? Which changes are safe to apply automatically? What exactly changed after a repair? Can the system prove that its storage and indexes are still consistent? I built Atlas to answer those questions inside the database engine itself. Atlas is a zero-dependency embedded database for semi-structured data. It stores records, builds a full-text search index, infers schema, analyzes data quality, proposes safe repairs, preserves uncertain records, and records an audit trail of applied changes. It does not use SQLite or SQL. It is not intended to replace SQLite for relational workloads. Instead, Atlas focuses on a gap that is usually handled by external scripts and tools: Data inspection, diagnosis, and safe repair as first-class database capabilities. That is the problem Atlas was built to solve. Why data quality belongs inside the database engine Most databases are very good at storing and retrieving data. That is necessary, but real-world data work rarely stops there.** Operational records, imported JSON, CSV files, event payloads, and semi-structured documents often arrive with problems: {"id": "T-1", "title": " Connection timeout ", "priority": "HIGH"} {"id": "T-1", "title": "connection timeout", "priority": "high"} {"id": "T-2", "title": "Unicode café search", "priority": null} These records contain several potential issues: Duplicate logical identifiers Leading or trailing whitespace Inconsistent capitalization Null-like values Missing fields Mixed data types Malformed email addresses Different date formats Inconsistent structures across records A storage engine can preserve these values perfectly while still leaving the data difficult to understand and use. The usual response is to add external tools: A schema profiler A data-quality script A search engine A cleanup job A validation pipeline An audit system A quarantine workflow This creates a fragmented system. The external tools may use different assumptions from the database. Repairs may not update the search index. Scripts may mutate records without a durable audit trail. Diagnosis may happen outside the storage boundary, where the system cannot guarantee that the result still matches the data. The fundamental problem I chose was this: Most embedded databases provide storage and query primitives, but inspection, diagnosis, and safe repair are usually handled by external tools. Atlas brings those capabilities into the database engine itself. The Atlas data-quality contract Atlas is built around a simple operational contract: Data should be stored durably, made searchable, inspected structurally, repaired explicitly, and verified afterward. That contract creates a complete workflow: INGEST | v STORE | v INDEX | v UNDERSTAND | v REPAIR | v VERIFY Each stage has a clear responsibility. Ingest Atlas accepts JSON arrays, JSON Lines, and CSV files. CSV values receive conservative type coercion, and JSONL and CSV inputs can be processed incrementally. Store Records are written to an append-only storage file with checksums and write-ahead journaling. Index Atlas maintains a persisted inverted index for full-text search and updates affected documents incrementally. Understand The engine infers field types, profiles field statistics, measures coverage and null values, and calculates an explainable health score. Repair Atlas detects and classifies issues. It does not modify records during diagnosis or dry-run repair. Verify The system checks storage blocks, journal state, index consistency, and audit-chain integrity. The goal is not simply to add more commands to a database CLI. The goal is to keep the entire data lifecycle connected. Why Atlas does not use SQLite SQLite is an excellent database. It is mature, reliable, portable, and extremely useful for embedded relational applications. SQLite is optimized for: Tables SQL queries Transactions Joins Relational application state A mature embedded database ecosystem Atlas targets a different optimization point. Atlas is designed for: Semi-structured JSON-like records Built-in schema inference Full-text BM25 search Explainable data-quality analysis Safe repair proposals Quarantine of uncertain records Repair audit trails Direct inspection of the storage lifecycle So the comparison is not that Atlas is universally better than SQLite. The more accurate claim is: Atlas is better suited when the question is not only “Can this record be stored?” but also “What is this data, what is wrong with it, which changes are safe, and how can those changes be proven?” Atlas does not use SQLite or SQL because the project is intentionally built around a different data model and workflow. SQLite remains an excellent choice for relational application data. Atlas is designed for a different class of problem: semi-structured data that must be stored, searched, understood, repaired, and verified in one place. The write path: durable storage with a visible trail Atlas uses append-only binary storage with a write-ahead journal. The write path is: Acquire lock | v Write transaction to journal | v Synchronize journal | v Apply committed blocks to data | v Synchronize data | v Truncate journal | v Release lock More precisely: Atlas acquires the database lock. It writes a transaction-begin marker. It appends record or tombstone blocks. It writes a transaction-commit marker. It synchronizes the journal. It replays committed entries into the main data file. It synchronizes the data file. It truncates the journal. It releases the lock. The journal is not treated as temporary noise. It is the recovery boundary. If the process stops during a write, Atlas can distinguish committed transaction work from incomplete work. On the next open, committed transactions can be replayed while unfinished transactions are discarded. Each binary block contains a checksum: [MAGIC] [VERSION] [BLOCK TYPE] [SEQUENCE] [PAYLOAD LENGTH] [PAYLOAD] [SHA-256 CHECKSUM] The checksum covers the block header and payload. A damaged block is detected rather than silently accepted. This design also makes the storage system inspectable. The database is not just a black box that returns records. It has a visible trail of how those records reached disk. The database directory is part of the design An Atlas database directory contains separate files for separate responsibilities: mydb/ data.atf primary append-only record log journal.atf write-ahead journal index.atj persisted inverted index audit.atf chained repair audit trail quarantine.json uncertain records meta.json human-readable metadata lock platform-native lock file Each file answers a different question. data.atf: What records are currently stored? journal.atf: What committed work is recoverable? index.atj: How are records searchable? audit.atf: What repairs have been applied? quarantine.json: Which records were preserved for review? meta.json: What does the database know about itself? The directory is not just an implementation detail. It is part of Atlas’s inspectability model. Search that explains its ranking Atlas includes a full-text search engine based on a persisted inverted index and Okapi BM25 ranking. The index stores information such as: { "doc_lengths": {}, "doc_fields": {}, "postings": {} } The index tracks: Term frequency Document frequency Document length Token positions Field-specific tokens Incremental document updates Atlas supports: Term searches Phrase searches Boolean expressions Field filters Numeric ranges Date-like ranges Explainable scoring For example, the query language can express: "connection timeout" (database OR storage) AND recovery priority:[2 TO 5] Search is not implemented as a full scan for ordinary term, phrase, and field queries. Atlas uses its persisted index to find candidate records, then applies BM25 scoring to rank them. The result is not just a list of matches. With explanation enabled, Atlas can show how search terms contributed to the ranking. That is important for a database whose goal is understanding. A search result should be relevant, but it should also be explainable. Understanding data without a predefined schema Semi-structured data does not always arrive with a reliable schema. Atlas infers field structure from the records it receives. It recognizes categories such as: Null Boolean Integer Number String Date-like value Array Object The profiler also reports: Field coverage Null counts Distinct values Common values Numeric minimums and maximums Averages Nested field paths This allows Atlas to answer questions that a basic storage layer cannot answer: Which fields are present in most records? Which fields contain mixed types? Which values are mostly null? Which fields have unexpectedly high cardinality? Which records contain nested structures? Which fields might need a repair rule? Atlas then combines multiple signals into a health score covering: Storage integrity Schema consistency Data quality Search-index health The score is intentionally explainable. Atlas does not treat health as a mysterious number. It exposes the components behind the result. Safe repair is a database operation The most important difference between Atlas and a typical cleanup script is that repair is part of the engine’s controlled workflow. Atlas does not mutate records simply because it found something suspicious. The repair lifecycle is: DIAGNOSE | v DRY RUN | v APPLY SAFE CHANGES | v AUDIT AND VERIFY The repair engine classifies proposals by confidence and action. The important boundary is explicit mutation. Diagnosis does not modify records. Dry-run repair does not modify records. Only an explicit apply operation changes records, and only proposals classified as safe are automatically applied. Uncertain records are not silently deleted or forced into a guessed format. They are preserved in quarantine.json for review. That makes quarantine a safety feature, not an error dump. Repair must also update the rest of the system A repair is not complete if it changes only the record file. Suppose Atlas trims whitespace from a title field. The repair must also ensure that: The updated record is persisted The search index reflects the new content The audit trail records the change The health calculation sees the updated state Future verification remains consistent That is why repair belongs inside the database engine rather than in an external script. The engine understands the relationship between storage, indexes, profiling, and audit history. Every applied repair is recorded with: Timestamp Record ID Field Repair rule Result Before-value hash After-value hash Previous audit-chain hash The audit log does not store raw before-and-after values. It stores hashes and chain links that make tampering detectable without duplicating sensitive content. What zero dependencies actually means Atlas has no external runtime dependencies. The requirements.txt file is intentionally empty. That means the implementation owns more of the system directly. Instead of relying on: A database wrapper A search package A schema library A data-quality framework A repair engine A file-locking package A CLI framework Atlas uses Python’s standard library: struct for binary framing hashlib for checksums json for record and metadata serialization csv for CSV ingestion argparse for the CLI re and unicodedata for tokenization fcntl on Linux and macOS msvcrt on Windows os.replace for atomic file replacement Zero dependencies did not make the project simpler automatically. It moved more design responsibility into the project itself. Atlas had to define: The binary block format The checksum boundary The WAL transaction markers The repair confidence rules The index representation The audit-chain format The cross-platform locking behavior The recovery behavior The advantage is control. Each important behavior is visible in the codebase and documented as part of the engine’s design. Cross-platform without adding a runtime package Atlas runs on Linux, macOS, and Windows. The platform-specific difference is file locking: Linux/macOS → fcntl Windows → msvcrt Both modules are part of Python’s standard library. On Linux and macOS, Atlas supports shared reader locks and exclusive writer locks. On Windows, the standard-library approach uses an exclusive lock for both readers and writers. This slightly reduces read concurrency, but preserves safer single-access behavior without adding a third-party package. The database format itself remains the same across platforms. Verification is part of the product Atlas is designed around the idea that claims should be testable. The project includes: Record codec tests Checksum corruption tests CRUD persistence tests WAL recovery tests Search and BM25 tests Schema inference tests Profiling tests Repair classification tests Audit-chain tests Quarantine tests Import/export tests CLI tests Cross-process lock tests Embedded self-tests The current project checks include: 161 unittest cases 20 embedded self-tests 0 runtime dependencies The release builder also creates normalized archives and verifies reproducibility by comparing independently generated release ZIP files. That matters because “zero dependency” and “reproducible release” are not useful claims unless the project can demonstrate them. Honest limits Atlas is deliberately focused. It is not: A network database server A distributed database An MVCC engine A replacement for every SQLite workload A relational query engine An encrypted storage system A high-concurrency database A stemming or fuzzy-search engine Atlas uses a single-writer model. Range queries use linear scans. On Windows, reads are serialized through the same exclusive locking approach used for writes. Repair confidence values are design heuristics, not statistical certainty. A proposal classified as REVIEW still requires human judgment. These limits are not hidden from the user. They are part of the project’s design boundary. A focused system is easier to understand when it is honest about what it does not try to be. The design principles behind Atlas If I designed Atlas again, I would still start with the same principles. Define the data-quality boundary first Before adding repair rules, define exactly what the engine is allowed to change automatically and what must be reviewed. Separate diagnosis from mutation A diagnostic command should not change records. A dry run should not change records. Mutation should always be explicit. Treat storage and data quality as connected systems Repairing a record without updating its index, health score, or audit trail is an incomplete repair. Preserve uncertainty A system that guesses silently is more dangerous than one that asks for review. Quarantine keeps uncertain data available without pretending it is correct. Make limitations visible The goal is not to claim that Atlas solves every database problem. The goal is to make a specific problem smaller and more trustworthy. The fundamental idea Atlas is not simply another embedded database. Its value is the combination of capabilities that are normally split across several tools: Storage + Search + Schema inference + Profiling + Health analysis + Safe repair + Quarantine + Auditability + Verification The core idea is straightforward: Data inspection, diagnosis, and safe repair should not be external afterthoughts. They should be capabilities of the database engine that stores the data. SQLite remains the right choice for many relational applications. Search engines remain the right choice for large-scale distributed search. Data-quality platforms remain valuable for enterprise pipelines. Atlas occupies a narrower space: A transparent, zero-dependency, embedded engine for semi-structured data that needs to be stored, searched, understood, repaired, and verified in one place. It does not just store data. It helps users understand what they stored, decide what can be safely changed, preserve what is uncertain, and prove what happened afterward. Repository: Atlas GitHub repository Demo: Atlas Demo Video Atlas: Store it. Search it. Understand it. Repair it.