Why I built DataHarvester

While working on a larger data collection project, I ran into a recurring problem: downloading files was easy, but keeping track of what had been downloaded, verifying file integrity, and maintaining a clear record of the process was much less convenient.

For small tasks, a browser or a simple download script can be enough. But when collecting multiple public datasets or maintaining a repeatable workflow, I wanted something more structured.

That's why I started developing DataHarvester.

What is DataHarvester?

DataHarvester is a portable Windows application designed to collect files from public HTTPS sources while providing integrity verification and traceable reports.

The goal isn't to replace every download manager or web scraping framework. It's to offer a focused tool for people who need to retrieve files and keep a verifiable record of the collection process.

Main features

  • HTTPS file collection: Retrieve files from supported public HTTPS sources.
  • SHA-256 verification: Use cryptographic checksums to verify file integrity.
  • Traceable reports: Keep records of collection results for later reference.
  • Portable Windows application: No cloud account or additional installation required.
  • English and French: Available in both languages.

An important distinction: SHA-256 can help verify that a file hasn't changed, but a checksum alone doesn't establish whether its original source is trustworthy.

Who is it for?

I designed DataHarvester with several potential use cases in mind:

  • Researchers collecting publicly available datasets.
  • Developers maintaining repeatable download workflows.
  • People working with public statistical files.
  • Users who want more transparency about downloaded files.

It's particularly relevant when being able to explain what was collected and whether its integrity was verified matters as much as obtaining the files themselves.

Current development status

DataHarvester is still an early-stage project.

The current 0.1-F.5 release focuses on reliability, clearer error reporting, improved handling of download and storage failures, and consistency across the application.

I'm keeping the scope deliberately focused rather than adding a large number of features before the core workflow is reliable.

Availability

DataHarvester is currently available for Windows x64 on itch.io.

Price: $4.99 CAD (one-time purchase).

Project page: https://waffle913.itch.io/dataharvester

Looking for feedback

I'm interested in hearing from developers, researchers, and people who regularly collect public data files.

A few questions I'd love your thoughts on:

  • How do you currently manage downloads of public datasets?
  • Do you keep integrity checks and collection records?
  • Would a dedicated desktop tool be useful in your workflow, or do you prefer scripts?

I'm especially interested in understanding which parts of the process people find unnecessarily repetitive.

Thanks for reading!