Recently I had an idea for a research tool, I realized I'd need to be able to find websites programmatically and the webs data is behind the largest corporations, not something we reliably have access to for being able to develop around.

So it got me thinking, how lightweight could you first build a crawler if you only cared about getting meta data (titles, description and the url)? Then it got me thinking, could we not share the load, create a distributed peer to peer network and make this data decentralized?

So what I have built is the open web directory. A network that crawls and builds a yellow pages like directory for web pages online.

You can setup your own node and contribute to the network by simply running:

git clone https://github.com/idev-games/the-open-web-directory.git cd the-open-web-directory npm start

And open port 80 for that device by port forwarding.

I think this is going to be a fascinating experiment, it's early days so expect bugs and issues but give it ago and let's see if we can index the web.

Check out more info here:

https://github.com/iDev-Games/The-Open-Web-Directory

The site listed on github is the live front end (you can host the html files in the public folder anywhere to open another frontend on the web).

submitted by /u/iDev_Games
[link] [留言]