Yes, it's called python.

Write a crawler that examines all local links, and builds a graph (it needn't be anything fancy; a collection of nodes and edges is enough, but you'll have to detect cycles when building the graph), and then print it out in graphviz .dot language.

Then you can feed it into neato [either through a command-line pipe, or construct one using Python's subprocess module], et voilà, instant pagemap! :)

You can also use some of the rendering modes and scrape position data for the rendered nodes, if you want to generate a clickable imagemap.

Here's a first cut implementation in ruby. Requires the "hpricot" gem. Outputs a dot file to stdout. http://gist.github.com/35841

Output from the above is at: http://jey.kottalam.net/tmp/obgraph.out

I tried rendering this with "dot -Tcmapx -oob.map -Tgif -oob.gif" but dot segfaulted after 70 minutes. The code as posted at gist only outputs nodes for articles written by Eliezer in an attempt to make the dot file a manageable size. Tips/fixes appreciated.

Not sure if it will work, but try:

1) Rendering an undirected graph via graphviz's neato tool.

2) Rendering to svg, which might let graphviz not have to keep as much rendering information in memory at any one time.

3) Asking about it on the grapviz-interest mailing list. https://mailman.research.att.com/mailman/listinfo/graphviz-i...

Yeah, it's that thing with the segfault that I worried about. "Highly interconnected." Thanks for the graph, which I may be able to use for my own purposes even if I can't show it.

There's got to be standard graphing tools for things that are highly connected...

When Daniel J. Bernstein made his DAGs of standard crypto algorithms (http://cr.yp.to/cipherdag/cipherdag-20070630.pdf) he mentioned that they'd crashed every standard drawing tool he tried. He makes a reference to "...my own drawing tools, which are much more careful in their use of memory." Unfortunately, I'm not sure where to obtain DJB's drawing tools.

Yup, Graphviz: http://www.graphviz.org

Graphviz only renders a graph which is represented as per dot format. There are other programs: dot, neato etc which can produce jpeg,png,gif... formats through command line tools.

I guess what he meant was tools which can do this for a particular site.

I can't think of any but as a previous poster said - a crawl of desired depth for all local pages, then feed the access log to "statviz" to generate a neat dot file along with the links.

I am home for winter break and would be willing to write this for you in the name of science. Plus, it seems that you know Marcello Herreshoff, with whom I took Computer Science in high school. Email me the requirements for your tool to <zackster>:at:<gmail.com> please.