Crawling

• From the textbook on Information Retrieval
  • web crawling and indexing
  • https://nlp.stanford.edu/IR-book/html/htmledition/crawling-1.html
  • https://nlp.stanford.edu/IR-book/html/htmledition/the-url-frontier-1.html
• https://www.ccs.neu.edu/home/vip/teach/IRcourse/IR_surveys/olston-najork@web-crawling10-crop.pdf
• Overview of various webcrawler architectures
• Estimating page freshness
• Compaq Research Paper with a video presentation
• Web Crawler Architecture, Marc Najork

Queues

• The most efficient Non-Blocking Queue algorithms along with pseudocode
• Implementing Lock-Free Queues
• An Optimistic Approach to Lock-Free FIFO Queues

Webpage Caching

• Detecting Near-Duplicates for Web Crawling

Graph Algorithms:

• On Estimating Average Degree (Anirban Dasgupta, et al.)
• https://www.youtube.com/watch?v=Tnu_Ws7Llo4 numberphile: A Breakthrough in graph theory
• https://www.youtube.com/watch?v=CDMQR422LGM talk on algebraic graph theory
• https://www.youtube.com/watch?v=wOeRfcglBng

3D Animations/Rendering:

• https://www.youtube.com/watch?v=Q7AOvWpIVHU

Go Profiling Tutorials:

• https://www.youtube.com/watch?v=xxDZuPEgbBU
• https://www.youtube.com/watch?v=ydWFpcoYraU

Names

• lared: “la red” in spanish means network
• reef: there are many biological networks that are supported by a coral reef
  • remora
  • reefnet
  • reefhop
• quest

The Graph UI

Empower users by handing the graph to them. Google has been successful by crawling and indexing the web then computing complex graph algorithms on the web of the internet in order to find what users need. But what if we sent the graph to the users, give them the power to traverse the graph of links themselves enabled by tools created by me. In essence, it is dissolving the centralized brain of google and distributing it’s parts across its entire user base. The brain of the web should not be behind corporate firewalls, but in the hands of the people.