# TL;DR: Automated Summaries

Like being in the know but not spending all that time reading to be in the know? We need to know, but we know we need more time. How can we save time to know when we need to read to know? Ya know?

[TL;DR: Automated Summaries](https://tldr.kmw.moe) can help.

The past few years have shown us staying knowledgeable is critical. However, with so many data sources, there's too much to consume. Reading headlines is not enough.

Presenting [TL;DR: Automated Summaries](https://tldr.kmw.moe). TL;DR finds what's relevant in an article and surfaces it up. TL;DR has a simple API to request the summaries.

## Building TL;DR

I used [NodeJS](https://nodejs.org) with [natural](https://github.com/NaturalNode/natural), [sylvester](https://github.com/NaturalNode/node-sylvester), [gramma](https://caderek.github.io/gramma/), [expressJS](https://expressjs.com/), [ejs](https://ejs.co/), [bootstrap](https://getbootstrap.com), and [LanguageTool](https://languagetool.org). I built this for the [Linode x Hashnode Hackathon](https://townhall.hashnode.com/build-with-linode-hackathon-june-2022), so it's running on [Linode](https://linode.com).

Natural is a Natural Language Processing library for nodeJS. Sylvester is a math library. ExpressJS and ejs provide the web framework and templating library. Gramma is an interface to Language Tool. Language Tool provides word replacements and sentence checks.

### Building Summaries

When thinking of surfacing relevant information, [PageRank](https://en.wikipedia.org/wiki/PageRank) is a popular algorithm. Using Natural's [stemmers](http://naturalnode.github.io/natural/stemmers.html) and [TF-IDF](http://naturalnode.github.io/natural/tfidf.html), we convert documents into a transition matrix we can run PageRank on.

![PageRank Visualized](https://twirp.km-w.net/hashnode/pagerank.png)
*PageRank visualized: The size of each face is proportional to the total size of the other faces which are pointing to it. [Art draw drawn by FML](https://commons.wikimedia.org/wiki/File:PageRank-hi-res.png).*

Important sentences rise up in score. We can grab the top scoring sentences for our summary.

Before doing all of this, we need to sanitize our inputs. Articles are commonly written in HTML or Markdown. TL;DR [strips markdown and HTML](https://github.com/TwiRp/tldr/blob/a3f74f818060dca682355368a6dbc43c54415e70/lib/summarize.js#L193). Language Tool [helps remove fluffy words](https://github.com/TwiRp/tldr/blob/a3f74f818060dca682355368a6dbc43c54415e70/lib/summarize.js#L172). Sites like [Hashnode](https://hashnode.com) contain blog entries and technical posts. LanguageTool helps [remove technical sentences that don't belong in summaries](https://github.com/TwiRp/tldr/blob/a3f74f818060dca682355368a6dbc43c54415e70/lib/summarize.js#L182).

Language Tool slows down processing. The [API allows for enabling or disabling](https://tldr.kmw.moe/deets) Language Tool.

### The Hardware

[Linode](https://www.linode.com) was kind enough to provide $100 for Hackathon participants.

[TL;DR](https://tldr.kmw.moe) runs on an [Linode 8GB Shared CPU](https://www.linode.com/docs/products/compute/shared-cpu/) instance. This instance hosts TL;DR and the [Language Tool server](https://dev.languagetool.org/http-server).

[PM2](https://pm2.keymetrics.io) and [Caddy](https://caddyserver.com) run in front of TL;DR. PM2 handles monitoring, startup, and restarting TL;DR. Caddy acts as a proxy in front of PM2 and handles SSL certificates.

#### Going Forward

Ideally, we would have multiple Language Tool instances running behind a [Node Balancer](https://www.linode.com/products/nodebalancers/) or in a [Kubernetes cluster](https://www.linode.com/products/kubernetes/). TL;DR could distribute processing a document across these instances. For productizing TL;DR, a similar setup is recommended. If you change our [Hashnode Example](https://github.com/TwiRp/tldr/blob/a3f74f818060dca682355368a6dbc43c54415e70/public/js/hashnode.js#L48) to enable Language Tool, you'll see rendering time increases. A distributed setup on [Linode](https://www.linode.com) allows for the best summaries in near real time.

## Examples

[![Homepage Tool](https://twirp.km-w.net/hashnode/tldr-homepage.jpg align="center")](https://tldr.kmw.moe)

[TL;DR's Homepage](https://tldr.kmw.moe) provides an interactive form for playing with the tool.

### Hashnode Summarized

![Hashnode TL;DR'd](https://twirp.km-w.net/hashnode/tldr-hashnode.jpg align="center")

[Browse Hashnode Summarized](https://tldr.kmw.moe/hashnode). These summaries are using TL;DR **without** Language Tool. 3 sentences summarize each post.

#### With vs Without Language Tool

Using Language Tool for filters can improve our summaries. Let's look at [What is Test Driven Development](https://fluttergamedev.com/what-is-test-driven-development) by Stephan E.G. Veenstra of [fluttergamedev.com](https://fluttergamedev.com).

##### With Language Tool

> Test Driven Development, or TDD, is a software development method where one writes the tests before the implementation. This article is the first in the Introduction to TDD series in which we will build the logic for the game Tic TAC Toe using TDD. We will do this by writing the minimal implementation to make the test pass, even if it means to return a hard-coded value.

##### Without Language Tool

> Test Driven Development, or TDD, is a software development method where one writes the tests before the implementation. By not thinking ahead you will really be guided by TDD. Writing tests in Dart could not have been easier.

With Language Tool, we have a little more context. Both grab the definition of TDD, but without Language Tool, the sentences are disconnected. With Language Tool, we know the article covers building Tic Tac Toe and making tests pass.

### The Hackathon Summarized

![Summary of Hashnode x Linode Hackathon](https://twirp.km-w.net/hashnode/summary-header.jpg align="center")

The [Build with Linode Hackathon](https://townhall.hashnode.com/build-with-linode-hackathon-june-2022?source=hashnode_countdown) summarized in 5 sentences becomes:

> Build an interesting open-source project using Linode and win up to 1000 USD and cool swag. We're super excited to announce the Linode Hackathon — Build, deploy, and scale your application easily and cost-effectively in the cloud with Linode. Linode products, services, and people enable developers and businesses to build, deploy, and scale applications more easily and cost-effectively in the cloud. Build an exciting open-source app of your choice using Linode and its products during the whole of June. This time around, we've got 5 Grand Prizes and 10 Runner Up Prizes on the line.

## Conclusion

And now you know, ya know?

Thanks to [Linode](https://linode.com) and [Hashnode](https://hashnode.com) for encouraging me to build something for the [Linode x Hashnode Hackathon](https://townhall.hashnode.com/build-with-linode-hackathon-june-2022). The code is [on GitHub](https://github.com/TwiRp/tldr).

Looking towards the future, I'm hoping to utilize Part of Speech Tagging and [Link Grammar](https://en.wikipedia.org/wiki/Link_grammar) to shorten sentences.