First of all: this is based on the idea of https://simonwillison.net/2020/Oct/9/git-scraping/ and https://github.com/simonw/ca-fires-history

The gist:

  • every 20min
    • scrape hackernews frontpage items
    • save them in hn.json
    • commit and push

Find the git repository here at christian-fei/hn-history

The schedule is done with the following GitHub Workflow file:

name: scrape hn data

on:
push:
workflow_dispatch:
schedule:
- cron: '5,25,45 * * * *'

jobs:
scheduled:
runs-on: ubuntu-latest
steps:
- name: checkout
uses: actions/[email protected]
- name: fetch hndata
run: |-
npx @christian_fei/hn --json | jq . > hn.json
- name: Commit and push if it changed
run: |-
git config user.name "Automated"
git config user.email "[email protected]"
git add -A
timestamp=$(date -u)
git commit -m "Latest data: ${timestamp}" || exit 0
git push

The data is scraped with @christian_fei/hn using the --json flag.

Read more about it here

Git diff FTW

The diff looks actually quite comprehensive:

   {
     "title": "Hooking Up Our Custom OS to a Standard Library",
     "url": "https://blog.stephenmarz.com/2020/10/25/hooking-up-our-custom-os-to-a-standard-library/",
-    "upvotes": 18,
+    "upvotes": 22,
     "author": "azhenley",
     "comments": null,
     "link": "https://news.ycombinator.com/item?id=25008953"
   },
+  {
+    "title": "HP ends its customers' lives [redefines 'lifetime' deal]",
+    "url": "https://pluralistic.net/2020/11/06/horrible-products/#inkwars",
+    "upvotes": 143,
+    "author": "samizdis",
+    "comments": 66,
+    "link": "https://news.ycombinator.com/item?id=25008894"
+  },
   {
     "title": "Gron – Make JSON Greppable",
     "url": "https://github.com/tomnomnom/gron",
-    "upvotes": 247,
+    "upvotes": 252,
     "author": "capableweb",
-    "comments": 58,
+    "comments": 59,
     "link": "https://news.ycombinator.com/item?id=25006277"
   },
   {
     "title": "Double Robotics",
     "url": "https://angel.co/company/double-robotics/jobs/73583-operations-and-supply-chain-manager"
   },
-  {
-    "title": "HP ends its customers' lives [redefines 'lifetime' deal]",
-    "url": "https://pluralistic.net/2020/11/06/horrible-products/#inkwars",
-    "upvotes": 118,
-    "author": "samizdis",
-    "comments": 47,
-    "link": "https://news.ycombinator.com/item?id=25008894"
-  },
   {
     "title": "Ask HN: As a person, what can I do to improve a city?",
     "url": "item?id=25007697",
-    "upvotes": 231,
[diff] Changes to 'hn.json' - line 74 of 346                                                                                                                                                                                               34%