how to automate real estate or job searches on a mac

recently i found myself waking up every day with computer chores: check websites and see if they had any new information for me. i wanted to free up this time so i built out an automated solution: once a day, if these sites change, send me an email.

here is how i did it on macosx big sur 11.4

open terminal.app, and set an editor:

echo 'export EDITOR=nano' >> ~/.zshrc 
source ~/.zshrc

install urlwatch:

pip3 install urlwatch

find a page you want to monitor, in my case here is an example:

https://deshow.com/advance-search/?operation=en-venta&type=all&subtipo=all&location=isabela&status=all&keyword=&price_range_min=0&price_range_max=3000000&bathrooms=&bedrooms=&pageid=25409

open google chrome developer tools and get the xpath of where the new content will be displayed. see how I do this here

now add the details into urlwatch

urlwatch --edit

content:

name: "real estate watcher"
url: "https://deshow.com/advance-search/?operation=en-venta&type=all&subtipo=all&location=isabela&status=all&keyword=&price_range_min=0&price_range_max=3000000&bathrooms=&bedrooms=&pageid=25409"
filter:
  - xpath: /html/body/div[1]/div/div/div/div[3]

let’s get email notifications working using gmail:

urlwatch --edit-config

content:

report:
  discord:
    embed: false
    enabled: false
    max_message_length: 2000
    subject: '{count} changes: {jobs}'
    webhook_url: ''
  email:
    enabled: true
    from: 'YOUREMAIL'
    html: false
    method: smtp
    sendmail:
      path: sendmail
    smtp:
      keyring: true
      host: smtp.gmail.com
      port: 587
      starttls: true
      user: 'YOUREMAIL'
    subject: '{count} changes: {jobs}'
    to: 'YOUREMAIL'

turn on gmail app passwords (requires two factor) then type password here:

urlwatch --smtp-login

run urlwatch for the first time to get current state:

urlwatch

run it again to see if you have a good stable baseline:

urlwatch

great! lets set up apple launch daemon:

sudo su
nano /Library/Scripts/watch.sh

script content:

#!/bin/bash
	/usr/local/bin/urlwatch

exit root:

exit

apple loves plists, so let’s set one up:

nano ~/Library/LaunchAgents/com.watch.plist

content:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.watch</string>
    <key>Program</key>
    <string>/Library/Scripts/watch.sh</string>
    <key>StandardOutPath</key>
    <string>/tmp/watch.sh.stdout</string>
    <key>StandardErrorPath</key>
    <string>/tmp/watch.sh.stderr</string>
    <key>StartCalendarInterval</key>
    <dict>
        <key>Hour</key>
        <integer>12</integer>
        <key>Minute</key>
        <integer>10</integer>
    </dict>
</dict>
</plist>

now let’s add our script to launchcontrol:

sudo launchctl load -w ~/Library/LaunchAgents/com.watch.plist

voilà! now you have a zero cost website monitor that runs once a day, even if your laptop is asleep.

Written on June 24, 2021