# Install required packages
!pip install --upgrade --quiet playwright
!pip install --upgrade --quiet beautifulsoup4
!pip install --upgrade --quiet lxml
!pip install --upgrade --quiet html5lib
!pip install --upgrade --quiet pandas
!pip install --upgrade --quiet nest_asyncio
print('✓ Packages installed!')
Slides: browser-automation.pdf
In this example we are going to scrape locksmiths from Maryland's licensing queries site.
Traditionally Python programmers use BeautifulSoup to scrape content from the interent. Instead of being traditional, we're going to use Playwright, a browser automation tool! This means you actually control the browser! Filling out forms, clicking buttons, downloading documents... it's magic!!!✨✨✨
Maryland locksmiths¶
- Inspecting the page
- Filling in a text box
- Working through a list of inputs (zip codes, in this case)
- Combining dataframes
- Back button
Installation¶
We need to install a few tools first! Remove the # and run the cell to install the Python packages and browsers that we'll need for our scraping adventure.
# %pip install --quiet lxml html5lib beautifulsoup4 pandas
# %pip install --quiet playwright
# !playwright install-deps
# !playwright install chromium firefox
And we'll set it up so Playwright will be sure to work on Windows.
# Detect if we're running in Google Colab
import os
IN_COLAB = 'COLAB_GPU' in os.environ or 'COLAB_RELEASE_TAG' in os.environ
import platform
import asyncio
import nest_asyncio
if platform.system() == "Windows":
asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
try:
asyncio.get_running_loop()
nest_asyncio.apply()
except RuntimeError:
pass
Opening up the browser and visiting our destination¶
from playwright.async_api import async_playwright
# "Hey, open up a browser"
playwright = await async_playwright().start()
# Colab can't open a visible browser, so we run headless there
if IN_COLAB:
use_headless = True
else:
use_headless = False
browser = await playwright.chromium.launch(headless=use_headless)
# Create a new browser window
page = await browser.new_page()
await page.goto("https://www.dllr.state.md.us/cgi-bin/ElectronicLicensing/OP_Search/OP_search.cgi?calling_app=LOCKSMITH::LOCKSMITH_personal_location")
from IPython.display import Image
Image(await page.screenshot())
Filling in a text box¶
You always start with await page.locator("input").fill("whatever you want"). You'll probably get an error because there are multiple inputs on the page, but Playwright doesn't know which one you want to use! Just read the error and figure out the right one.
# 20601
# 20602
# 20603
# 20606
# 20607
# 20608
# 20609
# await page.locator("input").fill("20602")
await page.locator("[name='zip']").fill("20602")
# await page.get_by_text("Search").click()
await page.get_by_role("button", name="Search").click()
Grab the tables from the page¶
Pandas is the Python equivalent to Excel, and it's great at dealing with tabular data! Often the data on a web page that looks like a spreadsheet can be read with pd.read_html.
You use await page.content() to save the contents of the page, then feed it to read_html to find the tables. len(tables) checks the number of tables you have, then you manually poke around to see which one is the one you're interested in. tables[0] is the first one, tables[1] is the second one, and so on...
import pandas as pd
from io import StringIO
html = await page.content()
await page.wait_for_selector("table", timeout=5000)
tables = pd.read_html(StringIO(html))
len(tables)
tables[0]
await page.go_back()
Fill out the ZIP code field again and again and again¶
I found a list of zipcodes on the internet! I pasted them below, then used .split() to make them into something we could use in Python.
zipcodes = """20906
21234
20878
21740
20874
21122
21222
21117
20904
20744
21061
21215
20902
20772
21207
20850
21206
20774
20783
21228
20854
20852
21043
21702
21218
21044
21921
20910
21224
21229""".split("\n")
print(zipcodes)
Now we fill out the form for each and every zip code, one by one, pulling out the tables and saving them and adding them to the list.
import pandas as pd
from io import StringIO
all_data = pd.DataFrame()
# Go to the front page
await page.goto("https://www.dllr.state.md.us/cgi-bin/ElectronicLicensing/OP_Search/OP_search.cgi?calling_app=LOCKSMITH::LOCKSMITH_personal_location")
# Search for each zipcode
for zipcode in zipcodes:
print("Searching for", zipcode)
# Fill out the form and search
await page.locator("[name='zip']").fill(zipcode)
await page.get_by_role("button", name="Search").click()
# try:
# Get all of the tables on the page
try:
await page.wait_for_selector("table", timeout=5000)
html = await page.content()
tables = pd.read_html(StringIO(html))
except:
tables = []
# Get the table (and edit if necessary)
if len(tables) > 0:
df = tables[0]
print("Found", len(df))
# Add the tables on this page to
all_data = pd.concat([all_data, df], ignore_index = True)
else:
print("Nothing found")
# Save after each zip code in case something breaks
all_data.to_csv("output.csv", index=False)
# Go back and start again
await page.go_back()
len(all_data)
all_data.head()
Saving the results¶
Now we'll save it to a CSV file! Easy peasy.
all_data.to_csv("output.csv", index=False)