A pixelated FOIA scan with an invisible table

This PDF holds wait-time data for a state agency call center, released through a public records request. It’s a heavily pixelated scan — no text layer at all — and the table on page one has ruling lines the scanner half-dissolved. Reading the numbers is hard for a human; for a machine it takes OCR plus hand-drawn column boundaries.

One wrinkle before the PDF itself: this host rejects downloads from Python’s built-in urllib (a plain PDF(url) gets a 403), so fetch the bytes with requests and hand them over — PDF() accepts a path, a URL, or any file-like object.

from io import BytesIO

import requests
from natural_pdf import PDF

url = "https://pub-4e99d31d19cb404d8d4f5f7efa51ef6e.r2.dev/pdfs/statecallcenterdata_redacted/statecallcenterdata_redacted.pdf"
pdf = PDF(BytesIO(requests.get(url).content))
page = pdf.pages[0]
page.show()

The pages are images, so there’s no text to extract — always worth double-checking before blaming your selectors:

# Empty? Needs OCR.
print(repr(page.extract_text()))

''

Apply OCR

apply_ocr() uses RapidOCR by default — its models ship inside the package, so there’s no download on first use. Two ways to check the results: look at where it found text, and look at what the text says.

page.apply_ocr()
page.find_all('text').show(crop=True)

print(page.extract_text(layout=True))

On-Demand Interviews-Interim and Final Reporting
                    Complete all shaded fields
                          Figure    Comments'   State:
        Average call wait time for interriew.n.minutes 19Minutes Repart Start Date 9/1/2019
        Number of all calls that result in a completed interview 34,396 Repart End Date 3/31/2020
                   Percent 98.31%
                                                Numberof applications
          Average callcompletion.time in minutes 29 filed  34438
                                                Numberof recertifications
              Number of dropped.calls 627       filed      17604
                                These were the completed Face to face Numberof applications
        Number of requests for an in-person interview 16982 ntervews.There is no way to track in the intervlewed on 1st day 22409
        Number of NOMIs sent for failu re to complete:
             initial applicationinterview 7809
                   Percent 22.60%
        Number of NOMis sent for failure to complete
              recertificetion interview. 13007
                   Percent 73.80%
         Number olapplcations denied tor fallure to
            complete the interview in 30 days 3885
                   Percent 45.00%
        Number of recertific ations denied for fallure to
            ompete the inteiewn 30 days 74
                   Percent 10.00%
      Please use the comments fied for any claritications or context that are needed for any-data points.
                      Notes on Measures
      vi r .cti krcllo
      begin theinterview.
      2umbr and prntf allcallsthatna compld inviwnce abandon and opped calnthe enatr when caclaingth
      nercentage.
      3.Average call completon tine in minutes Completion time means the ful duration of time the client sperds on the cal, beginning when the client.
      n thcllcnrqu the tnomplted
      4.Nmber of dropped allsInclude all cas iscoanectd due to call enter eror, ck of call enter capacity,etcDono inctude abandone calls
      duingwhich the intermiat the cllfora complation
      nterwwhrough the callcerterbt reque sted anin-personterviw insted
      .Numbe and prnf N stor a ocomp ilapon niwne all apcaons csst csadinthe
      denominator when cairalating the percentage
      7umand p s sor rcomp hetiiconvin al tiacrothecasdnthe
      8.Number and percent of all applications denied that were denied due to fallure tc compete the interview ir 30 days: include all applicatio ns scross
      tha cacolcad forwhicha donal wat kcuod ntha donominator whoncalcuating tho percentage
      umbe and pecent of al ecerticatios denied that were denieddue to fallure to complte the sterviev in 30daysInclude afl recertfications
     cross the caselesd for whichs donis was ieued in the donomintor when cleulating tho percentage.

Legible, with the usual scan-quality noise (interriew.n.minutes). The structure survived, and that’s what the extraction below leans on.

Isolate the table area

The table runs from the “Figure” header down to the “Please use the comments field” instruction:

table_area = (
    page
    .find('text:contains(Figure)')
    .below(
        until='text:contains(Please use the comments)',
        include_endpoint=False
    )
)
table_area.show(crop='wide')

That’s the right rows but the full page width. Cut in from the right (the table only occupies the left 40% of the page), trim the left margin, and add a hair at the bottom. These are hand-picked values — with OCR boxes there’s often no cleaner anchor to snap to:

table_area = (
    page
    .find('text:contains(Figure)')
    .below(
        until='text:contains(Please use the comments)',
        include_endpoint=False
    )
    .expand(
        right=-(page.width * 0.58),
        left=-30,
        bottom=3
    )
)
table_area.show(crop='wide')

Confirm the area holds the right text elements:

table_area.find_all('text').show(crop=True)

Draw the grid

extract_table() alone can’t split these columns — the gaps between them are too inconsistent after OCR. So drop three vertical dividers, then shuffle them into the whitespace so they don’t cut through any text. The rows are easier: the scan still has enough of its ruling lines for pixel detection to find them.

from natural_pdf.guides import Guides

guide = Guides(table_area)
guide.vertical.divide(3)
guide.vertical.snap_to_whitespace(detection_method='text')
guide.horizontal.from_lines(detection_method='pixels')
guide.show()

And now the table comes out as data:

df = (
  guide
  .extract_table()
  .to_df(
    header=['value', 'amount', 'comments']
  )
)
df

	value	amount	comments
0	Average call wait time for interriew.n.minutes	19Minutes	None
1	Number of all calls that result in a completed...	34,396	None
2	Percent	98.31%	None
3	Average callcompletion.time in minutes	29	None
4	Number of dropped.calls	627	None
5	Number of requests for an in-person interview	16982	These were the completed Face to face\nntervew...
6	Number of NOMIs sent for failu re to complete:...	7809	None
7	Percent	22.60%	None
8	Number of NOMis sent for failure to complete\n...	13007	None
9	Percent	73.80%	None
10	Number olapplcations denied tor fallure to\nco...	3885	None
11	Percent	45.00%	None
12	Number of recertific ations denied for fallure...	74	None
13	Percent	10.00%	None

The numbers carry OCR artifacts (19Minutes, merged words) — that’s cleanup work for pandas, not a reason to re-run extraction. If a value is unreadable, page.compare_ocr(engines=[...]) shows what other engines make of the same crop.

The later pages of this PDF are a different beast — multi-year pivot grids at the same scan quality:

pdf.pages[1].show()

That one needs its own strategy (and possibly a better copy of the document).