# Install required packages
!pip install --quiet ipywidgets pymupdf pandas sentence-transformers
print('✓ Packages installed!')
Now let's work with some data.
We'll start with some semantic search, like in the Luanda Leaks investigation
from pathlib import Path
import fitz
import pandas as pd
from sentence_transformers import SentenceTransformer
pd.options.display.max_colwidth = 500
PDF_DIR = Path("data/pdfs")
rows = []
for pdf_file in sorted(PDF_DIR.rglob("*.pdf")):
with fitz.open(pdf_file) as pdf:
for page_number, page in enumerate(pdf, start=1):
text = " ".join(page.get_text().split())
if text:
rows.append({
"file": pdf_file.name,
"page": page_number,
"text": text,
})
pages = pd.DataFrame(rows)
pages.head()
| file | page | text | |
|---|---|---|---|
| 0 | 1016119-kristin-peterson-case-amended-complaint.pdf | 1 | Peter D. Hoffman, Esq. (PH-8306) Jamie Mattice, Esq. (JM-0307) Catherine Laney, Esq. (CL-6340) Melanie Williams, Esq. (MW-2602) Law Office of Peter D. Hoffman, P.C. Attorneys for Plaintiffs 200 Katonah Ave. Katonah, NY 10536 (914) 232-2242 phone facsimile (914) 232-2245 pdh2@pdhoffmanlaw.com e-mail UNITED STATES DISTRICT COURT SOUTHERN DISTRICT OF NEW YORK Kristin M. Peterson, Plaintiffs, -against- Katonah Lewisboro School District; Katonah Lewisboro School District Board of Education, Paul ... |
| 1 | 1016119-kristin-peterson-case-amended-complaint.pdf | 2 | 2. disability and in retaliation for Peterson's advocacy on behalf of special needs students in her charge. Plaintiffs claims include but are not limited to intentional infliction of emotional distress, defamation, negligent supervision, negligent hiring, negligent credentialing, retaliation, hostile work environment, discrimination based on disability, violations of civil rights under both the state and federal law, violation of her civil rights as pursuant to Americans With Disabilities ci... |
| 2 | 1016119-kristin-peterson-case-amended-complaint.pdf | 3 | Waccabuc and Vista. Its administrative offices are located at P.O. Box 387, Katonah, NY 10536. 4. Defendant KLSD Board of Education ("BOE") is the board of education for the above mentioned KLSD, located at P.O. Box 387, Katonah, NY 10536. The KLSD BOE members include: Mark Lipton, Charles Day, Peter Breslin, Janet Harckham, Marjorie Schiff, Stephanie Tobin and Peter Treyz. 5. Dr. Paul Kreutzer ("Kreutzer") is the Superintendent of Schools for KLSD. He is sued in his individual and corporate... |
| 3 | 1016119-kristin-peterson-case-amended-complaint.pdf | 4 | exercise supplementary jurrisdiction over state claims pursuant to 28 U.S.C. §1367. 12. Jurisdiction is appropriate because Plaintiff filed a claim with the Equal Employment Opportunity Commission and a Right to Sue was issued on December 27,2012, EEOC Charge No.520201201700. The same was received by Plaintiff on December 31, 2012. 13.. Jurisdiction is further invoked by Defendants' Notice of Removal. FACTS PERSONAL HISTORY AND EXPERIENCE Plaintiff Peterson currently is employed as a from a ... |
| 4 | 1016119-kristin-peterson-case-amended-complaint.pdf | 5 | discrimination and retaliation, as based on disability, advocacy on behalf of her students and her speech. This conduct is ongoing through the date of this complaint. 21, Since sometime in September 2011, the District and all Defendants have retaliated against Plaintiff as it concerns her advocacy on behalf of her students, her being a witness in a federal litigation before this Court, and for related to the her disability. advocacy and complaints 22., Since sometime in September 2011 Plaint... |
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
embeddings = model.encode(
pages["text"].tolist(),
normalize_embeddings=True,
)
pages["embedding"] = list(embeddings)
pages.head()
Loading weights: 0%| | 0/103 [00:00<?, ?it/s]
| file | page | text | embedding | score | |
|---|---|---|---|---|---|
| 0 | 1016119-kristin-peterson-case-amended-complaint.pdf | 1 | Peter D. Hoffman, Esq. (PH-8306) Jamie Mattice, Esq. (JM-0307) Catherine Laney, Esq. (CL-6340) Melanie Williams, Esq. (MW-2602) Law Office of Peter D. Hoffman, P.C. Attorneys for Plaintiffs 200 Katonah Ave. Katonah, NY 10536 (914) 232-2242 phone facsimile (914) 232-2245 pdh2@pdhoffmanlaw.com e-mail UNITED STATES DISTRICT COURT SOUTHERN DISTRICT OF NEW YORK Kristin M. Peterson, Plaintiffs, -against- Katonah Lewisboro School District; Katonah Lewisboro School District Board of Education, Paul ... | [-0.07744623, 0.09072887, -0.0004004485, -0.0051214853, -0.0027211215, 0.013183706, -0.020780003, 0.016688656, 0.0342256, 0.06389175, 0.063314505, 0.020075949, 0.018136352, 0.006131184, 0.038879734, 0.08067422, 0.09912529, 0.030090163, -0.08837984, -0.018547278, 0.016594667, 0.010660298, -0.0026030466, 0.036807563, -0.08744563, 0.06732471, 0.024465473, -0.07606792, -0.02574135, -0.055101804, -0.05106238, -0.050301507, -0.06312121, 0.018399904, 0.059393387, 0.12351488, 0.06971986, 0.11833442,... | 0.210745 |
| 1 | 1016119-kristin-peterson-case-amended-complaint.pdf | 2 | 2. disability and in retaliation for Peterson's advocacy on behalf of special needs students in her charge. Plaintiffs claims include but are not limited to intentional infliction of emotional distress, defamation, negligent supervision, negligent hiring, negligent credentialing, retaliation, hostile work environment, discrimination based on disability, violations of civil rights under both the state and federal law, violation of her civil rights as pursuant to Americans With Disabilities ci... | [-0.020636197, 0.038847182, -0.031254467, 0.10141908, 0.019957587, 0.1156925, 0.025005948, 0.033193097, -0.024900632, -0.019641371, 0.03047271, -0.010507121, -0.03374784, 0.044401724, 0.0010828539, 0.070589066, 0.07218502, 0.047929693, -0.00915258, 0.03644709, 0.026810024, 0.01309823, -0.071218476, -0.0003581742, -0.09282795, -0.015275545, -0.06814902, -0.060921073, 0.018677471, 0.034956943, -0.08796535, -0.02415037, -0.009773831, 0.09239155, 0.07710754, 0.027503146, 0.03855012, 0.07448419, ... | 0.315574 |
| 2 | 1016119-kristin-peterson-case-amended-complaint.pdf | 3 | Waccabuc and Vista. Its administrative offices are located at P.O. Box 387, Katonah, NY 10536. 4. Defendant KLSD Board of Education ("BOE") is the board of education for the above mentioned KLSD, located at P.O. Box 387, Katonah, NY 10536. The KLSD BOE members include: Mark Lipton, Charles Day, Peter Breslin, Janet Harckham, Marjorie Schiff, Stephanie Tobin and Peter Treyz. 5. Dr. Paul Kreutzer ("Kreutzer") is the Superintendent of Schools for KLSD. He is sued in his individual and corporate... | [-0.0821188, -0.0022547084, 0.03564755, -0.031478677, -0.048447054, 0.053189907, 0.021373166, -0.035540946, 0.10609911, 0.072753415, -0.0117344605, 0.033416115, 0.0126290005, 0.012465783, -0.04687067, -0.031334445, -0.017814375, 0.03095199, -0.07347055, -0.045563832, 0.03181553, -0.0075045684, 0.015843274, 0.005670609, -0.06899838, -0.0098223435, 0.07756801, -0.068226986, 0.010307242, -0.051110562, 0.010049241, -0.022165615, -0.050075155, -0.012985409, 0.07344942, 0.056460656, 0.08203787, 0.... | 0.126130 |
| 3 | 1016119-kristin-peterson-case-amended-complaint.pdf | 4 | exercise supplementary jurrisdiction over state claims pursuant to 28 U.S.C. §1367. 12. Jurisdiction is appropriate because Plaintiff filed a claim with the Equal Employment Opportunity Commission and a Right to Sue was issued on December 27,2012, EEOC Charge No.520201201700. The same was received by Plaintiff on December 31, 2012. 13.. Jurisdiction is further invoked by Defendants' Notice of Removal. FACTS PERSONAL HISTORY AND EXPERIENCE Plaintiff Peterson currently is employed as a from a ... | [-0.09988051, 0.08519423, 0.045242846, 0.026878005, 0.019499475, 0.12228931, -0.034660377, 0.0069440263, 0.035077564, 0.063032106, 0.07624761, 0.011396352, -0.059652306, 0.020509237, 0.01994301, 0.040922392, 0.0383941, 0.06711617, 0.013109952, -0.03216053, 0.02541797, -0.055816613, -0.023790661, 0.020030433, 0.03649771, -0.02883284, 0.030514803, -0.045460816, 0.051936097, 0.03325806, -0.036858976, 0.043835316, -0.029568644, 0.01621314, 0.08867518, 0.03924013, 0.005968093, 0.06334705, -0.0506... | 0.390931 |
| 4 | 1016119-kristin-peterson-case-amended-complaint.pdf | 5 | discrimination and retaliation, as based on disability, advocacy on behalf of her students and her speech. This conduct is ongoing through the date of this complaint. 21, Since sometime in September 2011, the District and all Defendants have retaliated against Plaintiff as it concerns her advocacy on behalf of her students, her being a witness in a federal litigation before this Court, and for related to the her disability. advocacy and complaints 22., Since sometime in September 2011 Plaint... | [-0.0042881654, 0.05473239, -0.025942776, -0.046387196, -0.018789876, 0.020053677, 0.08319826, 0.06251034, -0.035859536, 0.05488037, 0.047510363, 0.066196494, -0.028870234, -0.021695174, -0.048249166, 0.063006096, 0.07640012, 0.07334448, -0.07030505, 0.068065576, -0.042870495, 0.058684804, -0.048461158, 0.043108854, -0.014551013, 0.012546345, 0.021128988, -0.064531796, -0.022991423, 0.03239436, -0.004105635, 0.023135452, -0.16799109, 0.052964922, 0.05040604, 0.04945842, -0.039399933, 0.00119... | 0.233624 |
query = "police misconduct"
query_embedding = model.encode(
[query],
normalize_embeddings=True,
)[0]
pages["score"] = pages["embedding"].apply(
lambda embedding: embedding @ query_embedding
)
(
pages
.sort_values("score", ascending=False)
.drop('embedding', axis=1)
.head(10)
)
| file | page | text | score | |
|---|---|---|---|---|
| 2529 | 2780518-jessica-benn-civil-action.pdf | 11 | 11 of the City and Department. The policymakers for the City and the Department knew of these problems, allowed them to continue, and made decisions not to implement adequate policies, training, or supervision. 39. The constitutional violations complained of by Ms. Benn were a highly obvious and predictable consequence of a failure to equip Denver police officers with the specific tools—including policies, training and supervision—to handle the recurring situation of how to handle civilians ... | 0.651971 |
| 2973 | 3288688-michele-vandegrift-civil-action.pdf | 27 | 27 conducted an investigation resulting in only one alleged harasser—Detective Ruth—charged with violating City policy. The City has not yet disciplined Detective Ruth for his charged misconduct. Captain Derbyshire admits he could have required harassment training, but he did not do so. Instead, the City transferred Ms. Vandegrift to a different squad and then later to a different division. Ms. Vandegrift’s expert on workplace investigations recognized many deficiencies in the investigation ... | 0.629623 |
| 2526 | 2780518-jessica-benn-civil-action.pdf | 8 | 8 have tended to disprove allegations of police misconduct, and sometimes they have tended to prove allegations of police misconduct. 31. Public protests and demonstrations are newsworthy events, and arrests taking place at these events are especially newsworthy. Recordings often provide the only objective evidence of the behavior of demonstrators, protestors, and police officers. 32. The right of citizens to record police officers performing their public duties is so important that the Unit... | 0.626292 |
| 2533 | 2780518-jessica-benn-civil-action.pdf | 15 | 15 52. This First Amendment right to gather, receive, record, and disseminate information includes the right to audio and video record police officers performing their duties in public. 53. Police officers performing their public duties in public places have no reasonable expectation that their conduct is private and will not be recorded, published, and disseminated. 54. In the manner described more fully above, Defendant Lopez’s actions caused her injury that would chill a person of ordinar... | 0.624630 |
| 2530 | 2780518-jessica-benn-civil-action.pdf | 12 | 12 42. Furthermore, in the immediate aftermath of the Denver police shooting of Jessica Hernandez in January 2015, a witness tried to video-record the officers’ actions in public while she was standing on her own property. The officers threatened the witness and ordered her not to record them. 43. The above-described widespread practice was so well-settled as to constitute de facto policy in the Denver Police Department, and it was allowed to exist because municipal policymakers with authori... | 0.614012 |
| 2947 | 3288688-michele-vandegrift-civil-action.pdf | 1 | IN THE UNITED STATES DISTRICT COURT FOR THE EASTERN DISTRICT OF PENNSYLVANIA MICHELE VANDEGRIFT : : CIVIL ACTION : v. : NO. 16-2999 : CITY OF PHILADELPHIA : MEMORANDUM KEARNEY, J. January 11, 2017 When a female detective complains about specific sexual assaults and harassment creating a hostile work environment involving certain officers, the police department must recognize, like any employer, its obligation to comprehensively and impartially address and evaluate appropriate remedies. The f... | 0.599236 |
| 4540 | 6882535-second-amended-complaint-10-5-18.pdf | 7 | SECOND AMENDED COMPLAINT AND JURY DEMAND – 7 (2:18-ccv-00506-MJP) Williams, Kastner & Gibbs PLLC 601 Union Street, Suite 4100 Seattle, Washington 98101-2380 (206) 628-6600 6407264.3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 55. The officers believe that through civil discovery—which Sawant frenetically stonewalled and sought secrecy orders in relation to, in Superior Court—will uncover a pattern of culpable conduct and further defamatory statements that she made with ... | 0.595451 |
| 2527 | 2780518-jessica-benn-civil-action.pdf | 9 | 9 Garcia v. Montgomery County, Maryland, et al., 8:12-CV-3592-JFM (D. Md. Mar. 4, 2013). Defendant City’s Failure to Provide Any Policies, Training, or Supervision, and the Widespread Practices of Denver Police Officers 33. The misconduct described herein was undertaken pursuant to the policy of the Defendant City and its Police Department in that, at the time of the events in this case, the City and the Department had inadequate policies guiding Denver police officers on civilians’ First Am... | 0.591431 |
| 4535 | 6882535-second-amended-complaint-10-5-18.pdf | 2 | SECOND AMENDED COMPLAINT AND JURY DEMAND – 2 (2:18-ccv-00506-MJP) Williams, Kastner & Gibbs PLLC 601 Union Street, Suite 4100 Seattle, Washington 98101-2380 (206) 628-6600 6407264.3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 4. In February 2016, Officer Spaulding and Detective Miller were apprehending Che Taylor—a violent felon and rapist—when he began pulling a gun on them. 5. The officers were forced to fire, defending their lives and ending his. In less than a week,... | 0.580136 |
| 2983 | 3288688-michele-vandegrift-civil-action.pdf | 37 | 37 blind, or at least severely impaired, eye to specific complaints of sexual assault and harassment by identified officers upon the policewomen proudly serving our community. Ms. Vandegrift adduces facts necessary to create a genuine issue of material fact as to her claims of gender discrimination, hostile work environment, and retaliation under Title VII, the PHRA, and the Philadelphia Fair Practices Ordinance. Ms. Vandegrift also provides sufficient evidence of a well-settled custom of se... | 0.578670 |
Great. But what do we do with that???? Horrible interface!