Cheap First, Ask the LLM Later? Cost and Accuracy of Rule-Based, Native-PDF, and Text-Based LLM Extraction of Syllabus Deadlines

1. Introduction

Course syllabi are the main source of assessment deadlines for university students, but they arrive in inconsistent formats: tables, week-by-week schedules, calendar grids, slides, and prose. Several student tools now extract deadlines automatically by sending syllabus content to an LLM. Because LLM calls cost money per token, a natural engineering choice is a cheap-first design: run deterministic rules first, and call the LLM only when the rules seem to fail. The same idea underlies LLM cascades, which route easy queries to cheaper models [1].

Whether cheap-first works depends on two things: how much the rules save, and how reliably the system can tell when the rules have failed. We found no controlled study that measures both cost and accuracy for syllabus deadline extraction. Practitioner write-ups claim large savings from sending extracted text instead of native PDFs, but without matched accuracy measurements.

This paper makes three contributions:

1.

A labelled benchmark of 16 public syllabi from eight North American institutions with 151 hand-verified deadlines, including deliberately difficult cases (calendar grids, a syllabus with no fixed dates, and source-document errors).

2.

A controlled comparison of five extraction strategies on token cost, latency, and accuracy, with three repeats per LLM strategy and significance tests.

3.

An error analysis showing that wrong-year inference dominates rule-based failures, that text extraction loses dates printed in calendar grids, and that a rule’s output size is a poor signal of its correctness.

The extraction code under test comes from GRYPH OS, an open-source student planning tool built during a 24-hour hackathon. Strategies B and C reuse its code unchanged so that the results describe a real deployed design.

2. Related Work

LLM cascades. FrugalGPT [1] routes queries through a sequence of models of increasing cost and stops when a learned scorer judges an answer reliable, reporting large cost reductions. RouteLLM [7] likewise learns routers from preference data to choose between a stronger and a weaker model. Our hybrid strategy is a simpler cascade whose first stage is a rule system and whose stopping rule is a fixed threshold rather than a learned scorer. We test whether such a simple rule is enough.

Temporal expression extraction. Rule-based temporal taggers such as HeidelTime [2] and SUTime [3] recognise and normalise date expressions in text. We include the Python library dateparser [4] as a runnable general-purpose baseline. Deadline extraction is harder than temporal tagging because a document contains many dates that are not deadlines.

LLM extraction from PDFs. Recent work extracts structured metadata from scientific PDFs with LLMs and compares parsers on speed and accuracy [5], and hybrid OCR–LLM pipelines show that parser choice trades speed against accuracy, with tables breaking fast parsers [6]. Docling [8] is an open-source toolkit that converts PDFs into structured text with layout and table recognition. These studies do not compare native-PDF LLM input with extracted-text input on both cost and accuracy.

LLM information extraction and prompt cost. A survey of generative information extraction with LLMs [9] reviews how LLMs are used to extract structured information from text, and LLMLingua [10] reduces inference cost by compressing prompts. We do not compress prompts; we compare input formats and routing rules for one concrete extraction task.

Syllabus tools. We found no academic work on syllabus deadline extraction. Practitioner reports list common parser failures, including dates hidden in week labels, exam windows, and multi-part projects [12].

3. Strategies

All strategies output a list of events, each with a due date and a type (assignment, quiz, midterm, final, project, reading, or other).

• A: PDF→LLM. The binary PDF is sent to the model with GRYPH OS’s extraction prompt.
• B: text→LLM. Text is extracted with pypdf and sent to the model with the same prompt (capped at 30,000 characters). This is the pipeline GRYPH OS ships.
• C: regex. GRYPH OS’s rules: every line of extracted text is scanned with date patterns, and assessment keywords on the line set the event type. No LLM call.
• D: regex→LLM. Run C; if it finds fewer than three events, run B instead. This is the cheap-first design.
• E: dateparser. The same line loop as C, but dates are found with dateparser’s search_dates (day and month required). No LLM call.

The prompt asks the model to return JSON and to “infer the most likely upcoming year” when a year is missing. The rules (C, E) fill a missing year with the current calendar year. Both behaviours are inherited unchanged from GRYPH OS.

4. Method

4.1. Dataset

We collected 16 publicly accessible syllabi (Table 1) from Carleton University, the University of Calgary, Western University, the University of Texas at Austin, Midwestern State University, Richland Community College, Lamar State College Port Arthur, and the University of Wisconsin–Madison. They cover terms from 2016 to 2026, one to nine pages, and formats including tables, week-by-week schedules, calendar grids, lecture slides, and prose. The PDFs are not redistributed; the release lists each source URL with its SHA-256 checksum.

4.2. Labels

Gold labels contain only graded items with a specific date. Readings, lectures, withdrawal and refund dates, “no late work after” cut-offs, items marked TBD, and implicit recurring deadlines (“due every Sunday”) are excluded. For a window (“opens Feb 17, closes Feb 23”) the close date is used, and the year is taken from the stated term. Labels were pre-annotated by an LLM that is not under test (Anthropic Claude) and then checked against each PDF and corrected by the author. One syllabus (Wisconsin) prints no year; its year was taken from the source URL.

4.3. Metrics

We report precision, recall, and F1 on the set of due dates per syllabus, then average across syllabi. Because both the prompt and the rules guess missing years, which penalises past syllabi, we also report month-day F1, which ignores the year. For the syllabus with no fixed dates, F1 is 1 if nothing is predicted and 0 otherwise. Cost is measured as input plus output tokens from the API’s usage metadata; latency is wall-clock time.

4.4. Setup

All LLM calls used gemini-3.1-flash-lite, a model from Google’s Gemini family [11], through the google-genai SDK (2.33.0), with pypdf 6.21.2 and dateparser 1.4.3. All 240 jobs (16 syllabi × 5 strategies × 3 repeats) completed without errors on 25 October 2026. Differences are tested with a paired sign-flip permutation test (two-sided, 20,000 permutations) on per-syllabus F1, with bootstrap 95% confidence intervals (seed 0).

4.5. Use of AI Tools

Anthropic’s Claude (through Claude Code) was used in three ways: to pre-annotate the gold labels, which the author then checked line by line against each PDF and corrected; to write and run the benchmark code under the author’s direction; and to draft the text of this paper. Google’s Gemini model is the system under study. The author reviewed all text, checked every reported number against the saved outputs in the released repository, and is responsible for the final content. AI tools are not listed as authors.

5. Results

Table 2 and Figure 1 summarise the results.

Native PDF versus extracted text. Over the 16 syllabi, strategy A used 43,360 tokens and strategy B used 57,490, so sending extracted text cost 32.6% more than sending the PDF. A scored 0.051 higher in F1, but this difference was not significant (Table 3). In a pilot on five syllabi with a different model (gemini-3.8-flash), text input was about 8% cheaper, so the direction of the cost difference depends on the model.

Cheap-first routing. The hybrid D used 57.8% fewer tokens than B, because the rules were trusted in 69% of runs. Its F1 fell from 0.657 to 0.370, a significant loss. When the rules were trusted, F1 was 0.179; when the hybrid fell back to the LLM, F1 was 0.789. The rules’ event count therefore did not indicate whether the rules were right.

Routing threshold. Because D’s fallback is exactly B, we simulated D for other thresholds k from the saved predictions of B and C, without new API calls. At $k=3$ the simulation reproduces the measured D (F1 0.369 versus 0.370). Table 4 and Figure 2 show that F1 rises steadily as more syllabi are sent to the LLM: no threshold achieves both low cost and high accuracy. Adding a cheap check that all rule-extracted dates fall in one calendar year raises F1 at $k=3$ from 0.369 to 0.441 while calling the LLM on 44% of runs.

Stability. Across the three repeats, the mean standard deviation of per-syllabus F1 was 0.010 for A and 0.029 for B, so the LLM results were stable.

6. Error Analysis and Discussion

Wrong year. Counting predicted dates whose month and day match a gold deadline, the year was wrong for 58% of matches from the rules (66 of 114) and 85% from dateparser (261 of 306), compared with 2% for A and B (9 of 415 and 9 of 385) and 18% for D. Missing years are filled with the current year, so a Spring 2025 syllabus processed in 2026 receives 2026 dates. The rules’ output therefore changes with the day they are run.

Rule failures. The rules also read a week range as a year (“11/24–28” became 24 November 2028), did not match year-less numeric dates such as “9/26” at all, misread day-first dates such as “17 February”, and kept only one event per date. dateparser found dates more often but could not tell deadlines from other dates, giving the lowest precision (0.049).

Calendar grids. Two Carleton syllabi print assignment due dates only inside a calendar grid. Text extraction keeps the label “Due Friday” but loses the date. Recall on these two syllabi was 0.78 for A against 0.52 and 0.11 for B, and 0 for every rule-based strategy.

False precision. One syllabus contains no fixed dates: tests are “TBD between October 16–18” and “November 27–29”, and the final is “TBD December 12–23”. In every repeat, all three LLM strategies returned exactly one date per window: B and D always chose the first day of each window, and A chose the last day in two of three repeats and the first day in the third. These dates are not invented, but they present an uncertain window as a fixed deadline, and which day is chosen varies with the input format and between runs. The rules correctly returned nothing.

Definition of a deadline. On the two Richland syllabi the LLM strategies also returned withdrawal and refund dates, and on the Lamar syllabus they returned implied weekly deadlines. Our labels exclude both, which lowered LLM precision; scores are sensitive to how “deadline” is defined.

Robustness to source errors. One schedule is headed “Spring 2025” but lists only August–December dates, and contains a typo (“9/30/1”). Strategies A and B still scored 1.00 and 0.98 on it.

Engineering observations. GRYPH OS’s default model (gemini-2.0-flash) had been retired by the time of the study, and its fallback to rules only covers rate-limit errors, so the deployed pipeline fails rather than degrading. Changing the pypdf version from 6.14.2 to 6.21.2 changed the number of rule-extracted events on the same PDFs several-fold (for example, from 2 to 27), because line breaks in the extracted text changed.

7. Limitations

The benchmark is small (16 syllabi), labelled by one annotator, mostly from computer science and science courses, in English, and from Canada and the United States. We tested one LLM family and one model in the main study; free-tier quotas and retired models limited the choice. Latency was measured on a free tier and is noisy. Implicit recurring deadlines were excluded from the labels, and one syllabus’s year came from its URL rather than its text. The hybrid’s fallback is fixed to one threshold rule; learned routers [1] may perform differently.

8. Conclusion

For syllabus deadline extraction, cheap-first routing was not free: running rules first saved 57.8% of tokens but lowered F1 from 0.657 to 0.370, and no routing threshold avoided this trade-off because the rules’ output size did not reveal when they had failed. Rule-based extraction failed mostly on years and layouts. Sending the native PDF to the LLM was at least as accurate as sending extracted text and, with the model tested, used fewer tokens. Systems that use rules to save LLM cost need a confidence signal that actually tracks correctness, such as the single-year check that improved the hybrid here.