The source argues that treating a large language model (LLM) as a direct classifier—feeding a prompt and taking the label as a hard output—has several drawbacks. It notes that LLM verdicts are often hard labels, and while token log probabilities can be extracted, there is no guarantee they are well‑calibrated. Asking the model for its own confidence does not solve this issue, making it difficult to trade off precision and recall in a principled way. Additionally, the source points out that LLMs may not incorporate all available information reliably; structured data pasted into a prompt might be ignored, and the model’s internal priors may not match the target population’s class prevalence without explicit, population‑specific context that is hard to verify. Interpretability is also limited because, although the prompt is readable prose, it is unclear which parts of the prompt the LLM actually follows. The source then proposes a reframing: treat the LLM’s output as a feature rather than the final decision. By wrapping the LLM verdict in a simple logistic regression, the model becomes p(y=1|x)=σ(α+β·LLM(x)). This approach yields two key benefits. First, using the LLM prediction as a feature provides calibrated probabilities in expectation, allowing the practitioner to set thresholds to balance precision and recall. Second, the logistic regression framework can easily incorporate additional covariates, adapt to baseline class frequencies from the training data, and be reweighted to target other distributions. Interpretability improves because the contribution of the LLM feature to the final decision becomes explicit, especially when other features are present. To improve performance, the source suggests the usual machine‑learning levers: collect more labeled data, engineer better LLM‑derived features (e.g., using token log probabilities, multiple runs, or sub‑verdicts), and swap the underlying model (e.g., to XGBoost or a neural network) while keeping the LLM as a feature generator. The source illustrates the idea with a test case using the SemEval 2018 Task 3 irony detection dataset, which contains 4,618 tweets (3,834 for training, 784 for testing). A zero‑shot prompt asking the model to label each tweet as “Ironic” or “Not” produced a true positive rate of 0.965, a false negative rate of 0.035, a Brier score of 0.259, and an F1 score of 0.747 at the 0.5 threshold. The source describes these results as remarkable for a one‑shot approach but notes there is still room for improvement via the feature‑engineering workflow outlined above.
