You are not logged in.
- Topics: Active | Unanswered
#26 Yesterday 13:38:18
- aegis_eclipse
- Member
- Registered: 2025-10-26
- Posts: 26
Re: TrustSight: automated AUR PKGBUILD review, seeking technical criticism
I’ll add one more thing, because I think seth’s caution is exactly right and I want to be clear that I’m not trying to wave it away.
The warning in the wiki is there for good reason: no tool can replace reading the PKGBUILD, and TrustSight will never claim otherwise. The tool itself prints a note to that effect, and the LLM verdict doesn’t hide the score or the evidence. If someone chooses to ignore the diff entirely, that’s a mistake, but it’s a mistake that’s already happening at scale, with or without TrustSight. What I’m hoping is that for the user who won’t read the PKGBUILD, a loud “Critical: remote execution detected” warning is better than the silence they’re currently getting. And for those who do read, it’s a fast triage step that points them to the right file in seconds. I’m not asking the Arch old guard to lower their standards (and they shouldn't) I’m just trying to build a safety net that catches people before they hit the ground. Imperfect, but better than nothing.
Offline
#27 Yesterday 14:43:53
- seth
- Member
- From: Won't reply 2 private help req
- Registered: 2012-09-03
- Posts: 77,002
Re: TrustSight: automated AUR PKGBUILD review, seeking technical criticism
Yes, in reality people will misbehave and reap the consequences of that and then whine about it.
But you're using the anticipated misbehavior as reason for the feature, ie. the feature is meant to provoke and support the misbehavior.
Don't.
A good tool will do one of three things here:
1. "the update only changes the version and checksum"
2. "line 18 will download a known virus, please report the package"
3. "line 12,13 & 15 look shady because [reason] and you need to look into that"
The idea that "oh, it's only 30% likely malicious so it's gonna be fine" is nuts.
Offline
#28 Yesterday 15:25:04
- aegis_eclipse
- Member
- Registered: 2025-10-26
- Posts: 26
Re: TrustSight: automated AUR PKGBUILD review, seeking technical criticism
A good tool will do one of three things here:
1. "the update only changes the version and checksum"
2. "line 18 will download a known virus, please report the package"
3. "line 12,13 & 15 look shady because [reason] and you need to look into that"
The idea that "oh, it's only 30% likely malicious so it's gonna be fine" is nuts.
I think we're closer than it looks, because TrustSight already does exactly those three things:
"Version bump, no structural changes": 81% of benign updates score 0 and produce exactly that verdict.
"Line 18 will download a known virus": When a CRITICAL rule fires (R001: curl | bash, R012: prompt injection, R013: bidi attack), the score goes to 80–100 and the output says something like "Remote script execution detected on line 18." That's case 2.
"Lines 12, 13, 15 look shady because [reason]": When a mix of lower-severity rules or novelty signals fire, the inspect view shows exactly which lines, which rules, and why. The score is the weighted sum of those reasons, not a probability. A score of 35 means "there are unusual patterns here, look at lines 12 and 15," not "35% chance of malice." I can see how calling it a "risk score" might suggest a probability, and I'll tighten the language in the docs to make that distinction clearer.
On the misbehaviour point: I'm not building a tool to encourage blind trust. I'm building a tool for the reality that millions of people already install from the AUR without reading a single line. That's not something I can fix, but I can give them a red flag that says "CRITICAL: remote execution on line 18: do not install until you understand this." That's better than the silence they get now. It's harm reduction, not endorsement. I do take your warning seriously. The tool already prints a note that manual review is required, and I'll make sure the output and docs are crystal clear that the score is input, not verdict. If there's a specific place you think the messaging is still ambiguous, I'd genuinely like to fix it.
I want to be clear about who TrustSight is for. The Arch wiki's advice to read every PKGBUILD is correct, and for the experienced users who built this community, it's both sufficient and non‑negotiable. No tool replaces that. TrustSight isn't trying to replace manual review for those users; it's a triage step that saves them time by dismissing the trivial updates so they can focus on the suspicious ones.
But the AUR's user base has changed. Today, a large portion of AUR users come through Arch‑based distros aimed at gamers and newcomers (CachyOS, SteamOS, EndeavourOS, or even newbie Arch users), and many of them aren't developers. They don't know what a PKGBUILD is, and they're not going to learn it overnight. For that audience, the choice isn't between “use TrustSight” and “read the PKGBUILD”; it's between “see a loud red flag when something is obviously wrong” and “run yay -Syu completely blind.” TrustSight is built for that second group: to be the warning they otherwise wouldn't get. It's harm reduction, not a replacement for expertise. The tool does not, and will never, claim to be a substitute for reading. It just tries to put a safety barrier where there's currently nothing at all.
Offline
#29 Today 06:14:28
- carlosplanchon
- Member
- From: Dolores, Uruguay
- Registered: 2020-09-23
- Posts: 1
- Website
Re: TrustSight: automated AUR PKGBUILD review, seeking technical criticism
Hi! I think this is a really interesting design problem to explore.
I think Seth's model is that the analyzer should report concrete findings:
- only the version and checksum changed
- a known malicious payload was found
- specific lines look suspicious for specific reasons
The user, or another tool, then decides what those findings mean.
Aegis's concern is different: many users will not inspect the PKGBUILD at all, so some user-facing layer should turn those findings into an obvious warning.
Those goals do not necessarily conflict, but they may belong in separate layers:
For example: a policy-neutral core that emits structured findings and evidence,
and an optional triage layer that applies a scoring or warning policy for less experienced users.
That would allow experienced users, distributions and third-party tools to consume the same findings without having to accept one particular scoring model.
Other frontends could implement stricter or different policies on top of the same core.
The architectural question then is not whether inexperienced users deserve a warning (they clearly do) but whether evidence collection and the final risk judgement should be owned by the same component.
Nice work!
Any sufficiently advanced technology is indistinguishable from biology.
Offline
#30 Today 10:49:13
- Lone_Wolf
- Administrator
- From: Netherlands, Europe
- Registered: 2005-10-04
- Posts: 15,246
Re: TrustSight: automated AUR PKGBUILD review, seeking technical criticism
# 2. (Optional) Configure an LLM for plain-English verdicts
# Run the interactive wizard — it tests the connection before saving:The LLM is optional. The score is always deterministic and calculated locally; the LLM only translates the breakdown into a sentence. Without one, you get template verdicts like "Version bump. Checksum disabled. Signals: R004 (HIGH)."
The task is to be converting the 'verdicts' accurately and reliably to plain english.
Those 2 needs disqualify LLMs.
An agentic solution chosen by you/ archlinux community that's pinned at a known-good version is much more likely to satisfy those needs.
Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.
clean chroot building not flexible enough ?
Try clean chroot manager by graysky
Offline
#31 Today 12:34:19
- aegis_eclipse
- Member
- Registered: 2025-10-26
- Posts: 26
Re: TrustSight: automated AUR PKGBUILD review, seeking technical criticism
First, I've thought about it (the solution came to me in a dream) and maybe we can find a middle ground:
some-pkg 1.2.3 → 1.2.4
R085 post_install writes to /etc/systemd/system/ [0.2% of AUR]
R081 post_install runs `npm install lockfile-helper` [0.1% of AUR]
R071 maintainer changed (new account, first package) [1.4% of AUR]
3 findings. Run with --explain for a plain-English summary.No score shown. No verdict. Three checkable statements, each with its corpus rarity so the reader can calibrate for themselves rather than trusting my weighting.
Hiding the score doesn't address the actual objection. I suspect the pushback isn't really "don't show a number", it's "who are you to assign this number, and what does 45 mean?", which is true. Then --score for people who want the aggregate, --explain for the LLM summary (opt-in, which also solves the "I don't want an LLM near my package manager" objection entirely), and --json carrying everything for tooling.
This is only achievable using the --full-aur flag, so it may become a default/mandatory, or at least necessary to sync, every once in a while, the AUR metadata.
The task is to be converting the 'verdicts' accurately and reliably to plain english.
Those 2 needs disqualify LLMs.
Every possible input is known at build time. A template does it accurately, reliably, identically every run, offline, with no API key, no cost, no latency, and no verdict-integrity assertions needed because there's nothing to assert against.
R085: "{position} writes to {path}, a systemd unit directory"
R081: "{position} runs `{command}`, fetching from the {ecosystem} registry"
R071: "maintainer changed from {old} to {new} ({tenure})"A template is deterministic (same input → identical string, forever) and accurate (it can't hallucinate a detail the rule didn't produce).
Some feedback about this approach would be cool before I refactor the tool. Ty to all for the feedback!
Last edited by aegis_eclipse (Today 12:48:22)
Offline
#32 Today 13:15:29
- seth
- Member
- From: Won't reply 2 private help req
- Registered: 2012-09-03
- Posts: 77,002
Re: TrustSight: automated AUR PKGBUILD review, seeking technical criticism
The update is not trivial, you will have to review it.
I found 4 things I can highlight:
PKGBUILD lLine 13: post_install writes to /etc/systemd/system/
PKGBUILD line 28: post_install runs `npm install lockfile-helper`
And the maintainer changed (new account, first package)
A new file "icon.png" was added.
4 findings. Run with --explain for an AI summary.The corpus rarity is a rather meaningless detail - or very user-unfriendly score of sorts.
Be transparent about the "plain english" source.
The result is inconclusive (not 100% harmless or 100% bad) so you decide on behalf of the user that they are going to look at the file.
But the AUR's user base has changed. Today, a large portion of AUR users come through Arch‑based distros aimed at gamers and newcomers
So it's everybody elses job to show them around. "Welcome to archlinux, this is how things are done. Get a helmet."
Another reason I might have to point out is that those tools easily allow to architect malware around them (obscurity, ie. throwing a PKGBUILD at a blackbox and get an uncommented traffic light as result would actually be more tedious to overcome) - they're inherently not reliable in their verdict because the malicious actor explicitly will seek to fool them.
Offline
#33 Today 16:19:38
- aegis_eclipse
- Member
- Registered: 2025-10-26
- Posts: 26
Re: TrustSight: automated AUR PKGBUILD review, seeking technical criticism
Three of you identified the same problem from different angles: I need to separate evidence from judgement, and make the evidence the product.
1.
No score, no verdict, no risk label.
some-pkg 1.2.3 → 1.2.4
The update is not trivial. Review it.
PKGBUILD line 13 post_install writes to /etc/systemd/system/ [R085]
PKGBUILD line 28 post_install runs `npm install lockfile-helper` [R081]
.SRCINFO maintainer changed: oldmain → newmain [R071]
new file icon.png
trivial-pkg 2.0 → 2.1
Only pkgver and sha256sums changed. Review the diff before building.Line numbers are the substantive change, not the cosmetics. A finding you can open the file and check in five seconds is a different object from a score you have to trust. Neutral changes (new files, dependency additions) are reported to, this is a diff summary that flags things, not a threat list, as seth (and the AUR) recommend for tools of this kind.
2.
Then, Score becomes opt-in. --score restores the aggregate for anyone who wants it; it stays internal for CI exit codes and threshold alerting. It's no longer the face of the tool, and nobody has to accept my weighting to use it. Even better, as weights are set per criticality level, then could even be configurable.
3.
No more LLM per Lone_Wolf recomendation: Every input is a known rule ID with known fields, so generation buys nothing and costs determinism. Templates are identical every run, work offline, need no API key. openai comes out of the dependency list. If anyone wants LLM summarisation they can pipe --json into their own tool. It stays out of the trust path.
R085: "{position} writes to {path}, a systemd unit directory"
R081: "{position} runs `{command}`, fetching from the {ecosystem} registry"
R071: "maintainer changed from {old} to {new} ({tenure})"4.
Policy-neutral core, per carlosplanchon. --json emits findings, locations, rule IDs and evidence with no aggregate score unless --score is passed. Other tools, distros, or stricter policies consume the same evidence without inheriting my judgement. Evidence collection and risk judgement stop living in the same component.
5.
Known-payload matching. Seth listed "line 18 will download a known virus, please report the package" as something a good tool does. Trustsight currently do that, because my rules detect patterns rather than known-bad artifacts. Adding a versioned IOC list (confirmed campaign package names, domains, hashes) would let Trustsight say exactly that, definitively, with no scoring involved.
Seth is right that published rules are evadable, and that happens with every Open Source project. I don't think obscurity is the answer though, an uncheckable traffic light is worse precisely because nobody can verify it. The one thing that genuinely doesn't have a heuristic to route around is IOC matching: a known-bad package name or artifact hash is either present or it isn't. Beyond that, the corpus rules raise the cost of evasion without eliminating it. An attacker can avoid any single pattern I match, but avoiding shared infrastructure across the packages they're attacking is harder, because coordinating an attack at scale means reusing something. That's a cost increase, not a defence, and the docs will say so.
Offline
#34 Today 17:07:18
- 5hridhyan
- Member
- From: A fake and corrupt democracy
- Registered: 2025-12-25
- Posts: 966
- Website
Re: TrustSight: automated AUR PKGBUILD review, seeking technical criticism
Look man, Im really the type of guy who prefers to use python thingy in a virtual environment, and installing python-openai just for it, I kinda don't like that.
Any other way to let TrustSight do review w/o that module?
Edit:
Missed that reply, Guess Ill have to wait till the "next" release...
Last edited by 5hridhyan (Today 17:11:00)
Offline
#35 Today 17:21:53
- aegis_eclipse
- Member
- Registered: 2025-10-26
- Posts: 26
Re: TrustSight: automated AUR PKGBUILD review, seeking technical criticism
Any other way to let TrustSight do review w/o that module?
Yes! You can just install Trustsight and run it without setting the API key and that will fall back to templates (older templates though). You can still run --verbose to see the full rules, or check the docs.
Offline
#36 Today 18:07:10
- 5hridhyan
- Member
- From: A fake and corrupt democracy
- Registered: 2025-12-25
- Posts: 966
- Website
Re: TrustSight: automated AUR PKGBUILD review, seeking technical criticism
Am I doing smtg wrong?
┌──(shridhyan@ATRNXA)-[~]
└─$ trustsight review
Reviewing packages... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
╭──────────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────────╮
│ /usr/lib/python3.14/site-packages/trustsight/cli.py:323 in review │
│ │
│ 320 │ │ │ _print_colored("No outdated packages found.", "green") │
│ 321 │ │ return │
│ 322 │ │
│ ❱ 323 │ _run_analysis_loop(outdated_pkgs, effective_limit, verbose, quiet, json_output, total_installed, all_packages, simple) │
│ 324 │
│ 325 │
│ 326 def _run_analysis_loop( │
│ │
│ /usr/lib/python3.14/site-packages/trustsight/cli.py:362 in _run_analysis_loop │
│ │
│ 359 │ │ │ │ else: │
│ 360 │ │ │ │ │ progress.update(task, description=description) │
│ 361 │ │ │ │
│ ❱ 362 │ │ │ results = _analyze_outdated_batch(limited, on_progress, verbose, simple) │
│ 363 │ │ │ progress.update(task, visible=False) │
│ 364 │ elif json_output: │
│ 365 │ │ def on_progress(_current, total, description): │
│ │
│ /usr/lib/python3.14/site-packages/trustsight/cli.py:616 in _analyze_outdated_batch │
│ │
│ 613 │ │ done_count = 0 │
│ 614 │ │ for future in as_completed(futures): │
│ 615 │ │ │ idx = futures[future] │
│ ❱ 616 │ │ │ result = future.result() │
│ 617 │ │ │ status = result[0] │
│ 618 │ │ │ entry = result[1] │
│ 619 │ │ │ done_count += 1 │
│ │
│ /usr/lib/python3.14/concurrent/futures/_base.py:447 in result │
│ │
│ 444 │ │ │ │ if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]: │
│ 445 │ │ │ │ │ raise CancelledError() │
│ 446 │ │ │ │ elif self._state == FINISHED: │
│ ❱ 447 │ │ │ │ │ return self.__get_result() │
│ 448 │ │ │ │ │
│ 449 │ │ │ │ self._condition.wait(timeout) │
│ 450 │
│ │
│ /usr/lib/python3.14/concurrent/futures/_base.py:396 in __get_result │
│ │
│ 393 │ def __get_result(self): │
│ 394 │ │ if self._exception is not None: │
│ 395 │ │ │ try: │
│ ❱ 396 │ │ │ │ raise self._exception │
│ 397 │ │ │ finally: │
│ 398 │ │ │ │ # Break a reference cycle with the exception in self._exception │
│ 399 │ │ │ │ self = None │
│ │
│ /usr/lib/python3.14/concurrent/futures/thread.py:86 in run │
│ │
│ 83 │ │ │ return │
│ 84 │ │ │
│ 85 │ │ try: │
│ ❱ 86 │ │ │ result = ctx.run(self.task) │
│ 87 │ │ except BaseException as exc: │
│ 88 │ │ │ self.future.set_exception(exc) │
│ 89 │ │ │ # Break a reference cycle with the exception 'exc' │
│ │
│ /usr/lib/python3.14/concurrent/futures/thread.py:73 in run │
│ │
│ 70 │ │
│ 71 │ def run(self, task): │
│ 72 │ │ fn, args, kwargs = task │
│ ❱ 73 │ │ return fn(*args, **kwargs) │
│ 74 │
│ 75 │
│ 76 class _WorkItem: │
│ │
│ /usr/lib/python3.14/site-packages/trustsight/cli.py:603 in _pipeline_one │
│ │
│ 600 │ │ │ log.warning("analysis of %s failed unexpectedly", name, exc_info=True) │
│ 601 │ │ │ return ("fail", entry, None, None, exc, 0, 0) │
│ 602 │ │ t1 = time.perf_counter() │
│ ❱ 603 │ │ verdict = _verdict_for(fact, simple) │
│ 604 │ │ t2 = time.perf_counter() │
│ 605 │ │ return ("ok", entry, fact, verdict, None, t1 - t0, t2 - t1) │
│ 606 │
│ │
│ /usr/lib/python3.14/site-packages/trustsight/cli.py:543 in _verdict_for │
│ │
│ 540 │ nothing suspicious was found and the deterministic verdict is sufficient. │
│ 541 │ Pass ``simple=True`` to skip the LLM entirely and always use the fallback. │
│ 542 │ """ │
│ ❱ 543 │ from .llm import fallback_verdict, generate_verdict │
│ 544 │ │
│ 545 │ if simple or fact.final_score <= 0: │
│ 546 │ │ return fallback_verdict(fact) │
│ │
│ /usr/lib/python3.14/site-packages/trustsight/llm.py:8 in <module> │
│ │
│ 5 import sys │
│ 6 from typing import Optional │
│ 7 │
│ ❱ 8 from openai import OpenAI │
│ 9 │
│ 10 from .config import load_config │
│ 11 from .scoring import risk_level │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: No module named 'openai'and
└─$ trustsight --verbose
Usage: trustsight [OPTIONS] COMMAND [ARGS]...
Try 'trustsight --help' for help.
╭─ Error ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ No such option: --verbose (Possible options: --version) │
╰────────────────────────────────────────────────────────────or check the docs.
uhm, will look into that...
and AUR servers are unfriendly w/ it?
RemoteDisconnected: Remote end closed connection without responseEdit:
is --verbose a command-specific flag, not global? IDK I can't check it because the code cannot be imported w/o that module...
Last edited by 5hridhyan (Today 18:14:33)
Offline
#37 Today 18:12:13
- aegis_eclipse
- Member
- Registered: 2025-10-26
- Posts: 26
Re: TrustSight: automated AUR PKGBUILD review, seeking technical criticism
Simply amazing, error is on my end, let me look into it, I'll probably just make it optional as to be truly opt-in.
The --verbose flag is to be used with the review command. Still, the error is because i made OpenAI dependency mandatory.
Last edited by aegis_eclipse (Today 18:14:17)
Offline
#38 Today 20:47:43
- aegis_eclipse
- Member
- Registered: 2025-10-26
- Posts: 26
Re: TrustSight: automated AUR PKGBUILD review, seeking technical criticism
Okay, that has been resolved in v0.9.0. LLM has been fully removed.
Offline