Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: A MitM proxy to see what your LLM tools are sending

github.com

Post reply on HN
Print view View original

71–80 of 128 posts

Re: Show HN: A MitM proxy to see what your LLM tools are sending

#71

Post by catlifeonmars »

jmuncor wrote: post #67
catlifeonmars wrote: post #54

This tool looks like it unconditionally disables tls verification for upstream requests. It shells out to mitmproxy with "--set", "ssl_insecure=true" This took all of 5 minutes to find reading through main.py on my phone. https://github.com/jmuncor/sherlock/blob/fb76605fabbda351828... Edit: In case it’s not clear, you should not use this.

Just fixed it and implemented a simple http relay, eliminating the mitmproxy and the ssl_insecure=true. The new implementation uses TLS verification, doing last tests and merging it... After the merge can you check it out and tell me if I earned your star? :D

I’m not sure you fully understand the implications of the misconfiguration of mitmproxy there. Effectively you provided an easily accessible front door for remote code execution on a user’s machine.

No offense, but I wouldn’t trust anything else you published.

I think it’s great that you are learning and it is difficult to put yourself out there and publish code, but what you originally wrote had serious implications and could have caused real harm to users.

Re: Show HN: A MitM proxy to see what your LLM tools are sending

#72

Post by throwaway277432 »

jmuncor wrote: post #67
catlifeonmars wrote: post #54

This tool looks like it unconditionally disables tls verification for upstream requests. It shells out to mitmproxy with "--set", "ssl_insecure=true" This took all of 5 minutes to find reading through main.py on my phone. https://github.com/jmuncor/sherlock/blob/fb76605fabbda351828... Edit: In case it’s not clear, you should not use this.

Just fixed it and implemented a simple http relay, eliminating the mitmproxy and the ssl_insecure=true. The new implementation uses TLS verification, doing last tests and merging it... After the merge can you check it out and tell me if I earned your star? :D

>tell me if I earned your star

Since you asked: Not in a million years, no.

A bug of this type is either an honest typo or a sign that the author(s) don't take security seriously. Even if it were a typo, any serious author would've put a large FIXME right there when adding that line disabling verification. I know I would. In any case a huge red flag for a mitm tool.

Seeing that it's vibe coded leads me believe it's due to AI slop, not a simple typo from debugging.

Re: Show HN: A MitM proxy to see what your LLM tools are sending

#73

Post by jmuncor »

throwaway277432 wrote: post #72
jmuncor wrote: post #67

Earlier quoted context omitted.

Just fixed it and implemented a simple http relay, eliminating the mitmproxy and the ssl_insecure=true. The new implementation uses TLS verification, doing last tests and merging it... After the merge can you check it out and tell me if I earned your star? :D

>tell me if I earned your star Since you asked: Not in a million years, no. A bug of this type is either an honest typo or a sign that the author(s) don't take security seriously. Even if it were a typo, any serious author would've put a large FIXME right there when adding that line disabling verification. I know I would. In any case a huge red flag for a mitm tool. Seeing that it's vibe coded leads me believe it's d…

I love the real feedback tbh, I am still learning, and want to learn as much as possible. Would love if you can review it and tell me bluntly either in the repo or here the things that should be improved. I would love to learn more from you and get better :D

Re: Show HN: A MitM proxy to see what your LLM tools are sending

#74

Post by jmuncor »

catlifeonmars wrote: post #71
jmuncor wrote: post #67

Earlier quoted context omitted.

Just fixed it and implemented a simple http relay, eliminating the mitmproxy and the ssl_insecure=true. The new implementation uses TLS verification, doing last tests and merging it... After the merge can you check it out and tell me if I earned your star? :D

I’m not sure you fully understand the implications of the misconfiguration of mitmproxy there. Effectively you provided an easily accessible front door for remote code execution on a user’s machine. No offense, but I wouldn’t trust anything else you published. I think it’s great that you are learning and it is difficult to put yourself out there and publish code, but what you originally wrote had serious implications…

Ohh my, no offense taken... The next time I will be a lot more careful with the stuff that I put out there. Learning and getting the hang of it, would love if you either comment on the code or here any other things you think could be improved. I am in the process of getting better and appreciate all the blunt and transparent feedback. No one grows out of praise.

Re: Show HN: A MitM proxy to see what your LLM tools are sending

#75

Post by jamespo »

jmuncor wrote: post #73
throwaway277432 wrote: post #72

Earlier quoted context omitted.

>tell me if I earned your star Since you asked: Not in a million years, no. A bug of this type is either an honest typo or a sign that the author(s) don't take security seriously. Even if it were a typo, any serious author would've put a large FIXME right there when adding that line disabling verification. I know I would. In any case a huge red flag for a mitm tool. Seeing that it's vibe coded leads me believe it's d…

I love the real feedback tbh, I am still learning, and want to learn as much as possible. Would love if you can review it and tell me bluntly either in the repo or here the things that should be improved. I would love to learn more from you and get better :D

unlikely to get that from a throwaway

Re: Show HN: A MitM proxy to see what your LLM tools are sending

#76

Post by jmuncor »

Retr0id wrote: post #55
mikehotel wrote: post #35

Earlier quoted context omitted.

Addon to mitmproxy: https://docs.mitmproxy.org/stable/addons/overview/

Looks like it's implemented that way already? https://github.com/jmuncor/sherlock/blob/fb76605fabbda351828...

Yes... Got it patched up and fixed some security issues it had. Let me know what you think of the new version.

Re: Show HN: A MitM proxy to see what your LLM tools are sending

#77

Post by jmuncor »

jamespo wrote: post #75
jmuncor wrote: post #73

Earlier quoted context omitted.

I love the real feedback tbh, I am still learning, and want to learn as much as possible. Would love if you can review it and tell me bluntly either in the repo or here the things that should be improved. I would love to learn more from you and get better :D

unlikely to get that from a throwaway

You can always try right?

Re: Show HN: A MitM proxy to see what your LLM tools are sending

#79

Post by jmuncor »

daxfohl wrote: post #36
david_shaw wrote: post #3

Nice work! I'm sure the data gleaned here is illuminating for many users. I'm surprised that there isn't a stronger demand for enterprise-wide tools like this. Yes, there are a few solutions, but when you contrast the new standard of "give everyone at the company agentic AI capabilities" with the prior paradigm of strong data governance (at least at larger orgs), it's a stark difference. I think we're not far from th…

Yes, I was just thinking about how, as engineers, we're trained to document every thought that has ever crossed our minds, for liability and future reference. Yet once an LLM is done with its task, the "hit by a bus" scenario takes place immediately.

Yes, I think you can actually later store this in a database and start querying and optimizing what is happening there. Even you can start using these files or a destilation of these as long term memory.

Re: Show HN: A MitM proxy to see what your LLM tools are sending

#80

Post by jmuncor »

vitorbaptistaa wrote: post #52
jmuncor wrote: post #50

Earlier quoted context omitted.

Something like sherlock start --otel-endpoint?

Yes. It can get a bit more complex as some otels require authentication. You can check Pydantic AI Gateway, Cloudflare AI Gateway or LiteLLM itself. They do similar things. One advantage of yours would be simplicity.

I love this idea... Going to look into it, thank you!
Post reply on HN