Brave Leo now uses Mixtral 8x7B as default

How fast is it with a setup like this?

I think this perspective comes from a lack of historical experience and hands-on experience overall.

Nvidia more broadly has very impressive support for their GPUs. If you look at the support lifecycles for their Jetson hardware over time it's significantly worse. I encourage you to look at what support lifecycles have looked like, with the most "egregious" example being dropping of support for the Jetson Nano in from what I recall was within a couple of years.

Another consideration - Jetson is optimized for power efficiency/form-factor and on a per $ basis CUDA performance is terrible. The power efficiency and form-factor come at significant cost. See this discussion from one of my projects[0]. I evaluated the use of WIS on an Orin Nano that I have and it was nearly 10x slower than a GTX 1070 which is seven years old and is still supported by the latest drivers and CUDA 12 on whatever OS you want.

Nvidia knows what they're doing in terms of productization and the Jetson line should not be seen as some kind of secret hack/unlock for getting CUDA performance with gobs of RAM. In the case of LLMs I wouldn't be surprised at all if CPU beats it and at that point pickup 256GB of RAM or whatever for equivalent cost.

In the end what do I care what people use, I'm offering the perspective and experience of someone who has actually used the Jetson line for many years and frequently struggled with all of these issues and more.

[0] - https://github.com/toverainc/willow-inference-server/discuss...

I have a Jetson as well, and you are sorely mistaken. Just reading the doc pages everything seems nice and well, but Nvidia deprecates these little boards like no other. No support after you've bought the thing, and everything is kept frozen. (ie no new python, no new python dependencies, etc) What they aren't telling you is that specific sub-versions within each jetson/orin family board have differing support (ie not what they say on that website you are reading), and it's up to you to figure it out.

I've gotten my Jetson to work well using Yocto to build my own linux distros with correct updated dependencies, libraries and updated jetpack, but it's not for the faint of heart, and that's a whole other ball of yarn. It also takes a few hours to generate a new build every time I need to update some dependency that depends on other dependencies (Yocto maintenance is a full time job in many embedded development shops - you're basically authoring your own distribution).

Treat these devices as what they are: embedded target boards for fixed industrial development (for example, to go into a robot or a car - once that design is finished, Nvidia will expect you to NEVER update any part of the system with an embedded jetson or orin system for years, until you replace the whole thing with their newest model that you buy off the shelf again).

This is standard fare in embedded and robotics space. Do not use these boards for any kind of rapidly moving software development, because it's the wrong tool for the job.

> No, I want people interested in this feature to track the bug or contribute the fix themselves since it is an open source project.

> That is correct in my understanding. This means that the open source community needs to dedicate their own resources if they want to accelerate progress here.

OR... hear me out: we could recognize that Chrome, a browser owned by one of if not the most powerful tech companies in the entire world, does not need charity, and that it's not a community owned project (if it was, MV2 would not be getting deprecated), and that the community has no say in how the project gets updated or maintained and is fact very regularly treated with hostility by the project, and that instead of trying to fix an issue that Google probably isn't even interested in getting help with, we could instead go use and contribute to Open Source browsers that aren't transparently trying to kill adblockers.

This is wild. You went from "MV3 is fine, there's just a lot of misinformation" to "MV3 is trying, but the devs are overworked" to "nobody is allowed to complain about this unless they're giving hecking Google free labor."

Google made a decision to deprecate a working API, developers correctly pointed out that the API hobbles adblockers, Google has not fixed the API but is launching it anyway, and your answer is that the open source community needs to dedicate their own resources. This problem is not our fault, we didn't make it.

How about Google not completely rewrite an API out of the blue if they don't have the internal resources necessary to do so? Is there some hiring freeze on developers, has Google lost the ability to hire and pay people? Google creates a problem, and you're blaming the Open Source community for not fixing Google's problem for them, a problem that exists entirely of Google's own free will, because Google ignored the Open Source community that was desperately trying to stop Google from making the problem.

The Open Source community has a solution to this; don't deprecate blockingWebRequest. Like most community feedback, Google isn't interested in hearing that.

I would suggest that if Open Source developers need to contribute somewhere, they could try contributing to Firefox, a browser that has none of these problems because somehow a company that is so small its entire corporate profits would barely register in Google's spreadsheets is still better at building extension APIs than Google is.

----

> Assuming no events are dropped then I don't see it as an issue.

It's an issue because extensions sometimes like to respond to events when they happen. If the browser stopped allowing synchronous responses to click events and I said, "well as long as they eventually trigger there's no issue" I would be rightly laughed out of the room.

The issue is that a user can update filters and they have no guarantee that the filters will be updated before the requests that they mean to block are fired off. If an adblocking extension does decide to update filter rules on launch, they have no guarantee that the update will take effect before the page loads.

> It doesn't sound that wild if it speeds up launch time by not blocking on third party extensions having to load.

Not loading the extension at all would speed up launch time even more, Google should just get rid of extensions entirely /s

People use extensions as a way to increase security and to build reliable features into the browser. There are many use-cases from tab-control to feature toggles where having tabs and pages load before extensions breaks functionality. It is wild to suggest that breaking functionality in userspace is preferable to adding an extra second to a browser launch.

For many people, adblocking is a security issue. To suggest that the browser should occasionally just send requests anyway before an extension has time to update and apply rules is like suggesting that a desktop should start loading and displaying user settings before it checks the user's login password.

> This is not an issue as the extension does not need to be running for the browser to apply the rules.

See above.

----

> Considering that the deadline keeps being pushed back, I think they recognize that it has low maturity.

This is once again a pretty big shift from "MV3 doesn't stop adblockers" to "MV3 isn't ready yet and they should keep pushing it back." It's also optimistic given that Google has given no indication yet that they're going to push back the June deadline.

But most importantly, it also doesn't change anything about my point -- we are seeing that Google does not have the resources to keep pace with adblocking innovations. That's not going to change. This is not a question of MV3 becoming ready and then everything becoming fine: "ready" is a moving target, and Google is not moving fast enough to keep pace with it.

What these issues demonstrate is that the declarativeNetRequest API is poorly designed -- it is poorly designed because it greatly increases the support requirements from Google to a level where the company is unable to meet them. And when Google is unable to meet those support requirements, user security and privacy suffers as a result.

This is an intrinsic problem that is baked into the design of declarativeNetRequests, and it's not going away even if Google does manage to close a couple of bugs. Because more of these issues are coming every time that adblockers need to innovate. If an API increases support burdens to the point where the dominant browser owned by the most powerful tech company has to come crawling to the Open Source community for help triaging 3-year-old issues, then it's a bad API.

And underlying all of the criticism of MV3 is that idea -- that it is not tenable for extension developers to have to request support for every individual feature that adblockers need. A better API would be a more flexible API that allows extension developers to innovate without asking permission. blockingWebRequest is an imperfect API and it has issues, but it allows extension developers to innovate without filing support tickets for every new feature and then waiting 3 years for a response because the API is flexible enough that extension developers can use it in creative ways that the authors may not have fully anticipated or known about.

I've used Jetson for a few projects as a hobby. Made an I2S Sodar array with a TX2. And some robotics projects with a Jetson AGX Xavier that I got to evaluate and then to work on. And a few both, professional and toy projects with versions of Jetson Nano kit and Xavier. But this was between 2017 and 2021 or so.

About a year back, I took that very early version of AGX Xavier, that got released years ago. It wasn't even the version that was officially released. Yet I was able to refresh it to newer Ubuntu without any issues.

Wheels are often not pre-built for aarch64, yes. If you want to compile directly on Nano, disk performance is very important. Sometimes you get I/O bound.

Orin Nano being that slow in [0], it looks like you've been trying it in Aug 2023. It maybe worth re-evaluating on the latest Jetpack, it had transitioned to CUDA 12.2, TensorRT 8.6, cuDNN 8.9. I would expect that recent popularity of ASR/TTS pipelines and LLMs was not completely missed by Jetpack maintainers (there are some tutorials here - https://www.jetson-ai-lab.com ). And recently released JetPack could be optimized a lot more for these workflows.

And your project is very cool! I'd suggest sharing it and your performance numbers (!) with the maintainers of: https://developer.nvidia.com/embedded/community/jetson-proje...