The skills you need to guide LLMs though comes from practical application/experience writing code in the industry though, so my concern is how people are going to get those skills now. You don’t get them from college.

Yeah… it’s going to be really rough for new engineers for a while until we figure out a new way to train people. I think there will be people who are self taught and AI native who will do really well, but they will be the minority.

I don’t feel comfortable telling anyone to go into software engineering right now. I think the future is really uncertain. That said, I’m also not convinced this is a bad time.

The need for software engineering isn’t going away. LLMs are actually not very good at thinking and solving problems at a high level, so we still need people for that. LLMs are more of a performance multiplier than a replacement for human software engineers.

That does mean that fewer engineers will be needed to do the same work. But also, I think there’s a chance it’s a Jevons Paradox situation, and maybe if the cost of software drops we’ll see even more demand for it - like how we found tons of new uses for electricity when prices dropped. Maybe we’re entering a golden age of bespoke software. Unclear.

This is pretty off-base for what OP is asking about.

React and C# are open source.

WebM, WebP, and AV1 are open standards that have mature and well developed open source implementations.

Literally every item OP asked about is already OSS. OP just doesn’t like that they originated at Meta, Microsoft, and Google.

Not everything made by big tech is bad.

React is great. It’s open source. The governance is reasonable. Why should we replace it?

WebP, WebM and AV1 are great open technical formats. We should use them.

If a standard is truly open, then you should judge it on its technical merits, not some weird purity test about where it came from.

I’m not saying it’s great.

I just don’t see this ending without a bunch of people having their lives ruined, so whatever we can do to pull the bandaid off faster is the best we can hope for.

You know, this might not be for the worst.

People really need to get used to the idea that everything you see online, and especially on Facebook is fake. Maybe it’s for the best if they don’t moderate things well enough to let people pretend otherwise.

Is France… ok?

They really want people scared, ehh?

You’d better give us money and not try to regulate us.

Actually, right now, the western AI companies desperately want regulation.

An unregulated environment is one where they’re forced to compete on price against the Chinese companies and startups running open-weight models. All signs point to them losing that battle.

OpenAI/Anthropic’s wish list right now is a bunch of regulation to prevent the use of foreign models, and to require expensive to implement “safety” requirements that make it hard for startups to compete.

You absolutely shouldn’t be angry!

The existence of powerful open-weight models is the single thing that’s going to prevent us from descending into the full corporate dystopia everyone’s afraid of.

It’s going to hit the stock market hard once everyone figures out what’s going on though.

The hilarious part is, yes, I don’t care how fast my launcher is.

But… also, Steam is by far the slowest, least responsive program on my computer.

“Faster than Steam” feels like a joke.

I don’t use a lot of non-open source applications (and not a lot of desktop applications generally). Have they gotten so bad that the slowness of Steam doesn’t feel bad to people anymore?

the models only need to be right some of the time and they still constitute a threat.

Right!?!

And with how cheap and fast they are to run, they don’t have to be right that often as long as a separate run can verify/validate claims. In a lot of ways what the LLMs do should probably be thought of as just a very sophisticated sort of brute force approach. You’re effectively just searching the space of combinations of known vulnerabilities. It doesn’t require particular cleverness, just an encyclopedic knowledge, a reasonable way to prune the search space, and the ability to put in the compute time.

I must be missing something, because you’re getting a bunch of upvotes. I don’t disagree with anything you said, but it feels like a total non-sequitur to what I was saying.

Yes - proper air-gapping is a good idea. But asking the big AI labs to air gap is kind of pointless. Liability already gives them legal incentives to avoid incidents. If that didn’t stop them, a letter signed by a bunch of academics sure won’t. More importantly, open-weight models capable of doing all the things that we’re concerned about are already in the hands of bad actors around the world.

This letter is asking us to lock the front door of the house when the back wall is missing.

The ones in Russia definitely are. They don’t have to be local models - you run them in the cloud like anyone else.

The idea that the frontier labs have LLMs with special capabilities that make them dramatically better than the top open-weight models is largely marketing.

The reality is that most systems are poorly secured, and LLMs make exploiting vulnerabilities fast, cheap, and easy. What would have taken a trained security researcher weeks before can be done by a much less skilled attacker with an LLM in a few hours.

This is stupid. Open-weight models that can largely do what the OpenAI/Anthropic models did are already out in the wild.

Do these people not understand that you can’t put the cat back in the bag?

Doesn’t mean they’re wrong!

I’ll just upvote the both of you. We all need to be more aware of our echo chambers.

I think most the serious investors know. But, in a bubble, stock value can keep going up long past the point that makes sense. If you think that’s going to happen, it can make sense to invest even if you think it’s a losing play in the long run.

This should really make the timing more urgent for Anthropic. I was legitimately surprised when they decided to delay the IPO.

Also there’s still a real chance that Anthropic has some real value proposition that I’m not understanding, though I’ve yet to find someone who can make a coherent argument for what that looks like.

Right? This is going to be an interesting story to follow.

On the one hand, they don’t want to IPO right now, probably because they don’t think investors are keen to buy at the price they’re asking.

On the other hand, I don’t really see things getting better for them as long as cheap open-weight models that are almost as good as Anthropic’s exist. As more people figure this out, I think it’s going to get worse not better. So I don’t think they can afford to wait long.

I think it’s important to call out that this isn’t an issue with AI being brilliant at hacking. It’s that these systems are basically totally unsecured, and the main protection is that no one with the technical skill is bothering to try to get access.

This isn’t to say this isn’t a serious problem. We’re moving from a world where you need a trained security expert to spend non-trivial time to get access to these systems to one where anyone with access to an un-restricted agentic LLM can do this trivially and quickly.

As a civilization, we need to start taking security seriously.

“The outputs of this opt-in vulnerability scanner will be fully model-generated, without human review or triage,” Anthropic explained. “This will enable faster and more frequent scanning, but means that it is possible reports will be incorrect or invalid.”

When I saw the headline, I was wondering about this specifically. This may make this service not super useful.

My experience with AI security reviews is that they’re fantastic at finding faults, but they always find a list of things to complain about. If there are no real/serious faults they’ll start finding things that kind of have the same shape as a security issue, but really aren’t if you dig into them. I’ve regularly had an LLM generate a list of 10-15 issues ranging in severity from “nits” to “critical” where none of them were actual issues.

Periodic reviews seem like they could get annoying really quickly, becoming more of a maintenance burden than a help.