How do you think about this risks in this domain? If it's possible to get an AI Agent to answer the question about the stock price, then surely it's also possible to (a) ask the agent to hack a target system (b) attempt to set up a phishing/scamming operation, or (c) setup copies of itself (if you have weights access) on cloud VMs, which will recursively take the same action with the final goal of DDOS'ing a website, leading to exponential proliferation of agents.
These capabilities seem fairly hard to control, to say the least!
I think the risks are overblown. Marc Andreessen has the right perspective, quoting him from memory (from his appearance on a Stratechery podcast):
"When we launched Mosaic people were worried sick about spam and stealing credit card numbers and stuff... And all of those things happened! But they weren't what was important about the Internet"
They could certainly be overblown, but I can't see how to defeat the argument that with increasing model capabilities eventually we will have LLM agents capable of the kinds of attacks I mentioned.
I agree that spam is an example of worry not turning out justified, but that's not an argument against this particular situation turning out differently.
Right, so my objection is not an argument about LLMs - it's a meta-argument. We don't know the full consequences of ANY new technology! Maybe LLM agents will make attacks. Maybe they will provide better defenses. I have no idea.
All I know is that humans are really biased towards imagining horrible consequences of any new tech but so far technological development has been great for humans.
Yes, I agree. We don't know the full consequences so we should rigorously evaluate the capabilities and risks of a variety of models over time to keep track of potential attacks and defenses, so we have early warning of any problems.
Sadly there is not really any institution doing this for LLMs, but maybe soon :)
I argue that the most valuable problem is guaranteeing an unbiased solution, no matter the budget or time constraint. I don't believe it can be solved or guaranteed. It is a snapshot in time, and any historical problem solving completed can be reprogrammed/retrained or provided bad input at any moment, in any fashion. Dense individuals will continue to mistake speed for accuracy. Just like "safe and effective" at "warp speed".
How do you think about this risks in this domain? If it's possible to get an AI Agent to answer the question about the stock price, then surely it's also possible to (a) ask the agent to hack a target system (b) attempt to set up a phishing/scamming operation, or (c) setup copies of itself (if you have weights access) on cloud VMs, which will recursively take the same action with the final goal of DDOS'ing a website, leading to exponential proliferation of agents.
These capabilities seem fairly hard to control, to say the least!
I think the risks are overblown. Marc Andreessen has the right perspective, quoting him from memory (from his appearance on a Stratechery podcast):
"When we launched Mosaic people were worried sick about spam and stealing credit card numbers and stuff... And all of those things happened! But they weren't what was important about the Internet"
They could certainly be overblown, but I can't see how to defeat the argument that with increasing model capabilities eventually we will have LLM agents capable of the kinds of attacks I mentioned.
I agree that spam is an example of worry not turning out justified, but that's not an argument against this particular situation turning out differently.
Right, so my objection is not an argument about LLMs - it's a meta-argument. We don't know the full consequences of ANY new technology! Maybe LLM agents will make attacks. Maybe they will provide better defenses. I have no idea.
All I know is that humans are really biased towards imagining horrible consequences of any new tech but so far technological development has been great for humans.
Yes, I agree. We don't know the full consequences so we should rigorously evaluate the capabilities and risks of a variety of models over time to keep track of potential attacks and defenses, so we have early warning of any problems.
Sadly there is not really any institution doing this for LLMs, but maybe soon :)
AI cannot think and its not even close to it yet. We need a technology design breakthrough to leap to the next level of ASI or singularity
I argue that the most valuable problem is guaranteeing an unbiased solution, no matter the budget or time constraint. I don't believe it can be solved or guaranteed. It is a snapshot in time, and any historical problem solving completed can be reprogrammed/retrained or provided bad input at any moment, in any fashion. Dense individuals will continue to mistake speed for accuracy. Just like "safe and effective" at "warp speed".