Such an agency is a must and it has to be done fast, but there is no reason why it should be “a government agency.”
In fact it has to be instated by Congress or at international level by the UN but it has to be independent and composed of people who are no apparatchiks and no bureaucrats but specialists from the very AI field. Otherwise we run into the danger of having this field sterilized by bureaucrats or politicians. Look at what is happening with climate change in the USA right now.
