An that can only make http requests is dangerous enough. One that also honors other URL schemes is worse, because those schemes reach beyond the web and can turn a fetch into file disclosure or arbitrary traffic to internal services.
file:// reads the local disk
The file:// scheme addresses the local filesystem rather than a network host. If a fetcher follows it, a request for file:///etc/passwd or a configuration file becomes local file disclosure, with no network involved at all. There is no destination IP to classify; the danger is the scheme itself.
gopher:// crafts raw bytes
The gopher:// scheme is the most abused, because it lets an attacker place nearly arbitrary bytes into a TCP stream to a chosen host and port. That turns SSRF into a way to speak other protocols: a crafted gopher URL can send commands to a Redis instance, forge an (Simple Mail Transfer Protocol) message, or drive any simple line-based service listening on the internal network. It is protocol smuggling through a URL fetcher.
The others
dict:// can probe and interact with services and leak banners. ftp://, tftp://, ldap://, sftp://, and jar:// each reach services a plain web request cannot, and several have been used to move from a fetch to an internal action.
The defense is a scheme allow-list
Because these schemes provide capabilities far beyond fetching a web page, the fix is simple and strict: allow only http and https, and reject every other scheme before the request is built. SSRF URL classifier flags a dangerous scheme as soon as it sees one, raising the risk to high regardless of the host, because the scheme alone is enough to matter.
The scheme list is an allow-list, or it is nothing
Enumerating the dangerous schemes is the wrong shape of defence, because the set is open-ended: file, gopher, dict, ftp, ldap, jar, and whatever the HTTP library you upgraded to last month decided to support.
A library's scheme support is a dependency, not a constant. A fetch that could only speak HTTP in one version can speak more in the next, and nothing in the release notes will be addressed to your blocklist.
So the rule is the same one that governs address checking: decide what is permitted rather than what is forbidden. http and https and nothing else is a policy that survives an upgrade; a list of the schemes you happened to know about in the year you wrote it does not.