For whatever reason, WordPress does not yet natively support using an SVG as a favicon; fortunately, it is a very basic snippet.

<?php 

// Inject SVG Favicon into website head
add_action('wp_head', function() {
    echo '<link rel="icon" href="full local path to svg">';
});
// Inject SVG Favicon into admin dashboard head. 
add_action('admin_head', function() {
    echo '<link rel="icon" href="full local path to svg">';
});

Just add this to your functions.php or snippet manager.

There is a significant change to the UI in WP 7.1, and the second snippet serves no purpose.