WordPress: tax query of X number of posts related by tag first, then by category if not enough in tag-related

[admin]

<h1>Question</h1>

How to output tag related posts first, then IF there are less than 4 tag related posts, fill the rest of the 4 spots with category related posts?

<h2>Scenario</h2>

Sometimes a post has very few tags, or the tags it is labeled with have very fews posts. When outputting "related posts by tag," the area is either very sparse with 1 or 2 posts, or outright empty.

To solve this it would be good to show posts from related categories, if there are not enough tag related posts to satisfy

posts_per_page => X

.

So, something like this:

<hr>

Scenario A -- if more than 4 tag-related posts exist, then:

Related Posts:

Show the below posts:

1. tag-related post #1
2. tag-related post #2
3. tag-related post #3
4. tag-related post #4

Do Not show the below posts:

5. tag-related post #5
6. tag-related post #6
7. tag-related post #7
...

<hr>

Scenario B -- if only 2 tag-related posts exist, then:

Related Posts:

Show the below posts:

1. tag-related post #1
2. tag-related post #2  
3. category-related post #1
4. category-related post #2

Do Not show the below posts:

5. category-related post #3
6. category-related post #4
7. category-related post #5
...

<hr>

<h2>What I've tried</h2>

The tax query I am using:

// start of the tax_query arguments
$args = array( 'posts_per_page'=>4, 'post__not_in' => array($post->ID), 'tax_query' => array( 'relation' => 'OR' ) );

// get current post tags
$tags = wp_get_object_terms( $post->ID, 'post_tag', array( 'fields' => 'ids' ) );

if ( !empty( $tags ) ) {
    $args['tax_query'][] = array(
        'taxonomy' => 'post_tag',
        'field'    => 'id',
        'terms'    => $tags
    );
}

// get current post categories
$categories = wp_get_object_terms( $post->ID, 'category', array( 'fields' => 'ids' ) );

if ( !empty( $categories ) ) {
    $args['tax_query'][] = array(
        'taxonomy' => 'category',
        'field'    => 'id',
        'terms'    => $categories
    );
}

// the query
$related_query = new WP_Query( $args );

As I understand it, that tax query says "get posts that are in the same categories, then get posts that are in the same tags, then output posts until 4 are on screen."

Yet it keeps outputting category posts first (of which there are many) which satisfies the 4 posts on screens rule and leaves out the most important tag-related posts. I've tried moving the code around, using

AND

instead of

OR

, which didn't work and made no sense to me anyways.

I've also seen these posts: <a href="https://stackoverflow.com/questions...st-of-posts-filtered-by-tag-and-then-category">WordPress - producing a list of posts filtered by tag and then category</a> and <a href="https://stackoverflow.com/questions/29215181/wordpress-query-posts-by-tag-and-category">Wordpress query posts by tag and category</a>, but they are about outputting a list of posts that are filtered by tag AND category. I need posts first related by tag, and if 4 or more, then just output those top 4. If less than 4, then output up to as many category-related posts to meet the 4 posts criteria.

Clearly I am misunderstanding the query and/or the problem, so any help would be appreciated.