Drupal 9 - Add new terms to existing field programatically

I'm having trouble adding a new term to an existing field on a Drupal 9 website. Attempts to set field values work for fields I've tested (ie: text and boolean) but not for taxonomy term reference fields. Not sure exactly what I'm missing.

Below is the code I'm currently working with.

  $nids =[123, 124, 125];

  foreach ($nids as $nid) {
    /** @var \Drupal\node\NodeInterface $node */
    $node = \Drupal::entityTypeManager()->getStorage('node')->load($nid);
    $field_category = $node->get('field_category')->getValue();
    $categories = [];
    $new_category = "100647";

    foreach ($field_category as $category) {
      $categories[] = $category['target_id'];
    };

    if (!in_array($new_category, $categories)) {
      $categories[] = $new_category;

      try {
        $node->set('field_category', $categories);
        $node->save();
      }
      catch (\Exception $e) {}
    }
  }

source https://stackoverflow.com/questions/73872166/drupal-9-add-new-terms-to-existing-field-programatically