Following the migration of repositories and reviewer settings to Bitbucket Cloud, the next logical step in ensuring a smooth transition is migrating groups and their memberships. This post, part of our series on Bitbucket Cloud migration, focuses on automating the migration of groups and group memberships from Bitbucket Server (Data Center) to Bitbucket Cloud. Like the previous posts, this guide relies on the config.py file for accessing Bitbucket credentials.

Introduction

Groups in Bitbucket Server are a vital part of managing permissions and access controls efficiently. Migrating these groups and their memberships manually to Bitbucket Cloud can be tedious and prone to errors, especially for organizations with a significant number of groups. This post introduces a two-part script solution to automate this process, ensuring your team’s access controls are seamlessly transferred to the cloud.

Preparing Group Membership Data

Querying the Bitbucket Server Database

Before running our scripts to create groups and add members in Bitbucket Cloud, we must first generate the group-membership.csv file. This file is derived from querying your Bitbucket Server’s database to list all group memberships.

SQL Query:

SELECT u.user_name,
       u.display_name,
       g.group_name
FROM cwd_membership m
INNER JOIN cwd_user u ON m.child_id = u.id
INNER JOIN cwd_group g ON m.parent_id = g.id
INNER JOIN cwd_directory d ON g.directory_id = d.id
ORDER BY d.directory_name, u.user_name, g.group_name;

This query joins a few tables to compile a comprehensive list of users, their display names, and the groups to which they belong. Ensure you have the necessary permissions to execute this query against your Bitbucket Server database.

Result Example:

"user_name","display_name","group_name"
"rbortolin","Rodolfo Bortolin","group-1"

Generating group-membership.csv

Execute the above SQL query against your Bitbucket Server database and export the results into a CSV file named group-membership.csv. This file will be used by our scripts to automate the migration process.

Automating Group and Membership Migration

The migration process involves two scripts: the first to create groups in Bitbucket Cloud based on a CSV input, and the second to populate these groups with members, also driven by CSV data.

Script 1: Creating Groups in Bitbucket Cloud

This script reads a list of group names from a CSV file (group-membership.csv) and creates these groups in Bitbucket Cloud. It ensures that all your required groups exist in the cloud before adding members to them.

import csv
import os
import requests
import logging
from requests.auth import HTTPBasicAuth
from config import cloud

# Initialize logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s: %(message)s')

# Define file paths directly
script_location = os.path.dirname(os.path.abspath(__file__))
membership_csv = os.path.join(script_location, "group-membership.csv")

workspace_id = cloud['workspace']

# Function to create group in Bitbucket Cloud
def create_group(workspace_id, group_name):
    try:
        response = requests.post(
            f"https://api.bitbucket.org/1.0/groups/{workspace_id}",
            auth=HTTPBasicAuth(cloud['username'], cloud['token']),  # Authentication setup
            data={"name": group_name}  # Data payload
        )
        return response
    except Exception as e:
        logging.exception(f"Failed to create group: {group_name}")
        return None

# Function to load group names from CSV and return a set of unique group names
def load_unique_groups(file_path):
    unique_groups = set()
    try:
        with open(file_path, encoding='utf-8') as f:
            reader = csv.DictReader(f)
            for row in reader:
                unique_groups.add(row['group_name'])
    except Exception as e:
        logging.exception("Failed to load unique groups from CSV")
    return unique_groups

# Main function to create groups in Bitbucket Cloud
def create_groups_from_csv(workspace_id, membership_csv):
    unique_groups = load_unique_groups(membership_csv)
    for group_name in unique_groups:
        response = create_group(workspace_id, group_name)
        if response and response.ok:
            logging.info(f"Successfully created group: {group_name}")
        elif response:
            logging.error(f"Failed to create group: {group_name}, Reason: {response.text}")
        else:
            logging.error(f"Failed to create group: {group_name}, no response object.")

# Create groups from CSV file
create_groups_from_csv(workspace_id, membership_csv)

Script 2: Adding Members to Groups

After creating the necessary groups in Bitbucket Cloud, the next step is to add members to these groups. This script uses two CSV files: one containing the mappings between Bitbucket Server and Cloud users (bitbucket_users_match.csv, generated in previous posts), and another specifying which users belong to which groups (group-membership.csv).

import csv
import requests
import os
import logging
from requests.auth import HTTPBasicAuth
from config import cloud

# Setup logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

# Define file paths directly
script_location = os.path.dirname(os.path.abspath(__file__))
users_file = os.path.join(script_location, "bitbucket_users_match.csv")
membership_csv = os.path.join(script_location, "group-membership.csv")

# Read the 'bitbucket_users_match.csv' and create a mapping from display_name to cloud_uuid
def create_displayname_to_uuid_map(users_file):
    displayname_to_uuid_map = {}
    try:
        with open(users_file, encoding='utf-8') as f:
            reader = csv.DictReader(f)
            for row in reader:
                display_name = row['cloud_display_name']
                cloud_uuid = row['cloud_uuid']
                displayname_to_uuid_map[display_name] = cloud_uuid
    except Exception as e:
        logging.error(f"Error reading users file {users_file}: {e}")
    return displayname_to_uuid_map

# Get group slugs from Bitbucket API
def get_group_slugs(workspace_id):
    url = f"https://api.bitbucket.org/1.0/groups/{workspace_id}/"
    auth = HTTPBasicAuth(cloud['username'], cloud['token'])
    try:
        response = requests.get(url, auth=auth)
        group_slugs = {}
        if response.ok:
            for group in response.json():
                group_slugs[group['name']] = group['slug']
            logging.info(f"Retrieved group slugs for workspace: {workspace_id}")
        else:
            logging.error(f"Failed to fetch group slugs. Response: {response.text}")
    except Exception as e:
        logging.error(f"Exception occurred while fetching group slugs: {e}")
    return group_slugs

# Function to add a user to a group in Bitbucket Cloud
def add_user_to_group(workspace_id, group_slug, user_uuid):
    try:
        response = requests.put(
            f"https://api.bitbucket.org/1.0/groups/{workspace_id}/{group_slug}/members/{user_uuid}/",
            auth=HTTPBasicAuth(cloud['username'], cloud['token']),  # Authentication setup
            data='{}'  # Empty data payload as per the API requirement
        )
        if response.ok:
            logging.info(f"Successfully added user {user_uuid} to group {group_slug}")
        else:
            logging.warning(f"Failed to add user {user_uuid} to group {group_slug}. Response: {response.text}")
    except Exception as e:
        logging.error(f"Exception occurred while adding user to group: {e}")

# Main function to add users to groups based on the CSV files
def add_users_to_groups(workspace_id, users_file, membership_csv, group_slugs):
    # Create the mapping from display names to UUIDs
    displayname_to_uuid_map = create_displayname_to_uuid_map(users_file)
    
    try:
        with open(membership_csv, encoding='utf-8') as f:
            reader = csv.DictReader(f)
            for row in reader:
                display_name = row['display_name']
                group_name = row['group_name']
                group_slug = group_slugs.get(group_name)
                user_uuid = displayname_to_uuid_map.get(display_name)
                
                if group_slug and user_uuid:
                    add_user_to_group(workspace_id, group_slug, user_uuid)
                    logging.info(f"{display_name} added to group {group_slug}")
                else:
                    logging.warning(f"Missing group slug or user UUID for {display_name} in group {group_name}.")
    except Exception as e:
        logging.error(f"Error processing membership CSV: {e}")

workspace_id = cloud['workspace'] 
group_slugs = get_group_slugs(workspace_id)
add_users_to_groups(workspace_id, users_file, membership_csv, group_slugs)

Running the Scripts

  1. Prepare your environment: Ensure you have completed the previous posts’ steps, setting the stage for this migration process.
  2. Run the first script to create groups in Bitbucket Cloud based on your CSV file.
  3. Execute the second script to add users to the groups, leveraging the mappings and group slugs retrieved by the script.

Conclusion

Migrating groups and their memberships is a crucial aspect of maintaining your project’s access control and permissions structure in Bitbucket Cloud. By automating this process, you ensure a more accurate and efficient transition, minimizing manual effort and potential errors. This guide has walked you through automating the migration of groups and group memberships, further smoothing your transition to Bitbucket Cloud.