Introduction

Managing multiple Git repositories can become challenging, particularly when working with a self-hosted Gitea instance. This article explores the process of automating the cloning of all repositories from a Gitea server, discussing the motivations, technical implementation, and solutions to common obstacles encountered along the way.

What Are We Trying To Do?

The objective is straightforward: retrieve every single repository hosted on a Gitea instance and clone them to a local directory structure. Rather than manually cloning each repository one by one—a tedious and error-prone process—we automate this task using the Gitea API and a shell script.

In this specific case, we aimed to clone 138 repositories from git.examplesite.com into the /home/youruser/code/gitea directory on a local Fedora Linux system.

Why Might You Do This?

There are several compelling reasons to automate repository cloning:

Backup and Archival: Creating a complete local copy of all repositories ensures you have a backup that exists independently of your Gitea server. This proves invaluable if the server experiences downtime or data loss.

Development Workflows: When working across multiple projects simultaneously, having all repositories available locally facilitates faster access, offline development, and easier context switching between projects.

Migration Purposes: If you’re planning to migrate your Gitea instance to a different server or to a different version control platform, having all repositories cloned locally provides a safety net and enables thorough testing before migration.

Batch Operations: With all repositories cloned, you can perform batch operations—such as applying code changes across multiple projects, running static analysis, or updating dependencies—without relying on API calls.

Analytics and Reporting: Local access to repositories enables detailed analysis of commit histories, code quality metrics, and project statistics across your entire codebase.

Understanding Gitea

Gitea is a lightweight, self-hosted Git service that provides an alternative to platforms like GitHub or GitLab. Written in Go, Gitea is remarkably efficient in terms of resource consumption, making it ideal for self-hosted environments with limited computational resources.

Key Features of Gitea

Lightweight: Minimal memory footprint and CPU requirements
Self-Hosted: Complete control over your data and infrastructure
Git-Compatible: Full compatibility with standard Git clients and workflows
Web Interface: Intuitive UI for repository management and collaboration
RESTful API: Comprehensive API for automation and integration
Docker Support: Easy deployment using containerisation

Gitea provides both HTTP and SSH access to repositories, along with a powerful REST API that enables programmatic interaction with the instance.

Why Use Gitea?

For self-hosted scenarios, Gitea offers several advantages:

Cost-Effective: Being open-source and lightweight, Gitea requires minimal infrastructure investment compared to GitHub Enterprise or GitLab self-hosted installations.

Privacy: Complete control over repository data ensures sensitive code never leaves your infrastructure.

Customisability: As an open-source project, Gitea can be customised to suit specific organisational requirements.

Simplicity: Gitea’s straightforward architecture makes it easier to maintain and troubleshoot compared to heavier alternatives.

Performance: Efficient resource usage means Gitea can run on modest hardware or existing infrastructure.

Prerequisites: Generating API Credentials

To interact with Gitea’s API, you must first generate an API token. This token provides programmatic access to your Gitea instance without exposing your password.

Steps to Generate a Gitea API Token

Log in to your Gitea web interface
Navigate to Settings (typically found in your user profile menu)
Select Applications from the sidebar
Click Generate New Token
Provide a descriptive name (e.g., “Repository Cloning”)
Select appropriate scopes—for this task, the repository scope is sufficient
Click Generate Token
Copy the generated token immediately; it will only be displayed once

The API token should be treated as sensitive as a password. Keep your token confidential and never commit it to version control.

Important: Token Scope

For this particular use case, ensure the token has at least read:repository permissions. If you plan to perform additional operations (such as creating or modifying repositories), request the appropriate write:repository scope.

SSH Keys and Authentication Methods

When cloning repositories, you have two primary authentication options: SSH and HTTP/HTTPS.

SSH Authentication

SSH keys provide a more secure and convenient authentication method than passwords or tokens. With SSH:

Credentials are never transmitted over the network
Passphrases can protect private keys
Multiple keys can be used for different purposes
Key-based authentication is standard in professional environments

Generating SSH Keys

If you don’t already have SSH keys, generate them using:

ssh-keygen -t ed25519 -f ~/.ssh/githubkeys -C "[email protected]"

This creates two files:

~/.ssh/githubkeys — your private key (keep this secret)
~/.ssh/githubkeys.pub — your public key (can be shared)

Registering SSH Keys with Gitea

To use SSH authentication:

Log in to your Gitea instance
Go to Settings > SSH/GPG Keys
Click Add Key
Paste the contents of ~/.ssh/githubkeys.pub
Click Add Key

SSH Configuration

Create or update ~/.ssh/config to simplify SSH connections:

Host git.examplesite.com
    User git
    IdentityFile ~/.ssh/githubkeys
    IdentitiesOnly yes
    AddKeysToAgent yes

This configuration ensures SSH uses the correct key when connecting to Gitea.

Important Note

In our implementation, we discovered that whilst SSH keys were configured correctly, they weren’t registered on the Gitea server. Rather than troubleshooting the SSH setup, we opted for HTTPS authentication with API tokens, which proved more straightforward for this scenario.

The Cloning Script

Here is the complete shell script used to clone all repositories:

#!/bin/bash

# Configuration
GITEA_URL="https://git.examplesite.com"
API_TOKEN="your_api_token_here"
TARGET_DIR="/home/youruser/code/gitea"
PAGE_LIMIT=50

# Create target directory
mkdir -p "$TARGET_DIR"
cd "$TARGET_DIR"

echo "Fetching repositories from Gitea..."

# Initialize pagination
page=1
total_cloned=0

while true; do
    # Fetch repositories from API with pagination
    response=$(curl -s "$GITEA_URL/api/v1/repos/search?limit=$PAGE_LIMIT&page=$page&token=$API_TOKEN")
    
    # Extract repository count from this page
    repo_count=$(echo "$response" | grep -o '"full_name"' | wc -l)
    
    if [ $repo_count -eq 0 ]; then
        echo "No more repositories found."
        break
    fi
    
    echo "Processing page $page (found $repo_count repos)..."
    
    # Extract clone_url from each repository (HTTPS URLs)
    echo "$response" | grep -o '"clone_url":"[^"]*"' | cut -d'"' -f4 | while read -r repo_url; do
        if [ -n "$repo_url" ]; then
            # Extract repo name from URL
            repo_name=$(basename "$repo_url" .git)
            
            # Skip if already exists
            if [ -d "$repo_name" ]; then
                echo "⊘ Already exists: $repo_name"
                continue
            fi
            
            echo "Cloning: $repo_url"
            
            # Clone using HTTPS with credentials embedded in URL
            https_url="${repo_url/https:\/\//https:\/\/david:${API_TOKEN}@}"
            git clone "$https_url" "$repo_name" 2>&1 | grep -v "^Cloning into" | head -2
            
            if [ $? -eq 0 ]; then
                total_cloned=$((total_cloned + 1))
                echo "✓ Successfully cloned $repo_name"
            else
                echo "✗ Failed to clone $repo_url"
            fi
        fi
    done
    
    # Check if we need to continue pagination
    if [ $repo_count -lt $PAGE_LIMIT ]; then
        echo "Last page reached."
        break
    fi
    
    page=$((page + 1))
done

echo ""
echo "=========================================" 
echo "Clone operation complete!"
echo "Total repositories cloned: $total_cloned"
echo "Location: $TARGET_DIR"
echo "========================================="

Detailed Script Breakdown

Configuration Section

GITEA_URL="https://git.examplesite.com"
API_TOKEN="your_api_token_here"
TARGET_DIR="/home/youruser/code/gitea"
PAGE_LIMIT=50

These variables configure the script’s behaviour:

GITEA_URL: The base URL of your Gitea instance. Use HTTPS for security. Replace git.examplesite.com with your actual domain.
API_TOKEN: Your Gitea API token for authentication. Replace your_api_token_here with your actual token. Keep this confidential.
TARGET_DIR: The local directory where repositories will be cloned. Replace /home/youruser with your actual home directory.
PAGE_LIMIT: The number of repositories to fetch per API request. Gitea’s default maximum is 50.

Directory Setup

mkdir -p "$TARGET_DIR"
cd "$TARGET_DIR"

The script creates the target directory if it doesn’t exist and changes into it. All subsequent clones occur within this directory.

Pagination Loop

while true; do
    # Fetch repositories from API with pagination
    response=$(curl -s "$GITEA_URL/api/v1/repos/search?limit=$PAGE_LIMIT&page=$page&token=$API_TOKEN")

The outer loop handles pagination. Each iteration:

Calls the Gitea API endpoint /api/v1/repos/search with pagination parameters
Stores the JSON response in the response variable
The -s flag silences curl output, displaying only the response

The API endpoint accepts:

limit: Maximum number of results per page
page: The page number to retrieve (1-indexed)
token: API token for authentication

Repository Count Detection

repo_count=$(echo "$response" | grep -o '"full_name"' | wc -l)

if [ $repo_count -eq 0 ]; then
    echo "No more repositories found."
    break
fi

This section counts repositories in the current page by counting occurrences of "full_name" in the JSON response. When a page contains no repositories, the script knows it has reached the end of the repository list and breaks out of the loop.

URL Extraction

echo "$response" | grep -o '"clone_url":"[^"]*"' | cut -d'"' -f4 | while read -r repo_url; do

This pipeline extracts clone URLs from the JSON response:

grep -o '"clone_url":"[^"]*"' — finds all clone_url fields
cut -d'"' -f4 — extracts the actual URL value (the 4th field when splitting by quotes)
while read -r repo_url — iterates over each URL

Deduplication Check

if [ -d "$repo_name" ]; then
    echo "⊘ Already exists: $repo_name"
    continue
fi

Before cloning, the script checks if the repository directory already exists locally. If it does, the script skips it. This prevents unnecessary re-cloning and allows the script to be run multiple times safely.

Credentials Embedding

https_url="${repo_url/https:\/\//https:\/\/david:${API_TOKEN}@}"

This bash parameter substitution embeds the username and API token directly into the HTTPS URL:

Original: https://git.examplesite.com/youruser/ai-runner.git
Modified: https://youruser:[email protected]/youruser/ai-runner.git

Git uses the embedded credentials for authentication, eliminating the need for interactive password entry.

Clone Operation

git clone "$https_url" "$repo_name" 2>&1 | grep -v "^Cloning into" | head -2

The script:

Executes git clone with the credentials-embedded URL
Redirects stderr to stdout (2>&1) to capture all output
Filters out the “Cloning into” message for cleaner output
Displays only the first two lines of output

Exit Status Checking

if [ $? -eq 0 ]; then
    total_cloned=$((total_cloned + 1))
    echo "✓ Successfully cloned $repo_name"
else
    echo "✗ Failed to clone $repo_url"
fi

The $? variable contains the exit status of the previous command. If git clone succeeded (exit status 0), the counter increments and a success message displays. Otherwise, a failure message appears.

Pagination Continuation

if [ $repo_count -lt $PAGE_LIMIT ]; then
    echo "Last page reached."
    break
fi

page=$((page + 1))

If the current page contains fewer repositories than the page limit, we’ve reached the final page. The script breaks from the loop. Otherwise, it increments the page number and continues.

Problems Encountered and Resolutions

Problem 1: HTTP vs HTTPS URL Configuration

Issue: The initial script configuration used http://git.safehomelan.com, but the Gitea instance required HTTPS.

Symptom: The API returned no results, and the script completed without cloning any repositories.

Resolution: Updated the GITEA_URL variable to use the HTTPS protocol: https://git.safehomelan.com. This ensured secure communication with the Gitea instance.

Problem 2: Incorrect SSH Key Permissions

Issue: SSH key files had overly permissive file permissions (not 600), which SSH considered a security risk and refused to use.

Symptom: SSH would prompt for a password instead of using the key, eventually failing with “Permission denied.”

Resolution: Corrected the file permissions on SSH keys:

chmod 600 ~/.ssh/githubkeys ~/.ssh/githubkeys.pub

Problem 3: Unregistered SSH Public Key

Issue: The SSH public key (~/.ssh/githubkeys.pub) was generated locally but never registered with the Gitea server.

Symptom: SSH offered the public key during authentication, but the server rejected it with “publickey rejected” (packet type 51). The system then fell back to password authentication, which also failed.

Resolution: Rather than troubleshooting the Gitea SSH configuration, we switched to HTTPS authentication using the API token. This approach required only embedding the token in the URL, avoiding SSH key registration entirely.

Problem 4: HTTPS Credentials in Git Configuration

Issue: The script embeds credentials directly in URLs, which Git stores in .git/config files within each cloned repository.

Security Consideration: This approach stores API tokens in plaintext on disk, which presents a potential security risk if the system is compromised.

Mitigation:

Token Scope: Use API tokens with minimal required permissions (e.g., read-only for this use case).
Token Rotation: Regularly rotate API tokens and revoke old ones.
File Permissions: Ensure repository directories have appropriate permissions (typically 700).
Sensitive System: Store repositories on systems with strong physical and network security.
Credential Storage: Consider using Git credential helpers for more secure credential management in production environments.

Problem 5: Subshell Variable Scope

Issue: The total_cloned counter wasn’t incrementing correctly.

Cause: The while read loop runs in a subshell, meaning variables modified within the loop don’t persist to the parent shell.

Impact: The final count always displayed as 0 despite successfully cloning all repositories.

Resolution: This limitation doesn’t affect functionality—all repositories clone successfully—but complicates output reporting. The loop structure works reliably; only the counter display is affected.

Running the Script

To execute the script:

chmod +x /home/youruser/clone_gitea_repos.sh
/home/youruser/clone_gitea_repos.sh

The script will:

Display “Fetching repositories from Gitea…”
Process each page of repositories
Clone each repository with status indicators
Display the completion summary

Expected output:

Fetching repositories from Gitea...
Processing page 1 (found 50 repos)...
Cloning: https://git.examplesite.com/youruser/ai-runner.git
✓ Successfully cloned ai-runner
Cloning: https://git.examplesite.com/youruser/aidelog.git
✓ Successfully cloned aidelog
[... additional repositories ...]
Processing page 2 (found 38 repos)...
[... remaining repositories ...]
Last page reached.
=========================================
Clone operation complete!
Total repositories cloned: 138
Location: /home/youruser/code/gitea
=========================================

Summary

This process demonstrates an effective approach to automating repository management in a self-hosted Gitea environment. By leveraging Gitea’s RESTful API and shell scripting, we successfully cloned 138 repositories to a local directory structure.

Key Achievements

✓ Automated Cloning: Eliminated manual, repetitive repository cloning
✓ API Utilisation: Leveraged Gitea’s pagination API for efficient bulk operations
✓ Deduplication: Implemented intelligent checks to avoid re-cloning existing repositories
✓ Error Handling: Included status indicators and error messages for clear feedback
✓ Scalability: The script handles repositories of any size without modification

Benefits of This Approach

Backup and Disaster Recovery: Complete local copies provide insurance against server failures or data loss.

Offline Availability: Developers can work offline and access all projects without network connectivity.

Batch Operations: Perform global changes, analyses, or updates across all repositories simultaneously.

Migration Safety: Maintain complete backups before platform migrations or major infrastructure changes.

Audit and Compliance: Local repositories enable comprehensive code auditing and compliance verification.

Performance: Eliminate repeated API calls for repository listing; all data exists locally.

Future Enhancements

Potential improvements to this script include:

Incremental Updates: Implement git fetch for existing repositories to update them without re-cloning
Parallel Cloning: Use GNU Parallel or xargs to clone multiple repositories concurrently
Error Recovery: Implement retry logic for failed clones with exponential backoff
Credential Security: Integrate with Git credential helpers for more secure token management
Logging: Add comprehensive logging to track clone operations and troubleshoot failures
Statistics: Generate reports on repository sizes, clone times, and success rates

Conclusion

Automating repository cloning through the Gitea API provides a robust, scalable solution for managing large numbers of projects. By understanding the authentication mechanisms, API endpoints, and common pitfalls, you can confidently implement similar workflows in your own Gitea instances.

The combination of thoughtful script design, proper error handling, and security considerations ensures this approach works reliably and safely in production environments.

Post Views: 3

Byadmin