diff --git a/util/DbSeederUtility/README.md b/util/DbSeederUtility/README.md new file mode 100644 index 0000000000..e3ab154e39 --- /dev/null +++ b/util/DbSeederUtility/README.md @@ -0,0 +1,141 @@ +# Bitwarden Database Seeder Utility + +A command-line utility for generating and managing test data for Bitwarden databases. + +## Overview + +DbSeederUtility is an executable wrapper around the Seeder class library that provides a convenient command-line interface for: + +1. **Generating** test data as JSON files +2. **Loading** test data into a database +3. **Extracting** database data into seed files +4. **Generating and loading** data in a single operation + +## Commands + +The utility provides the following commands: + +### generate + +Generates seed data as JSON files. + +``` +DbSeeder.exe generate --users --ciphers-per-user --seed-name +``` + +Options: +- `-u, --users`: Number of users to generate +- `-c, --ciphers-per-user`: Number of ciphers per user to generate +- `-n, --seed-name`: Name for the seed data files + +Example: +``` +DbSeeder.exe generate --users 10 --ciphers-per-user 5 --seed-name test_data +``` + +### load + +Loads seed data from JSON files into the database. + +``` +DbSeeder.exe load --seed-name [--timestamp ] [--dry-run] +``` + +Options: +- `-n, --seed-name`: Name of the seed data to load +- `-t, --timestamp`: Specific timestamp of the seed data to load (defaults to most recent) +- `-d, --dry-run`: Validate the seed data without actually loading it + +Example: +``` +DbSeeder.exe load --seed-name test_data +``` + +### generate-direct-load + +Generates seed data and loads it directly into the database without creating JSON files. + +``` +DbSeeder.exe generate-direct-load --users --ciphers-per-user --seed-name +``` + +Options: +- `-u, --users`: Number of users to generate +- `-c, --ciphers-per-user`: Number of ciphers per user to generate +- `-n, --seed-name`: Name identifier for this seed operation + +Example: +``` +DbSeeder.exe generate-direct-load --users 3 --ciphers-per-user 5 --seed-name direct_test_data +``` + +### extract + +Extracts data from the database into seed files. + +``` +DbSeeder.exe extract --seed-name +``` + +Options: +- `-n, --seed-name`: Name for the extracted seed + +Example: +``` +DbSeeder.exe extract --seed-name extracted_data +``` + +## Configuration + +DbSeederUtility uses the same configuration as the Seeder library. See the Seeder README for details on configuration options and file structure. + +## Installation + +The utility can be built and run as a .NET 8 application: + +``` +dotnet build +dotnet run -- [options] +``` + +Or directly using the compiled executable: + +``` +DbSeeder.exe [options] +``` + +## Examples + +### Generate and load test data + +```bash +# Generate 10 users, each with 5 ciphers +DbSeeder.exe generate --users 10 --ciphers-per-user 5 --seed-name demo_data + +# Load the generated data +DbSeeder.exe load --seed-name demo_data +``` + +### Extract and reload data + +```bash +# Extract existing data +DbSeeder.exe extract --seed-name production_backup + +# Load the extracted data +DbSeeder.exe load --seed-name production_backup +``` + +### One-step generation and loading + +```bash +# Generate and load in one step +DbSeeder.exe generate-direct-load --users 5 --ciphers-per-user 10 --seed-name quick_test +``` + +## Dependencies + +This utility depends on: +- The Seeder class library +- CommandDotNet for command-line parsing +- .NET 8.0 runtime \ No newline at end of file diff --git a/util/seeder/README.md b/util/seeder/README.md index e6a9094d1e..1f7f7b4f30 100644 --- a/util/seeder/README.md +++ b/util/seeder/README.md @@ -1,98 +1,130 @@ # Bitwarden Database Seeder -The Bitwarden Database Seeder utility is a tool for generating and loading test data for Bitwarden databases. +A class library for generating, loading, and extracting test data for Bitwarden databases. -## Features +## Overview -- Generate random user accounts with associated ciphers (passwords, secure notes, etc.) -- Load generated seeds into the database -- Configure database connections using the same configuration approach as the Bitwarden Migrator +The Seeder library provides functionality to: + +1. **Generate** realistic test data for Bitwarden, including users and ciphers +2. **Load** previously generated test data into the database +3. **Extract** existing data from a database into seed files +4. **Generate and load** data in a single operation + +## Project Structure + +The project is organized into these main components: + +### Commands + +- **ExtractCommand** - Extracts existing data from the database into seed files +- **GenerateCommand** - Generates new test data as seed files or directly loads it +- **LoadCommand** - Loads previously generated seed files into the database + +### Services + +- **DatabaseContext** - EF Core DbContext connecting to the configured database +- **DatabaseService** - Provides database operations (save, retrieve, clear data) +- **EncryptionService** - Handles security operations (password hashing, encryption) +- **SeederService** - Core service that generates realistic test data using Bogus + +### Settings + +- **GlobalSettings** - Configuration model for database connections +- **GlobalSettingsFactory** - Loads and caches settings from config sources ## Usage -The seeder utility can be run in two ways: +The Seeder library is designed to be used by the DbSeederUtility executable. For direct usage in code: -1. Using `dotnet run` from the source: - ``` - cd util/DbSeederUtility - dotnet run -- generate-seeds --users 10 --ciphers 5 --output ./seeds - dotnet run -- load-seeds --path ./seeds - ``` +```csharp +// Get seeder service from dependency injection +var seederService = serviceProvider.GetRequiredService(); -2. As a standalone executable: - ``` - DbSeeder.exe generate-seeds --users 10 --ciphers 5 --output ./seeds - DbSeeder.exe load-seeds --path ./seeds - ``` +// Generate seed data +await seederService.GenerateSeedsAsync( + userCount: 10, + ciphersPerUser: 5, + seedName: "test_data" +); -## Commands +// Load seed data +await seederService.LoadSeedsAsync( + seedName: "test_data", + timestamp: null // Use most recent if null +); -### Generate Seeds +// Extract data +await seederService.ExtractSeedsAsync( + seedName: "extracted_data" +); -Generates random seed data for users and ciphers. - -``` -DbSeeder.exe generate-seeds [options] - -Options: - -u, --users Number of users to generate (default: 10) - -c, --ciphers Number of ciphers per user (default: 5) - -o, --output Output directory for seed files (default: seeds) -``` - -### Load Seeds - -Loads generated seed data into the database. - -``` -DbSeeder.exe load-seeds [options] - -Options: - -p, --path Path to the directory containing seed data - --dry-run Preview the operation without making changes +// Generate and load in one step +await seederService.GenerateAndLoadSeedsAsync( + userCount: 10, + ciphersPerUser: 5, + seedName: "direct_load" +); ``` ## Configuration -The utility uses the same configuration approach as other Bitwarden utilities: +The library uses the following configuration sources (in order of precedence): -1. **User Secrets** - For local development -2. **appsettings.json** - For general settings -3. **Environment variables** - For deployment environments +1. Environment variables +2. User secrets (with ID "Bit.Seeder") +3. appsettings.{Environment}.json +4. appsettings.json -### Database Configuration - -Configure the database connection in user secrets or appsettings.json: +The expected configuration structure is: ```json { "globalSettings": { - "databaseProvider": "postgresql", // or "sqlserver", "mysql", "sqlite" - "postgreSql": { - "connectionString": "Host=localhost;Port=5432;Database=vault_dev;Username=postgres;Password=YOURPASSWORD;Include Error Detail=true" - }, + "selfHosted": true, + "databaseProvider": "postgres", "sqlServer": { - "connectionString": "Data Source=localhost;Initial Catalog=vault_dev;Integrated Security=SSPI;MultipleActiveResultSets=true" + "connectionString": "..." + }, + "postgreSql": { + "connectionString": "..." + }, + "mySql": { + "connectionString": "..." + }, + "sqlite": { + "connectionString": "..." } } } ``` -## Building the Executable +## Seed File Structure -To build the standalone executable: +Seed files are organized as follows: ``` -cd util -.\publish-seeder.ps1 +seeds/ +├── {seed_name}/ +│ └── {timestamp}/ +│ ├── users/ +│ │ └── users.json +│ └── ciphers/ +│ ├── {user_id1}.json +│ ├── {user_id2}.json +│ └── ... ``` -This will create the executable in the `util/Seeder/publish/win-x64` directory. You can also specify a different runtime: +## Dependencies -``` -.\publish-seeder.ps1 -runtime linux-x64 -``` +- **EntityFrameworkCore** - Database access +- **Bogus** - Realistic test data generation +- **CommandDotNet** - Used by DbSeederUtility for CLI commands +- **DataProtection** - Used for secure data handling -## Integration with Bitwarden Server +## Best Practices -The seeder utility is designed to work seamlessly with Bitwarden Server. It uses the same database models and configuration approach as the server, ensuring compatibility with the core repository. \ No newline at end of file +- Clear the database before loading new seed data +- Use consistent seed names for related operations +- Store sensitive connection strings in user secrets or environment variables +- Use the DbSeederUtility executable for command-line operations \ No newline at end of file