skrazzo-public/mtc-project-sync-script
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update readme

Browse Source
This commit is contained in:
Leons Aleksandrovs
2025-12-07 23:21:23 +02:00
parent eeaf4a9a45
commit 646a278169
1 changed files with 104 additions and 492 deletions
Download Patch File Download Diff File Expand all files Collapse all files

596
README.md
View File

@@ -1,547 +1,159 @@
# MTC Project Sync Script # MTC Project Sync Script
A comprehensive automation toolkit for synchronizing and managing projects between remote servers and local development environments. Designed for developers working with remote servers who need efficient project setup and bidirectional file synchronization. A simple toolkit for synchronizing projects between remote servers and local development environments.
## Table of Contents ## Setup
- [Overview](#overview) ### 1. Clone the Repository
- [Features](#features)
- [Requirements](#requirements)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Scripts Reference](#scripts-reference)
- [sync-project.sh](#sync-projectsh)
- [local-remote.sh](#local-remotesh)
- [remote-local.sh](#remote-localsh)
- [sftp-watch.sh](#sftp-watchsh)
- [build-watch.sh](#build-watchsh)
- [Configuration](#configuration)
- [Troubleshooting](#troubleshooting)
- [License](#license)
## Overview
**MTC Project Sync Script** is a bash-based automation suite that streamlines remote project development workflows. It handles initial project setup, ongoing file synchronization, and integrates seamlessly with VS Code for SFTP-based development.
### What It Does
- Downloads complete projects from remote servers with one command
- Configures VS Code SFTP settings automatically
- Provides helper scripts for continuous sync and build workflows
- Manages bidirectional file synchronization between local and remote environments
- Integrates lazygit for convenient version control
## Features
- **One-Command Setup** - Initialize and download remote projects instantly
- **VS Code Integration** - Auto-generates SFTP configuration with optimized ignore patterns
- **Bidirectional Sync** - Push local changes to server or pull remote changes to local
- **Watch Mode Support** - Automatically sync files on changes
- **Build Integration** - Watch and upload build artifacts automatically
- **Error Handling** - Comprehensive validation at each step
- **Clean Architecture** - Modular scripts for different workflows
- **Version Control Ready** - Includes lazygit and proper .gitignore setup
## Requirements
### System Dependencies
- **Bash** (4.0 or higher)
- **SSH/SCP** - Secure shell and copy utilities
- **sshpass** - Password-based SSH authentication
- **tar** - Archive utility
- **rsync** - Fast file synchronization tool
- **watchexec** (optional) - For watch mode functionality
### Installation of Dependencies
#### macOS (using Homebrew)
```bash
brew install sshpass rsync watchexec
```
#### Ubuntu/Debian
```bash
sudo apt-get install sshpass rsync openssh-client
# For watch functionality:
cargo install watchexec-cli
# or download from: https://github.com/watchexec/watchexec/releases
```
#### CentOS/RHEL
```bash
sudo yum install sshpass rsync openssh-clients
```
### Server Requirements
- SSH access with username/password or key-based authentication
- Sufficient permissions to create/read files in project directory
- tar utility installed on remote server
## Installation
### Method 1: Clone Repository
```bash ```bash
cd ~/Documents/GitHub cd ~/Documents/GitHub
git clone <repository-url> mtc-project-sync-script git clone <repository-url> mtc-project-sync-script
cd mtc-project-sync-script
chmod +x sync-project.sh
``` ```
### Method 2: Create Symbolic Link (Recommended) ### 2. Create a Symlink in Your Projects Directory
For easy access from any project directory: Navigate to where you keep all your projects and create a symbolic link to the sync script:
```bash ```bash
# From your project directory cd ~/Documents/Projects # or wherever you keep your projects
ln -s ~/Documents/GitHub/mtc-project-sync-script/sync-project.sh sync-project.sh ln -s ~/Documents/GitHub/mtc-project-sync-script/sync-project.sh sync-project.sh
# Run the script
./sync-project.sh
``` ```
This creates a symbolic link that allows you to use the script without copying it to every project. **Why symlink?** The script uses its own location to find the `scripts/` folder and `lazygit` binary. A symlink preserves the connection to the original script location, so it can access these resources. This way, you only need one copy of the toolkit that all your projects can use.
## Quick Start ### 3. Run the Sync Script
1. **Navigate to your desired project directory:** From your projects directory, run:
```bash
mkdir my-project && cd my-project
```
2. **Create symbolic link or copy the script:**
```bash
ln -s ~/Documents/GitHub/mtc-project-sync-script/sync-project.sh .
```
3. **Run the sync script:**
```bash
./sync-project.sh
```
4. **Provide the requested information:**
- Server address (e.g., `username@server.com`)
- Server project path (e.g., `/home/username/public_html`)
- Local folder path (e.g., `./project-name`)
- SSH password when prompted
5. **Start developing!** The script will have:
- Downloaded all project files
- Created VS Code SFTP configuration
- Generated helper scripts for ongoing development
- Set up lazygit for version control
## Scripts Reference
### sync-project.sh
**Main synchronization script** - Downloads a complete project from a remote server and configures the local development environment.
#### Usage
```bash ```bash
./sync-project.sh ./sync-project.sh
``` ```
#### What It Does Or for SFTP-only setup (skips lazygit and .gitignore modifications):
1. **Prompts for Configuration:** ```bash
- Server credentials (user@host) ./sync-project.sh --sftp-only
- Remote project path ```
- Local destination folder
## What the Script Does
When you run `sync-project.sh`, it will:
1. **Ask for configuration:**
- Server user@host (e.g., `user@example.com`)
- Server project path (e.g., `/home/user/public_html`)
- Local folder path (e.g., `my-project`)
- SSH password - SSH password
2. **Creates Archive on Server:** 2. **Sync your project:**
- Compresses the entire project directory into `leo.tar.gz` - Creates a tar archive on the remote server
- Stores archive in parent directory temporarily - Downloads and extracts it to your local folder
- Cleans up temporary files
3. **Downloads Archive:** 3. **Generate configuration and scripts:**
- Uses SCP to securely transfer the archive - Creates `.vscode/sftp.json` for VS Code SFTP extension
- Shows progress during transfer - Generates helper scripts with your credentials pre-configured
- Adds scripts to `.gitignore` (unless using `--sftp-only`)
- Uploads lazygit to server (unless using `--sftp-only`)
4. **Extracts Locally:** ## Generated Scripts
- Unpacks the archive to specified local folder
- Preserves file permissions and structure
5. **Generates VS Code Configuration:** After running the sync script, you'll have these helper scripts in your project:
- Creates `.vscode/sftp.json` with:
- Server connection details
- Auto-upload on save enabled
- Optimized ignore patterns
6. **Creates Helper Scripts:** ### `remote-local.sh`
- Generates `local-remote.sh`, `remote-local.sh`, `sftp-watch.sh`, and `build-watch.sh` **Purpose:** Download files from server to local machine
- Pre-configures them with your server credentials
- Adds them to `.gitignore`
7. **Deploys lazygit:** **What it does:**
- Copies lazygit binary to local project - Uses rsync to pull files from remote server
- Uploads lazygit to remote server - Only downloads changed files
- Adds to `.gitignore` - Excludes common development folders (node_modules, vendor, .git, etc.)
8. **Cleanup:** **Use cases:**
- Removes temporary archive files from both local and remote - Get latest changes from server
#### Example Session
```bash
$ ./sync-project.sh
=== Project Sync Configuration ===
Server user@host (e.g., leo@server.com): john@myserver.com
Server project path (e.g., /home/leo/public_leo): /home/john/websites/shop
Local folder path (e.g., Delta Pharmacy): online-shop
SSH Password: [hidden]
Starting project sync...
Creating archive on server...
Archive created successfully
Copying archive to local machine...
Archive copied successfully
Extracting archive...
Archive extracted successfully
Cleaning up local archive...
Cleaning up server archive...
File sync complete!
.vscode/sftp.json created successfully
build-watch.sh created successfully
Added build-watch.sh to .gitignore
local-remote.sh created successfully
Added local-remote.sh to .gitignore
remote-local.sh created successfully
Added remote-local.sh to .gitignore
sftp-watch.sh created successfully
Added sftp-watch.sh to .gitignore
Uploading lazygit to server...
lazygit uploaded successfully
Added lazygit to .gitignore
```
---
### local-remote.sh
**Push local changes to remote server** - Synchronizes files from your local machine to the remote server.
#### Usage
```bash
./local-remote.sh
```
#### What It Does
- Uses `rsync` to upload local files to remote server
- Preserves timestamps and permissions
- Deletes files on server that don't exist locally (using `--delete` flag)
- Excludes common development directories (node_modules, vendor, storage, etc.)
- Only uploads changed files for efficiency
#### Excluded Directories
- `.git`
- `.vscode`
- `.DS_Store`
- `node_modules`
- `vendor`
- `storage`
- `uploads`
- `temp`
- `cache`
- `sitepress-multilingual-cms`
#### Use Cases
- Deploy local changes to staging/production
- Update server after local development session
- Restore server files from local backup
---
### remote-local.sh
**Pull remote changes to local** - Synchronizes files from the remote server to your local machine.
#### Usage
```bash
./remote-local.sh
```
#### What It Does
- Uses `rsync` to download files from remote server
- Preserves timestamps and permissions
- Does NOT delete local files (no `--delete` flag)
- Excludes common development directories
- Only downloads changed files for efficiency
#### Excluded Directories
- `.vscode`
- `.DS_Store`
- `node_modules`
- `vendor`
- `storage`
- `uploads`
- `temp`
- `cache`
- `sitepress-multilingual-cms`
#### Use Cases
- Update local environment with server changes
- Pull production data for debugging - Pull production data for debugging
- Sync changes made by other team members - Sync before starting work
- Backup server files to local machine
--- ### `local-remote.sh`
**Purpose:** Upload files from local machine to server
### sftp-watch.sh **What it does:**
- Uses rsync to push files to remote server
- Only uploads changed files
- Deletes remote files that don't exist locally
- Excludes common development folders
**Bidirectional sync with watch mode** - Continuously monitors file changes and syncs automatically. **Use cases:**
- Deploy local changes to server
- Push code updates
- Restore server from local backup
#### Usage ### `ssh.sh`
**Purpose:** SSH into the remote server
**What it does:**
- Opens an SSH connection to your server
- Changes directory to your project path
- Supports `--db` flag for MySQL port forwarding
**Usage:**
```bash ```bash
./sftp-watch.sh ./ssh.sh # Normal SSH connection
./ssh.sh --db # SSH with MySQL port forwarding (local:3306 -> remote:3306)
``` ```
#### What It Does **Use cases:**
- Run commands on the server
- Check logs
- Access remote database via localhost:3306
1. **Initial Sync:** ### `sftp-watch.sh`
- Runs `remote-local.sh` first to ensure local is up-to-date **Purpose:** Continuous bidirectional sync
- Displays "Syncing remote->local..." message
2. **Watch Mode:** **What it does:**
- Monitors current directory for file changes - First runs `remote-local.sh` to sync down
- Debounces changes (500ms delay) to avoid excessive syncing - Watches for local file changes
- Automatically runs `local-remote.sh` on any file modification - Automatically runs `local-remote.sh` when files change
#### Requirements **Requirements:** Needs `watchexec` installed (`brew install watchexec`)
- Requires `watchexec` to be installed
#### Use Cases **Use cases:**
- Development with immediate server deployment - Development with live server testing
- Real-time collaboration on remote server - Real-time sync during coding
- Continuous integration workflows
- Testing on remote environment while coding locally
#### Example Workflow ### `watch-build.sh`
```bash **Purpose:** Watch for changes, build, and upload
# Terminal 1: Start watch mode
./sftp-watch.sh
# Terminal 2: Make changes to your code **What it does:**
vim index.php - Monitors files for changes
# Changes automatically sync to server after save - Runs `npm run build` when changes detected
``` - Uploads built assets to server using `local-remote.sh`
--- **Requirements:**
- `watchexec` installed
- `package.json` with build script
- Node.js/npm installed
### build-watch.sh **Use cases:**
**Watch and build mode** - Monitors source files, rebuilds on changes, and deploys build output.
#### Usage
```bash
./build-watch.sh
```
#### What It Does
1. Monitors current directory for file changes (2-second debounce)
2. Runs `npm run build` when changes are detected
3. Automatically executes `local-remote.sh` after successful build
4. Uploads built assets to remote server
#### Requirements
- Requires `watchexec` to be installed
- Requires `package.json` with build script defined
- Node.js and npm installed
#### Use Cases
- Frontend development with build steps (React, Vue, etc.) - Frontend development with build steps (React, Vue, etc.)
- Asset compilation (Sass, Less, TypeScript) - Automatic deployment of compiled assets
- Continuous deployment of build artifacts - Continuous integration during development
- Development where server runs production builds
#### Example Workflow ## Requirements
- **bash** - Shell scripting
- **ssh/scp** - Remote server access
- **sshpass** - Password authentication
- **tar** - Archive creation
- **rsync** - File synchronization
- **watchexec** (optional) - For watch mode scripts
### Install on macOS:
```bash ```bash
# Terminal: Start build watch brew install sshpass rsync watchexec
./build-watch.sh
# Make changes to source files
# Watches for changes -> Runs npm run build -> Uploads to server
``` ```
#### package.json Example ### Install on Ubuntu/Debian:
```json
{
"scripts": {
"build": "webpack --mode production"
}
}
```
---
## Configuration
### VS Code SFTP Configuration
The sync script automatically generates `.vscode/sftp.json`:
```json
{
"name": "project-name",
"host": "server.com",
"protocol": "sftp",
"port": 22,
"username": "your-username",
"password": "your-password",
"remotePath": "/path/to/remote/project",
"uploadOnSave": true,
"useTempFile": false,
"openSsh": false,
"ignore": [
".vscode",
".DS_Store",
"node_modules",
"vendor",
"storage",
"uploads",
"temp",
"cache",
"sitepress-multilingual-cms"
]
}
```
### Customizing Ignore Patterns
Edit `.vscode/sftp.json` or the helper scripts to add/remove excluded directories:
```bash ```bash
# Edit local-remote.sh or remote-local.sh sudo apt-get install sshpass rsync openssh-client
--exclude custom_folder \
``` ```
### Security Considerations
1. **Credentials in Scripts:**
- Helper scripts contain plaintext passwords
- These scripts are automatically added to `.gitignore`
- Never commit these files to version control
2. **Alternative: SSH Keys**
- Consider setting up SSH key-based authentication
- Eliminates need for password storage
- More secure for production environments
```bash
# Generate SSH key
ssh-keygen -t rsa -b 4096
# Copy to server
ssh-copy-id user@server.com
```
## Troubleshooting
### Common Issues
#### "sshpass: command not found"
**Solution:** Install sshpass using your package manager (see [Requirements](#requirements))
#### "watchexec: command not found"
**Solution:** Install watchexec for watch mode functionality:
```bash
brew install watchexec # macOS
cargo install watchexec-cli # Cross-platform
```
#### "Permission denied" during SSH
**Solutions:**
- Verify username and password are correct
- Check that SSH is enabled on the server
- Ensure your user has permission to access the remote path
- Try connecting manually: `ssh user@server.com`
#### "Remote path not found"
**Solutions:**
- Verify the server project path exists
- Use absolute paths (e.g., `/home/user/project` not `~/project`)
- Check path spelling and case sensitivity
#### "Archive creation failed"
**Solutions:**
- Ensure you have write permissions in the parent directory
- Check if there's enough disk space on the server
- Verify tar is installed on the server
#### Files not syncing in watch mode
**Solutions:**
- Ensure watchexec is installed
- Check that you're editing files in the watched directory
- Try increasing debounce delay in the script
- Verify rsync is working manually
#### Build fails in build-watch.sh
**Solutions:**
- Verify `npm run build` works manually
- Check that Node.js and npm are installed
- Ensure package.json has a build script defined
- Review npm error messages for missing dependencies
### Debug Mode
Enable verbose output for rsync commands:
```bash
# Edit local-remote.sh or remote-local.sh
# Add -v flag to rsync
rsync -avzuv ...
```
### Manual Testing
Test components individually:
```bash
# Test SSH connection
ssh user@server.com
# Test SCP transfer
scp testfile.txt user@server.com:~/
# Test rsync
rsync -avzu --dry-run . user@server.com:~/test/
```
## Best Practices
1. **Initial Setup:**
- Run `sync-project.sh` once per project
- Keep the sync script in a central location and use symbolic links
2. **Development Workflow:**
- Use `sftp-watch.sh` for rapid development with live server testing
- Use `build-watch.sh` for projects with build steps
- Use `local-remote.sh` for manual deployments
3. **Team Collaboration:**
- Run `remote-local.sh` before starting work to get latest changes
- Communicate with team when making large changes
- Use version control (git) in parallel for code history
4. **Security:**
- Never commit generated scripts with credentials
- Use SSH keys for production servers
- Limit server access with appropriate permissions
## License
This project is provided as-is for development and synchronization purposes.
## Contributing
Contributions, issues, and feature requests are welcome. Feel free to check the issues page or submit pull requests.
---
**Note:** This toolkit is designed for development workflows. Always test in staging environments before deploying to production servers.