570 lines
18 KiB
Markdown
570 lines
18 KiB
Markdown
# Comparison Project - Backend Documentation
|
|
|
|
A Flask-based web application for comparing and managing construction project data (excavation and manhole work) between subcontractors and clients. The system imports data from Excel files, stores it in a database, and generates comparison reports.
|
|
|
|
## Table of Contents
|
|
|
|
- [Project Overview](#project-overview)
|
|
- [Tech Stack](#tech-stack)
|
|
- [Project Structure](#project-structure)
|
|
- [Database Schema](#database-schema)
|
|
- [Application Flow](#application-flow)
|
|
- [API Routes & Endpoints](#api-routes--endpoints)
|
|
- [Setup & Installation](#setup--installation)
|
|
|
|
---
|
|
|
|
## Project Overview
|
|
|
|
The Comparison Project is designed to:
|
|
- **Manage subcontractors** and their excavation data (Manhole Excavation, Trench Excavation, Domestic Chambers)
|
|
- **Import client-side project data** for comparison
|
|
- **Compare subcontractor work** against client specifications
|
|
- **Generate detailed reports** highlighting differences between client and subcontractor data
|
|
- **User authentication** with login/registration system
|
|
|
|
### Key Features
|
|
|
|
✅ User Authentication (Register/Login/Logout)
|
|
✅ Subcontractor Management (Add/Edit/List/Delete)
|
|
✅ File Import (Excel/CSV) for both Client and Subcontractor data
|
|
✅ Data validation and duplicate detection
|
|
✅ Comparison Reports with Excel export
|
|
✅ Dashboard with file management
|
|
|
|
---
|
|
|
|
## Tech Stack
|
|
|
|
**Backend Framework**: Flask
|
|
**Database**: SQL Database (MySQL/PostgreSQL/SQLite configured via environment variables)
|
|
**ORM**: SQLAlchemy
|
|
**File Processing**: Pandas, OpenPyXL, XlsxWriter
|
|
**Authentication**: Werkzeug (Password hashing)
|
|
**Configuration**: Python Dotenv
|
|
|
|
### Dependencies
|
|
|
|
```
|
|
Flask
|
|
pandas
|
|
openpyxl
|
|
xlrd
|
|
Werkzeug
|
|
python-dotenv
|
|
cryptography
|
|
xlsxwriter
|
|
```
|
|
|
|
---
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
Comparison_Project/
|
|
├── run.py # Application entry point
|
|
├── requirements.txt # Python dependencies
|
|
├── .env # Environment variables (DB, secrets)
|
|
├── README.md # This file
|
|
│
|
|
├── app/
|
|
│ ├── __init__.py # Flask app initialization
|
|
│ ├── config.py # Configuration settings
|
|
│ │
|
|
│ ├── models/ # Database models (SQLAlchemy ORM)
|
|
│ │ ├── user_model.py # User table schema
|
|
│ │ ├── subcontractor_model.py # Subcontractor table schema
|
|
│ │ ├── manhole_excavation_model.py # Subcontractor MH excavation
|
|
│ │ ├── trench_excavation_model.py # Subcontractor Trench excavation
|
|
│ │ ├── manhole_domestic_chamber_model.py # Subcontractor Domestic chamber
|
|
│ │ ├── mh_ex_client_model.py # Client MH excavation
|
|
│ │ ├── tr_ex_client_model.py # Client Trench excavation
|
|
│ │ └── mh_dc_client_model.py # Client Domestic chamber
|
|
│ │
|
|
│ ├── routes/ # Flask Blueprints (API endpoints)
|
|
│ │ ├── auth.py # Login, Register, Logout
|
|
│ │ ├── dashboard.py # Dashboard page
|
|
│ │ ├── file_import.py # File upload endpoints
|
|
│ │ ├── file_report.py # Fetch and format data for reports
|
|
│ │ ├── generate_comparison_report.py # Comparison logic & Excel export
|
|
│ │ ├── subcontractor_routes.py # CRUD for subcontractors
|
|
│ │ ├── user.py # User management routes
|
|
│ │ └── file_format.py # File format validation
|
|
│ │
|
|
│ ├── services/ # Business logic layer
|
|
│ │ ├── db_service.py # Database connection & SQLAlchemy init
|
|
│ │ ├── file_service.py # File parsing & data insertion
|
|
│ │ └── user_service.py # User authentication logic
|
|
│ │
|
|
│ ├── utils/ # Helper functions
|
|
│ │ ├── file_utils.py # File path utilities
|
|
│ │ ├── helpers.py # Decorators, common helpers
|
|
│ │ └── __pycache__/
|
|
│ │
|
|
│ ├── static/ # Static assets
|
|
│ │ ├── css/ # Stylesheets
|
|
│ │ ├── images/ # Images
|
|
│ │ ├── uploads/ # Uploaded files directory
|
|
│ │ │ ├── Client_Bill_1/
|
|
│ │ │ └── sub_1/
|
|
│ │ └── downloads/ # Generated reports
|
|
│ │
|
|
│ ├── templates/ # Jinja2 HTML templates
|
|
│ │ ├── base.html # Base template
|
|
│ │ ├── login.html
|
|
│ │ ├── register.html
|
|
│ │ ├── dashboard.html
|
|
│ │ ├── file_import.html
|
|
│ │ ├── file_import_client.html
|
|
│ │ ├── file_format.html
|
|
│ │ ├── file_report.html
|
|
│ │ ├── generate_comparison_report.html
|
|
│ │ ├── generate_comparison_client_vs_subcont.html
|
|
│ │ ├── list_user.html
|
|
│ │ ├── report.html
|
|
│ │ ├── users.htm
|
|
│ │ └── subcontractor/
|
|
│ │ ├── add.html
|
|
│ │ ├── edit.html
|
|
│ │ └── list.html
|
|
│ │
|
|
│ └── logs/ # Application logs
|
|
│
|
|
├── instance/ # Instance folder (DB, temp files)
|
|
└── logs/ # Root level logs
|
|
```
|
|
|
|
---
|
|
|
|
## Database Schema
|
|
|
|
### 1. **Users Table** (`users`)
|
|
Stores user authentication data.
|
|
|
|
| Column | Type | Constraints | Purpose |
|
|
|--------|------|-------------|---------|
|
|
| `id` | Integer | Primary Key | User identifier |
|
|
| `name` | String(120) | NOT NULL | User's full name |
|
|
| `email` | String(120) | UNIQUE, NOT NULL | User's email |
|
|
| `password_hash` | String(255) | NOT NULL | Hashed password (Werkzeug) |
|
|
|
|
**Key Methods:**
|
|
- `set_password(password)` - Hash and store password
|
|
- `check_password(password)` - Verify password
|
|
|
|
---
|
|
|
|
### 2. **Subcontractors Table** (`subcontractors`)
|
|
Stores subcontractor company information.
|
|
|
|
| Column | Type | Constraints | Purpose |
|
|
|--------|------|-------------|---------|
|
|
| `id` | Integer | Primary Key | Subcontractor ID |
|
|
| `subcontractor_name` | String(255) | NOT NULL | Company name |
|
|
| `address` | String(500) | - | Company address |
|
|
| `gst_no` | String(50) | - | GST number |
|
|
| `pan_no` | String(50) | - | PAN number |
|
|
| `mobile_no` | String(20) | - | Contact phone |
|
|
| `email_id` | String(150) | - | Contact email |
|
|
| `contact_person` | String(150) | - | Contact person name |
|
|
| `status` | String(20) | Default: "Active" | Active/Inactive status |
|
|
| `created_at` | DateTime | Default: today | Record creation date |
|
|
|
|
**Relationships:**
|
|
- One-to-Many with `manhole_excavation`
|
|
- One-to-Many with `trench_excavation`
|
|
- One-to-Many with `manhole_domestic_chamber`
|
|
|
|
---
|
|
|
|
### 3. **Manhole Excavation (Subcontractor)** (`manhole_excavation`)
|
|
Stores manhole excavation work data from subcontractors.
|
|
|
|
| Column Type | Purpose |
|
|
|-----|---------|
|
|
| `id` | Primary Key |
|
|
| `subcontractor_id` | Foreign Key → Subcontractor |
|
|
| `Location` | Worksite location |
|
|
| `MH_NO` | Manhole number |
|
|
| `RA_Bill_No` | RA Bill reference (for grouping) |
|
|
| `Upto_IL_Depth`, `Cutting_Depth`, `ID_of_MH_m`, `Ex_Dia_of_Manhole`, `Area_of_Manhole` | Dimension fields |
|
|
| **Excavation Categories** (by depth ranges): | |
|
|
| Soft_Murum_0_to_1_5, Soft_Murum_1_5_to_3_0, etc. | Material type + depth range |
|
|
| Hard_Murum_0_to_1_5, Hard_Murum_1_5_to_3_0, etc. | Hard mudstone layers |
|
|
| Soft_Rock_0_to_1_5, Soft_Rock_1_5_to_3_0, etc. | Soft rock layers |
|
|
| Hard_Rock_0_to_1_5 through Hard_Rock_6_0_to_7_5 | Hard rock layers |
|
|
| **Totals** | Sum per category |
|
|
| `Total` | Grand total excavation |
|
|
| `Remarks`, `created_at` | Notes and timestamp |
|
|
|
|
---
|
|
|
|
### 4. **Trench Excavation (Subcontractor)** (`trench_excavation`)
|
|
Stores trench/sewer line excavation work data from subcontractors.
|
|
|
|
**Similar structure to Manhole Excavation with additional fields:**
|
|
- Width categories: `Width_0_to_2_5`, `Width_2_5_to_3_0`, etc.
|
|
- `Avg_Depth`, `Actual_Trench_Length`, `Pipe_Dia_mm`
|
|
|
|
---
|
|
|
|
### 5. **Manhole Domestic Chamber (Subcontractor)** (`manhole_domestic_chamber`)
|
|
Stores domestic chamber construction data.
|
|
|
|
| Key Fields | Purpose |
|
|
|-----------|---------|
|
|
| `id`, `subcontractor_id` | Primary & Foreign Key |
|
|
| `Location`, `MH_NO`, `RA_Bill_No` | Identification |
|
|
| `Depth_of_MH`, `MH_TOP_LEVEL`, `MH_IL_LEVEL` | Dimensions |
|
|
| `d_0_to_1_5` through `d_6_0_to_6_5` | Depth-based excavation categories |
|
|
| `Domestic_Chambers` | Count of chambers |
|
|
| `DWC_Pipe_Length`, `UPVC_Pipe_Length` | Pipe lengths |
|
|
|
|
---
|
|
|
|
### 6-8. **Client Data Tables** (Similar structure to subcontractor models)
|
|
|
|
- **`mh_ex_client`** - Manhole Excavation (Client specifications)
|
|
- **`tr_ex_client`** - Trench Excavation (Client specifications)
|
|
- **`mh_dc_client`** - Manhole Domestic Chamber (Client specifications)
|
|
|
|
**Note:** Client tables do NOT have `subcontractor_id` as they represent client baseline data.
|
|
|
|
---
|
|
|
|
## Application Flow
|
|
|
|
### 1. **User Authentication Flow**
|
|
```
|
|
User Access (/)
|
|
↓
|
|
Redirect to /login
|
|
↓
|
|
[Login Page]
|
|
├─ POST /auth/login
|
|
│ ↓
|
|
│ UserService.validate_login(email, password)
|
|
│ ↓
|
|
│ Query User table, verify password
|
|
│ ↓
|
|
│ Set session["user_id"], session["user_name"]
|
|
↓
|
|
Redirect to /dashboard
|
|
```
|
|
|
|
### 2. **Subcontractor File Import Flow**
|
|
```
|
|
User → /file/import [GET]
|
|
↓
|
|
Display form with subcontractor dropdown
|
|
↓
|
|
User selects subcontractor & uploads Excel file
|
|
↓
|
|
POST /file/import
|
|
↓
|
|
FileService.handle_file_upload()
|
|
├─ Validate file type (xlsx, xls, csv)
|
|
├─ Create upload folder: static/uploads/sub_{id}/
|
|
├─ Save file
|
|
│
|
|
├─ Read Excel sheets:
|
|
│ ├─ "Tr.Ex." (Trench Excavation)
|
|
│ ├─ "MH Ex." (Manhole Excavation)
|
|
│ └─ "MH & DC" (Manhole & Domestic Chamber)
|
|
│
|
|
├─ Process each sheet:
|
|
│ ├─ Normalize column names
|
|
│ ├─ Clean data (handle NaN, empty values)
|
|
│ ├─ Check for duplicates by (Location, MH_NO, RA_Bill_No, subcontractor_id)
|
|
│ ├─ Insert records into respective tables
|
|
│ └─ Commit to database
|
|
│
|
|
└─ Return success/error message
|
|
```
|
|
|
|
### 3. **Client File Import Flow**
|
|
```
|
|
User → /file/import_client [GET]
|
|
↓
|
|
Display form for client file upload
|
|
↓
|
|
User uploads Excel file with client specifications
|
|
↓
|
|
POST /file/import_client
|
|
↓
|
|
FileService.handle_client_file_upload()
|
|
├─ Validate & save file
|
|
├─ Read sheets: "Tr.Ex.", "MH Ex.", "MH & DC"
|
|
├─ Process and insert into:
|
|
│ ├─ mh_ex_client
|
|
│ ├─ tr_ex_client
|
|
│ └─ mh_dc_client
|
|
└─ Return status
|
|
```
|
|
|
|
### 4. **Comparison Report Generation Flow**
|
|
```
|
|
User → /report/generate_comparison [GET]
|
|
↓
|
|
Display form to select RA Bill No.
|
|
↓
|
|
User selects bill number
|
|
↓
|
|
POST /report/generate_comparison
|
|
↓
|
|
generate_report_bp routes request
|
|
├─ Fetch client data (mh_ex_client, tr_ex_client, mh_dc_client)
|
|
├─ Fetch subcontractor data (by RA_Bill_No)
|
|
├─ For each record type:
|
|
│ ├─ Create lookup table by (Location, MH_NO)
|
|
│ ├─ Match client records with subcontractor records
|
|
│ ├─ Calculate totals for each category
|
|
│ ├─ Compute differences/variances
|
|
│ └─ Format for Excel output
|
|
│
|
|
├─ Generate Excel file with comparison results:
|
|
│ ├─ Summary sheet
|
|
│ ├─ Detailed Trench Excavation comparison
|
|
│ ├─ Detailed Manhole Excavation comparison
|
|
│ └─ Detailed Domestic Chamber comparison
|
|
│
|
|
└─ Send file to client as download
|
|
```
|
|
|
|
### 5. **Dashboard & Data Retrieval Flow**
|
|
```
|
|
User → /dashboard [GET]
|
|
↓
|
|
Check session["user_id"] (authentication)
|
|
↓
|
|
Render dashboard.html
|
|
├─ Display available reports
|
|
├─ List recent RA Bills
|
|
├─ Show subcontractor list
|
|
└─ Provide navigation to:
|
|
├─ File import
|
|
├─ Reports
|
|
├─ Subcontractor management
|
|
└─ Logout
|
|
```
|
|
|
|
---
|
|
|
|
## API Routes & Endpoints
|
|
|
|
### Authentication Routes (`/auth`)
|
|
| Method | Endpoint | Purpose | Auth |
|
|
|--------|----------|---------|------|
|
|
| GET/POST | `/login` | Login form & validation | ❌ |
|
|
| GET/POST | `/register` | User registration | ❌ |
|
|
| GET | `/logout` | Clear session & logout | ✅ |
|
|
|
|
### Dashboard Route (`/dashboard`)
|
|
| Method | Endpoint | Purpose | Auth |
|
|
|--------|----------|---------|------|
|
|
| GET | `/dashboard/` | Main dashboard | ✅ |
|
|
|
|
### Subcontractor Routes (`/subcontractor`)
|
|
| Method | Endpoint | Purpose | Auth |
|
|
|--------|----------|---------|------|
|
|
| GET | `/subcontractor/add` | Add form | ✅ |
|
|
| POST | `/subcontractor/save` | Save new subcontractor | ✅ |
|
|
| GET | `/subcontractor/list` | List all subcontractors | ✅ |
|
|
| GET | `/subcontractor/edit/<id>` | Edit form | ✅ |
|
|
| POST | `/subcontractor/update/<id>` | Update subcontractor | ✅ |
|
|
| GET | `/subcontractor/delete/<id>` | Delete subcontractor | ✅ |
|
|
|
|
### File Import Routes (`/file`)
|
|
| Method | Endpoint | Purpose | Auth |
|
|
|--------|----------|---------|------|
|
|
| GET/POST | `/file/import` | Subcontractor file upload | ✅ |
|
|
| GET/POST | `/file/import_client` | Client file upload | ✅ |
|
|
| GET/POST | `/file/report` | View imported data report | ✅ |
|
|
|
|
### Report Routes (`/report`)
|
|
| Method | Endpoint | Purpose | Auth |
|
|
|--------|----------|---------|------|
|
|
| GET/POST | `/report/generate_comparison` | Generate comparison report | ✅ |
|
|
| GET/POST | `/report/generate_comparison_client_vs_subcont` | Alternative comparison | ✅ |
|
|
|
|
### User Routes (`/user`)
|
|
| Method | Endpoint | Purpose | Auth |
|
|
|--------|----------|---------|------|
|
|
| GET | `/user/list` | List all users | ✅ |
|
|
| Other | (Check routes) | User management | ✅ |
|
|
|
|
### File Format Route (`/format`)
|
|
| Method | Endpoint | Purpose | Auth |
|
|
|--------|----------|---------|------|
|
|
| GET/POST | `/format/...` | File format reference | ✅ |
|
|
|
|
---
|
|
|
|
## Setup & Installation
|
|
|
|
### Prerequisites
|
|
- Python 3.7+
|
|
- MySQL/PostgreSQL (or SQLite for development)
|
|
- pip (Python package manager)
|
|
|
|
### 1. Clone & Install Dependencies
|
|
```bash
|
|
cd d:\New folder\Comparison\Comparison_Project
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
### 2. Configure Environment Variables
|
|
Create a `.env` file in the project root:
|
|
```env
|
|
# Flask Configuration
|
|
FLASK_HOST=127.0.0.1
|
|
FLASK_PORT=5000
|
|
FLASK_DEBUG=True
|
|
SECRET_KEY=your-secret-key-here
|
|
|
|
# Database Configuration
|
|
DB_DIALECT=mysql # or postgresql, sqlite
|
|
DB_DRIVER=pymysql # or psycopg2, etc.
|
|
DB_USER=root
|
|
DB_PASSWORD=your_password
|
|
DB_HOST=localhost
|
|
DB_PORT=3306
|
|
DB_NAME=comparison_project
|
|
```
|
|
|
|
### 3. Initialize Database
|
|
```bash
|
|
python run.py
|
|
```
|
|
This will:
|
|
- Load environment variables
|
|
- Create Flask app with configuration
|
|
- Initialize SQLAlchemy
|
|
- Create all tables (if not exist)
|
|
- Start Flask development server
|
|
|
|
### 4. Access Application
|
|
Open browser: `http://127.0.0.1:5000/`
|
|
- First time: Redirect to `/login`
|
|
- Register a new account
|
|
- Login and start using the application
|
|
|
|
---
|
|
|
|
## Key Services & Utilities
|
|
|
|
### FileService (`app/services/file_service.py`)
|
|
**Purpose:** Handles file uploads, parsing, and data validation
|
|
|
|
**Key Methods:**
|
|
- `handle_file_upload(file, subcontractor_id, RA_Bill_No)` - Upload & process subcontractor file
|
|
- `handle_client_file_upload(file, RA_Bill_No)` - Upload & process client file
|
|
- `process_trench_excavation(df, subcontractor_id, RA_Bill_No)` - Parse & insert trench data
|
|
- `process_manhole_excavation(df, subcontractor_id, RA_Bill_No)` - Parse & insert manhole data
|
|
- `process_manhole_domestic_chamber(df, subcontractor_id, RA_Bill_No)` - Parse & insert chamber data
|
|
|
|
### UserService (`app/services/user_service.py`)
|
|
**Purpose:** User authentication & management
|
|
|
|
**Key Methods:**
|
|
- `register_user(name, email, password)` - Register new user
|
|
- `validate_login(email, password)` - Authenticate user
|
|
- `get_all_users()` - Fetch all users
|
|
|
|
### DBService (`app/services/db_service.py`)
|
|
**Purpose:** Database connection & SQLAlchemy initialization
|
|
|
|
**Exports:**
|
|
- `db` - SQLAlchemy instance for ORM operations
|
|
- `migrate` - Flask-Migrate for schema migrations
|
|
|
|
---
|
|
|
|
## Authentication & Security
|
|
|
|
### Session Management
|
|
- Uses Flask `session` object
|
|
- Stores `user_id` and `user_name` after successful login
|
|
- `@login_required` decorator validates authenticated requests
|
|
|
|
### Password Security
|
|
- Passwords hashed using Werkzeug's `generate_password_hash()`
|
|
- Verification via `check_password_hash()`
|
|
- No plaintext passwords stored in database
|
|
|
|
### File Security
|
|
- File uploads sanitized with `secure_filename()`
|
|
- Only allowed extensions: `.xlsx`, `.xls`, `.csv`
|
|
- Files stored in user-specific subfolders
|
|
|
|
---
|
|
|
|
## Error Handling & Validation
|
|
|
|
### File Import Validation
|
|
1. **File Type Check** - Only CSV/XLS/XLSX allowed
|
|
2. **Sheet Validation** - Required sheets must exist
|
|
3. **Column Normalization** - Auto-fixes column name inconsistencies
|
|
4. **Data Type Conversion** - Converts to appropriate types
|
|
5. **Duplicate Detection** - Prevents duplicate records by (Location, MH_NO, RA_Bill_No)
|
|
6. **Null Handling** - Converts empty/NaN values to None
|
|
|
|
### Database Constraints
|
|
- Foreign key relationships enforced
|
|
- Unique email constraint on users
|
|
- NOT NULL constraints on critical fields
|
|
|
|
---
|
|
|
|
## Example: Typical User Workflow
|
|
|
|
```
|
|
1. User registers/logs in
|
|
→ POST /auth/register (new user) or /auth/login
|
|
|
|
2. User adds subcontractors
|
|
→ GET /subcontractor/add
|
|
→ POST /subcontractor/save
|
|
|
|
3. User uploads subcontractor data (Excel file with 3 sheets)
|
|
→ GET /file/import
|
|
→ POST /file/import (file, subcontractor_id, RA_Bill_No)
|
|
→ Database populated with excavation data
|
|
|
|
4. User uploads client specifications (Excel file with same 3 sheets)
|
|
→ GET /file/import_client
|
|
→ POST /file/import_client (file, RA_Bill_No)
|
|
→ Database populated with client data
|
|
|
|
5. User generates comparison report
|
|
→ GET /report/generate_comparison
|
|
→ POST /report/generate_comparison (RA_Bill_No)
|
|
→ System compares client vs subcontractor data
|
|
→ Generates Excel file showing differences
|
|
→ User downloads report
|
|
|
|
6. User logs out
|
|
→ GET /auth/logout
|
|
→ Session cleared, redirected to login
|
|
```
|
|
|
|
---
|
|
|
|
## Future Enhancements
|
|
|
|
- [ ] Email notifications for report generation
|
|
- [ ] Data visualization dashboards
|
|
- [ ] Role-based access control (Admin, User, Viewer)
|
|
- [ ] Bulk report generation
|
|
- [ ] Data export to PDF format
|
|
- [ ] Audit logs for data modifications
|
|
- [ ] API documentation (Swagger/OpenAPI)
|
|
- [ ] Unit & integration tests
|
|
|
|
---
|
|
|
|
## Support & Contribution
|
|
|
|
For issues, feature requests, or contributions, please contact the development team.
|
|
|
|
**Last Updated:** January 2026 |