A Flask-based web application for comparing and managing construction project data (excavation and manhole work) between subcontractors and clients. The system imports data from Excel files, stores it in a database, and generates comparison reports.

Project Overview
Tech Stack
Project Structure
Database Schema
Application Flow
API Routes & Endpoints
Setup & Installation

Project Overview

The Comparison Project is designed to:

Manage subcontractors and their excavation data (Manhole Excavation, Trench Excavation, Domestic Chambers)
Import client-side project data for comparison
Compare subcontractor work against client specifications
Generate detailed reports highlighting differences between client and subcontractor data
User authentication with login/registration system

Key Features

✅ User Authentication (Register/Login/Logout)
✅ Subcontractor Management (Add/Edit/List/Delete)
✅ File Import (Excel/CSV) for both Client and Subcontractor data
✅ Data validation and duplicate detection
✅ Comparison Reports with Excel export
✅ Dashboard with file management

Tech Stack

Backend Framework: Flask
Database: SQL Database (MySQL/PostgreSQL/SQLite configured via environment variables)
ORM: SQLAlchemy
File Processing: Pandas, OpenPyXL, XlsxWriter
Authentication: Werkzeug (Password hashing)
Configuration: Python Dotenv

Dependencies

Flask
pandas
openpyxl
xlrd
Werkzeug
python-dotenv
cryptography
xlsxwriter

Project Structure

Comparison_Project/
├── run.py                          # Application entry point
├── requirements.txt                # Python dependencies
├── .env                            # Environment variables (DB, secrets)
├── README.md                       # This file
│
├── app/
│   ├── __init__.py                # Flask app initialization
│   ├── config.py                  # Configuration settings
│   │
│   ├── models/                    # Database models (SQLAlchemy ORM)
│   │   ├── user_model.py          # User table schema
│   │   ├── subcontractor_model.py # Subcontractor table schema
│   │   ├── manhole_excavation_model.py        # Subcontractor MH excavation
│   │   ├── trench_excavation_model.py         # Subcontractor Trench excavation
│   │   ├── manhole_domestic_chamber_model.py  # Subcontractor Domestic chamber
│   │   ├── mh_ex_client_model.py   # Client MH excavation
│   │   ├── tr_ex_client_model.py   # Client Trench excavation
│   │   └── mh_dc_client_model.py   # Client Domestic chamber
│   │
│   ├── routes/                    # Flask Blueprints (API endpoints)
│   │   ├── auth.py                # Login, Register, Logout
│   │   ├── dashboard.py           # Dashboard page
│   │   ├── file_import.py         # File upload endpoints
│   │   ├── file_report.py         # Fetch and format data for reports
│   │   ├── generate_comparison_report.py  # Comparison logic & Excel export
│   │   ├── subcontractor_routes.py        # CRUD for subcontractors
│   │   ├── user.py                # User management routes
│   │   └── file_format.py         # File format validation
│   │
│   ├── services/                  # Business logic layer
│   │   ├── db_service.py          # Database connection & SQLAlchemy init
│   │   ├── file_service.py        # File parsing & data insertion
│   │   └── user_service.py        # User authentication logic
│   │
│   ├── utils/                     # Helper functions
│   │   ├── file_utils.py          # File path utilities
│   │   ├── helpers.py             # Decorators, common helpers
│   │   └── __pycache__/
│   │
│   ├── static/                    # Static assets
│   │   ├── css/                   # Stylesheets
│   │   ├── images/                # Images
│   │   ├── uploads/               # Uploaded files directory
│   │   │   ├── Client_Bill_1/
│   │   │   └── sub_1/
│   │   └── downloads/             # Generated reports
│   │
│   ├── templates/                 # Jinja2 HTML templates
│   │   ├── base.html              # Base template
│   │   ├── login.html
│   │   ├── register.html
│   │   ├── dashboard.html
│   │   ├── file_import.html
│   │   ├── file_import_client.html
│   │   ├── file_format.html
│   │   ├── file_report.html
│   │   ├── generate_comparison_report.html
│   │   ├── generate_comparison_client_vs_subcont.html
│   │   ├── list_user.html
│   │   ├── report.html
│   │   ├── users.htm
│   │   └── subcontractor/
│   │       ├── add.html
│   │       ├── edit.html
│   │       └── list.html
│   │
│   └── logs/                      # Application logs
│
├── instance/                      # Instance folder (DB, temp files)
└── logs/                          # Root level logs

Database Schema

1. Users Table (`users`)

Stores user authentication data.

Column	Type	Constraints	Purpose
`id`	Integer	Primary Key	User identifier
`name`	String(120)	NOT NULL	User's full name
`email`	String(120)	UNIQUE, NOT NULL	User's email
`password_hash`	String(255)	NOT NULL	Hashed password (Werkzeug)

Key Methods:

set_password(password) - Hash and store password
check_password(password) - Verify password

2. Subcontractors Table (`subcontractors`)

Stores subcontractor company information.

Column	Type	Constraints	Purpose
`id`	Integer	Primary Key	Subcontractor ID
`subcontractor_name`	String(255)	NOT NULL	Company name
`address`	String(500)	-	Company address
`gst_no`	String(50)	-	GST number
`pan_no`	String(50)	-	PAN number
`mobile_no`	String(20)	-	Contact phone
`email_id`	String(150)	-	Contact email
`contact_person`	String(150)	-	Contact person name
`status`	String(20)	Default: "Active"	Active/Inactive status
`created_at`	DateTime	Default: today	Record creation date

Relationships:

One-to-Many with manhole_excavation
One-to-Many with trench_excavation
One-to-Many with manhole_domestic_chamber

3. Manhole Excavation (Subcontractor) (`manhole_excavation`)

Stores manhole excavation work data from subcontractors.

Column Type	Purpose
`id`	Primary Key
`subcontractor_id`	Foreign Key → Subcontractor
`Location`	Worksite location
`MH_NO`	Manhole number
`RA_Bill_No`	RA Bill reference (for grouping)
`Upto_IL_Depth`, `Cutting_Depth`, `ID_of_MH_m`, `Ex_Dia_of_Manhole`, `Area_of_Manhole`	Dimension fields
Excavation Categories (by depth ranges):
Soft_Murum_0_to_1_5, Soft_Murum_1_5_to_3_0, etc.	Material type + depth range
Hard_Murum_0_to_1_5, Hard_Murum_1_5_to_3_0, etc.	Hard mudstone layers
Soft_Rock_0_to_1_5, Soft_Rock_1_5_to_3_0, etc.	Soft rock layers
Hard_Rock_0_to_1_5 through Hard_Rock_6_0_to_7_5	Hard rock layers
Totals	Sum per category
`Total`	Grand total excavation
`Remarks`, `created_at`	Notes and timestamp

4. Trench Excavation (Subcontractor) (`trench_excavation`)

Stores trench/sewer line excavation work data from subcontractors.

Similar structure to Manhole Excavation with additional fields:

Width categories: Width_0_to_2_5, Width_2_5_to_3_0, etc.
Avg_Depth, Actual_Trench_Length, Pipe_Dia_mm

5. Manhole Domestic Chamber (Subcontractor) (`manhole_domestic_chamber`)

Stores domestic chamber construction data.

Key Fields	Purpose
`id`, `subcontractor_id`	Primary & Foreign Key
`Location`, `MH_NO`, `RA_Bill_No`	Identification
`Depth_of_MH`, `MH_TOP_LEVEL`, `MH_IL_LEVEL`	Dimensions
`d_0_to_1_5` through `d_6_0_to_6_5`	Depth-based excavation categories
`Domestic_Chambers`	Count of chambers
`DWC_Pipe_Length`, `UPVC_Pipe_Length`	Pipe lengths

6-8. Client Data Tables (Similar structure to subcontractor models)

mh_ex_client - Manhole Excavation (Client specifications)
tr_ex_client - Trench Excavation (Client specifications)
mh_dc_client - Manhole Domestic Chamber (Client specifications)

Note: Client tables do NOT have subcontractor_id as they represent client baseline data.

Application Flow

1. User Authentication Flow

User Access (/) 
  ↓
Redirect to /login
  ↓
[Login Page]
  ├─ POST /auth/login
  │   ↓
  │   UserService.validate_login(email, password)
  │   ↓
  │   Query User table, verify password
  │   ↓
  │   Set session["user_id"], session["user_name"]
  ↓
Redirect to /dashboard

2. Subcontractor File Import Flow

User → /file/import [GET]
  ↓
Display form with subcontractor dropdown
  ↓
User selects subcontractor & uploads Excel file
  ↓
POST /file/import
  ↓
FileService.handle_file_upload()
  ├─ Validate file type (xlsx, xls, csv)
  ├─ Create upload folder: static/uploads/sub_{id}/
  ├─ Save file
  │
  ├─ Read Excel sheets:
  │  ├─ "Tr.Ex." (Trench Excavation)
  │  ├─ "MH Ex." (Manhole Excavation)
  │  └─ "MH & DC" (Manhole & Domestic Chamber)
  │
  ├─ Process each sheet:
  │  ├─ Normalize column names
  │  ├─ Clean data (handle NaN, empty values)
  │  ├─ Check for duplicates by (Location, MH_NO, RA_Bill_No, subcontractor_id)
  │  ├─ Insert records into respective tables
  │  └─ Commit to database
  │
  └─ Return success/error message

3. Client File Import Flow

User → /file/import_client [GET]
  ↓
Display form for client file upload
  ↓
User uploads Excel file with client specifications
  ↓
POST /file/import_client
  ↓
FileService.handle_client_file_upload()
  ├─ Validate & save file
  ├─ Read sheets: "Tr.Ex.", "MH Ex.", "MH & DC"
  ├─ Process and insert into:
  │  ├─ mh_ex_client
  │  ├─ tr_ex_client
  │  └─ mh_dc_client
  └─ Return status

4. Comparison Report Generation Flow

User → /report/generate_comparison [GET]
  ↓
Display form to select RA Bill No.
  ↓
User selects bill number
  ↓
POST /report/generate_comparison
  ↓
generate_report_bp routes request
  ├─ Fetch client data (mh_ex_client, tr_ex_client, mh_dc_client)
  ├─ Fetch subcontractor data (by RA_Bill_No)
  ├─ For each record type:
  │  ├─ Create lookup table by (Location, MH_NO)
  │  ├─ Match client records with subcontractor records
  │  ├─ Calculate totals for each category
  │  ├─ Compute differences/variances
  │  └─ Format for Excel output
  │
  ├─ Generate Excel file with comparison results:
  │  ├─ Summary sheet
  │  ├─ Detailed Trench Excavation comparison
  │  ├─ Detailed Manhole Excavation comparison
  │  └─ Detailed Domestic Chamber comparison
  │
  └─ Send file to client as download

5. Dashboard & Data Retrieval Flow

User → /dashboard [GET]
  ↓
Check session["user_id"] (authentication)
  ↓
Render dashboard.html
  ├─ Display available reports
  ├─ List recent RA Bills
  ├─ Show subcontractor list
  └─ Provide navigation to:
     ├─ File import
     ├─ Reports
     ├─ Subcontractor management
     └─ Logout

API Routes & Endpoints

Authentication Routes (`/auth`)

Method	Endpoint	Purpose	Auth
GET/POST	`/login`	Login form & validation	❌
GET/POST	`/register`	User registration	❌
GET	`/logout`	Clear session & logout	✅

Dashboard Route (`/dashboard`)

Method	Endpoint	Purpose	Auth
GET	`/dashboard/`	Main dashboard	✅

Subcontractor Routes (`/subcontractor`)

Method	Endpoint	Purpose	Auth
GET	`/subcontractor/add`	Add form	✅
POST	`/subcontractor/save`	Save new subcontractor	✅
GET	`/subcontractor/list`	List all subcontractors	✅
GET	`/subcontractor/edit/<id>`	Edit form	✅
POST	`/subcontractor/update/<id>`	Update subcontractor	✅
GET	`/subcontractor/delete/<id>`	Delete subcontractor	✅

File Import Routes (`/file`)

Method	Endpoint	Purpose	Auth
GET/POST	`/file/import`	Subcontractor file upload	✅
GET/POST	`/file/import_client`	Client file upload	✅
GET/POST	`/file/report`	View imported data report	✅

Report Routes (`/report`)

Method	Endpoint	Purpose	Auth
GET/POST	`/report/generate_comparison`	Generate comparison report	✅
GET/POST	`/report/generate_comparison_client_vs_subcont`	Alternative comparison	✅

User Routes (`/user`)

Method	Endpoint	Purpose	Auth
GET	`/user/list`	List all users	✅
Other	(Check routes)	User management	✅

File Format Route (`/format`)

Method	Endpoint	Purpose	Auth
GET/POST	`/format/...`	File format reference	✅

Setup & Installation

Prerequisites

Python 3.7+
MySQL/PostgreSQL (or SQLite for development)
pip (Python package manager)

1. Clone & Install Dependencies

cd d:\New folder\Comparison\Comparison_Project
pip install -r requirements.txt

2. Configure Environment Variables

Create a .env file in the project root:

# Flask Configuration
FLASK_HOST=127.0.0.1
FLASK_PORT=5000
FLASK_DEBUG=True
SECRET_KEY=your-secret-key-here

# Database Configuration
DB_DIALECT=mysql          # or postgresql, sqlite
DB_DRIVER=pymysql         # or psycopg2, etc.
DB_USER=root
DB_PASSWORD=your_password
DB_HOST=localhost
DB_PORT=3306
DB_NAME=comparison_project

3. Initialize Database

python run.py

This will:

Load environment variables
Create Flask app with configuration
Initialize SQLAlchemy
Create all tables (if not exist)
Start Flask development server

4. Access Application

Open browser: http://127.0.0.1:5000/

First time: Redirect to /login
Register a new account
Login and start using the application

Key Services & Utilities

FileService (`app/services/file_service.py`)

Purpose: Handles file uploads, parsing, and data validation

Key Methods:

handle_file_upload(file, subcontractor_id, RA_Bill_No) - Upload & process subcontractor file
handle_client_file_upload(file, RA_Bill_No) - Upload & process client file
process_trench_excavation(df, subcontractor_id, RA_Bill_No) - Parse & insert trench data
process_manhole_excavation(df, subcontractor_id, RA_Bill_No) - Parse & insert manhole data
process_manhole_domestic_chamber(df, subcontractor_id, RA_Bill_No) - Parse & insert chamber data

UserService (`app/services/user_service.py`)

Purpose: User authentication & management

Key Methods:

register_user(name, email, password) - Register new user
validate_login(email, password) - Authenticate user
get_all_users() - Fetch all users

DBService (`app/services/db_service.py`)

Purpose: Database connection & SQLAlchemy initialization

Exports:

db - SQLAlchemy instance for ORM operations
migrate - Flask-Migrate for schema migrations

Authentication & Security

Session Management

Uses Flask session object
Stores user_id and user_name after successful login
@login_required decorator validates authenticated requests

Password Security

Passwords hashed using Werkzeug's generate_password_hash()
Verification via check_password_hash()
No plaintext passwords stored in database

File Security

File uploads sanitized with secure_filename()
Only allowed extensions: .xlsx, .xls, .csv
Files stored in user-specific subfolders

Error Handling & Validation

File Import Validation

File Type Check - Only CSV/XLS/XLSX allowed
Sheet Validation - Required sheets must exist
Column Normalization - Auto-fixes column name inconsistencies
Data Type Conversion - Converts to appropriate types
Duplicate Detection - Prevents duplicate records by (Location, MH_NO, RA_Bill_No)
Null Handling - Converts empty/NaN values to None

Database Constraints

Foreign key relationships enforced
Unique email constraint on users
NOT NULL constraints on critical fields

Example: Typical User Workflow

1. User registers/logs in
   → POST /auth/register (new user) or /auth/login
   
2. User adds subcontractors
   → GET /subcontractor/add
   → POST /subcontractor/save
   
3. User uploads subcontractor data (Excel file with 3 sheets)
   → GET /file/import
   → POST /file/import (file, subcontractor_id, RA_Bill_No)
   → Database populated with excavation data
   
4. User uploads client specifications (Excel file with same 3 sheets)
   → GET /file/import_client
   → POST /file/import_client (file, RA_Bill_No)
   → Database populated with client data
   
5. User generates comparison report
   → GET /report/generate_comparison
   → POST /report/generate_comparison (RA_Bill_No)
   → System compares client vs subcontractor data
   → Generates Excel file showing differences
   → User downloads report
   
6. User logs out
   → GET /auth/logout
   → Session cleared, redirected to login

Future Enhancements

Email notifications for report generation
Data visualization dashboards
Role-based access control (Admin, User, Viewer)
Bulk report generation
Data export to PDF format
Audit logs for data modifications
API documentation (Swagger/OpenAPI)
Unit & integration tests

Support & Contribution

For issues, feature requests, or contributions, please contact the development team.

Last Updated: January 2026

README.md

Comparison Project - Backend Documentation

Table of Contents

Project Overview

Key Features

Tech Stack

Dependencies

Project Structure

Database Schema

1. Users Table (users)

2. Subcontractors Table (subcontractors)

3. Manhole Excavation (Subcontractor) (manhole_excavation)

4. Trench Excavation (Subcontractor) (trench_excavation)

5. Manhole Domestic Chamber (Subcontractor) (manhole_domestic_chamber)

6-8. Client Data Tables (Similar structure to subcontractor models)

Application Flow

1. User Authentication Flow

2. Subcontractor File Import Flow

3. Client File Import Flow

4. Comparison Report Generation Flow

5. Dashboard & Data Retrieval Flow

API Routes & Endpoints

Authentication Routes (/auth)

Dashboard Route (/dashboard)

Subcontractor Routes (/subcontractor)

File Import Routes (/file)

Report Routes (/report)

User Routes (/user)

File Format Route (/format)

Setup & Installation

Prerequisites

1. Clone & Install Dependencies

2. Configure Environment Variables

3. Initialize Database

4. Access Application

Key Services & Utilities

FileService (app/services/file_service.py)

UserService (app/services/user_service.py)

DBService (app/services/db_service.py)

Authentication & Security

Session Management

Password Security

File Security

Error Handling & Validation

File Import Validation

Database Constraints

Example: Typical User Workflow

Future Enhancements

Support & Contribution

1. Users Table (`users`)

2. Subcontractors Table (`subcontractors`)

3. Manhole Excavation (Subcontractor) (`manhole_excavation`)

4. Trench Excavation (Subcontractor) (`trench_excavation`)

5. Manhole Domestic Chamber (Subcontractor) (`manhole_domestic_chamber`)

Authentication Routes (`/auth`)

Dashboard Route (`/dashboard`)

Subcontractor Routes (`/subcontractor`)

File Import Routes (`/file`)

Report Routes (`/report`)

User Routes (`/user`)

File Format Route (`/format`)

FileService (`app/services/file_service.py`)

UserService (`app/services/user_service.py`)

DBService (`app/services/db_service.py`)