Cyber Services / All lessons / Python Programming / Directory Structures

Directory Structures

Module 24 of 31 · 4 min read · Free to read

MODULES ↓ Command blocks have a COPY button
Organizing Python Projects for Scalability, Maintainability, and Collaboration

Why Directory Structure Matters

A well-organized directory structure is the backbone of any professional Python project. It enhances readability, simplifies collaboration, enables smooth testing and deployment, and makes your codebase scalable. Whether you're building a small script or a large web application, thoughtful project layout pays dividends in long-term maintainability.

Core Principles of Good Structure

  • Separation of Concerns: Group related functionality together (e.g., models, views, utilities).
  • Modularity: Break code into reusable, independent modules and packages.
  • Testability: Keep tests separate but aligned with source code.
  • Environment Isolation: Use virtual environments and avoid hardcoding paths.
  • Standardization: Follow community conventions (e.g., PEP 8, packaging standards).

Basic Project Layout (Small to Medium Projects)

Typical Structure for a CLI or Library
my_project/
├── README.md
├── requirements.txt
├── setup.py (or pyproject.toml)
├── .gitignore
├── my_package/
│   ├── __init__.py
│   ├── core.py
│   ├── utils.py
│   └── helpers/
│       ├── __init__.py
│       └── validators.py
├── tests/
│   ├── __init__.py
│   ├── test_core.py
│   └── test_utils.py
└── scripts/
    └── deploy.sh

Key Components Explained

  • README.md: Project overview, installation, and usage instructions.
  • requirements.txt: Lists dependencies (e.g., requests==2.31.0).
  • setup.py or pyproject.toml: For packaging and distribution (modern projects prefer pyproject.toml).
  • my_package/: The main Python package—must contain __init__.py to be importable.
  • tests/: Mirrors the package structure for easy test discovery.
  • scripts/: Deployment, migration, or utility shell/Python scripts.

Advanced Structure (Web Applications or Large Systems)

Django-Style or Enterprise Layout
ecommerce_app/
├── README.md
├── requirements/
│   ├── base.txt
│   ├── dev.txt
│   └── prod.txt
├── pyproject.toml
├── .env.example
├── .gitignore
├── manage.py
├── app/
│   ├── __init__.py
│   ├── settings/
│   │   ├── __init__.py
│   │   ├── base.py
│   │   ├── dev.py
│   │   └── prod.py
│   ├── urls.py
│   └── wsgi.py
├── core/
│   ├── __init__.py
│   ├── models.py
│   ├── views.py
│   └── utils.py
├── products/
│   ├── __init__.py
│   ├── models.py
│   ├── views.py
│   └── admin.py
├── users/
│   ├── __init__.py
│   ├── models.py
│   └── views.py
├── static/
│   └── css/
├── templates/
│   └── base.html
├── tests/
│   ├── __init__.py
│   ├── conftest.py
│   ├── unit/
│   └── integration/
└── docs/
    └── architecture.md

Working with Directories in Python Code

Python provides built-in modules to interact with the file system and manage paths in a cross-platform way.

Key Modules: os, os.path, and pathlib (Modern Approach)

Using pathlib (Recommended in Python 3.4+)
from pathlib import Path

# Get current working directory
cwd = Path.cwd()
print("Current dir:", cwd)

# Create a path to a file
config_path = Path("config") / "settings.json"
print("Config path:", config_path)

# Check if file/directory exists
if config_path.exists():
    print("Config exists!")

# Create directories (parents=True creates intermediate dirs)
logs_dir = Path("logs")
logs_dir.mkdir(exist_ok=True, parents=True)

# List all .py files in a directory
py_files = list(Path("my_package").glob("*.py"))
for f in py_files:
    print(f.name)

# Resolve absolute path
abs_path = config_path.resolve()
print("Absolute path:", abs_path)
Legacy: Using os and os.path
import os

# Current working directory
cwd = os.getcwd()

# Join paths safely (cross-platform)
config_path = os.path.join("config", "settings.json")

# Check existence
if os.path.exists(config_path):
    print("Config found!")

# Create directory
os.makedirs("logs", exist_ok=True)

# List files
py_files = [f for f in os.listdir("my_package") if f.endswith(".py")]
print(py_files)

Best Practices for Directory Management

  • ✅ Use pathlib.Path instead of os.path for new projects—it’s more readable and object-oriented.
  • ✅ Never hardcode absolute paths like C:\Users\... or /home/user/....
  • ✅ Use relative paths from the project root or resolve paths dynamically using __file__ or Path(__file__).parent.
  • ✅ Keep sensitive data (API keys, passwords) out of code—use environment variables or .env files (with python-dotenv).
  • ✅ Include a .gitignore to exclude __pycache__, virtual environments (venv/), and logs.

Common Pitfalls & Anti-Patterns

  • ❌ Flat structure with all files in one directory—becomes unmaintainable beyond 5–10 files.
  • ❌ Mixing source code, tests, and data files in the same folder.
  • ❌ Using spaces or special characters in directory/file names—use snake_case or kebab-case.
  • ❌ Ignoring __init__.py—without it, Python won’t recognize the folder as a package.
  • ❌ Committing virtual environments or IDE files (.vscode/, .idea/) to version control.

Real-World Example: Building a CLI Tool

Project: Weather CLI Tool
weather_cli/
├── README.md
├── requirements.txt
├── pyproject.toml
├── weather/
│   ├── __init__.py
│   ├── main.py          # Entry point: def cli()
│   ├── api.py           # Fetches data from OpenWeather
│   ├── formatter.py     # Formats output
│   └── cache.py         # Caches responses
├── tests/
│   ├── __init__.py
│   ├── test_api.py
│   └── test_formatter.py
└── .env

Entry Point (weather/main.py):

from pathlib import Path
from weather.api import get_weather
from weather.formatter import format_output

def cli(city: str):
    data = get_weather(city)
    output = format_output(data)
    print(output)

# Usage in terminal:
# $ python -m weather.main "Dubai"

Automating Structure Creation

Use tools to scaffold projects consistently:

  • cookiecutter: Template-based project generator.
  • poetry new my_project: Creates a modern Python project with pyproject.toml.
  • django-admin startproject: For Django apps.
Using Poetry to Initialize
# Install Poetry first
pip install poetry

# Create new project
poetry new analytics_tool

# Resulting structure:
analytics_tool/
├── pyproject.toml
├── README.md
├── analytics_tool/
│   ├── __init__.py
│   └── __main__.py
└── tests/
    ├── __init__.py
    └── test_analytics_tool.py

Key Takeaways

  • A clean directory structure is critical for professional Python development.
  • Use pathlib for modern, readable path operations.
  • Separate source code, tests, configuration, and assets logically.
  • Follow community standards—your future self and teammates will thank you.
  • Automate scaffolding with tools like Poetry or Cookiecutter to ensure consistency.
Modules (31) ← Python Programming