context.txt

Directory Structure:
./
    env/
    get_context.egg-info/
    get_context/
    .git/ (contents omitted)
    .contextignore/
    LICENSE/
    requirements.txt/
    MANIFEST.in/
    README.md/
    setup.py/
    .gitignore/
    known_issues.md/
    context.txt/
env/
    bin/ (contents omitted)
    include/ (contents omitted)
    lib/ (contents omitted)
    pyvenv.cfg/
get_context.egg-info/
    PKG-INFO/
    SOURCES.txt/
    entry_points.txt/
    requires.txt/
    top_level.txt/
    dependency_links.txt/
get_context/
    __pycache__/
    directory_structure.py/
    __init__.py/
    cli.py/
    file_utils.py/
    ignore_utils.py/
    merger.py/
    __pycache__/
        ignore_utils.cpython-312.pyc/
        cli.cpython-312.pyc/
        merger.cpython-312.pyc/
        __init__.cpython-312.pyc/
        file_utils.cpython-312.pyc/
        directory_structure.cpython-312.pyc/


==================================================

.contextignore
Text not generated, file is not human-readable

==================================================

LICENSE
Text not generated, file is not human-readable

==================================================

requirements.txt
pathspec>=0.9.0

==================================================

MANIFEST.in
include README.md
include LICENSE

==================================================

README.md
# GET CONTEXT

## About
**Get Context** is a directory context generator designed to provide a structured overview of the contents of a directory and its subdirectories. It generates a single text file that lists the directory structure and the contents of text or program files, while excluding non-human-readable files such as binary or multimedia files. This tool is particularly useful for organizing and reviewing large codebases or data directories. It can also
be used to easily pass the contents of a directory to a large language AI model for further analysis.

## Installation and Usage
This program is available as the Python package "get_context". If Python is installed on your system, you should be able to run:

```
pip install get_context
```
After the package is installed, the program can be run on the command line by typing:

```
get_context <directory> [file_extensions...]
```

The program takes the directory as the first argument and can optionally filter files by specified extensions. For example:

```
get_context . .py .txt
```

This will generate a text file containing the directory structure and the contents of `.py` and `.txt` files within the specified directory.

## Disclaimer
This program is still in development and is currently buggy. Further testing and implementation is required before it can be used as a
fully-fledged, reliable product. **For this reason, it is currently unavailable as a pip package.** 

## Attributions
This program uses both Python’s built-in libraries and external libraries for various functionalities:

* **File and Directory Handling:** The program uses Python’s built-in `os` and `mimetypes` libraries for file and directory manipulation.
* **Text Encoding and Decoding:** The program uses Python's built-in `codecs` and `io` libraries to handle different text encodings.
* **Pattern Matching:** The program uses the external `pathspec` library to handle file exclusion patterns, specifically for interpreting and applying `.gitignore`-style patterns.


## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.


==================================================

setup.py
# setup.py

# Copyright (c) Aiden R. McCormack. All rights reserved.
# Licensed under the MIT License. See LICENSE file in the project root for more information.

# This file keeps track of the metadata for the package.

from setuptools import setup, find_packages

# Read the requirements from requirements.txt
def read_requirements():
    with open('requirements.txt') as req_file:
        return req_file.read().splitlines()

setup(

    # Package information
    name='get_context',  # the name of the package
    version='0.1.0',  # the current version
    description='Generates a single text document for a directory',  # short description
    
    # Read the contents of your README.md file for a full-length description
    long_description=open('README.md').read(),  # file should be opened and read
    long_description_content_type='text/markdown',  # specifies that the long description is markdown

    # URL to the source code or documentation
    url='https://github.com/Aiden2244/get-context',  # source code location

    # Classifiers: help others find the package based on metadata
    classifiers=[
        'Programming Language :: Python :: 3',  # specifies Python 3
        'License :: OSI Approved :: MIT License',  # your chosen license
        'Operating System :: OS Independent',  # works on any OS
    ],

    # Author information
    author='Aiden R. McCormack',  # your name
    author_email='aidenm2244@proton.me',  # your email
    
    # Entry point: where the program starts executing
    entry_points={
        'console_scripts': [
            'get_context=get_context.cli:main',  # links command to main function
        ],
    },

    # Packages: automatically discovers all modules and packages
    packages=find_packages(),  # finds all Python packages recursively in the project

    # Dependencies: external libraries your project needs (empty here)
    install_requires=read_requirements(),

    # Additional metadata (optional, you can add if needed)
    include_package_data=True,  # include non-Python files like README.md
)



==================================================

.gitignore
Text not generated, file is not human-readable

==================================================

known_issues.md
# Known Issues
### Structural
- "merger.py" could afford some refactoring

### File Exclusion
- Commented lines (beggining with a #) are not properly processed in ".contextignore"
- __pycache__ is still being included in "context.txt" file, even when listed in ".contextignore"

==================================================

get_context.egg-info/PKG-INFO
Text not generated, file is not human-readable

==================================================

get_context.egg-info/SOURCES.txt
LICENSE
MANIFEST.in
README.md
setup.py
get_context/__init__.py
get_context/cli.py
get_context/directory_structure.py
get_context/file_utils.py
get_context/ignore_utils.py
get_context/merger.py
get_context.egg-info/PKG-INFO
get_context.egg-info/SOURCES.txt
get_context.egg-info/dependency_links.txt
get_context.egg-info/entry_points.txt
get_context.egg-info/requires.txt
get_context.egg-info/top_level.txt

==================================================

get_context.egg-info/entry_points.txt
[console_scripts]
get_context = get_context.cli:main


==================================================

get_context.egg-info/requires.txt
pathspec>=0.9.0


==================================================

get_context.egg-info/top_level.txt
get_context


==================================================

get_context.egg-info/dependency_links.txt



==================================================

get_context/directory_structure.py
# directory_structure.py

# Copyright (c) Aiden R. McCormack. All rights reserved.
# Licensed under the MIT License. See LICENSE file in the project root for more information.

# this file handles the logic for printing the directory structure at the top of the context file.

import os

def write_directory_structure(directory, outfile, max_items=20, ignore_patterns=None, exclude_dirs=None):
    """Orchestrates writing the directory structure to the output file."""
    for root, dirs, files in os.walk(directory):
        relative_root = os.path.relpath(root, directory)
        level = relative_root.count(os.sep)
        indent = ' ' * 4 * level
        current_dir = os.path.basename(root) or os.path.basename(directory)

        # Determine whether the current directory should be excluded
        if should_exclude_directory(relative_root, current_dir, ignore_patterns, exclude_dirs):
            write_directory_entry(outfile, current_dir, indent, excluded=True)
            dirs[:] = []  # Prevent os.walk from traversing excluded directories
            continue

        # Write the directory entry
        write_directory_entry(outfile, current_dir, indent, excluded=False)

        # Display subdirectories and files
        sub_indent = ' ' * 4 * (level + 1)
        items = dirs + files
        limited_items = limit_display_items(items, max_items)

        for item in limited_items:
            item_path = os.path.join(root, item)
            relative_item_path = os.path.relpath(item_path, directory)

            if os.path.isdir(item_path):
                # Check if the subdirectory should be excluded
                if should_exclude_directory(relative_item_path, item, ignore_patterns, exclude_dirs):
                    write_directory_entry(outfile, item, sub_indent, excluded=True)
                    dirs.remove(item)
                else:
                    write_directory_entry(outfile, item, sub_indent, excluded=False)
            else:
                write_directory_entry(outfile, item, sub_indent, excluded=False)

    outfile.write("\n\n" + "=" * 50 + "\n\n")


def should_exclude_directory(relative_path, directory_name, ignore_patterns, exclude_dirs):
    """Determine whether a directory should be excluded based on ignore patterns or default exclusions."""
    if ignore_patterns and ignore_patterns.match_file(relative_path):
        return True
    if exclude_dirs and directory_name in exclude_dirs:
        return True
    return False


def write_directory_entry(outfile, entry_name, indent, excluded=False):
    """Write a directory or file entry to the output file."""
    if excluded:
        outfile.write(f"{indent}{entry_name}/ (contents omitted)\n")
    else:
        outfile.write(f"{indent}{entry_name}/\n")


def limit_display_items(items, max_items):
    """Limit the number of displayed items in a directory."""
    if len(items) > max_items:
        return items[:max_items] + [f"... (truncated {len(items) - max_items} more items)"]
    return items


==================================================

get_context/__init__.py
# This file is intentionally blank.
# It is just here because Python needs it to recognize this library as a package.

==================================================

get_context/cli.py
# cli.py

# Copyright (c) Aiden R. McCormack. All rights reserved.
# Licensed under the MIT License. See LICENSE file in the project root for more information.

# This file handles the CLI logic for the context generator.

import sys
import os
from .merger import merge_files_in_directory

def print_usage():
    print("Usage:")
    print("  python3 -m get_context <directory> [file_extensions...]")
    print("Description:")
    print("  This program processes all human-readable files in the given directory and its subdirectories.")
    print("  Optionally, you can specify file extensions to filter which files to include.")
    print("Examples:")
    print("  python3 -m get_context .         # Process all human-readable files.")
    print("  python3 -m get_context . .py .txt  # Process only .py and .txt files.")

def main():
    if len(sys.argv) < 2 or not os.path.isdir(sys.argv[1]):
        print("Error: Invalid CLI arguments")
        print_usage()
        sys.exit(1)

    print(f"Generating context file for directory {os.getcwd()}")
    directory = sys.argv[1]
    if len(sys.argv) > 2:
        extensions = sys.argv[2:]
        if all(ext.startswith('.') for ext in extensions):
            print(f"Reading only files with extensions {extensions}")
            merge_files_in_directory(directory, valid_extensions=extensions)
        else:
            print("Error: Invalid CLI arguments")
            print_usage()
            sys.exit(1)
    else:
        merge_files_in_directory(directory)


==================================================

get_context/file_utils.py
# file_utils.py

# Copyright (c) Aiden R. McCormack. All rights reserved.
# Licensed under the MIT License. See LICENSE file in the project root for more information.

# This file handles logic for determining if files are human-readable and have valid extensions.


import mimetypes

def is_human_readable(file_path):
    mime_type, _ = mimetypes.guess_type(file_path)
    if mime_type:
        return mime_type.startswith('text') or 'application/javascript' in mime_type or 'application/xml' in mime_type
    return False

def has_valid_extension(file_name, valid_extensions):
    return any(file_name.endswith(ext) for ext in valid_extensions)


==================================================

get_context/ignore_utils.py
# ignore_utils.py

# Copyright (c) Aiden R. McCormack. All rights reserved.
# Licensed under the MIT License. See LICENSE file in the project root for more information.

# This file handles logic for loading ignore patterns

import os
import pathspec

def load_ignore_patterns(filename='.contextignore'):
    """Load ignore patterns from .contextignore (or .gitignore, if present)."""
    if os.path.exists(filename):
        with open(filename, 'r') as file:
            patterns = file.read().splitlines()
        patterns = [line for line in patterns if line.strip() and not line.strip().startswith('#')]
        return pathspec.PathSpec.from_lines(pathspec.patterns.GitWildMatchPattern, patterns)
    return None


def get_ignored_items(file_path):
    """
    Reads a file and returns a list of lines that begin with a letter or a '.' or '_' character.
    Other lines are ignored.
    """
    if not os.path.exists(file_path):
        return []

    ignored_items = []
    
    with open(file_path, 'r') as file:
        for line in file:
            stripped_line = line.strip()
            # Check if the line starts with a letter or '.' and ignore comments or empty lines
            if stripped_line and (stripped_line[0].isalpha() or stripped_line[0] == '.' or stripped_line[0] == "_"):
                ignored_items.append(stripped_line)
    
    return ignored_items


==================================================

get_context/merger.py
# merger.py

# Copyright (c) Aiden R. McCormack. All rights reserved.
# Licensed under the MIT License. See LICENSE file in the project root for more information.

# This file handles the core functionality of the program.

import os
from .directory_structure import write_directory_structure
from .file_utils import is_human_readable, has_valid_extension
from .ignore_utils import load_ignore_patterns, get_ignored_items


def setup_output_file(output_file):
    """Set up the output file by removing the old file if it exists."""
    if os.path.exists(output_file):
        os.remove(output_file)
    return open(output_file, 'w')


def get_ignore_patterns():
    """Get ignore patterns from .contextignore or .gitignore, if available."""
    if os.path.exists(".contextignore"):
        ignore_dir = ".contextignore"
    elif os.path.exists(".gitignore"):
        ignore_dir = ".gitignore"
    else:
        return None, None

    ignore_patterns = load_ignore_patterns(ignore_dir)
    print(f"Using ignore patterns from {ignore_dir} for brevity:")
    print(get_ignored_items(ignore_dir))
    return ignore_patterns, ignore_dir


def exclude_directories(dirs, root, ignore_patterns, exclude_dirs):
    """Modify dirs in-place to exclude directories based on ignore patterns or default list."""
    if ignore_patterns:
        dirs[:] = [d for d in dirs if not ignore_patterns.match_file(os.path.join(root, d))]
    else:
        dirs[:] = [d for d in dirs if d not in exclude_dirs]


def process_file(file_path, directory, outfile):
    """Process an individual file: check readability, write to the output file."""
    relative_path = os.path.relpath(file_path, directory)

    # Check if the file is human-readable
    if is_human_readable(file_path):
        try:
            with open(file_path, 'r', encoding='utf-8') as infile:
                outfile.write(f"{relative_path}\n")
                outfile.write(infile.read())
                outfile.write("\n\n" + "=" * 50 + "\n\n")
        except (UnicodeDecodeError, IOError):
            outfile.write(f"{relative_path}\n")
            outfile.write("Text not generated, file is not human-readable or could not be decoded\n\n")
            outfile.write("=" * 50 + "\n\n")
    else:
        outfile.write(f"{relative_path}\n")
        outfile.write("Text not generated, file is not human-readable\n\n")
        outfile.write("=" * 50 + "\n\n")


def merge_files_in_directory(directory, valid_extensions=None, output_file='context.txt'):
    """Main function that orchestrates the merging of files in the directory."""
    # Setup: delete the old output file if it exists and open a new one
    outfile = setup_output_file(output_file)

    # Get ignore patterns, or use default exclusions
    ignore_patterns, ignore_dir = get_ignore_patterns()
    default_exclude_dirs = ['env', 'venv', '__pycache__', '.git', 'build', 'dist']
    exclude_dirs = default_exclude_dirs if ignore_patterns is None else []

    # Write the directory structure at the top of the file
    outfile.write("Directory Structure:\n")
    write_directory_structure(directory, outfile, ignore_patterns=ignore_patterns, exclude_dirs=exclude_dirs)

    # Walk through all files and subdirectories
    for root, dirs, files in os.walk(directory):
        exclude_directories(dirs, root, ignore_patterns, exclude_dirs)

        for file in files:
            file_path = os.path.join(root, file)

            # Skip the output file (context.txt) itself
            if file == output_file:
                continue

            # Skip files based on ignore patterns
            if ignore_patterns and ignore_patterns.match_file(file_path):
                continue

            # If valid extensions are provided, only include matching files
            if valid_extensions and not has_valid_extension(file, valid_extensions):
                continue

            # Process the file (read, check human readability, write to output)
            process_file(file_path, directory, outfile)

    # Close the output file after processing
    outfile.close()
    print(f"Generated 'context.txt' at '{os.getcwd()}/context.txt'")


==================================================