API
A utility for managing logging directories.
logdir.LogDir
A utility for managing logging directories.
Creates a logging directory on startup; comes with a handful of other handy methods.
logdir.LogDir.datetime: datetime
property
readonly
Date and time of this class's creation.
logdir.LogDir.logdir: Path
property
readonly
Path to the directory itself.
logdir.LogDir.name: str
property
readonly
The name passed in at initialization time.
logdir.LogDir.__init__(self, name, rootdir='./logs', custom_dir=None, slugify_kwargs=None, uuid=None)
special
Initializes by creating the logging directory.
The directory is created under rootdir
(which is created if it does
not exist). Logging directory is named with the date, followed by the
time, followed by the slugified name
, followed by a UUID (if
uuid
is provided). For example:
2020-02-14_18-01-45_my-logging-dir_16fd2706-8baf-433b-82eb-8c7fada847da
A custom directory may also be passed in via custom_dir
.
name
is slugified with
python-slugify. Pass options
to slugify
with slugify_kwargs
.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name |
str |
Name to associate with this directory. This is used in creating the logging directory name. It is also used in places like the README. |
required |
rootdir |
str or pathlib.Path |
Root directory for all logging
directories, e.g. |
'./logs' |
custom_dir |
str or pathlib.Path |
If passed in, this directory
will be used instead of automatically generating one in
|
None |
slugify_kwargs |
|
kwargs for slugify. |
None |
uuid |
bool or int |
If passed in, generates a
UUID
and appends it to the directory name, i.e., |
None |
Source code in logdir/__init__.py
def __init__(self,
name,
rootdir="./logs",
custom_dir=None,
slugify_kwargs=None,
uuid=None):
"""Initializes by creating the logging directory.
The directory is created under `rootdir` (which is created if it does
not exist). Logging directory is named with the date, followed by the
time, followed by the slugified `name`, followed by a UUID (if
`uuid` is provided). For example:
```
2020-02-14_18-01-45_my-logging-dir_16fd2706-8baf-433b-82eb-8c7fada847da
```
A custom directory may also be passed in via `custom_dir`.
`name` is slugified with
[python-slugify](https://github.com/un33k/python-slugify). Pass options
to `slugify` with `slugify_kwargs`.
Args:
name (str): Name to associate with this directory. This is used in
creating the logging directory name. It is also used in places
like the README.
rootdir (str or pathlib.Path): Root directory for all logging
directories, e.g. `./logs/`
custom_dir (str or pathlib.Path): If passed in, this directory
will be used instead of automatically generating one in
`rootdir`. This directory can be one that already exists. If it
does not exist, it will be created.
slugify_kwargs: kwargs for
[slugify](https://github.com/un33k/python-slugify#options).
uuid (bool or int): If passed in, generates a
[UUID](https://en.wikipedia.org/wiki/Universally_unique_identifier)
and appends it to the directory name, i.e., `..._[UUID]`. This
can be useful if you are generating multiple logging directories
at the same time -- by default, directories will have name
conflicts if they are generated during the same second in time.
When this parameter is an int, it indicates the length of the
UUID (e.g. `uuid=8` indicates a UUID of length 8). A "falsy"
value (e.g., 0, False, None) indicates no UUID should be
generated. Passing in True will indicate a UUID of default
length 16. UUID generation is done randomly with
[shortuuid](https://github.com/skorokithakis/shortuuid/).
"""
self._name = name
self._datetime = datetime.datetime.now()
if custom_dir is None:
if uuid:
if isinstance(uuid, bool):
uuid_len = self.DEFAULT_UUID_LEN
elif isinstance(uuid, int):
uuid_len = uuid
else:
raise ValueError(
f"Expected `uuid` to be a bool or int but got {uuid}")
uuid_str = f"_{shortuuid.ShortUUID().random(length=uuid_len)}"
else:
# If the uuid is 0, we go here because `if uuid` is False.
uuid_str = ""
# Automatically generate directory in `rootdir`.
rootdir = Path(rootdir)
slugify_kwargs = {} if slugify_kwargs is None else slugify_kwargs
name_slug = slugify.slugify(name, **slugify_kwargs)
dirname = (self._datetime.strftime("%Y-%m-%d_%H-%M-%S") + "_" +
name_slug + uuid_str)
self._logdir = rootdir / Path(dirname)
else:
# Use custom directory.
self._logdir = Path(custom_dir)
if not self._logdir.exists():
# Creates intermediate directories as well if needed.
self._logdir.mkdir(parents=True)
logdir.LogDir.copy(self, src, dest)
Copies a file into the logging directory.
Examples:
The following copies foobar.txt
in the current directory to
new/foobar2.txt
within the logging directory.
logdir = LogDir("logdir")
logdir.copy("foobar.txt", "new/foobar2.txt")
Parameters:
Name | Type | Description | Default |
---|---|---|---|
src |
str or pathlib.Path |
The source file. It is evaluated relative to the current working directory of the program (it could also be absolute). |
required |
dest |
str or pathlib.Path |
Destination location within the logging directory. Intermediate directories are created. |
required |
Source code in logdir/__init__.py
def copy(self, src, dest):
"""Copies a file into the logging directory.
Example:
The following copies `foobar.txt` in the current directory to
`new/foobar2.txt` _within the logging directory_.
```python
logdir = LogDir("logdir")
logdir.copy("foobar.txt", "new/foobar2.txt")
```
Args:
src (str or pathlib.Path): The source file. It is evaluated relative
to the current working directory of the program (it could also
be absolute).
dest (str or pathlib.Path): Destination location _within the
logging directory_. Intermediate directories are created.
"""
shutil.copy(str(src), self.file(dest))
logdir.LogDir.dir(self, dirname, touch=False, touch_inter=True)
Returns a string path to the given directory.
By default, intermediate directories are created if they do not exist.
However the directory itself is not created. This can be changed by
passing in touch=True
.
Examples:
logdir = LogDir("logdir")
logdir.filepath("mydir") # "..._logdir/mydir/"
Parameters:
Name | Type | Description | Default |
---|---|---|---|
dirname |
str or pathlib.Path |
The name of the directory. |
required |
touch |
bool |
Whether to automatically create the directory. |
False |
touch_inter |
bool |
Whether to automatically create intermediate
directories. If this option is set to False, |
True |
Returns:
Type | Description |
---|---|
str |
Path to the new directory in the logging directory. |
Source code in logdir/__init__.py
def dir(self, dirname, touch=False, touch_inter=True):
"""Returns a string path to the given directory.
By default, intermediate directories are created if they do not exist.
However the directory itself is not created. This can be changed by
passing in `touch=True`.
Example:
```python
logdir = LogDir("logdir")
logdir.filepath("mydir") # "..._logdir/mydir/"
```
Args:
dirname (str or pathlib.Path): The name of the directory.
touch (bool): Whether to automatically create the directory.
touch_inter (bool): Whether to automatically create intermediate
directories. If this option is set to False, `touch` is also set
to False.
Returns:
str: Path to the new directory in the logging directory.
"""
return str(self.pdir(dirname, touch, touch_inter))
logdir.LogDir.file(self, filename, touch=False, touch_inter=True)
Returns a string path to the given file.
By default, intermediate directories are created if they do not exist.
However, the file itself it not created (this helps avoid confusing
situations where one is checking for the existence of a file). This can
be changed by passing in touch=True
.
Examples:
Basic usage:
logdir = LogDir("logdir")
logdir.filepath("file.txt") # "..._logdir/file.txt"
logdir = LogDir("logdir")
logdir.filepath("newdir/file.txt") # "..._logdir/newdir/file.txt"
Parameters:
Name | Type | Description | Default |
---|---|---|---|
filename |
str or pathlib.Path |
The name of the file. |
required |
touch |
bool |
Whether to automatically touch the file so that an empty file exists even if it is not opened. |
False |
touch_inter |
bool |
Whether to automatically create intermediate
directories. If this option is set to False, |
True |
Returns:
Type | Description |
---|---|
str |
Path to the new file in the logging directory. |
Source code in logdir/__init__.py
def file(self, filename, touch=False, touch_inter=True):
"""Returns a string path to the given file.
By default, intermediate directories are created if they do not exist.
However, the file itself it not created (this helps avoid confusing
situations where one is checking for the existence of a file). This can
be changed by passing in `touch=True`.
Example:
Basic usage:
```python
logdir = LogDir("logdir")
logdir.filepath("file.txt") # "..._logdir/file.txt"
```
Creating intermediate directories:
```python
logdir = LogDir("logdir")
logdir.filepath("newdir/file.txt") # "..._logdir/newdir/file.txt"
```
Args:
filename (str or pathlib.Path): The name of the file.
touch (bool): Whether to automatically touch the file so that an
empty file exists even if it is not opened.
touch_inter (bool): Whether to automatically create intermediate
directories. If this option is set to False, `touch` is also set
to False.
Returns:
str: Path to the new file in the logging directory.
"""
return str(self.pfile(filename, touch, touch_inter))
logdir.LogDir.pdir(self, dirname, touch=False, touch_inter=True)
Same as dir, but returns pathlib.Path.
See dir for args.
Returns:
Type | Description |
---|---|
pathlib.Path |
Path to the new directory in the logging directory. |
Source code in logdir/__init__.py
def pdir(self, dirname, touch=False, touch_inter=True):
"""Same as dir, but returns pathlib.Path.
See dir for args.
Returns:
pathlib.Path: Path to the new directory in the logging directory.
"""
touch = False if not touch_inter else touch
dirname = self._logdir / Path(dirname)
if touch_inter and not dirname.parent.exists():
dirname.parent.mkdir(parents=True)
if touch and not dirname.exists():
dirname.mkdir()
return dirname
logdir.LogDir.pfile(self, filename, touch=False, touch_inter=True)
Same as file, but returns pathlib.Path.
See file for args.
Returns:
Type | Description |
---|---|
pathlib.Path |
Path to the new file in the logging directory. |
Source code in logdir/__init__.py
def pfile(self, filename, touch=False, touch_inter=True):
"""Same as file, but returns pathlib.Path.
See file for args.
Returns:
pathlib.Path: Path to the new file in the logging directory.
"""
touch = False if not touch_inter else touch
filename = self._logdir / Path(filename)
if touch_inter and not filename.parent.exists():
filename.parent.mkdir(parents=True)
if touch and not filename.exists():
filename.touch()
return filename
logdir.LogDir.readme(self, date=True, git_commit=False, git_path='.', info=())
Adds a README.md with useful info.
The README consists of the name of the directory (passed in at init time), followed by a bulleted list with various pieces of info.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
date |
bool |
Add the date and time in the bulleted list of info. |
True |
git_commit |
bool |
Add the current git commit hash in the bulleted list of info. |
False |
git_path |
str or pathlib.Path |
The path to the git repo (i.e. a
directory that contains |
'.' |
info |
list of str |
A list of additional bullets to add in the README. |
() |
Returns:
Type | Description |
---|---|
pathlib.path |
Full path to the README. |
Source code in logdir/__init__.py
def readme(self, date=True, git_commit=False, git_path=".", info=()):
"""Adds a README.md with useful info.
The README consists of the name of the directory (passed in at init
time), followed by a bulleted list with various pieces of info.
Args:
date (bool): Add the date and time in the bulleted list of info.
git_commit (bool): Add the current git commit hash in the bulleted
list of info.
git_path (str or pathlib.Path): The path to the git repo (i.e. a
directory that contains `.git`). Only applicable if `git_commit`
is True. If the path given is not to a Git repo, a warning is
issued, and the Git Commit is replaced with `"(no repo found)"`
info (list of str): A list of additional bullets to add in the
README.
Returns:
pathlib.path: Full path to the README.
"""
readme_path = self.pfile("README.md")
with readme_path.open("w") as file:
lines = [f"# {self._name}", ""]
if date:
date_str = self._datetime.strftime("%Y-%m-%d %H:%M:%S")
lines.append(f"- Date: {date_str}")
if git_commit:
git_path = Path(git_path)
if (git_path / ".git").exists():
repo = Repo(str(git_path))
commit_hash = repo.head().decode("utf-8")
else:
warnings.warn("No Git repo found")
commit_hash = "(no repo found)"
lines.append(f"- Git Commit: {commit_hash}")
lines.extend(map(lambda s: f"- {s}", info))
file.write("\n".join(lines) + "\n")
return readme_path
logdir.LogDir.save_data(self, data, filename)
Saves data to filename
in the log directory.
This is particularly useful when saving configuration options or other pieces of data that are dict's or lists.
Supported file types are:
- JSON (
*.json
) - YAML (
*.yml
,*.yaml
) - TOML (
*.toml
) - Pickle (
*.pkl
,*.pickle
)
Parameters:
Name | Type | Description | Default |
---|---|---|---|
data |
object |
Data to save. Typically a dict or list for JSON, YAML, and TOML, and any pickle-able object for pickle. |
required |
filename |
str or pathlib.Path |
The name of the file; we will create it under the logdir using pfile. If the filetype is unsupported, we will default to pickle and raise a warning. |
required |
Returns:
Type | Description |
---|---|
pathlib.path |
Full path to the config file. |
Source code in logdir/__init__.py
def save_data(self, data, filename):
"""Saves data to `filename` in the log directory.
This is particularly useful when saving configuration options or other
pieces of data that are dict's or lists.
Supported file types are:
- JSON (`*.json`)
- YAML (`*.yml`, `*.yaml`)
- TOML (`*.toml`)
- Pickle (`*.pkl`, `*.pickle`)
Args:
data (object): Data to save. Typically a dict or list for JSON,
YAML, and TOML, and any pickle-able object for pickle.
filename (str or pathlib.Path): The name of the file; we will create
it under the logdir using pfile. If the
filetype is unsupported, we will default to pickle and raise a
warning.
Returns:
pathlib.path: Full path to the config file.
"""
filepath = self.pfile(filename)
ext = filepath.suffix[1:]
if ext == "json":
with filepath.open("w") as file:
json.dump(data, file)
elif ext in ("yml", "yaml"):
with filepath.open("w") as file:
yaml_worker = yaml.YAML(typ="unsafe")
yaml_worker.dump(data, file)
elif ext == "toml":
with filepath.open("w") as file:
toml.dump(data, file)
else:
# Pickle is default if file extension cannot be identified.
with filepath.open("wb") as file:
pickle.dump(data, file)
if ext not in ("pkl", "pickle"):
warnings.warn(f"Filetype {ext} not found. Used pickle instead.")
return filepath