How Checks are Run

In Flake8 2.x, Flake8 delegated check running to pep8. In 3.0 Flake8 takes on that responsibility. This has allowed for simpler handling of the --jobs parameter (using multiprocessing) and simplified our fallback if something goes awry with concurrency. At the lowest level we have a FileChecker. Instances of FileChecker are created for each file to be analyzed by Flake8. Each instance, has a copy of all of the plugins registered with setuptools in the flake8.extension entry-point group.

The FileChecker instances are managed by an instance of Manager. The Manager instance handles creating sub-processes with multiprocessing module and falling back to running checks in serial if an operating system level error arises. When creating FileChecker instances, the Manager is responsible for determining if a particular file has been excluded.

Processing Files

Unfortunately, since Flake8 took over check running from pep8/pycodestyle, it also had to take over parsing and processing files for the checkers to use. Since it couldn’t reuse pycodestyle’s functionality (since it did not separate cleanly the processing from check running) that function was isolated into the FileProcessor class. We moved several helper functions into the flake8.processor module (see also Processor Utility Functions).

API Reference

class flake8.checker.FileChecker(*, filename, plugins, options)[source]

Manage running checks for a file and aggregate the results.

Parameters:
check_physical_eol(token, prev_physical)[source]

Run physical checks if and only if it is at the end of the line.

Parameters:
  • token (TokenInfo) –

  • prev_physical (str) –

Return type:

None

handle_newline(token_type)[source]

Handle the logic when encountering a newline token.

Parameters:

token_type (int) –

Return type:

None

process_tokens()[source]

Process tokens and trigger checks.

Instead of using this directly, you should use flake8.checker.FileChecker.run_checks().

Return type:

None

report(error_code, line_number, column, text)[source]

Report an error by storing it in the results list.

Parameters:
  • error_code (str | None) –

  • line_number (int) –

  • column (int) –

  • text (str) –

Return type:

str

run_ast_checks()[source]

Run all checks expecting an abstract syntax tree.

Return type:

None

run_check(plugin, **arguments)[source]

Run the check in a single plugin.

Parameters:
  • plugin (LoadedPlugin) –

  • arguments (Any) –

Return type:

Any

run_checks()[source]

Run checks against the file.

Return type:

tuple[str, List[Tuple[str, int, int, str, str | None]], dict[str, int]]

run_logical_checks()[source]

Run all checks expecting a logical line.

Return type:

None

run_physical_checks(physical_line)[source]

Run all checks for a given physical line.

A single physical check may return multiple errors.

Parameters:

physical_line (str) –

Return type:

None

class flake8.checker.Manager(style_guide, plugins, argv)[source]

Manage the parallelism and checker instances for each plugin and file.

This class will be responsible for the following:

  • Determining the parallelism of Flake8, e.g.:

    • Do we use multiprocessing or is it unavailable?

    • Do we automatically decide on the number of jobs to use or did the user provide that?

  • Falling back to a serial way of processing files if we run into an OSError related to multiprocessing

  • Organizing the results of each checker so we can group the output together and make our output deterministic.

Parameters:
  • style_guide (StyleGuideManager) –

  • plugins (Checkers) –

  • argv (Sequence[str]) –

report()[source]

Report all of the errors found in the managed file checkers.

This iterates over each of the checkers and reports the errors sorted by line number.

Returns:

A tuple of the total results found and the results reported.

Return type:

tuple[int, int]

run()[source]

Run all the checkers.

This will intelligently decide whether to run the checks in parallel or whether to run them in serial.

If running the checks in parallel causes a problem (e.g., #117) this also implements fallback to serial processing.

Return type:

None

run_parallel()[source]

Run the checkers in parallel.

Return type:

None

run_serial()[source]

Run the checkers in serial.

Return type:

None

start()[source]

Start checking files.

Parameters:

paths – Path names to check. This is passed directly to make_checkers().

Return type:

None

stop()[source]

Stop checking files.

Return type:

None

class flake8.processor.FileProcessor(filename, options, lines=None)[source]

Processes a file and holds state.

This processes a file by generating tokens, logical and physical lines, and AST trees. This also provides a way of passing state about the file to checks expecting that state. Any public attribute on this object can be requested by a plugin. The known public attributes are:

Parameters:
blank_before

Number of preceding blank lines

blank_lines

Number of blank lines

build_ast()[source]

Build an abstract syntax tree from the list of lines.

Return type:

AST

build_logical_line()[source]

Build a logical line from the current tokens list.

Return type:

tuple[str, str, List[Tuple[int, Tuple[int, int]]]]

build_logical_line_tokens()[source]

Build the mapping, comments, and logical line lists.

Return type:

Tuple[List[str], List[str], List[Tuple[int, Tuple[int, int]]]]

checker_state: dict[Any, Any]

Current checker state

delete_first_token()[source]

Delete the first token in the list of tokens.

Return type:

None

property file_tokens: list[TokenInfo]

Return the complete set of tokens for a file.

fstring_start(lineno)[source]

Signal the beginning of an fstring.

Parameters:

lineno (int) –

Return type:

None

generate_tokens()[source]

Tokenize the file and yield the tokens.

Return type:

Generator[TokenInfo, None, None]

hang_closing

User provided option for hang closing

indent_char: str | None

Character used for indentation

indent_level

Current level of indentation

indent_size

Number of spaces used for indentation

keyword_arguments_for(parameters, arguments)[source]

Generate the keyword arguments for a list of parameters.

Parameters:
Return type:

dict[str, Any]

line_number

Line number in the file

logical_line

Current logical line

max_doc_length

Maximum docstring / comment line length as configured by the user

max_line_length

Maximum line length as configured by the user

multiline

Whether the current physical line is multiline

multiline_string(token)[source]

Iterate through the lines of a multiline string.

Parameters:

token (TokenInfo) –

Return type:

Generator[str, None, None]

next_line()[source]

Get the next line from the list.

Return type:

str

next_logical_line()[source]

Record the previous logical line.

This also resets the tokens list and the blank_lines count.

Return type:

None

noqa = False

always False, included for compatibility

noqa_line_for(line_number)[source]

Retrieve the line which will be used to determine noqa.

Parameters:

line_number (int) –

Return type:

str | None

previous_indent_level

Previous level of indentation

previous_logical

Previous logical line

previous_unindented_logical_line

Previous unindented (i.e. top-level) logical line

read_lines()[source]

Read the lines for this file checker.

Return type:

list[str]

read_lines_from_filename()[source]

Read the lines for a file.

Return type:

list[str]

read_lines_from_stdin()[source]

Read the lines from standard in.

Return type:

list[str]

reset_blank_before()[source]

Reset the blank_before attribute to zero.

Return type:

None

should_ignore_file()[source]

Check if flake8: noqa is in the file to be ignored.

Returns:

True if a line matches defaults.NOQA_FILE, otherwise False

Return type:

bool

statistics

Statistics dictionary

strip_utf_bom()[source]

Strip the UTF bom from the lines of the file.

Return type:

None

tokens: list[TokenInfo]

Current set of tokens

total_lines

Total number of lines in the file

update_checker_state_for(plugin)[source]

Update the checker_state attribute for the plugin.

Parameters:

plugin (LoadedPlugin) –

Return type:

None

update_state(mapping)[source]

Update the indent level based on the logical line mapping.

Parameters:

mapping (List[Tuple[int, Tuple[int, int]]]) –

Return type:

None

verbose

Verbosity level of Flake8

visited_new_blank_line()[source]

Note that we visited a new blank line.

Return type:

None

Utility Functions

flake8.processor.count_parentheses(current_parentheses_count, token_text)[source]

Count the number of parentheses.

Parameters:
  • current_parentheses_count (int) –

  • token_text (str) –

Return type:

int

flake8.processor.expand_indent(line)[source]

Return the amount of indentation.

Tabs are expanded to the next multiple of 8.

>>> expand_indent('    ')
4
>>> expand_indent('\t')
8
>>> expand_indent('       \t')
8
>>> expand_indent('        \t')
16
Parameters:

line (str) –

Return type:

int

flake8.processor.is_eol_token(token)[source]

Check if the token is an end-of-line token.

Parameters:

token (TokenInfo) –

Return type:

bool

flake8.processor.is_multiline_string(token)[source]

Check if this is a multiline string.

Parameters:

token (TokenInfo) –

Return type:

bool

flake8.processor.mutate_string(text)[source]

Replace contents with ‘xxx’ to prevent syntax matching.

>>> mutate_string('"abc"')
'"xxx"'
>>> mutate_string("'''abc'''")
"'''xxx'''"
>>> mutate_string("r'abc'")
"r'xxx'"
Parameters:

text (str) –

Return type:

str

flake8.processor.token_is_newline(token)[source]

Check if the token type is a newline token type.

Parameters:

token (TokenInfo) –

Return type:

bool