Network Status Documents¶
Parsing for Tor network status documents. This supports both the v2 and v3 dir-spec. Documents can be obtained from a few sources…
- The ‘cached-consensus’ file in Tor’s data directory.
- Archived descriptors provided by CollecTor.
- Directory authorities and mirrors via their DirPort.
… and contain the following sections…
- document header
- list of
stem.descriptor.networkstatus.DirectoryAuthority
- list of
stem.descriptor.router_status_entry.RouterStatusEntry
- document footer
For a great graphical overview see Jordan Wright’s chart describing the anatomy of the consensus.
Of these, the router status entry section can be quite large (on the order of
hundreds of kilobytes). As such we provide a couple of methods for reading
network status documents through parse_file()
.
For more information see DocumentHandler()
…
from stem.descriptor import parse_file, DocumentHandler
with open('.tor/cached-consensus', 'rb') as consensus_file:
# Processes the routers as we read them in. The routers refer to a document
# with an unset 'routers' attribute.
for router in parse_file(consensus_file, 'network-status-consensus-3 1.0', document_handler = DocumentHandler.ENTRIES):
print router.nickname
Module Overview:
NetworkStatusDocument - Network status document
|- NetworkStatusDocumentV2 - Version 2 network status document
|- NetworkStatusDocumentV3 - Version 3 network status document
+- BridgeNetworkStatusDocument - Version 3 network status document for bridges
KeyCertificate - Certificate used to authenticate an authority
DocumentSignature - Signature of a document by a directory authority
DirectoryAuthority - Directory authority as defined in a v3 network status document
-
class
stem.descriptor.networkstatus.
PackageVersion
[source]¶ Bases:
stem.descriptor.networkstatus.PackageVersion
Latest recommended version of a package that’s available.
Variables: - name (str) – name of the package
- version (str) – latest recommended version
- url (str) – package’s url
- digests (dict) – mapping of digest types to their value
Bases:
stem.descriptor.networkstatus.SharedRandomnessCommitment
Directory authority’s commitment for generating the next shared random value.
Variables: - version (int) – shared randomness protocol version
- algorithm (str) – hash algorithm used to make the commitment
- identity (str) – authority’s sha1 identity fingerprint
- commit (str) – base64 encoded commitment hash to the shared random value
- reveal (str) – base64 encoded commitment to the shared random value, None of not provided
-
class
stem.descriptor.networkstatus.
NetworkStatusDocument
(contents, lazy_load=False)[source]¶ Bases:
stem.descriptor.Descriptor
Common parent for network status documents.
-
class
stem.descriptor.networkstatus.
NetworkStatusDocumentV2
(raw_content, validate=False)[source]¶ Bases:
stem.descriptor.networkstatus.NetworkStatusDocument
Version 2 network status document. These have been deprecated and are no longer generated by Tor.
Variables: - routers (dict) – fingerprints to
RouterStatusEntryV2
contained in the document - version (int) – * document version
- hostname (str) – * hostname of the authority
- address (str) – * authority’s IP address
- dir_port (int) – * authority’s DirPort
- fingerprint (str) – * authority’s fingerprint
- contact (str) – * authority’s contact information
- signing_key (str) – * authority’s public signing key
- client_versions (list) – list of recommended client tor version strings
- server_versions (list) – list of recommended server tor version strings
- published (datetime) – * time when the document was published
- options (list) – * list of things that this authority decides
- signing_authority (str) – * name of the authority signing the document
- signature (str) – * authority’s signature for the document
* attribute is either required when we’re parsed with validation or has a default value, others are left as None if undefined
-
classmethod
content
(attr=None, exclude=(), sign=False)[source]¶ Creates descriptor content with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
Parameters: - attr (dict) – keyword/value mappings to be included in the descriptor
- exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
- sign (bool) – includes cryptographic signatures and digests if True
Returns: str with the content of a descriptor
Raises: - ImportError if cryptography is unavailable and sign is True
- NotImplementedError if not implemented for this descriptor type
- routers (dict) – fingerprints to
-
class
stem.descriptor.networkstatus.
NetworkStatusDocumentV3
(raw_content, validate=False, default_params=True)[source]¶ Bases:
stem.descriptor.networkstatus.NetworkStatusDocument
Version 3 network status document. This could be either a vote or consensus.
Variables: - routers (dict) – fingerprint to
RouterStatusEntryV3
mapping for relays contained in the document - version (int) – * document version
- version_flavor (str) – * flavor associated with the document (such as ‘microdesc’)
- is_consensus (bool) – * True if the document is a consensus
- is_vote (bool) – * True if the document is a vote
- is_microdescriptor (bool) – * True if this is a microdescriptor flavored document, False otherwise
- valid_after (datetime) – * time when the consensus became valid
- fresh_until (datetime) – * time when the next consensus should be produced
- valid_until (datetime) – * time when this consensus becomes obsolete
- vote_delay (int) – * number of seconds allowed for collecting votes from all authorities
- dist_delay (int) – * number of seconds allowed for collecting signatures from all authorities
- client_versions (list) – list of recommended client tor versions
- server_versions (list) – list of recommended server tor versions
- packages (list) – * list of
PackageVersion
entries - known_flags (list) – * list of
Flag
for the router’s flags - params (dict) – * dict of parameter(str) => value(int) mappings
- directory_authorities (list) – * list of
DirectoryAuthority
objects that have generated this document - signatures (list) – *
DocumentSignature
of the authorities that have signed the document
Consensus Attributes:
Variables: - consensus_method (int) – method version used to generate this consensus
- bandwidth_weights (dict) – dict of weight(str) => value(int) mappings
- shared_randomness_current_reveal_count (int) – number of commitments used to generate the current shared random value
- shared_randomness_current_value (str) – base64 encoded current shared random value
- shared_randomness_previous_reveal_count (int) – number of commitments used to generate the last shared random value
- shared_randomness_previous_value (str) – base64 encoded last shared random value
Vote Attributes:
Variables: - consensus_methods (list) – list of ints for the supported method versions
- published (datetime) – time when the document was published
- flag_thresholds (dict) – * mapping of internal performance thresholds used while making the vote, values are ints or floats
- recommended_client_protocols (dict) – recommended protocols for clients
- recommended_relay_protocols (dict) – recommended protocols for relays
- required_client_protocols (dict) – required protocols for clients
- required_relay_protocols (dict) – required protocols for relays
- bandwidth_file_headers (dict) – headers from the bandwidth authority that generated this vote
* attribute is either required when we’re parsed with validation or has a default value, others are left as None if undefined
Changed in version 1.4.0: Added the packages attribute.
Changed in version 1.5.0: Added the is_shared_randomness_participate, shared_randomness_commitments, shared_randomness_previous_reveal_count, shared_randomness_previous_value, shared_randomness_current_reveal_count, and shared_randomness_current_value attributes.
Changed in version 1.6.0: Added the recommended_client_protocols, recommended_relay_protocols, required_client_protocols, and required_relay_protocols attributes.
Changed in version 1.6.0: The is_shared_randomness_participate and shared_randomness_commitments were misdocumented in the tor spec and as such never set. They’re now an attribute of votes in the directory_authorities.
Changed in version 1.7.0: The shared_randomness_current_reveal_count and shared_randomness_previous_reveal_count attributes were undocumented and not provided properly if retrieved before their shred_randomness_*_value counterpart.
Changed in version 1.7.0: Added the bandwidth_file_headers attributbute.
-
HEADER_PARSER_FOR_LINE
= {'bandwidth-file-headers': <function _parse_bandwidth_file_headers at 0xf5d4d1dc>, 'client-versions': <function _parse_versions_line.<locals>._parse at 0xf5d4d2fc>, 'consensus-method': <function _parse_header_consensus_method_line at 0xf5e6fe84>, 'consensus-methods': <function _parse_header_consensus_methods_line at 0xf5e6fe3c>, 'flag-thresholds': <function _parse_header_flag_thresholds_line at 0xf5e6ff5c>, 'fresh-until': <function _parse_timestamp_line.<locals>._parse at 0xf5d4d26c>, 'known-flags': <function _parse_simple_line.<locals>._parse at 0xf5d4d3d4>, 'network-status-version': <function _parse_header_network_status_version_line at 0xf5e6fcd4>, 'package': <function _parse_package_line at 0xf5d4d0bc>, 'params': <function _parse_header_parameters_line at 0xf5e6ffa4>, 'published': <function _parse_timestamp_line.<locals>._parse at 0xf5e6fbb4>, 'recommended-client-protocols': <function _parse_protocol_line.<locals>._parse at 0xf5d4d4f4>, 'recommended-relay-protocols': <function _parse_protocol_line.<locals>._parse at 0xf5d4d53c>, 'required-client-protocols': <function _parse_protocol_line.<locals>._parse at 0xf5d4d584>, 'required-relay-protocols': <function _parse_protocol_line.<locals>._parse at 0xf5d4d5cc>, 'server-versions': <function _parse_versions_line.<locals>._parse at 0xf5d4d344>, 'shared-rand-current-value': <function _parse_shared_rand_current_value at 0xf5d4d194>, 'shared-rand-previous-value': <function _parse_shared_rand_previous_value at 0xf5d4d14c>, 'valid-after': <function _parse_timestamp_line.<locals>._parse at 0xf5d4d224>, 'valid-until': <function _parse_timestamp_line.<locals>._parse at 0xf5d4d2b4>, 'vote-status': <function _parse_header_vote_status_line at 0xf5e6fdf4>, 'voting-delay': <function _parse_header_voting_delay_line at 0xf5e6fecc>}¶
-
FOOTER_PARSER_FOR_LINE
= {'bandwidth-weights': <function _parse_simple_line.<locals>._parse at 0xf5d4d464>, 'directory-footer': <function _parse_directory_footer_line at 0xf5d4d02c>, 'directory-signature': <function _parse_footer_directory_signature_line at 0xf5d4d074>}¶
-
classmethod
content
(attr=None, exclude=(), sign=False, authorities=None, routers=None)[source]¶ Creates descriptor content with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
Parameters: - attr (dict) – keyword/value mappings to be included in the descriptor
- exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
- sign (bool) – includes cryptographic signatures and digests if True
Returns: str with the content of a descriptor
Raises: - ImportError if cryptography is unavailable and sign is True
- NotImplementedError if not implemented for this descriptor type
-
classmethod
create
(attr=None, exclude=(), validate=True, sign=False, authorities=None, routers=None)[source]¶ Creates a descriptor with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
Parameters: - attr (dict) – keyword/value mappings to be included in the descriptor
- exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
- validate (bool) – checks the validity of the descriptor’s content if True, skips these checks otherwise
- sign (bool) – includes cryptographic signatures and digests if True
Returns: Descriptor
subclassRaises: - ValueError if the contents is malformed and validate is True
- ImportError if cryptography is unavailable and sign is True
- NotImplementedError if not implemented for this descriptor type
-
validate_signatures
(key_certs)[source]¶ Validates we’re properly signed by the signing certificates.
New in version 1.6.0.
Parameters: key_certs (list) – KeyCertificates
to validate the consensus againstRaises: ValueError if an insufficient number of valid signatures are present.
-
get_unrecognized_lines
()[source]¶ Provides a list of lines that were either ignored or had data that we did not know how to process. This is most common due to new descriptor fields that this library does not yet know how to process. Patches welcome!
Returns: list of lines of unrecognized content
-
meets_consensus_method
(method)[source]¶ Checks if we meet the given consensus-method. This works for both votes and consensuses, checking our ‘consensus-method’ and ‘consensus-methods’ entries.
Parameters: method (int) – consensus-method to check for Returns: True if we meet the given consensus-method, and False otherwise
- routers (dict) – fingerprint to
-
class
stem.descriptor.networkstatus.
DirectoryAuthority
(raw_content, validate=False, is_vote=False)[source]¶ Bases:
stem.descriptor.Descriptor
Directory authority information obtained from a v3 network status document.
Authorities can optionally use a legacy format. These are no longer found in practice, but have the following differences…
- The authority’s nickname ends with ‘-legacy’.
- There’s no contact or vote_digest attribute.
Variables: - nickname (str) – * authority’s nickname
- v3ident (str) – * identity key fingerprint used to sign votes and consensus
- hostname (str) – * hostname of the authority
- address (str) – * authority’s IP address
- dir_port (int) – * authority’s DirPort
- or_port (int) – * authority’s ORPort
- is_legacy (bool) – * if the authority’s using the legacy format
- contact (str) – contact information, this is included if is_legacy is False
Consensus Attributes:
Variables: vote_digest (str) – digest of the authority that contributed to the consensus, this is included if is_legacy is False Vote Attributes:
Variables: - legacy_dir_key (str) – fingerprint of and obsolete identity key
- key_certificate (stem.descriptor.networkstatus.KeyCertificate) – * authority’s key certificate
- is_shared_randomness_participate (bool) – * True if this authority participates in establishing a shared random value, False otherwise
- shared_randomness_commitments (list) – * list of
SharedRandomnessCommitment
entries - shared_randomness_previous_reveal_count (int) – number of commitments used to generate the last shared random value
- shared_randomness_previous_value (str) – base64 encoded last shared random value
- shared_randomness_current_reveal_count (int) – number of commitments used to generate the current shared random value
- shared_randomness_current_value (str) – base64 encoded current shared random value
* mandatory attribute
Changed in version 1.4.0: Renamed our ‘fingerprint’ attribute to ‘v3ident’ (prior attribute exists for backward compatability, but is deprecated).
Changed in version 1.6.0: Added the is_shared_randomness_participate, shared_randomness_commitments, shared_randomness_previous_reveal_count, shared_randomness_previous_value, shared_randomness_current_reveal_count, and shared_randomness_current_value attributes.
-
classmethod
content
(attr=None, exclude=(), sign=False, is_vote=False)[source]¶ Creates descriptor content with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
Parameters: - attr (dict) – keyword/value mappings to be included in the descriptor
- exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
- sign (bool) – includes cryptographic signatures and digests if True
Returns: str with the content of a descriptor
Raises: - ImportError if cryptography is unavailable and sign is True
- NotImplementedError if not implemented for this descriptor type
-
classmethod
create
(attr=None, exclude=(), validate=True, sign=False, is_vote=False)[source]¶ Creates a descriptor with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
Parameters: - attr (dict) – keyword/value mappings to be included in the descriptor
- exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
- validate (bool) – checks the validity of the descriptor’s content if True, skips these checks otherwise
- sign (bool) – includes cryptographic signatures and digests if True
Returns: Descriptor
subclassRaises: - ValueError if the contents is malformed and validate is True
- ImportError if cryptography is unavailable and sign is True
- NotImplementedError if not implemented for this descriptor type
-
class
stem.descriptor.networkstatus.
KeyCertificate
(raw_content, validate=False)[source]¶ Bases:
stem.descriptor.Descriptor
Directory key certificate for a v3 network status document.
Variables: - version (int) – * version of the key certificate
- address (str) – authority’s IP address
- dir_port (int) – authority’s DirPort
- fingerprint (str) – * authority’s fingerprint
- identity_key (str) – * long term authority identity key
- published (datetime) – * time when this key was generated
- expires (datetime) – * time after which this key becomes invalid
- signing_key (str) – * directory server’s public signing key
- crosscert (str) – signature made using certificate’s signing key
- certification (str) – * signature of this key certificate signed with the identity key
* mandatory attribute
-
classmethod
content
(attr=None, exclude=(), sign=False)[source]¶ Creates descriptor content with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
Parameters: - attr (dict) – keyword/value mappings to be included in the descriptor
- exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
- sign (bool) – includes cryptographic signatures and digests if True
Returns: str with the content of a descriptor
Raises: - ImportError if cryptography is unavailable and sign is True
- NotImplementedError if not implemented for this descriptor type
-
class
stem.descriptor.networkstatus.
DocumentSignature
(method, identity, key_digest, signature, validate=False)[source]¶ Bases:
object
Directory signature of a v3 network status document.
Variables: - method (str) – algorithm used to make the signature
- identity (str) – fingerprint of the authority that made the signature
- key_digest (str) – digest of the signing key
- signature (str) – document signature
Parameters: validate (bool) – checks validity if True
Raises: ValueError if a validity check fails
-
class
stem.descriptor.networkstatus.
BridgeNetworkStatusDocument
(raw_content, validate=False)[source]¶ Bases:
stem.descriptor.networkstatus.NetworkStatusDocument
Network status document containing bridges. This is only available through the metrics site.
Variables: - routers (dict) – fingerprint to
RouterStatusEntryV3
mapping for relays contained in the document - published (datetime) – time when the document was published
- routers (dict) – fingerprint to