Bump version

Add documentation to commands
Merge pull request #157 from blockchain-etl/feature/remove-legacy-scripts
2026-01-12 23:28:02 -05:00 · 2019-03-03 14:48:23 +07:00 · 2019-03-03 14:48:10 +07:00 · 2019-03-03 14:38:20 +07:00 · 2019-03-03 14:37:25 +07:00 · 2019-03-03 14:34:34 +07:00
58 changed files with 9562 additions and 598 deletions
--- a/.dockerignore
+++ b/.dockerignore
@@ -0,0 +1,3 @@
+.*
+last_synced_block.txt
+output
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,7 +1,14 @@
 language: python
-python:
-  - "3.6"
+dist: xenial
+matrix:
+  include:
+    - python: "3.5"
+      env: TOX_POSARGS="-e py35"
+    - python: "3.6"
+      env: TOX_POSARGS="-e py36"
+    - python: "3.7"
+      env: TOX_POSARGS="-e py37"
 install:
-  - travis_retry pip install -r requirements.txt
+  - travis_retry pip install tox
 script:
-  - pytest -vv
+  - tox $TOX_POSARGS
--- a/7
+++ b/7
@@ -4,9 +4,8 @@ ENV PROJECT_DIR=ethereum-etl

 RUN mkdir /$PROJECT_DIR
 WORKDIR /$PROJECT_DIR
-COPY requirements.txt .
-RUN apk add --no-cache gcc musl-dev  #for C libraries: <limits.h> <stdio.h>
-RUN pip install --upgrade pip && pip install -r /$PROJECT_DIR/requirements.txt
 COPY . .
+RUN apk add --no-cache gcc musl-dev  #for C libraries: <limits.h> <stdio.h>
+RUN pip install --upgrade pip && pip install -e /$PROJECT_DIR/

-ENTRYPOINT ["python", "export_all.py"]
+ENTRYPOINT ["python", "ethereumetl"]
--- a/2
+++ b/2
@@ -1,6 +1,6 @@
 MIT License

-Copyright (c) 2018 Evgeny Medvedev, evge.medvedev@gmail.com
+Copyright (c) 2018 Evgeny Medvedev, evge.medvedev@gmail.com, https://twitter.com/EvgeMedvedev

 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
--- a/README.md
+++ b/README.md
@@ -7,7 +7,7 @@
 Install Ethereum ETL:

 ```bash
-pip install ethereum-etl
+pip3 install ethereum-etl
 ```

 Export blocks and transactions ([Schema](#blockscsv), [Reference](#export_blocks_and_transactions)):
@@ -47,8 +47,8 @@ Export traces ([Schema](#tracescsv), [Reference](#export_traces)):

 For the latest version, check out the repo and call 
 ```bash
-> pip install -e . 
-> python ethereumetl.py
+> pip3 install -e . 
+> python3 ethereumetl.py
 ```

 [LIMITATIONS](#limitations)
@@ -67,17 +67,20 @@ For the latest version, check out the repo and call
 - [Exporting the Blockchain](#exporting-the-blockchain)
  - [Export in 2 Hours](#export-in-2-hours)
  - [Command Reference](#command-reference)
+- [Ethereum Classic Support](#ethereum-classic-support)
 - [Querying in Amazon Athena](#querying-in-amazon-athena)
 - [Querying in Google BigQuery](#querying-in-google-bigquery)
  - [Public Dataset](#public-dataset)
+  - [How to Query Balances for all Ethereum Addresses](#how-to-query-balances-for-all-ethereum-addresses)
+  - [Building Token Recommender in Google Cloud Platform](#building-token-recommender-in-google-cloud-platform)


 ## Schema

 ### blocks.csv

-Column                  | Type               |
------------------------|--------------------|
+Column            | Type               |
+------------------|--------------------|
 number            | bigint             |
 hash              | hex_string         |
 parent_hash       | hex_string         |
@@ -99,8 +102,8 @@ transaction_count | bigint             |

 ### transactions.csv

-Column              |    Type     |
--------------------|-------------|
+Column           |    Type     |
+-----------------|-------------|
 hash             | hex_string  |
 nonce            | bigint      |
 block_hash       | hex_string  |
@@ -141,9 +144,9 @@ status                       | bigint      |

 ### logs.csv

-Column                       |    Type     |
-----------------------------|-------------|
-log_index                    | bigint      |
+Column                   |    Type     |
+-------------------------|-------------|
+log_index                | bigint      |
 transaction_hash         | hex_string  |
 transaction_index        | bigint      |
 block_hash               | hex_string  |
@@ -215,20 +218,9 @@ because numeric types there can't handle 32-byte integers. You should use
 [fallback function](https://solidity.readthedocs.io/en/v0.4.21/contracts.html#fallback-function) that returns a `boolean`
 will have `0` or `1` in the `decimals` column in the CSVs.

-### Differences between geth and parity traces.csv
-
- `to_address` field differs for `callcode` trace (geth seems to return correct value, as parity value of `to_address` is same as `to_address` of parent call);
- geth output doesn't have `reward` traces;
- geth output doesn't have `to_address`, `from_address`, `value` for `suicide traces;
- `error` field contains human readable error message, which might differ in geth/parity output;
- geth output doesn't have `transaction_hash`;
- `gas_used` is 0 on traces with error in geth, empty in parity;
- zero output of subcalls is `0x000...` in geth, `0x` in parity;
-
-
 ## Exporting the Blockchain

-1. Install python 3.6 https://www.python.org/downloads/ (3.5 and 3.7 are not supported by this tool for now)
+1. Install python 3.5.3+ https://www.python.org/downloads/

 1. You can use Infura if you don't need ERC20 transfers (Infura doesn't support eth_getFilterLogs JSON RPC method).
 For that use `-p https://mainnet.infura.io` option for the commands below. If you need ERC20 transfers or want to
@@ -245,7 +237,7 @@ and token details; for those you need to wait until the full sync).
 1. Install Ethereum ETL:

    ```bash
-    > pip install ethereum-etl
+    > pip3 install ethereum-etl
    ```

 1. Export all:
@@ -254,6 +246,8 @@ and token details; for those you need to wait until the full sync).
    > ethereumetl export_all --help
    > ethereumetl export_all -s 0 -e 5999999 -b 100000 -p file://$HOME/Library/Ethereum/geth.ipc -o output
    ```
+    
+    In case `ethereumetl` command is not available in PATH, use `python3 -m ethereumetl` instead.

    The result will be in the `output` subdirectory, partitioned in Hive style:

@@ -270,10 +264,9 @@ and token details; for those you need to wait until the full sync).
 Should work with geth and parity, on Linux, Mac, Windows.
 If you use Parity you should disable warp mode with `--no-warp` option because warp mode
 does not place all of the block or receipt data into the database https://wiki.parity.io/Getting-Synced
-Tested with Python 3.6, geth 1.8.7, Ubuntu 16.04.4

 If you see weird behavior, e.g. wrong number of rows in the CSV files or corrupted files,
-check this issue: https://github.com/medvedev1088/ethereum-etl/issues/28
+check out this issue: https://github.com/medvedev1088/ethereum-etl/issues/28

 ### Export in 2 Hours

@@ -292,8 +285,8 @@ Read this article for details https://medium.com/@medvedev1088/how-to-export-the

 1. Run a container out of the image
    ```bash
-    > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest -s 0 -e 5499999 -b 100000 -p https://mainnet.infura.io
-    > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest -s 2018-01-01 -e 2018-01-01 -b 100000 -p https://mainnet.infura.io
+    > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 0 -e 5499999 -b 100000 -p https://mainnet.infura.io
+    > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://mainnet.infura.io
    ```

 ### Command Reference
@@ -443,8 +436,11 @@ You can tune `--max-workers` for performance.

 #### export_traces

+Also called internal transactions.
 The API used in this command is not supported by Infura, 
-so you will need a local Parity archive node (`parity --tracing on`).
+so you will need a local Parity archive node (`parity --tracing on`). 
+Make sure your node has at least 8GB of memory, or else you will face timeout errors. 
+See [this issue](https://github.com/blockchain-etl/ethereum-etl/issues/137) 

 ```bash
 > ethereumetl export_traces --start-block 0 --end-block 500000 \
@@ -455,6 +451,8 @@ You can tune `--batch-size`, `--max-workers` for performance.

 #### export_geth_traces

+Read [Differences between geth and parity traces.csv](#differences-between-geth-and-parity-tracescsv)
+
 The API used in this command is not supported by Infura, 
 so you will need a local Geth archive node (`geth --gcmode archive --syncmode full --ipcapi debug`).
 When using rpc, add `--rpc --rpcapi debug` options.
@@ -488,14 +486,36 @@ You can tune `--batch-size`, `--max-workers` for performance.
 0xa9059cbb2ab09eb219583f4a59a5d0623ade346d962bcd4e46b11da047c9049b
 ```

-#### Running Tests
+### Running Tests

 ```bash
-> pip install -e . -r requirements.txt
+> pip3 install -e .[dev]
 > export ETHEREUM_ETL_RUN_SLOW_TESTS=True
 > pytest -vv
 ```

+### Running Tox Tests
+
+```bash
+> pip3 install tox
+> tox
+```
+
+### Ethereum Classic Support
+
+For getting ETC csv files, make sure you pass in the `--chain classic` param where it's required for the scripts you want to export. 
+ETC won't run if your `--provider-uri` is Infura. It will provide a warning and change the provider-uri to `https://ethereumclassic.network` instead. For faster performance, run a client instead locally for classic such as `parity chain=classic` and Geth-classic.
+
+### Differences between geth and parity traces.csv
+
+- `to_address` field differs for `callcode` trace (geth seems to return correct value, as parity value of `to_address` is same as `to_address` of parent call);
+- geth output doesn't have `reward` traces;
+- geth output doesn't have `to_address`, `from_address`, `value` for `suicide` traces;
+- `error` field contains human readable error message, which might differ in geth/parity output;
+- geth output doesn't have `transaction_hash`;
+- `gas_used` is 0 on traces with error in geth, empty in parity;
+- zero output of subcalls is `0x000...` in geth, `0x` in parity;
+
 ## Querying in Amazon Athena

 - Upload the files to S3:
@@ -542,3 +562,13 @@ Refer to https://github.com/medvedev1088/ethereum-etl-airflow for the instructio

 You can query the data that's updated daily in the public BigQuery dataset
 https://medium.com/@medvedev1088/ethereum-blockchain-on-google-bigquery-283fb300f579
+
+### How to Query Balances for all Ethereum Addresses
+
+Read this article 
+https://medium.com/google-cloud/how-to-query-balances-for-all-ethereum-addresses-in-bigquery-fb594e4034a7
+
+### Building Token Recommender in Google Cloud Platform
+
+Read this article 
+https://medium.com/google-cloud/building-token-recommender-in-google-cloud-platform-1be5a54698eb
--- a/ethereumetl/cli/init.py
+++ b/ethereumetl/cli/init.py
@@ -40,6 +40,7 @@ from ethereumetl.cli.get_keccak_hash import get_keccak_hash


@click.group()
+@click.version_option(version='1.2.4')
@click.pass_context
 def cli(ctx):
    pass
--- a/ethereumetl/cli/export_all.py
+++ b/ethereumetl/cli/export_all.py
@@ -30,6 +30,7 @@ from web3 import Web3
 from ethereumetl.jobs.export_all_common import export_all_common
 from ethereumetl.providers.auto import get_provider_from_uri
 from ethereumetl.service.eth_service import EthService
+from ethereumetl.utils import check_classic_provider_uri


 def is_date_range(start, end):
@@ -74,7 +75,7 @@ def get_partitions(start, end, partition_batch_size, provider_uri):

        while start_date <= end_date:
            batch_start_block, batch_end_block = eth_service.get_block_range_for_date(start_date)
-            partition_dir = f'/date={str(start_date)}/'
+            partition_dir = '/date={start_date!s}/'.format(start_date=start_date)
            yield batch_start_block, batch_end_block, partition_dir
            start_date += day

@@ -89,7 +90,10 @@ def get_partitions(start, end, partition_batch_size, provider_uri):

            padded_batch_start_block = str(batch_start_block).zfill(8)
            padded_batch_end_block = str(batch_end_block).zfill(8)
-            partition_dir = f'/start_block={padded_batch_start_block}/end_block={padded_batch_end_block}'
+            partition_dir = '/start_block={padded_batch_start_block}/end_block={padded_batch_end_block}'.format(
+                padded_batch_start_block=padded_batch_start_block,
+                padded_batch_end_block=padded_batch_end_block,
+            )
            yield batch_start_block, batch_end_block, partition_dir

    else:
@@ -107,7 +111,10 @@ def get_partitions(start, end, partition_batch_size, provider_uri):
@click.option('-o', '--output-dir', default='output', type=str, help='Output directory, partitioned in Hive style.')
@click.option('-w', '--max-workers', default=5, type=int, help='The maximum number of workers.')
@click.option('-B', '--export-batch-size', default=100, type=int, help='The number of requests in JSON RPC batches.')
-def export_all(start, end, partition_batch_size, provider_uri, output_dir, max_workers, export_batch_size):
-    """Exports all for a range of blocks."""
+@click.option('-c', '--chain', default='ethereum', type=str, help='The chain network to connect to.')
+def export_all(start, end, partition_batch_size, provider_uri, output_dir, max_workers, export_batch_size,
+               chain='ethereum'):
+    """Exports all data for a range of blocks."""
+    provider_uri = check_classic_provider_uri(chain, provider_uri)
    export_all_common(get_partitions(start, end, partition_batch_size, provider_uri),
                      output_dir, provider_uri, max_workers, export_batch_size)
--- a/ethereumetl/cli/export_blocks_and_transactions.py
+++ b/ethereumetl/cli/export_blocks_and_transactions.py
@@ -28,6 +28,7 @@ from ethereumetl.jobs.exporters.blocks_and_transactions_item_exporter import blo
 from ethereumetl.logging_utils import logging_basic_config
 from ethereumetl.providers.auto import get_provider_from_uri
 from ethereumetl.thread_local_proxy import ThreadLocalProxy
+from ethereumetl.utils import check_classic_provider_uri

 logging_basic_config()

@@ -45,8 +46,11 @@ logging_basic_config()
@click.option('--transactions-output', default=None, type=str,
              help='The output file for transactions. '
                   'If not provided transactions will not be exported. Use "-" for stdout')
-def export_blocks_and_transactions(start_block, end_block, batch_size, provider_uri, max_workers, blocks_output, transactions_output):
+@click.option('-c', '--chain', default='ethereum', type=str, help='The chain network to connect to.')
+def export_blocks_and_transactions(start_block, end_block, batch_size, provider_uri, max_workers, blocks_output,
+                                   transactions_output, chain='ethereum'):
    """Exports blocks and transactions."""
+    provider_uri = check_classic_provider_uri(chain, provider_uri)
    if blocks_output is None and transactions_output is None:
        raise ValueError('Either --blocks-output or --transactions-output options must be provided')

--- a/ethereumetl/cli/export_contracts.py
+++ b/ethereumetl/cli/export_contracts.py
@@ -29,6 +29,7 @@ from ethereumetl.jobs.exporters.contracts_item_exporter import contracts_item_ex
 from ethereumetl.logging_utils import logging_basic_config
 from ethereumetl.thread_local_proxy import ThreadLocalProxy
 from ethereumetl.providers.auto import get_provider_from_uri
+from ethereumetl.utils import check_classic_provider_uri

 logging_basic_config()

@@ -42,8 +43,10 @@ logging_basic_config()
@click.option('-p', '--provider-uri', default='https://mainnet.infura.io', type=str,
              help='The URI of the web3 provider e.g. '
                   'file://$HOME/Library/Ethereum/geth.ipc or https://mainnet.infura.io')
-def export_contracts(batch_size, contract_addresses, output, max_workers, provider_uri):
+@click.option('-c', '--chain', default='ethereum', type=str, help='The chain network to connect to.')
+def export_contracts(batch_size, contract_addresses, output, max_workers, provider_uri, chain='ethereum'):
    """Exports contracts bytecode and sighashes."""
+    check_classic_provider_uri(chain, provider_uri)
    with smart_open(contract_addresses, 'r') as contract_addresses_file:
        contract_addresses = (contract_address.strip() for contract_address in contract_addresses_file
                              if contract_address.strip())
--- a/ethereumetl/cli/export_receipts_and_logs.py
+++ b/ethereumetl/cli/export_receipts_and_logs.py
@@ -29,6 +29,7 @@ from ethereumetl.jobs.exporters.receipts_and_logs_item_exporter import receipts_
 from ethereumetl.logging_utils import logging_basic_config
 from ethereumetl.thread_local_proxy import ThreadLocalProxy
 from ethereumetl.providers.auto import get_provider_from_uri
+from ethereumetl.utils import check_classic_provider_uri

 logging_basic_config()

@@ -46,8 +47,11 @@ logging_basic_config()
@click.option('--logs-output', default=None, type=str,
              help='The output file for receipt logs. '
                   'aIf not provided receipt logs will not be exported. Use "-" for stdout')
-def export_receipts_and_logs(batch_size, transaction_hashes, provider_uri, max_workers, receipts_output, logs_output):
+@click.option('-c', '--chain', default='ethereum', type=str, help='The chain network to connect to.')
+def export_receipts_and_logs(batch_size, transaction_hashes, provider_uri, max_workers, receipts_output, logs_output,
+                             chain='ethereum'):
    """Exports receipts and logs."""
+    provider_uri = check_classic_provider_uri(chain, provider_uri)
    with smart_open(transaction_hashes, 'r') as transaction_hashes_file:
        job = ExportReceiptsJob(
            transaction_hashes_iterable=(transaction_hash.strip() for transaction_hash in transaction_hashes_file),
--- a/ethereumetl/cli/export_tokens.py
+++ b/ethereumetl/cli/export_tokens.py
@@ -31,19 +31,23 @@ from ethereumetl.jobs.exporters.tokens_item_exporter import tokens_item_exporter
 from ethereumetl.logging_utils import logging_basic_config
 from ethereumetl.thread_local_proxy import ThreadLocalProxy
 from ethereumetl.providers.auto import get_provider_from_uri
+from ethereumetl.utils import check_classic_provider_uri

 logging_basic_config()


@click.command(context_settings=dict(help_option_names=['-h', '--help']))
-@click.option('-t', '--token-addresses', type=str, help='The file containing token addresses, one per line.')
+@click.option('-t', '--token-addresses', required=True, type=str,
+              help='The file containing token addresses, one per line.')
@click.option('-o', '--output', default='-', type=str, help='The output file. If not specified stdout is used.')
@click.option('-w', '--max-workers', default=5, type=int, help='The maximum number of workers.')
@click.option('-p', '--provider-uri', default='https://mainnet.infura.io', type=str,
              help='The URI of the web3 provider e.g. '
                   'file://$HOME/Library/Ethereum/geth.ipc or https://mainnet.infura.io')
-def export_tokens(token_addresses, output, max_workers, provider_uri):
+@click.option('-c', '--chain', default='ethereum', type=str, help='The chain network to connect to.')
+def export_tokens(token_addresses, output, max_workers, provider_uri, chain='ethereum'):
    """Exports ERC20/ERC721 tokens."""
+    provider_uri = check_classic_provider_uri(chain, provider_uri)
    with smart_open(token_addresses, 'r') as token_addresses_file:
        job = ExportTokensJob(
            token_addresses_iterable=(token_address.strip() for token_address in token_addresses_file),
--- a/ethereumetl/cli/export_traces.py
+++ b/ethereumetl/cli/export_traces.py
@@ -43,14 +43,24 @@ logging_basic_config()
@click.option('-p', '--provider-uri', required=True, type=str,
              help='The URI of the web3 provider e.g. '
                   'file://$HOME/.local/share/io.parity.ethereum/jsonrpc.ipc or http://localhost:8545/')
-def export_traces(start_block, end_block, batch_size, output, max_workers, provider_uri):
+@click.option('--genesis-traces/--no-genesis-traces', default=False, help='Whether to include genesis traces')
+@click.option('--daofork-traces/--no-daofork-traces', default=False, help='Whether to include daofork traces')
+@click.option('-t', '--timeout', default=60, type=int, help='IPC or HTTP request timeout.')
+@click.option('-c', '--chain', default='ethereum', type=str, help='The chain network to connect to.')
+def export_traces(start_block, end_block, batch_size, output, max_workers, provider_uri,
+                  genesis_traces, daofork_traces, timeout=60, chain='ethereum'):
    """Exports traces from parity node."""
+    if chain == 'classic' and daofork_traces == True:
+        raise ValueError(
+            'Classic chain does not include daofork traces. Disable daofork traces with --no-daofork-traces option.')
    job = ExportTracesJob(
        start_block=start_block,
        end_block=end_block,
        batch_size=batch_size,
-        web3=ThreadLocalProxy(lambda: Web3(get_provider_from_uri(provider_uri))),
+        web3=ThreadLocalProxy(lambda: Web3(get_provider_from_uri(provider_uri, timeout=timeout))),
        item_exporter=traces_item_exporter(output),
-        max_workers=max_workers)
+        max_workers=max_workers,
+        include_genesis_traces=genesis_traces,
+        include_daofork_traces=daofork_traces)

    job.run()
--- a/ethereumetl/cli/extract_csv_column.py
+++ b/ethereumetl/cli/extract_csv_column.py
@@ -33,7 +33,7 @@ from ethereumetl.file_utils import smart_open
@click.option('-o', '--output', default='-', type=str, help='The output file. If not specified stdout is used.')
@click.option('-c', '--column', required=True, type=str, help='The csv column name to extract.')
 def extract_csv_column(input, output, column):
-    """Extracts column from given CSV file."""
+    """Extracts column from given CSV file. Deprecated - use extract_field."""
    set_max_field_size_limit()

    with smart_open(input, 'r') as input_file, smart_open(output, 'w') as output_file:
--- a/ethereumetl/cli/extract_field.py
+++ b/ethereumetl/cli/extract_field.py
@@ -21,11 +21,9 @@
 # SOFTWARE.


-import json
-
 import click

-from ethereumetl.file_utils import smart_open
+from ethereumetl import misc_utils


@click.command(context_settings=dict(help_option_names=['-h', '--help']))
@@ -33,9 +31,5 @@ from ethereumetl.file_utils import smart_open
@click.option('-o', '--output', default='-', type=str, help='The output file. If not specified stdout is used.')
@click.option('-f', '--field', required=True, type=str, help='The field name to extract.')
 def extract_field(input, output, field):
-    """Extracts field from given JSON lines file."""
-    # TODO: Add support for CSV
-    with smart_open(input, 'r') as input_file, smart_open(output, 'w') as output_file:
-        for line in input_file:
-            item = json.loads(line)
-            output_file.write(item[field] + '\n')
+    """Extracts field from given CSV or JSON newline-delimited file."""
+    misc_utils.extract_field(input, output, field)
--- a/ethereumetl/cli/filter_items.py
+++ b/ethereumetl/cli/filter_items.py
@@ -20,12 +20,9 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.

-
-import json
-
 import click

-from ethereumetl.file_utils import smart_open
+from ethereumetl import misc_utils


@click.command(context_settings=dict(help_option_names=['-h', '--help']))
@@ -34,10 +31,7 @@ from ethereumetl.file_utils import smart_open
@click.option('-p', '--predicate', required=True, type=str,
              help='Predicate in Python code e.g. "item[\'is_erc20\']".')
 def filter_items(input, output, predicate):
-    """Filters given JSON lines file by predicate."""
-    # TODO: Add support for CSV
-    with smart_open(input, 'r') as input_file, smart_open(output, 'w') as output_file:
-        for line in input_file:
-            item = json.loads(line)
-            if eval(predicate, globals(), {'item': item}):
-                output_file.write(json.dumps(item) + '\n')
+    """Filters rows in given CSV or JSON newline-delimited file."""
+    def evaluated_predicate(item):
+        return eval(predicate, globals(), {'item': item})
+    misc_utils.filter_items(input, output, evaluated_predicate)
--- a/ethereumetl/cli/get_block_range_for_date.py
+++ b/ethereumetl/cli/get_block_range_for_date.py
@@ -21,14 +21,16 @@
 # SOFTWARE.


-from datetime import datetime
-
 import click
+
+from datetime import datetime
 from web3 import Web3

+from ethereumetl.file_utils import smart_open
 from ethereumetl.logging_utils import logging_basic_config
-from ethereumetl.providers.auto import get_provider_from_uri
 from ethereumetl.service.eth_service import EthService
+from ethereumetl.providers.auto import get_provider_from_uri
+from ethereumetl.utils import check_classic_provider_uri

 logging_basic_config()

@@ -40,13 +42,15 @@ logging_basic_config()
@click.option('-d', '--date', required=True, type=lambda d: datetime.strptime(d, '%Y-%m-%d'),
              help='The date e.g. 2018-01-01.')
@click.option('-o', '--output', default='-', type=str, help='The output file. If not specified stdout is used.')
-def get_block_range_for_date(provider_uri, date, output):
+@click.option('-c', '--chain', default='ethereum', type=str, help='The chain network to connect to.')
+def get_block_range_for_date(provider_uri, date, output, chain='ethereum'):
    """Outputs start and end blocks for given date."""
+    provider_uri = check_classic_provider_uri(chain, provider_uri)
    provider = get_provider_from_uri(provider_uri)
    web3 = Web3(provider)
    eth_service = EthService(web3)

    start_block, end_block = eth_service.get_block_range_for_date(date)

-    with click.open_file(output, 'w') as output_file:
+    with smart_open(output, 'w') as output_file:
        output_file.write('{},{}\n'.format(start_block, end_block))
--- a/ethereumetl/cli/get_block_range_for_timestamps.py
+++ b/ethereumetl/cli/get_block_range_for_timestamps.py
@@ -29,6 +29,7 @@ from ethereumetl.file_utils import smart_open
 from ethereumetl.logging_utils import logging_basic_config
 from ethereumetl.providers.auto import get_provider_from_uri
 from ethereumetl.service.eth_service import EthService
+from ethereumetl.utils import check_classic_provider_uri

 logging_basic_config()

@@ -40,8 +41,10 @@ logging_basic_config()
@click.option('-s', '--start-timestamp', required=True, type=int, help='Start unix timestamp, in seconds.')
@click.option('-e', '--end-timestamp', required=True, type=int, help='End unix timestamp, in seconds.')
@click.option('-o', '--output', default='-', type=str, help='The output file. If not specified stdout is used.')
-def get_block_range_for_timestamps(provider_uri, start_timestamp, end_timestamp, output):
+@click.option('-c', '--chain', default='ethereum', type=str, help='The chain network to connect to.')
+def get_block_range_for_timestamps(provider_uri, start_timestamp, end_timestamp, output, chain='ethereum'):
    """Outputs start and end blocks for given timestamps."""
+    provider_uri = check_classic_provider_uri(chain, provider_uri)
    provider = get_provider_from_uri(provider_uri)
    web3 = Web3(provider)
    eth_service = EthService(web3)
--- a/ethereumetl/domain/trace.py
+++ b/ethereumetl/domain/trace.py
@@ -36,6 +36,6 @@ class EthTrace(object):
        self.reward_type = None
        self.gas = None
        self.gas_used = None
-        self.subtraces = None
+        self.subtraces = 0
        self.trace_address = None
        self.error = None
--- a/ethereumetl/jobs/export_all_common.py
+++ b/ethereumetl/jobs/export_all_common.py
@@ -76,21 +76,45 @@ def export_all_common(partitions, output_dir, provider_uri, max_workers, batch_s

        padded_batch_start_block = str(batch_start_block).zfill(8)
        padded_batch_end_block = str(batch_end_block).zfill(8)
-        block_range = f'{padded_batch_start_block}-{padded_batch_end_block}'
-        file_name_suffix = f'{padded_batch_start_block}_{padded_batch_end_block}'
+        block_range = '{padded_batch_start_block}-{padded_batch_end_block}'.format(
+            padded_batch_start_block=padded_batch_start_block,
+            padded_batch_end_block=padded_batch_end_block,
+        )
+        file_name_suffix = '{padded_batch_start_block}_{padded_batch_end_block}'.format(
+            padded_batch_start_block=padded_batch_start_block,
+            padded_batch_end_block=padded_batch_end_block,
+        )

        # # # blocks_and_transactions # # #

-        blocks_output_dir = f'{output_dir}/blocks{partition_dir}'
+        blocks_output_dir = '{output_dir}/blocks{partition_dir}'.format(
+            output_dir=output_dir,
+            partition_dir=partition_dir,
+        )
        os.makedirs(os.path.dirname(blocks_output_dir), exist_ok=True)

-        transactions_output_dir = f'{output_dir}/transactions{partition_dir}'
+        transactions_output_dir = '{output_dir}/transactions{partition_dir}'.format(
+            output_dir=output_dir,
+            partition_dir=partition_dir,
+        )
        os.makedirs(os.path.dirname(transactions_output_dir), exist_ok=True)

-        blocks_file = f'{blocks_output_dir}/blocks_{file_name_suffix}.csv'
-        transactions_file = f'{transactions_output_dir}/transactions_{file_name_suffix}.csv'
-        logger.info(f'Exporting blocks {block_range} to {blocks_file}')
-        logger.info(f'Exporting transactions from blocks {block_range} to {transactions_file}')
+        blocks_file = '{blocks_output_dir}/blocks_{file_name_suffix}.csv'.format(
+            blocks_output_dir=blocks_output_dir,
+            file_name_suffix=file_name_suffix,
+        )
+        transactions_file = '{transactions_output_dir}/transactions_{file_name_suffix}.csv'.format(
+            transactions_output_dir=transactions_output_dir,
+            file_name_suffix=file_name_suffix,
+        )
+        logger.info('Exporting blocks {block_range} to {blocks_file}'.format(
+            block_range=block_range,
+            blocks_file=blocks_file,
+        ))
+        logger.info('Exporting transactions from blocks {block_range} to {transactions_file}'.format(
+            block_range=block_range,
+            transactions_file=transactions_file,
+        ))

        job = ExportBlocksJob(
            start_block=batch_start_block,
@@ -107,11 +131,20 @@ def export_all_common(partitions, output_dir, provider_uri, max_workers, batch_s

        token_transfers_file = None
        if is_log_filter_supported(provider_uri):
-            token_transfers_output_dir = f'{output_dir}/token_transfers{partition_dir}'
+            token_transfers_output_dir = '{output_dir}/token_transfers{partition_dir}'.format(
+                output_dir=output_dir,
+                partition_dir=partition_dir,
+            )
            os.makedirs(os.path.dirname(token_transfers_output_dir), exist_ok=True)

-            token_transfers_file = f'{token_transfers_output_dir}/token_transfers_{file_name_suffix}.csv'
-            logger.info(f'Exporting ERC20 transfers from blocks {block_range} to {token_transfers_file}')
+            token_transfers_file = '{token_transfers_output_dir}/token_transfers_{file_name_suffix}.csv'.format(
+                token_transfers_output_dir=token_transfers_output_dir,
+                file_name_suffix=file_name_suffix,
+            )
+            logger.info('Exporting ERC20 transfers from blocks {block_range} to {token_transfers_file}'.format(
+                block_range=block_range,
+                token_transfers_file=token_transfers_file,
+            ))

            job = ExportTokenTransfersJob(
                start_block=batch_start_block,
@@ -124,22 +157,46 @@ def export_all_common(partitions, output_dir, provider_uri, max_workers, batch_s

        # # # receipts_and_logs # # #

-        cache_output_dir = f'{output_dir}/.tmp{partition_dir}'
+        cache_output_dir = '{output_dir}/.tmp{partition_dir}'.format(
+            output_dir=output_dir,
+            partition_dir=partition_dir,
+        )
        os.makedirs(os.path.dirname(cache_output_dir), exist_ok=True)

-        transaction_hashes_file = f'{cache_output_dir}/transaction_hashes_{file_name_suffix}.csv'
-        logger.info(f'Extracting hash column from transaction file {transactions_file}')
+        transaction_hashes_file = '{cache_output_dir}/transaction_hashes_{file_name_suffix}.csv'.format(
+            cache_output_dir=cache_output_dir,
+            file_name_suffix=file_name_suffix,
+        )
+        logger.info('Extracting hash column from transaction file {transactions_file}'.format(
+            transactions_file=transactions_file,
+        ))
        extract_csv_column_unique(transactions_file, transaction_hashes_file, 'hash')

-        receipts_output_dir = f'{output_dir}/receipts{partition_dir}'
+        receipts_output_dir = '{output_dir}/receipts{partition_dir}'.format(
+            output_dir=output_dir,
+            partition_dir=partition_dir,
+        )
        os.makedirs(os.path.dirname(receipts_output_dir), exist_ok=True)

-        logs_output_dir = f'{output_dir}/logs{partition_dir}'
+        logs_output_dir = '{output_dir}/logs{partition_dir}'.format(
+            output_dir=output_dir,
+            partition_dir=partition_dir,
+        )
        os.makedirs(os.path.dirname(logs_output_dir), exist_ok=True)

-        receipts_file = f'{receipts_output_dir}/receipts_{file_name_suffix}.csv'
-        logs_file = f'{logs_output_dir}/logs_{file_name_suffix}.csv'
-        logger.info(f'Exporting receipts and logs from blocks {block_range} to {receipts_file} and {logs_file}')
+        receipts_file = '{receipts_output_dir}/receipts_{file_name_suffix}.csv'.format(
+            receipts_output_dir=receipts_output_dir,
+            file_name_suffix=file_name_suffix,
+        )
+        logs_file = '{logs_output_dir}/logs_{file_name_suffix}.csv'.format(
+            logs_output_dir=logs_output_dir,
+            file_name_suffix=file_name_suffix,
+        )
+        logger.info('Exporting receipts and logs from blocks {block_range} to {receipts_file} and {logs_file}'.format(
+            block_range=block_range,
+            receipts_file=receipts_file,
+            logs_file=logs_file,
+        ))

        with smart_open(transaction_hashes_file, 'r') as transaction_hashes:
            job = ExportReceiptsJob(
@@ -154,15 +211,29 @@ def export_all_common(partitions, output_dir, provider_uri, max_workers, batch_s

        # # # contracts # # #

-        contract_addresses_file = f'{cache_output_dir}/contract_addresses_{file_name_suffix}.csv'
-        logger.info(f'Extracting contract_address from receipt file {receipts_file}')
+        contract_addresses_file = '{cache_output_dir}/contract_addresses_{file_name_suffix}.csv'.format(
+            cache_output_dir=cache_output_dir,
+            file_name_suffix=file_name_suffix,
+        )
+        logger.info('Extracting contract_address from receipt file {receipts_file}'.format(
+            receipts_file=receipts_file
+        ))
        extract_csv_column_unique(receipts_file, contract_addresses_file, 'contract_address')

-        contracts_output_dir = f'{output_dir}/contracts{partition_dir}'
+        contracts_output_dir = '{output_dir}/contracts{partition_dir}'.format(
+            output_dir=output_dir,
+            partition_dir=partition_dir,
+        )
        os.makedirs(os.path.dirname(contracts_output_dir), exist_ok=True)

-        contracts_file = f'{contracts_output_dir}/contracts_{file_name_suffix}.csv'
-        logger.info(f'Exporting contracts from blocks {block_range} to {contracts_file}')
+        contracts_file = '{contracts_output_dir}/contracts_{file_name_suffix}.csv'.format(
+            contracts_output_dir=contracts_output_dir,
+            file_name_suffix=file_name_suffix,
+        )
+        logger.info('Exporting contracts from blocks {block_range} to {contracts_file}'.format(
+            block_range=block_range,
+            contracts_file=contracts_file,
+        ))

        with smart_open(contract_addresses_file, 'r') as contract_addresses_file:
            contract_addresses = (contract_address.strip() for contract_address in contract_addresses_file
@@ -178,15 +249,29 @@ def export_all_common(partitions, output_dir, provider_uri, max_workers, batch_s
        # # # tokens # # #

        if token_transfers_file is not None:
-            token_addresses_file = f'{cache_output_dir}/token_addresses_{file_name_suffix}'
-            logger.info(f'Extracting token_address from token_transfers file {token_transfers_file}')
+            token_addresses_file = '{cache_output_dir}/token_addresses_{file_name_suffix}'.format(
+                cache_output_dir=cache_output_dir,
+                file_name_suffix=file_name_suffix,
+            )
+            logger.info('Extracting token_address from token_transfers file {token_transfers_file}'.format(
+                token_transfers_file=token_transfers_file,
+            ))
            extract_csv_column_unique(token_transfers_file, token_addresses_file, 'token_address')

-            tokens_output_dir = f'{output_dir}/tokens{partition_dir}'
+            tokens_output_dir = '{output_dir}/tokens{partition_dir}'.format(
+                output_dir=output_dir,
+                partition_dir=partition_dir,
+            )
            os.makedirs(os.path.dirname(tokens_output_dir), exist_ok=True)

-            tokens_file = f'{tokens_output_dir}/tokens_{file_name_suffix}.csv'
-            logger.info(f'Exporting tokens from blocks {block_range} to {tokens_file}')
+            tokens_file = '{tokens_output_dir}/tokens_{file_name_suffix}.csv'.format(
+                tokens_output_dir=tokens_output_dir,
+                file_name_suffix=file_name_suffix,
+            )
+            logger.info('Exporting tokens from blocks {block_range} to {tokens_file}'.format(
+                block_range=block_range,
+                tokens_file=tokens_file,
+            ))

            with smart_open(token_addresses_file, 'r') as token_addresses:
                job = ExportTokensJob(
@@ -200,4 +285,7 @@ def export_all_common(partitions, output_dir, provider_uri, max_workers, batch_s
        shutil.rmtree(os.path.dirname(cache_output_dir))
        end_time = time()
        time_diff = round(end_time - start_time, 5)
-        logger.info(f'Exporting blocks {block_range} took {time_diff} seconds')
+        logger.info('Exporting blocks {block_range} took {time_diff} seconds'.format(
+            block_range=block_range,
+            time_diff=time_diff,
+        ))
--- a/ethereumetl/jobs/export_traces_job.py
+++ b/ethereumetl/jobs/export_traces_job.py
@@ -22,8 +22,10 @@

 from ethereumetl.executors.batch_work_executor import BatchWorkExecutor
 from ethereumetl.jobs.base_job import BaseJob
-from ethereumetl.utils import validate_range
+from ethereumetl.mainnet_daofork_state_changes import DAOFORK_BLOCK_NUMBER
 from ethereumetl.mappers.trace_mapper import EthTraceMapper
+from ethereumetl.service.eth_special_trace_service import EthSpecialTraceService
+from ethereumetl.utils import validate_range


 class ExportTracesJob(BaseJob):
@@ -34,7 +36,9 @@ class ExportTracesJob(BaseJob):
            batch_size,
            web3,
            item_exporter,
-            max_workers):
+            max_workers,
+            include_genesis_traces=False,
+            include_daofork_traces=False):
        validate_range(start_block, end_block)
        self.start_block = start_block
        self.end_block = end_block
@@ -47,6 +51,10 @@ class ExportTracesJob(BaseJob):

        self.trace_mapper = EthTraceMapper()

+        self.special_trace_service = EthSpecialTraceService()
+        self.include_genesis_traces = include_genesis_traces
+        self.include_daofork_traces = include_daofork_traces
+
    def _start(self):
        self.item_exporter.open()

@@ -63,10 +71,23 @@ class ExportTracesJob(BaseJob):
        assert len(block_number_batch) == 1
        block_number = block_number_batch[0]

+        if self.include_genesis_traces and 0 in block_number_batch:
+            genesis_traces = self.special_trace_service.get_genesis_traces()
+            for trace in genesis_traces:
+                self.item_exporter.export_item(self.trace_mapper.trace_to_dict(trace))
+
+        if self.include_daofork_traces and DAOFORK_BLOCK_NUMBER in block_number_batch:
+            daofork_traces = self.special_trace_service.get_daofork_traces()
+            for trace in daofork_traces:
+                self.item_exporter.export_item(self.trace_mapper.trace_to_dict(trace))
+
        # TODO: Change to traceFilter when this issue is fixed
        # https://github.com/paritytech/parity-ethereum/issues/9822
        json_traces = self.web3.parity.traceBlock(block_number)

+        if json_traces is None:
+            raise ValueError('Response from the node is None. Is the node fully synced?')
+
        for json_trace in json_traces:
            trace = self.trace_mapper.json_dict_to_trace(json_trace)
            self.item_exporter.export_item(self.trace_mapper.trace_to_dict(trace))
--- a/ethereumetl/mainnet_daofork_state_changes.py
+++ b/ethereumetl/mainnet_daofork_state_changes.py
@@ -0,0 +1,149 @@
+# MIT License
+#
+# Copyright (c) 2018 Evgeny Medvedev, evge.medvedev@gmail.com
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+# https://github.com/ethereum/EIPs/blob/master/EIPS/eip-779.md
+# https://blog.ethereum.org/2016/07/20/hard-fork-completed/
+# https://gist.github.com/gavofyork/af747a034fbee2920f862ed352d32347
+# https://blog.ethereum.org/2016/07/15/to-fork-or-not-to-fork/
+
+DAOFORK_BLOCK_NUMBER = 1920000
+WITHDRAW_DAO_ADDRESS = '0xbf4ed7b27f1d666546e30d74d50d173d20bca754'
+
+MAINNET_DAOFORK_STATE_CHANGES = [
+    # from_address, to_address, value
+    ("0x005f5cee7a43331d5a3d3eec71305925a62f34b6", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x0101f3be8ebb4bbd39a2e3b9a3639d4259832fd9", WITHDRAW_DAO_ADDRESS, 559384955979606013894),
+    ("0x057b56736d32b86616a10f619859c6cd6f59092a", WITHDRAW_DAO_ADDRESS, 9900012824972102),
+    ("0x06706dd3f2c9abf0a21ddcc6941d9b86f0596936", WITHDRAW_DAO_ADDRESS, 1428573279216753537),
+    ("0x0737a6b837f97f46ebade41b9bc3e1c509c85c53", WITHDRAW_DAO_ADDRESS, 7144077587762826223),
+    ("0x07f5c1e1bc2c93e0402f23341973a0e043f7bf8a", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x0e0da70933f4c7849fc0d203f5d1d43b9ae4532d", WITHDRAW_DAO_ADDRESS, 19173240336954131945545),
+    ("0x0ff30d6de14a8224aa97b78aea5388d1c51c1f00", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x12e626b0eebfe86a56d633b9864e389b45dcb260", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x1591fc0f688c81fbeb17f5426a162a7024d430c2", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x17802f43a0137c506ba92291391a8a8f207f487d", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x1975bd06d486162d5dc297798dfc41edd5d160a7", WITHDRAW_DAO_ADDRESS, 989001281201758473335),
+    ("0x1ca6abd14d30affe533b24d7a21bff4c2d5e1f3b", WITHDRAW_DAO_ADDRESS, 76761842290232377901),
+    ("0x1cba23d343a983e9b5cfd19496b9a9701ada385f", WITHDRAW_DAO_ADDRESS, 68587370259945226),
+    ("0x200450f06520bdd6c527622a273333384d870efb", WITHDRAW_DAO_ADDRESS, 1250001619314659344457),
+    ("0x21c7fdb9ed8d291d79ffd82eb2c4356ec0d81241", WITHDRAW_DAO_ADDRESS, 27428797178668633),
+    ("0x23b75c2f6791eef49c69684db4c6c1f93bf49a50", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x24c4d950dfd4dd1902bbed3508144a54542bba94", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x253488078a4edf4d6f42f113d1e62836a942cf1a", WITHDRAW_DAO_ADDRESS, 3486036451558542464),
+    ("0x27b137a85656544b1ccb5a0f2e561a5703c6a68f", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x2a5ed960395e2a49b1c758cef4aa15213cfd874c", WITHDRAW_DAO_ADDRESS, 18693039890011849),
+    ("0x2b3455ec7fedf16e646268bf88846bd7a2319bb2", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x2c19c7f9ae8b751e37aeb2d93a699722395ae18f", WITHDRAW_DAO_ADDRESS, 8519214441755701),
+    ("0x304a554a310c7e546dfe434669c62820b7d83490", WITHDRAW_DAO_ADDRESS, 3642408527612792706899331),
+    ("0x319f70bab6845585f412ec7724b744fec6095c85", WITHDRAW_DAO_ADDRESS, 90658),
+    ("0x35a051a0010aba705c9008d7a7eff6fb88f6ea7b", WITHDRAW_DAO_ADDRESS, 15276059789372406985),
+    ("0x3ba4d81db016dc2890c81f3acec2454bff5aada5", WITHDRAW_DAO_ADDRESS, 1),
+    ("0x3c02a7bc0391e86d91b7d144e61c2c01a25a79c5", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x40b803a9abce16f50f36a77ba41180eb90023925", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x440c59b325d2997a134c2c7c60a8c61611212bad", WITHDRAW_DAO_ADDRESS, 266854104538362875475),
+    ("0x4486a3d68fac6967006d7a517b889fd3f98c102b", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x4613f3bca5c44ea06337a9e439fbc6d42e501d0a", WITHDRAW_DAO_ADDRESS, 28927603152430302650042),
+    ("0x47e7aa56d6bdf3f36be34619660de61275420af8", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x4863226780fe7c0356454236d3b1c8792785748d", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x492ea3bb0f3315521c31f273e565b868fc090f17", WITHDRAW_DAO_ADDRESS, 367380383063135344585),
+    ("0x4cb31628079fb14e4bc3cd5e30c2f7489b00960c", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x4deb0033bb26bc534b197e61d19e0733e5679784", WITHDRAW_DAO_ADDRESS, 1256101627216914882057),
+    ("0x4fa802324e929786dbda3b8820dc7834e9134a2a", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x4fd6ace747f06ece9c49699c7cabc62d02211f75", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x51e0ddd9998364a2eb38588679f0d2c42653e4a6", WITHDRAW_DAO_ADDRESS, 10000012954517274755),
+    ("0x52c5317c848ba20c7504cb2c8052abd1fde29d03", WITHDRAW_DAO_ADDRESS, 1996002585721648041229),
+    ("0x542a9515200d14b68e934e9830d91645a980dd7a", WITHDRAW_DAO_ADDRESS, 12548793143344641481996),
+    ("0x5524c55fb03cf21f549444ccbecb664d0acad706", WITHDRAW_DAO_ADDRESS, 6773243673260677597543),
+    ("0x579a80d909f346fbfb1189493f521d7f48d52238", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x58b95c9a9d5d26825e70a82b6adb139d3fd829eb", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x5c6e67ccd5849c0d29219c4f95f1a7a93b3f5dc5", WITHDRAW_DAO_ADDRESS, 1),
+    ("0x5c8536898fbb74fc7445814902fd08422eac56d0", WITHDRAW_DAO_ADDRESS, 205100000000392887672),
+    ("0x5d2b2e6fcbe3b11d26b525e085ff818dae332479", WITHDRAW_DAO_ADDRESS, 5000006477258637377),
+    ("0x5dc28b15dffed94048d73806ce4b7a4612a1d48f", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x5f9f3392e9f62f63b8eac0beb55541fc8627f42c", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x6131c42fa982e56929107413a9d526fd99405560", WITHDRAW_DAO_ADDRESS, 2121837249362469256186),
+    ("0x6231b6d0d5e77fe001c2a460bd9584fee60d409b", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x627a0a960c079c21c34f7612d5d230e01b4ad4c7", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x63ed5a272de2f6d968408b4acb9024f4cc208ebf", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x6966ab0d485353095148a2155858910e0965b6f9", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x6b0c4d41ba9ab8d8cfb5d379c69a612f2ced8ecb", WITHDRAW_DAO_ADDRESS, 854763543),
+    ("0x6d87578288b6cb5549d5076a207456a1f6a63dc0", WITHDRAW_DAO_ADDRESS, 1944767821345229848),
+    ("0x6f6704e5a10332af6672e50b3d9754dc460dfa4d", WITHDRAW_DAO_ADDRESS, 41173345768012804300),
+    ("0x7602b46df5390e432ef1c307d4f2c9ff6d65cc97", WITHDRAW_DAO_ADDRESS, 369231179004682274248),
+    ("0x779543a0491a837ca36ce8c635d6154e3c4911a6", WITHDRAW_DAO_ADDRESS, 100000000000000000),
+    ("0x77ca7b50b6cd7e2f3fa008e24ab793fd56cb15f6", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x782495b7b3355efb2833d56ecb34dc22ad7dfcc4", WITHDRAW_DAO_ADDRESS, 250000323862931868891),
+    ("0x807640a13483f8ac783c557fcdf27be11ea4ac7a", WITHDRAW_DAO_ADDRESS, 89472700),
+    ("0x8163e7fb499e90f8544ea62bbf80d21cd26d9efd", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x84ef4b2357079cd7a7c69fd7a37cd0609a679106", WITHDRAW_DAO_ADDRESS, 598974326560793095813484),
+    ("0x86af3e9626fce1957c82e88cbf04ddf3a2ed7915", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x8d9edb3054ce5c5774a420ac37ebae0ac02343c6", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x914d1b8b43e92723e64fd0a06f5bdb8dd9b10c79", WITHDRAW_DAO_ADDRESS, 285714295714285714286),
+    ("0x97f43a37f595ab5dd318fb46e7a155eae057317a", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x9aa008f65de0b923a2a4f02012ad034a5e2e2192", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x9c15b54878ba618f494b38f0ae7443db6af648ba", WITHDRAW_DAO_ADDRESS, 2236999142516500888),
+    ("0x9c50426be05db97f5d64fc54bf89eff947f0a321", WITHDRAW_DAO_ADDRESS, 0),
+    ("0x9da397b9e80755301a3b32173283a91c0ef6c87e", WITHDRAW_DAO_ADDRESS, 934889382511061152962),
+    ("0x9ea779f907f0b315b364b0cfc39a0fde5b02a416", WITHDRAW_DAO_ADDRESS, 15841461690131427090010),
+    ("0x9f27daea7aca0aa0446220b98d028715e3bc803d", WITHDRAW_DAO_ADDRESS, 99998647723253121277),
+    ("0x9fcd2deaff372a39cc679d5c5e4de7bafb0b1339", WITHDRAW_DAO_ADDRESS, 1409336722195117395464),
+    ("0xa2f1ccba9395d7fcb155bba8bc92db9bafaeade7", WITHDRAW_DAO_ADDRESS, 5000006477258637377),
+    ("0xa3acf3a1e16b1d7c315e23510fdd7847b48234f6", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xa5dc5acd6a7968a4554d89d65e59b7fd3bff0f90", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xa82f360a8d3455c5c41366975bde739c37bfeb8a", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xac1ecab32727358dba8962a0f3b261731aad9723", WITHDRAW_DAO_ADDRESS, 1),
+    ("0xaccc230e8a6e5be9160b8cdf2864dd2a001c28b6", WITHDRAW_DAO_ADDRESS, 23997787866533545896),
+    ("0xacd87e28b0c9d1254e868b81cba4cc20d9a32225", WITHDRAW_DAO_ADDRESS, 207153967008322399135),
+    ("0xadf80daec7ba8dcf15392f1ac611fff65d94f880", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xaeeb8ff27288bdabc0fa5ebb731b6f409507516c", WITHDRAW_DAO_ADDRESS, 859189750496835322093),
+    ("0xb136707642a4ea12fb4bae820f03d2562ebff487", WITHDRAW_DAO_ADDRESS, 7277385711515429122911683),
+    ("0xb2c6f0dfbb716ac562e2d85d6cb2f8d5ee87603e", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xb3fb0e5aba0e20e5c49d252dfd30e102b171a425", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xb52042c8ca3f8aa246fa79c3feaa3d959347c0ab", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xb9637156d330c0d605a791f1c31ba5890582fe1c", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xbb9bc244d798123fde783fcc1c72d3bb8c189413", WITHDRAW_DAO_ADDRESS, 1200000000000000001),
+    ("0xbc07118b9ac290e4622f5e77a0853539789effbe", WITHDRAW_DAO_ADDRESS, 5634097608979247392143),
+    ("0xbcf899e6c7d9d5a215ab1e3444c86806fa854c76", WITHDRAW_DAO_ADDRESS, 30696803822257124360133),
+    ("0xbe8539bfe837b67d1282b2b1d61c3f723966f049", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xc4bbd073882dd2add2424cf47d35213405b01324", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xca544e5c4687d109611d0f8f928b53a25af72448", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xcbb9d3703e651b0d496cdefb8b92c25aeb2171f7", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xcc34673c6c40e791051898567a1222daf90be287", WITHDRAW_DAO_ADDRESS, 60000077727103648),
+    ("0xceaeb481747ca6c540a000c1f3641f8cef161fa7", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xd131637d5275fd1a68a3200f4ad25c71a2a9522e", WITHDRAW_DAO_ADDRESS, 118886510785155274580),
+    ("0xd164b088bd9108b60d0ca3751da4bceb207b0782", WITHDRAW_DAO_ADDRESS, 1000001295451727475566),
+    ("0xd1ac8b1ef1b69ff51d1d401a476e7e612414f091", WITHDRAW_DAO_ADDRESS, 18387737083543350),
+    ("0xd343b217de44030afaa275f54d31a9317c7f441e", WITHDRAW_DAO_ADDRESS, 5192307692307692307692),
+    ("0xd4fe7bc31cedb7bfb8a345f31e668033056b2728", WITHDRAW_DAO_ADDRESS, 110000142499690430),
+    ("0xd9aef3a1e38a39c16b31d1ace71bca8ef58d315b", WITHDRAW_DAO_ADDRESS, 100000129545172747556),
+    ("0xda2fef9e4a3230988ff17df2165440f37e8b1708", WITHDRAW_DAO_ADDRESS, 73722042576599901129491),
+    ("0xdbe9b615a3ae8709af8b93336ce9b477e4ac0940", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xe308bd1ac5fda103967359b2712dd89deffb7973", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xe4ae1efdfc53b73893af49113d8694a057b9c0d1", WITHDRAW_DAO_ADDRESS, 5000006477258637377),
+    ("0xec8e57756626fdc07c63ad2eafbd28d08e7b0ca5", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xecd135fa4f61a655311e86238c92adcd779555d2", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xf0b1aa0eb660754448a7937c022e30aa692fe0c5", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xf1385fb24aad0cd7432824085e42aff90886fef5", WITHDRAW_DAO_ADDRESS, 0),
+    ("0xf14c14075d6c4ed84b86798af0956deef67365b5", WITHDRAW_DAO_ADDRESS, 2123311222366559138),
+    ("0xf4c64518ea10f995918a454158c6b61407ea345c", WITHDRAW_DAO_ADDRESS, 269565591797974102411594),
+    ("0xfe24cdd8648121a43a7c86d289be4dd2951ed49f", WITHDRAW_DAO_ADDRESS, 269833661813680507459)
+]
--- a/ethereumetl/mainnet_genesis_alloc.py
+++ b/ethereumetl/mainnet_genesis_alloc.py
--- a/ethereumetl/mappers/trace_mapper.py
+++ b/ethereumetl/mappers/trace_mapper.py
@@ -22,6 +22,7 @@


 from ethereumetl.domain.trace import EthTrace
+from ethereumetl.mainnet_daofork_state_changes import DAOFORK_BLOCK_NUMBER
 from ethereumetl.utils import hex_to_dec, to_normalized_address


@@ -93,6 +94,34 @@ class EthTraceMapper(object):

        return traces

+    def genesis_alloc_to_trace(self, allocation):
+        address = allocation[0]
+        value = allocation[1]
+
+        trace = EthTrace()
+
+        trace.block_number = 0
+        trace.to_address = address
+        trace.value = value
+        trace.trace_type = 'genesis'
+
+        return trace
+
+    def daofork_state_change_to_trace(self, state_change):
+        from_address = state_change[0]
+        to_address = state_change[1]
+        value = state_change[2]
+
+        trace = EthTrace()
+
+        trace.block_number = DAOFORK_BLOCK_NUMBER
+        trace.from_address = from_address
+        trace.to_address = to_address
+        trace.value = value
+        trace.trace_type = 'daofork'
+
+        return trace
+
    def _iterate_transaction_trace(self, block_number, tx_index, tx_trace, trace_address=[]):
        trace = EthTrace()

--- a/ethereumetl/misc_utils.py
+++ b/ethereumetl/misc_utils.py
@@ -0,0 +1,85 @@
+# MIT License
+#
+# Copyright (c) 2018 Evgeny Medvedev, evge.medvedev@gmail.com
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import contextlib
+import csv
+import json
+
+import six
+
+from ethereumetl.csv_utils import set_max_field_size_limit
+from ethereumetl.file_utils import get_file_handle, smart_open
+
+
+@contextlib.contextmanager
+def get_item_iterable(input_file):
+    fh = get_file_handle(input_file, 'r')
+
+    if input_file.endswith('.csv'):
+        set_max_field_size_limit()
+        reader = csv.DictReader(fh)
+    else:
+        reader = (json.loads(line) for line in fh)
+
+    try:
+        yield reader
+    finally:
+        fh.close()
+
+
+@contextlib.contextmanager
+def get_item_sink(output_file):
+    fh = get_file_handle(output_file, 'w')
+
+    if output_file.endswith('.csv'):
+        set_max_field_size_limit()
+
+        writer = None
+
+        def sink(item):
+            nonlocal writer
+            if writer is None:
+                fields = list(six.iterkeys(item))
+                writer = csv.DictWriter(fh, fieldnames=fields, extrasaction='ignore')
+                writer.writeheader()
+            writer.writerow(item)
+    else:
+        def sink(item):
+            fh.write(json.dumps(item) + '\n')
+
+    try:
+        yield sink
+    finally:
+        fh.close()
+
+
+def filter_items(input_file, output_file, predicate):
+    with get_item_iterable(input_file) as item_iterable, get_item_sink(output_file) as sink:
+        for item in item_iterable:
+            if predicate(item):
+                sink(item)
+
+
+def extract_field(input_file, output_file, field):
+    with get_item_iterable(input_file) as item_iterable, smart_open(output_file, 'w') as output:
+        for item in item_iterable:
+            output.write(item[field] + '\n')
--- a/ethereumetl/providers/auto.py
+++ b/ethereumetl/providers/auto.py
@@ -28,22 +28,22 @@ from web3 import IPCProvider, HTTPProvider
 from ethereumetl.providers.ipc import BatchIPCProvider
 from ethereumetl.providers.rpc import BatchHTTPProvider

-DEFAULT_IPC_TIMEOUT = 60
-DEFAULT_HTTP_REQUEST_KWARGS = {'timeout': 60}
+DEFAULT_TIMEOUT = 60


-def get_provider_from_uri(uri_string, batch=False):
+def get_provider_from_uri(uri_string, timeout=DEFAULT_TIMEOUT, batch=False):
    uri = urlparse(uri_string)
    if uri.scheme == 'file':
        if batch:
-            return BatchIPCProvider(uri.path, timeout=DEFAULT_IPC_TIMEOUT)
+            return BatchIPCProvider(uri.path, timeout=timeout)
        else:
-            return IPCProvider(uri.path, timeout=DEFAULT_IPC_TIMEOUT)
+            return IPCProvider(uri.path, timeout=timeout)
    elif uri.scheme == 'http' or uri.scheme == 'https':
+        request_kwargs = {'timeout': timeout}
        if batch:
-            return BatchHTTPProvider(uri_string, request_kwargs=DEFAULT_HTTP_REQUEST_KWARGS)
+            return BatchHTTPProvider(uri_string, request_kwargs=request_kwargs)
        else:
-            return HTTPProvider(uri_string, request_kwargs=DEFAULT_HTTP_REQUEST_KWARGS)
+            return HTTPProvider(uri_string, request_kwargs=request_kwargs)
    else:
        raise ValueError('Unknown uri scheme {}'.format(uri_string))

--- a/ethereumetl/service/eth_service.py
+++ b/ethereumetl/service/eth_service.py
@@ -32,8 +32,8 @@ class EthService(object):
        self._graph_operations = GraphOperations(graph)

    def get_block_range_for_date(self, date):
-        start_datetime = datetime.combine(date, datetime.min.time(), tzinfo=timezone.utc)
-        end_datetime = datetime.combine(date, datetime.max.time(), tzinfo=timezone.utc)
+        start_datetime = datetime.combine(date, datetime.min.time().replace(tzinfo=timezone.utc))
+        end_datetime = datetime.combine(date, datetime.max.time().replace(tzinfo=timezone.utc))
        return self.get_block_range_for_timestamps(start_datetime.timestamp(), end_datetime.timestamp())

    def get_block_range_for_timestamps(self, start_timestamp, end_timestamp):
--- a/ethereumetl/service/eth_special_trace_service.py
+++ b/ethereumetl/service/eth_special_trace_service.py
@@ -21,10 +21,22 @@
 # SOFTWARE.


-from ethereumetl.cli.export_all import export_all
+from ethereumetl.mappers.trace_mapper import EthTraceMapper

-print('========================================================================================')
-print('THIS SCRIPT IS DEPRECATED AND WILL BE REMOVED ON 2019-01-01. Use ethereumetl.py instead.')
-print('========================================================================================')

-export_all()
+class EthSpecialTraceService(object):
+
+    def __init__(self):
+        self.trace_mapper = EthTraceMapper()
+
+    def get_genesis_traces(self):
+        from ethereumetl.mainnet_genesis_alloc import MAINNET_GENESIS_ALLOC
+        genesis_traces = [self.trace_mapper.genesis_alloc_to_trace(alloc)
+                          for alloc in MAINNET_GENESIS_ALLOC]
+        return genesis_traces
+
+    def get_daofork_traces(self):
+        from ethereumetl.mainnet_daofork_state_changes import MAINNET_DAOFORK_STATE_CHANGES
+        daofork_traces = [self.trace_mapper.daofork_state_change_to_trace(change)
+                          for change in MAINNET_DAOFORK_STATE_CHANGES]
+        return daofork_traces
--- a/ethereumetl/service/eth_token_service.py
+++ b/ethereumetl/service/eth_token_service.py
@@ -19,13 +19,15 @@
 # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
-
+import logging

 from web3.exceptions import BadFunctionCallOutput

 from ethereumetl.domain.token import EthToken
 from ethereumetl.erc20_abi import ERC20_ABI

+logger = logging.getLogger('eth_token_service')
+

 class EthTokenService(object):
    def __init__(self, web3, function_call_result_transformer=None):
@@ -56,7 +58,7 @@ class EthTokenService(object):
        # OverflowError exception happens if the return type of the function doesn't match the expected type
        result = call_contract_function(
            func=func,
-            ignore_errors=(BadFunctionCallOutput, OverflowError),
+            ignore_errors=(BadFunctionCallOutput, OverflowError, ValueError),
            default_value=None)

        if self._function_call_result_transformer is not None:
@@ -71,6 +73,7 @@ def call_contract_function(func, ignore_errors, default_value=None):
        return result
    except Exception as ex:
        if type(ex) in ignore_errors:
+            logger.exception('An exception occurred. This exception can be safely ignored.')
            return default_value
        else:
            raise ex
--- a/ethereumetl/utils.py
+++ b/ethereumetl/utils.py
@@ -22,6 +22,7 @@


 import itertools
+import warnings


 def hex_to_dec(hex_string):
@@ -92,3 +93,9 @@ def pairwise(iterable):
    a, b = itertools.tee(iterable)
    next(b, None)
    return zip(a, b)
+
+def check_classic_provider_uri(chain, provider_uri):
+    if chain == 'classic' and provider_uri == 'https://mainnet.infura.io':
+        warnings.warn("ETC Chain not supported on Infura.io. Using https://ethereumclassic.network instead")
+        return 'https://ethereumclassic.network'
+    return provider_uri
--- a/export_blocks_and_transactions.py
+++ b/export_blocks_and_transactions.py
@@ -1,29 +0,0 @@
-# MIT License
-#
-# Copyright (c) 2018 Evgeny Medvedev, evge.medvedev@gmail.com
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-from ethereumetl.cli.export_blocks_and_transactions import export_blocks_and_transactions
-
-print('========================================================================================')
-print('THIS SCRIPT IS DEPRECATED AND WILL BE REMOVED ON 2019-01-01. Use ethereumetl.py instead.')
-print('========================================================================================')
-
-export_blocks_and_transactions()
--- a/export_contracts.py
+++ b/export_contracts.py
@@ -1,30 +0,0 @@
-# MIT License
-#
-# Copyright (c) 2018 Evgeny Medvedev, evge.medvedev@gmail.com
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-
-from ethereumetl.cli.export_contracts import export_contracts
-
-print('========================================================================================')
-print('THIS SCRIPT IS DEPRECATED AND WILL BE REMOVED ON 2019-01-01. Use ethereumetl.py instead.')
-print('========================================================================================')
-
-export_contracts()
--- a/export_geth_traces.py
+++ b/export_geth_traces.py
@@ -1,30 +0,0 @@
-# MIT License
-#
-# Copyright (c) 2018 Evgeniy Filatov, evgeniyfilatov@gmail.com
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-
-from ethereumetl.cli.export_geth_traces import export_geth_traces
-
-print('========================================================================================')
-print('THIS SCRIPT IS DEPRECATED AND WILL BE REMOVED ON 2019-01-01. Use ethereumetl.py instead.')
-print('========================================================================================')
-
-export_geth_traces()
--- a/export_receipts_and_logs.py
+++ b/export_receipts_and_logs.py
@@ -1,30 +0,0 @@
-# MIT License
-#
-# Copyright (c) 2018 Evgeny Medvedev, evge.medvedev@gmail.com
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-
-from ethereumetl.cli.export_receipts_and_logs import export_receipts_and_logs
-
-print('========================================================================================')
-print('THIS SCRIPT IS DEPRECATED AND WILL BE REMOVED ON 2019-01-01. Use ethereumetl.py instead.')
-print('========================================================================================')
-
-export_receipts_and_logs()
--- a/export_token_transfers.py
+++ b/export_token_transfers.py
@@ -1,30 +0,0 @@
-# MIT License
-#
-# Copyright (c) 2018 Evgeny Medvedev, evge.medvedev@gmail.com
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-
-from ethereumetl.cli.export_token_transfers import export_token_transfers
-
-print('========================================================================================')
-print('THIS SCRIPT IS DEPRECATED AND WILL BE REMOVED ON 2019-01-01. Use ethereumetl.py instead.')
-print('========================================================================================')
-
-export_token_transfers()
--- a/export_tokens.py
+++ b/export_tokens.py
@@ -1,30 +0,0 @@
-# MIT License
-#
-# Copyright (c) 2018 Evgeny Medvedev, evge.medvedev@gmail.com
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-
-from ethereumetl.cli.export_tokens import export_tokens
-
-print('========================================================================================')
-print('THIS SCRIPT IS DEPRECATED AND WILL BE REMOVED ON 2019-01-01. Use ethereumetl.py instead.')
-print('========================================================================================')
-
-export_tokens()
--- a/export_traces.py
+++ b/export_traces.py
@@ -1,30 +0,0 @@
-# MIT License
-#
-# Copyright (c) 2018 Evgeniy Filatov, evgeniyfilatov@gmail.com
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-
-from ethereumetl.cli.export_traces import export_traces
-
-print('========================================================================================')
-print('THIS SCRIPT IS DEPRECATED AND WILL BE REMOVED ON 2019-01-01. Use ethereumetl.py instead.')
-print('========================================================================================')
-
-export_traces()
--- a/extract_csv_column.py
+++ b/extract_csv_column.py
@@ -1,30 +0,0 @@
-# MIT License
-#
-# Copyright (c) 2018 Evgeny Medvedev, evge.medvedev@gmail.com
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-
-from ethereumetl.cli.extract_csv_column import extract_csv_column
-
-print('========================================================================================')
-print('THIS SCRIPT IS DEPRECATED AND WILL BE REMOVED ON 2019-01-01. Use ethereumetl.py instead.')
-print('========================================================================================')
-
-extract_csv_column()
--- a/extract_field.py
+++ b/extract_field.py
@@ -1,30 +0,0 @@
-# MIT License
-#
-# Copyright (c) 2018 Evgeny Medvedev, evge.medvedev@gmail.com
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-
-from ethereumetl.cli.extract_field import extract_field
-
-print('========================================================================================')
-print('THIS SCRIPT IS DEPRECATED AND WILL BE REMOVED ON 2019-01-01. Use ethereumetl.py instead.')
-print('========================================================================================')
-
-extract_field()
--- a/extract_geth_traces.py
+++ b/extract_geth_traces.py
@@ -1,30 +0,0 @@
-# MIT License
-#
-# Copyright (c) 2018 Evgeniy Filatov, evgebiyfilatov@gmail.com
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-
-from ethereumetl.cli.extract_geth_traces import extract_geth_traces
-
-print('========================================================================================')
-print('THIS SCRIPT IS DEPRECATED AND WILL BE REMOVED ON 2019-01-01. Use ethereumetl.py instead.')
-print('========================================================================================')
-
-extract_geth_traces()
--- a/extract_token_transfers.py
+++ b/extract_token_transfers.py
@@ -1,30 +0,0 @@
-# MIT License
-#
-# Copyright (c) 2018 Evgeny Medvedev, evge.medvedev@gmail.com
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-
-from ethereumetl.cli.extract_token_transfers import extract_token_transfers
-
-print('========================================================================================')
-print('THIS SCRIPT IS DEPRECATED AND WILL BE REMOVED ON 2019-01-01. Use ethereumetl.py instead.')
-print('========================================================================================')
-
-extract_token_transfers()
--- a/filter_items.py
+++ b/filter_items.py
@@ -1,30 +0,0 @@
-# MIT License
-#
-# Copyright (c) 2018 Evgeny Medvedev, evge.medvedev@gmail.com
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-
-from ethereumetl.cli.filter_items import filter_items
-
-print('========================================================================================')
-print('THIS SCRIPT IS DEPRECATED AND WILL BE REMOVED ON 2019-01-01. Use ethereumetl.py instead.')
-print('========================================================================================')
-
-filter_items()
--- a/get_block_range_for_date.py
+++ b/get_block_range_for_date.py
@@ -1,30 +0,0 @@
-# MIT License
-#
-# Copyright (c) 2018 Evgeny Medvedev, evge.medvedev@gmail.com
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-
-from ethereumetl.cli.get_block_range_for_date import get_block_range_for_date
-
-print('========================================================================================')
-print('THIS SCRIPT IS DEPRECATED AND WILL BE REMOVED ON 2019-01-01. Use ethereumetl.py instead.')
-print('========================================================================================')
-
-get_block_range_for_date()
--- a/get_block_range_for_timestamps.py
+++ b/get_block_range_for_timestamps.py
@@ -1,30 +0,0 @@
-# MIT License
-#
-# Copyright (c) 2018 Evgeny Medvedev, evge.medvedev@gmail.com
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-
-from ethereumetl.cli.get_block_range_for_timestamps import get_block_range_for_timestamps
-
-print('========================================================================================')
-print('THIS SCRIPT IS DEPRECATED AND WILL BE REMOVED ON 2019-01-01. Use ethereumetl.py instead.')
-print('========================================================================================')
-
-get_block_range_for_timestamps()
--- a/get_keccak_hash.py
+++ b/get_keccak_hash.py
@@ -1,30 +0,0 @@
-# MIT License
-#
-# Copyright (c) 2018 Evgeny Medvedev, evge.medvedev@gmail.com
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-# SOFTWARE.
-
-
-from ethereumetl.cli.get_keccak_hash import get_keccak_hash
-
-print('========================================================================================')
-print('THIS SCRIPT IS DEPRECATED AND WILL BE REMOVED ON 2019-01-01. Use ethereumetl.py instead.')
-print('========================================================================================')
-
-get_keccak_hash()
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,2 +0,0 @@
-.
-pytest~=3.2.0
--- a/setup.py
+++ b/setup.py
@@ -11,7 +11,7 @@ long_description = read('README.md') if os.path.isfile("README.md") else ""

 setup(
    name='ethereum-etl',
-    version='1.0.0',
+    version='1.2.4',
    author='Evgeny Medvedev',
    author_email='evge.medvedev@gmail.com',
    description='Tools for exporting Ethereum blockchain data to CSV or JSON',
@@ -24,19 +24,27 @@ setup(
        'Intended Audience :: Developers',
        'License :: OSI Approved :: MIT License',
        'Programming Language :: Python :: 3',
+        'Programming Language :: Python :: 3.5',
        'Programming Language :: Python :: 3.6',
+        'Programming Language :: Python :: 3.7'
    ],
    keywords='ethereum',
-    python_requires='>=3.6.0,<3.7.0',
+    # web3.py doesn't work on 3.5.2 and less (https://github.com/ethereum/web3.py/issues/1012)
+    python_requires='>=3.5.3,<3.8.0',
    install_requires=[
        'web3==4.7.2',
-        'eth-utils==1.2.0',
+        'eth-utils==1.3.0',
        'eth-abi==1.2.0',
        # TODO: This has to be removed when "ModuleNotFoundError: No module named 'eth_utils.toolz'" is fixed at eth-abi
        'python-dateutil==2.7.0',
        'click==6.7',
        'ethereum-dasm==0.1.4'
    ],
+    extras_require={
+        'dev': [
+            'pytest~=3.2.0',
+        ],
+    },
    entry_points={
        'console_scripts': [
            'ethereumetl=ethereumetl.cli:cli',
--- a/tests/ethereumetl/job/test_export_blocks_job.py
+++ b/tests/ethereumetl/job/test_export_blocks_job.py
@@ -46,8 +46,8 @@ def read_resource(resource_group, file_name):
    skip_if_slow_tests_disabled((47218, 47219, 2, 'blocks_with_transactions', 'infura')),
 ])
 def test_export_blocks_job(tmpdir, start_block, end_block, batch_size, resource_group, web3_provider_type):
-    blocks_output_file = tmpdir.join('actual_blocks.csv')
-    transactions_output_file = tmpdir.join('actual_transactions.csv')
+    blocks_output_file = str(tmpdir.join('actual_blocks.csv'))
+    transactions_output_file = str(tmpdir.join('actual_transactions.csv'))

    job = ExportBlocksJob(
        start_block=start_block, end_block=end_block, batch_size=batch_size,
--- a/tests/ethereumetl/job/test_export_contracts_job.py
+++ b/tests/ethereumetl/job/test_export_contracts_job.py
@@ -46,7 +46,7 @@ CONTRACT_ADDRESSES_UNDER_TEST = ['0x06012c8cf97bead5deae237070f9587f8e7a266d']
 ])
 def test_export_contracts_job(tmpdir, batch_size, contract_addresses, output_format, resource_group,
                              web3_provider_type):
-    contracts_output_file = tmpdir.join('actual_contracts.' + output_format)
+    contracts_output_file = str(tmpdir.join('actual_contracts.' + output_format))

    job = ExportContractsJob(
        contract_addresses_iterable=contract_addresses,
--- a/tests/ethereumetl/job/test_export_geth_traces_job.py
+++ b/tests/ethereumetl/job/test_export_geth_traces_job.py
@@ -47,7 +47,7 @@ def read_resource(resource_group, file_name):
    (1000895, 1000895, 'block_with_error', 'mock'),
 ])
 def test_export_geth_traces_job(tmpdir, start_block, end_block, resource_group, web3_provider_type):
-    traces_output_file = tmpdir.join('actual_geth_traces.json')
+    traces_output_file = str(tmpdir.join('actual_geth_traces.json'))

    job = ExportGethTracesJob(
        start_block=start_block, end_block=end_block, batch_size=1,
--- a/tests/ethereumetl/job/test_export_receipts_job.py
+++ b/tests/ethereumetl/job/test_export_receipts_job.py
@@ -51,8 +51,8 @@ DEFAULT_TX_HASHES = ['0x04cbcb236043d8fb7839e07bbc7f5eed692fb2ca55d897f1101eac3e
    skip_if_slow_tests_disabled((2, DEFAULT_TX_HASHES, 'json', 'receipts_with_logs', 'infura'))
 ])
 def test_export_receipts_job(tmpdir, batch_size, transaction_hashes, output_format, resource_group, web3_provider_type):
-    receipts_output_file = tmpdir.join('actual_receipts.' + output_format)
-    logs_output_file = tmpdir.join('actual_logs.' + output_format)
+    receipts_output_file = str(tmpdir.join('actual_receipts.' + output_format))
+    logs_output_file = str(tmpdir.join('actual_logs.' + output_format))

    job = ExportReceiptsJob(
        transaction_hashes_iterable=transaction_hashes,
--- a/tests/ethereumetl/job/test_export_token_transfers_job.py
+++ b/tests/ethereumetl/job/test_export_token_transfers_job.py
@@ -42,7 +42,7 @@ def read_resource(resource_group, file_name):
    (483920, 483920, 1, 'block_with_transfers', 'mock')
 ])
 def test_export_token_transfers_job(tmpdir, start_block, end_block, batch_size, resource_group, web3_provider_type):
-    output_file = tmpdir.join('token_transfers.csv')
+    output_file = str(tmpdir.join('token_transfers.csv'))

    job = ExportTokenTransfersJob(
        start_block=start_block, end_block=end_block, batch_size=batch_size,
--- a/tests/ethereumetl/job/test_export_tokens_job.py
+++ b/tests/ethereumetl/job/test_export_tokens_job.py
@@ -46,7 +46,7 @@ def read_resource(resource_group, file_name):
    )
 ])
 def test_export_tokens_job(tmpdir, token_addresses, resource_group, web3_provider_type):
-    output_file = tmpdir.join('tokens.csv')
+    output_file = str(tmpdir.join('tokens.csv'))

    job = ExportTokensJob(
        token_addresses_iterable=token_addresses,
--- a/tests/ethereumetl/job/test_export_traces_job.py
+++ b/tests/ethereumetl/job/test_export_traces_job.py
@@ -46,7 +46,7 @@ def read_resource(resource_group, file_name):
    (1000895, 1000895, 'block_with_error', 'mock'),
 ])
 def test_export_traces_job(tmpdir, start_block, end_block, resource_group, web3_provider_type):
-    traces_output_file = tmpdir.join('actual_traces.csv')
+    traces_output_file = str(tmpdir.join('actual_traces.csv'))

    job = ExportTracesJob(
        start_block=start_block, end_block=end_block, batch_size=1,
--- a/tests/ethereumetl/job/test_extract_geth_traces_job.py
+++ b/tests/ethereumetl/job/test_extract_geth_traces_job.py
@@ -44,7 +44,7 @@ def read_resource(resource_group, file_name):
    'block_with_error',
 ])
 def test_extract_traces_job(tmpdir, resource_group):
-    output_file = tmpdir.join('actual_traces.csv')
+    output_file = str(tmpdir.join('actual_traces.csv'))

    geth_traces_content = read_resource(resource_group, 'geth_traces.json')
    traces_iterable = (json.loads(line) for line in geth_traces_content.splitlines())
--- a/tests/ethereumetl/job/test_extract_token_transfers_job.py
+++ b/tests/ethereumetl/job/test_extract_token_transfers_job.py
@@ -41,7 +41,7 @@ def read_resource(resource_group, file_name):
    'logs'
 ])
 def test_export_token_transfers_job(tmpdir, resource_group):
-    output_file = tmpdir.join('token_transfers.csv')
+    output_file = str(tmpdir.join('token_transfers.csv'))

    logs_content = read_resource(resource_group, 'logs.csv')
    logs_csv_reader = csv.DictReader(io.StringIO(logs_content))
--- a/tests/ethereumetl/service/test_eth_service.py
+++ b/tests/ethereumetl/service/test_eth_service.py
@@ -22,11 +22,9 @@


 import pytest
-from click.testing import CliRunner
 from dateutil.parser import parse
 from web3 import HTTPProvider, Web3

-from ethereumetl.cli import get_block_range_for_date
 from ethereumetl.service.eth_service import EthService
 from ethereumetl.service.graph_operations import OutOfBoundsError
 from tests.helpers import skip_if_slow_tests_disabled
@@ -41,10 +39,10 @@ from tests.helpers import skip_if_slow_tests_disabled
    ('2018-06-10', 5761663, 5767303)
 ])
 def test_get_block_range_for_date(date, expected_start_block, expected_end_block):
-    runner = CliRunner()
-    result = runner.invoke(get_block_range_for_date, ['--date', date])
-    assert 0 == result.exit_code
-    assert '{},{}\n'.format(expected_start_block, expected_end_block) == result.output
+    eth_service = get_new_eth_service()
+    parsed_date = parse(date)
+    blocks = eth_service.get_block_range_for_date(parsed_date)
+    assert blocks == (expected_start_block, expected_end_block)


@skip_if_slow_tests_disabled
--- a/tests/helpers.py
+++ b/tests/helpers.py
@@ -22,15 +22,26 @@


 import os
+import json

 import pytest


+def sort_json(json_string):
+    return json.dumps(json.loads(json_string), sort_keys=True)
+
+
 def compare_lines_ignore_order(expected, actual):
    expected_lines = expected.splitlines()
    actual_lines = actual.splitlines()
    assert len(expected_lines) == len(actual_lines)

+    try:
+        expected_lines = [sort_json(line) for line in expected_lines]
+        actual_lines = [sort_json(line) for line in actual_lines]
+    except json.decoder.JSONDecodeError:
+        pass
+
    for expected_line, actual_line in zip(sorted(expected_lines), sorted(actual_lines)):
        assert expected_line == actual_line

--- a/tox.ini
+++ b/tox.ini
@@ -0,0 +1,16 @@
+[tox]
+envlist=
+    py{35,36,37}
+
+[testenv]
+usedevelop=True
+setenv=
+    ETHEREUM_ETL_RUN_SLOW_TESTS=True
+commands=
+    pytest {posargs}
+deps=
+    .[dev]
+basepython=
+    py35: python3.5
+    py36: python3.6
+    py37: python3.7
Author	SHA1	Message	Date
Evgeny Medvedev	930efe5a0e	Bump version	2019-03-03 14:48:23 +07:00
Evgeny Medvedev	aac00bf7d0	Add documentation to commands	2019-03-03 14:48:10 +07:00
Evgeny Medvedev	6f19ff0756	Merge pull request #157 from blockchain-etl/feature/remove-legacy-scripts Remove legacy files	2019-03-03 14:38:20 +07:00
Evgeny Medvedev	f18f303fa9	Merge pull request #156 from tpmccallum/patch-3 Update setup.py	2019-03-03 14:37:25 +07:00
Evgeny Medvedev	b5e290e2c1	Remove legacy files	2019-03-03 14:34:34 +07:00
Timothy McCallum	a10fb2fac9	Update eth utils in setup.py Updated from eth-utils>=1.2.0 to eth-utils==1.3.0 Also ran all of the installation and tests again and everything passed!	2019-03-03 17:34:23 +10:00
Evgeny Medvedev	83a7b5383f	Merge pull request #155 from tpmccallum/patch-2 Changing python and pip -> python3 and pip3	2019-03-03 14:12:11 +07:00
Timothy McCallum	978513efc0	Update setup.py I was getting this error ``` eth-keys 0.2.1 has requirement eth-utils<2.0.0,>=1.3.0, but you'll have eth-utils 1.2.0 which is incompatible. ``` It relates to the following issue https://github.com/blockchain-etl/ethereum-etl/issues/141 which has the following fix `bde116ad06` I just tested it and also created this PR which you can now merge.	2019-03-03 13:48:24 +10:00
Timothy McCallum	65f5de1df1	Changing python and pip -> python3 and pip3	2019-03-03 13:41:49 +10:00
Evgeny Medvedev	df10702486	Add links to articles	2019-02-27 20:00:06 +07:00
Evgeny Medvedev	a288b51b73	Merge pull request #152 from blockchain-etl/feature/export_traces_timeout_option Add timeout option to export_traces	2019-02-27 19:55:51 +07:00
Evgeny Medvedev	a6337d0817	Add timeout option to export_traces	2019-02-21 18:29:13 +07:00
Evgeny Medvedev	4fc495342b	Update LICENSE	2019-02-04 15:44:15 +07:00
Evgeny Medvedev	b0a5e02dd5	Update README	2019-02-04 15:25:52 +07:00
Evgeny Medvedev	f7af95d6c7	Update the version	2019-02-04 15:25:30 +07:00
Evgeny Medvedev	706eb8a9c9	Refactor misc utils	2019-01-30 16:04:23 +07:00
Evgeny Medvedev	3b866f4f32	Provide default value for chain argument for cli command callbacks	2019-01-17 17:48:20 +07:00
Evgeny Medvedev	d437f58eb9	Add exception logging to EthTokenService	2019-01-17 17:46:56 +07:00
medvedev1088	ecea237187	Merge pull request #115 from blockchain-etl/bug/value-error-export-tokens Fix ValueError when exporting contracts	2019-01-17 13:52:35 +07:00
Evgeny Medvedev	aa1a0ee32a	Update README	2018-12-13 16:29:59 +07:00
Evgeny Medvedev	4c3d67d442	Update README	2018-12-13 16:29:06 +07:00
Evgeny Medvedev	1e793f3d48	Add Ethereum Classic Support to table of contents	2018-11-18 23:05:45 +07:00
Evgeny Medvedev	3876957917	Update README	2018-11-18 23:03:37 +07:00
Evgeny Medvedev	76879e593d	Fix typo	2018-11-18 23:01:34 +07:00
medvedev1088	f9b353d803	Merge pull request #131 from YazzyYaz/develop Add chain command line argument for EVM Agnostic	2018-11-18 23:00:31 +07:00
Yaz Khoury	fb2c7fb149	feat: Default classic chain to https://ethereumclassic.network if not specified	2018-11-16 14:48:26 -05:00
Yaz Khoury	21808fb1c8	doc: Update README with classic info	2018-11-16 14:05:49 -05:00
Yaz Khoury	a4a15cb534	refactor: Move classic infura check function into utils.py	2018-11-16 14:01:50 -05:00
Yaz Khoury	04aa34dca4	feat: Add cli arg for chain network type	2018-11-16 12:33:24 -05:00
Evgeny Medvedev	5c98d95a5a	Bump version and update README	2018-11-16 01:51:36 +07:00
medvedev1088	49faafa3e0	Merge pull request #127 from evgeniuz/develop Python 3.5/3.7 compatibility and tox integration	2018-11-16 00:52:17 +07:00
Evgeny Medvedev	c8202d9533	Change export_all command description	2018-11-14 13:05:18 +07:00
Evgeniy Filatov	01c1792ca5	replaced f-strings with str.format	2018-11-12 16:12:06 +02:00
Evgeniy Filatov	32e7f593be	python 3.7 is not available on trusty	2018-11-12 15:53:12 +02:00
Evgeniy Filatov	538d841906	fixed typo	2018-11-12 15:49:37 +02:00
Evgeniy Filatov	3050f50893	updated travis to use tox	2018-11-12 15:47:32 +02:00
Evgeniy Filatov	49c6f042d7	added comment about minimum python version	2018-11-12 15:30:43 +02:00
Evgeniy Filatov	320f592e51	added tox tests, fixed some incompatibilities with python 3.5	2018-11-12 15:25:45 +02:00
medvedev1088	c0c8fd5845	Merge pull request #126 from blockchain-etl/feature/genesis-allocations Feature/genesis allocations	2018-11-10 23:30:31 +07:00
Evgeny Medvedev	7b9276c5a2	Add licence header	2018-11-10 23:26:01 +07:00
Evgeny Medvedev	e5e15b262d	Added daofork traces	2018-11-10 23:15:25 +07:00
Evgeny Medvedev	4092ce92b9	Add genesis traces support	2018-11-10 21:37:13 +07:00
Evgeny Medvedev	819f26e09e	Print deprecation warning to stderr	2018-11-09 17:02:38 +07:00
medvedev1088	b500542437	Merge pull request #125 from blockchain-etl/cli-changes Prepare for pip	2018-11-09 16:33:08 +07:00
Evgeny Medvedev	652193a2f2	Update Dockerfile	2018-11-09 16:23:10 +07:00
Evgeny Medvedev	7ecdfa4fb7	Fix ValueError when exporting contracts https://github.com/blockchain-etl/ethereum-etl/issues/113	2018-10-23 23:32:54 +07:00