{ "cells": [ { "cell_type": "markdown", "id": "abb2e8b6", "metadata": {}, "source": [ "# 03 · Large Data on Limited RAM\n", "\n", "Sooner or later a dataset will not fit comfortably in memory: a laptop has 8 to 16 GB, a small cloud\n", "server 2 to 4 GB, and pandas often needs several times the file size while loading. This module shows how\n", "to measure memory honestly and the techniques, and the alternative libraries, that keep it under control.\n", "\n", "**You will learn**\n", "1. How to measure memory: object sizes, and the **peak** memory of a whole process\n", "2. Shrinking a DataFrame: numeric downcasting and categoricals\n", "3. Streaming: process data in chunks or with generators instead of loading everything\n", "4. Columnar files: read only the columns and rows you need\n", "5. Alternatives to pandas: **Polars**, **DuckDB**, **PyArrow** (and when Dask or Spark make sense)\n", "6. A benchmark: the same query six ways, timed and with peak memory measured\n", "7. **PySpark** for data beyond one machine: the same queries with the DataFrame API and with SQL" ] }, { "cell_type": "markdown", "id": "41795620", "metadata": {}, "source": [ "## 1. Measuring memory\n", "`sys.getsizeof` gives the size of one Python object (not what it contains); `DataFrame.memory_usage(deep=True)`\n", "gives the real size of a table, strings included; `psutil` reports what the whole process uses." ] }, { "cell_type": "code", "execution_count": 1, "id": "1ddc28f8", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:00:00.737903Z", "iopub.status.busy": "2026-10-18T01:00:00.737283Z", "iopub.status.idle": "2026-10-18T01:00:01.845244Z", "shell.execute_reply": "2026-10-18T01:00:01.841049Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "this kernel uses 131 MB right now\n", "28 bytes for an int, 24 for a float, 42 for a 1-character str\n", "a NumPy array of 1M float64: 8 MB; a list of 1M floats: ~32 MB\n" ] } ], "source": [ "import sys, os, psutil\n", "import numpy as np\n", "import pandas as pd\n", "\n", "proc = psutil.Process(os.getpid())\n", "print(f\"this kernel uses {proc.memory_info().rss / 1e6:.0f} MB right now\")\n", "\n", "print(sys.getsizeof(1), \"bytes for an int,\", sys.getsizeof(1.0), \"for a float,\", sys.getsizeof(\"a\"), \"for a 1-character str\")\n", "print(f\"a NumPy array of 1M float64: {np.zeros(1_000_000).nbytes / 1e6:.0f} MB; a list of 1M floats: \"\n", " f\"~{(sys.getsizeof([0.0] * 1_000_000) + 1_000_000 * sys.getsizeof(0.0)) / 1e6:.0f} MB\")" ] }, { "cell_type": "markdown", "id": "04fa256a", "metadata": {}, "source": [ "## 2. Shrinking a DataFrame\n", "pandas defaults to 64-bit numbers and Python strings. Most columns need far less." ] }, { "cell_type": "code", "execution_count": 2, "id": "3c51f303", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:00:01.853759Z", "iopub.status.busy": "2026-10-18T01:00:01.852179Z", "iopub.status.idle": "2026-10-18T01:00:04.129897Z", "shell.execute_reply": "2026-10-18T01:00:04.126543Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'sensor_id': dtype('int64'), 'temperature': dtype('float64'), 'humidity': dtype('int64'), 'status': dtype('O'), 'site': dtype('O')}\n", "{'sensor_id': dtype('int16'), 'temperature': dtype('float32'), 'humidity': dtype('uint8'), 'status': CategoricalDtype(categories=['fault', 'ok', 'warning'], ordered=False, categories_dtype=object), 'site': CategoricalDtype(categories=['site_0', 'site_1', 'site_10', 'site_11', 'site_12',\n", " 'site_13', 'site_14', 'site_15', 'site_16', 'site_17',\n", " 'site_18', 'site_19', 'site_2', 'site_20', 'site_21',\n", " 'site_22', 'site_23', 'site_24', 'site_25', 'site_26',\n", " 'site_27', 'site_28', 'site_29', 'site_3', 'site_30',\n", " 'site_31', 'site_32', 'site_33', 'site_34', 'site_35',\n", " 'site_36', 'site_37', 'site_38', 'site_39', 'site_4',\n", " 'site_5', 'site_6', 'site_7', 'site_8', 'site_9'],\n", ", ordered=False, categories_dtype=object)}\n", "before 131 MB -> after 9 MB (14.6x smaller)\n" ] } ], "source": [ "rng = np.random.default_rng(0)\n", "n = 1_000_000\n", "df = pd.DataFrame({\n", " \"sensor_id\": rng.integers(1, 500, n), # fits in int16\n", " \"temperature\": rng.normal(21, 4, n), # float32 is plenty for a sensor\n", " \"humidity\": rng.integers(0, 101, n), # fits in uint8\n", " \"status\": rng.choice([\"ok\", \"warning\", \"fault\"], n, p=[0.9, 0.08, 0.02]), # 3 distinct values\n", " \"site\": rng.choice([f\"site_{i}\" for i in range(40)], n),\n", "})\n", "\n", "def mb(frame):\n", " return frame.memory_usage(deep=True).sum() / 1e6\n", "\n", "before = mb(df)\n", "small = df.assign(\n", " sensor_id=pd.to_numeric(df[\"sensor_id\"], downcast=\"integer\"),\n", " temperature=df[\"temperature\"].astype(\"float32\"),\n", " humidity=pd.to_numeric(df[\"humidity\"], downcast=\"unsigned\"),\n", " status=df[\"status\"].astype(\"category\"),\n", " site=df[\"site\"].astype(\"category\"),\n", ")\n", "print(df.dtypes.to_dict()); print(small.dtypes.to_dict())\n", "print(f\"before {before:.0f} MB -> after {mb(small):.0f} MB ({before / mb(small):.1f}x smaller)\")" ] }, { "cell_type": "markdown", "id": "1a6ce843", "metadata": {}, "source": [ "A **category** stores each distinct string once and keeps small integer codes per row. It is the single\n", "biggest win for text columns with few distinct values, and groupby on it is faster too:" ] }, { "cell_type": "code", "execution_count": 3, "id": "0689b62b", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:00:04.135019Z", "iopub.status.busy": "2026-10-18T01:00:04.134417Z", "iopub.status.idle": "2026-10-18T01:00:05.138971Z", "shell.execute_reply": "2026-10-18T01:00:05.137137Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "groupby on text: 147 ms, on category: 52 ms\n" ] } ], "source": [ "import timeit\n", "t_obj = timeit.timeit(lambda: df.groupby(\"site\")[\"temperature\"].mean(), number=5) / 5\n", "t_cat = timeit.timeit(lambda: small.groupby(\"site\", observed=True)[\"temperature\"].mean(), number=5) / 5\n", "print(f\"groupby on text: {t_obj * 1000:.0f} ms, on category: {t_cat * 1000:.0f} ms\")" ] }, { "cell_type": "markdown", "id": "d19d35fd", "metadata": {}, "source": [ "## 3. Streaming: chunks and generators\n", "If the result is small (a sum, a count, an average) there is no need to hold all rows at once." ] }, { "cell_type": "code", "execution_count": 4, "id": "1a75113d", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:00:05.145749Z", "iopub.status.busy": "2026-10-18T01:00:05.145256Z", "iopub.status.idle": "2026-10-18T01:00:29.102764Z", "shell.execute_reply": "2026-10-18T01:00:29.099401Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CSV 104 MB, Parquet 25 MB\n" ] } ], "source": [ "import tempfile\n", "tmp = tempfile.mkdtemp()\n", "\n", "# a 3-million-row dataset, written once as CSV and as Parquet, used for the rest of the module\n", "N = 3_000_000\n", "big = pd.DataFrame({\n", " \"sensor_id\": rng.integers(1, 500, N).astype(\"int16\"),\n", " \"temperature\": rng.normal(21, 4, N).astype(\"float32\"),\n", " \"status\": pd.Categorical(rng.choice([\"ok\", \"warning\", \"fault\"], N, p=[0.9, 0.08, 0.02])),\n", " \"site\": pd.Categorical(rng.choice([f\"site_{i}\" for i in range(40)], N)),\n", " \"note\": rng.choice([\"\", \"manual check\", \"recalibrated\", \"battery low\"], N),\n", "})\n", "CSV, PARQUET = os.path.join(tmp, \"big.csv\"), os.path.join(tmp, \"big.parquet\")\n", "big.to_csv(CSV, index=False)\n", "big.to_parquet(PARQUET, index=False, row_group_size=250_000)\n", "print(f\"CSV {os.path.getsize(CSV) / 1e6:.0f} MB, Parquet {os.path.getsize(PARQUET) / 1e6:.0f} MB\")\n", "del big" ] }, { "cell_type": "code", "execution_count": 5, "id": "b73e276a", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:00:29.109275Z", "iopub.status.busy": "2026-10-18T01:00:29.108579Z", "iopub.status.idle": "2026-10-18T01:00:32.243143Z", "shell.execute_reply": "2026-10-18T01:00:32.240739Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "site_0 21.057986\n", "site_1 21.014257\n", "site_10 20.881563\n", "site_11 20.891889\n", "site_12 21.071960\n", "dtype: float64\n" ] } ], "source": [ "# mean temperature of 'fault' readings per site, reading the CSV 250,000 rows at a time\n", "sums, counts = {}, {}\n", "for chunk in pd.read_csv(CSV, chunksize=250_000, usecols=[\"site\", \"status\", \"temperature\"]):\n", " faults = chunk[chunk[\"status\"] == \"fault\"]\n", " g = faults.groupby(\"site\")[\"temperature\"]\n", " for site, s in g.sum().items():\n", " sums[site] = sums.get(site, 0.0) + s\n", " for site, c in g.count().items():\n", " counts[site] = counts.get(site, 0) + c\n", "streamed = pd.Series({k: sums[k] / counts[k] for k in sums}).sort_index()\n", "print(streamed.head())" ] }, { "cell_type": "code", "execution_count": 6, "id": "9be7250d", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:00:32.248330Z", "iopub.status.busy": "2026-10-18T01:00:32.247498Z", "iopub.status.idle": "2026-10-18T01:00:35.474496Z", "shell.execute_reply": "2026-10-18T01:00:35.471778Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "fault rows: 59,835\n" ] } ], "source": [ "# the same idea with a plain generator: a pipeline that never holds the file in memory\n", "def read_lines(path):\n", " with open(path) as f:\n", " next(f) # skip the header\n", " for line in f:\n", " yield line.rstrip(\"\\n\").split(\",\")\n", "\n", "def faults_only(rows):\n", " return (r for r in rows if r[2] == \"fault\") # columns: sensor_id, temperature, status, site, note\n", "\n", "count = sum(1 for _ in faults_only(read_lines(CSV)))\n", "print(f\"fault rows: {count:,}\")" ] }, { "cell_type": "markdown", "id": "85a26f9d", "metadata": {}, "source": [ "## 4. Columnar files: read only what you need\n", "Parquet stores each column separately and keeps min/max statistics per block of rows (a *row group*).\n", "A reader can therefore skip columns it does not need (**column pruning**) and whole row groups that\n", "cannot match a filter (**predicate pushdown**)." ] }, { "cell_type": "code", "execution_count": 7, "id": "f1e6119b", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:00:35.481146Z", "iopub.status.busy": "2026-10-18T01:00:35.480522Z", "iopub.status.idle": "2026-10-18T01:00:35.647410Z", "shell.execute_reply": "2026-10-18T01:00:35.645831Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3,000,000 rows in 12 row groups, 5 columns\n", "(59835, 2) 0.3 MB loaded instead of the whole table\n" ] } ], "source": [ "import pyarrow.parquet as pq\n", "meta = pq.ParquetFile(PARQUET).metadata\n", "print(f\"{meta.num_rows:,} rows in {meta.num_row_groups} row groups, {meta.num_columns} columns\")\n", "\n", "subset = pd.read_parquet(PARQUET, columns=[\"site\", \"temperature\"], filters=[(\"status\", \"==\", \"fault\")])\n", "print(subset.shape, f\"{mb(subset):.1f} MB loaded instead of the whole table\")" ] }, { "cell_type": "markdown", "id": "28fd918b", "metadata": {}, "source": [ "## 5. Alternatives to pandas\n", "\n", "| Library | What it is | Strengths |\n", "|---|---|---|\n", "| **Polars** | DataFrame library written in Rust, built on Apache Arrow | multi-threaded, lazy queries with an optimiser, streaming for larger-than-RAM data |\n", "| **DuckDB** | an in-process analytical SQL database | SQL directly on Parquet/CSV files, out-of-core execution, very low memory |\n", "| **PyArrow** | the Python library for Apache Arrow, the columnar memory format the others share | zero-copy exchange between tools, datasets, compute functions |\n", "| **Dask** | pandas-like API split into partitions | scales pandas code to many cores or a cluster |\n", "| **Spark (PySpark)** | distributed engine | data that needs a cluster (hundreds of GB and more) |\n", "\n", "Rule of thumb: up to a few GB, pandas with good types is fine; beyond that, or when memory is tight,\n", "Polars or DuckDB on Parquet usually solve it on one machine; reach for Dask or Spark only when one\n", "machine is truly not enough." ] }, { "cell_type": "code", "execution_count": 8, "id": "28c9da77", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:00:35.655076Z", "iopub.status.busy": "2026-10-18T01:00:35.654535Z", "iopub.status.idle": "2026-10-18T01:00:36.994975Z", "shell.execute_reply": "2026-10-18T01:00:36.992882Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "shape: (3, 2)\n", "┌─────────┬─────────────┐\n", "│ site ┆ temperature │\n", "│ --- ┆ --- │\n", "│ cat ┆ f32 │\n", "╞═════════╪═════════════╡\n", "│ site_0 ┆ 21.057985 │\n", "│ site_1 ┆ 21.014257 │\n", "│ site_10 ┆ 20.881563 │\n", "└─────────┴─────────────┘\n", " site t\n", "0 site_0 21.057986\n", "1 site_1 21.014257\n", "2 site_10 20.881563\n" ] } ], "source": [ "import polars as pl, duckdb\n", "\n", "print(pl.scan_parquet(PARQUET) # lazy: nothing is read yet\n", " .filter(pl.col(\"status\") == \"fault\")\n", " .group_by(\"site\").agg(pl.col(\"temperature\").mean())\n", " .sort(\"site\").head(3).collect())\n", "\n", "print(duckdb.sql(f\"select site, avg(temperature) as t from '{PARQUET}' where status = 'fault' \"\n", " \"group by site order by site limit 3\").df())" ] }, { "cell_type": "markdown", "id": "4e9e627b", "metadata": {}, "source": [ "## 6. Benchmark: one query, six ways\n", "Each approach runs in a **fresh Python process** so its peak memory (the operating system's maximum\n", "resident set size) is measured on its own, without anything this notebook already holds. Native\n", "libraries allocate memory outside Python, so this is the only fair way to compare them." ] }, { "cell_type": "code", "execution_count": 9, "id": "a4268749", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:00:37.000914Z", "iopub.status.busy": "2026-10-18T01:00:37.000324Z", "iopub.status.idle": "2026-10-18T01:00:58.732706Z", "shell.execute_reply": "2026-10-18T01:00:58.731475Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "mean temperature of fault readings per site; peak memory above a bare 'import pandas' process (99 MB)\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
secondspeak_mb
approach
pandas read_csv (all columns)4.20229.27
pandas read_csv (3 columns, typed)2.8637.41
pandas read_csv in chunks3.4442.76
pandas read_parquet (pruned + filtered)0.21108.68
Polars lazy scan0.91101.42
DuckDB SQL0.2666.62
\n", "
" ], "text/plain": [ " seconds peak_mb\n", "approach \n", "pandas read_csv (all columns) 4.20 229.27\n", "pandas read_csv (3 columns, typed) 2.86 37.41\n", "pandas read_csv in chunks 3.44 42.76\n", "pandas read_parquet (pruned + filtered) 0.21 108.68\n", "Polars lazy scan 0.91 101.42\n", "DuckDB SQL 0.26 66.62" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import subprocess, json, textwrap\n", "\n", "QUERY = \"mean temperature of fault readings per site\"\n", "approaches = {\n", " \"pandas read_csv (all columns)\": f\"\"\"\n", "df = pd.read_csv({CSV!r})\n", "r = df[df.status == 'fault'].groupby('site').temperature.mean()\"\"\",\n", " \"pandas read_csv (3 columns, typed)\": f\"\"\"\n", "df = pd.read_csv({CSV!r}, usecols=['site', 'status', 'temperature'], dtype={{'site': 'category', 'status': 'category', 'temperature': 'float32'}})\n", "r = df[df.status == 'fault'].groupby('site', observed=True).temperature.mean()\"\"\",\n", " \"pandas read_csv in chunks\": f\"\"\"\n", "parts = [c[c.status == 'fault'] for c in pd.read_csv({CSV!r}, usecols=['site', 'status', 'temperature'], chunksize=250_000)]\n", "r = pd.concat(parts).groupby('site').temperature.mean()\"\"\",\n", " \"pandas read_parquet (pruned + filtered)\": f\"\"\"\n", "df = pd.read_parquet({PARQUET!r}, columns=['site', 'temperature'], filters=[('status', '==', 'fault')])\n", "r = df.groupby('site', observed=True).temperature.mean()\"\"\",\n", " \"Polars lazy scan\": f\"\"\"\n", "import polars as pl\n", "r = pl.scan_parquet({PARQUET!r}).filter(pl.col('status') == 'fault').group_by('site').agg(pl.col('temperature').mean()).collect()\"\"\",\n", " \"DuckDB SQL\": f\"\"\"\n", "import duckdb\n", "r = duckdb.sql(\"select site, avg(temperature) from '{PARQUET}' where status = 'fault' group by site\").df()\"\"\",\n", "}\n", "\n", "def run(snippet):\n", " # Peak memory = VmHWM (\"high water mark\") from /proc/self/status, in kB. Unlike getrusage's\n", " # ru_maxrss it belongs to the new program only: Linux carries ru_maxrss over from the parent\n", " # (this notebook) through fork and exec, which would hide every difference. (Linux only.)\n", " prog = (\"import time, pandas as pd\\nt0 = time.perf_counter()\\n\" + textwrap.dedent(snippet) +\n", " \"\\nhwm = next(l for l in open('/proc/self/status') if l.startswith('VmHWM')).split()[1]\"\n", " \"\\nprint(time.perf_counter() - t0, hwm)\")\n", " out = subprocess.run([sys.executable, \"-c\", prog], capture_output=True, text=True, check=True).stdout.split()\n", " return float(out[-2]), int(out[-1]) / 1024 # seconds, MB\n", "\n", "baseline_mb = run(\"pass\")[1] # what an empty python + pandas import costs\n", "bench = pd.DataFrame([dict(approach=k, seconds=s, peak_mb=m - baseline_mb) for k, (s, m) in\n", " ((k, run(v)) for k, v in approaches.items())]).set_index(\"approach\").round(2)\n", "print(f\"{QUERY}; peak memory above a bare 'import pandas' process ({baseline_mb:.0f} MB)\")\n", "bench" ] }, { "cell_type": "code", "execution_count": 10, "id": "2e8ea540", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:00:58.738434Z", "iopub.status.busy": "2026-10-18T01:00:58.737947Z", "iopub.status.idle": "2026-10-18T01:01:00.525586Z", "shell.execute_reply": "2026-10-18T01:01:00.522727Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABEIAAAFeCAYAAACB7ro+AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjIsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvgI3uAAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAebpJREFUeJzt3XdYFNf7NvCbXqRLVRBQLChYUSyoWLD3hr0Se4+JPWjUr9FYYknUWKLGgi0qdkWxgBV7AUFFQUABqUqHef/gZX6stAXRBff+XBdXMmfOnPPM2ZVhnz1zRkEQBAFERERERERERHJAUdYBEBERERERERF9K0yEEBEREREREZHcYCKEiIiIiIiIiOQGEyFEREREREREJDeYCCEiIiIiIiIiucFECBERERERERHJDSZCiIiIiIiIiEhuMBFCRERERERERHJDWdYBEBGVNVlZWQgPD4e2tjYUFBRkHQ4RERVBEAQkJiaiUqVKUFT88u/5eB0gIip/inMtYCKEiOgz4eHhsLCwkHUYRERUTKGhoTA3N//idngdICIqv6S5FjARQkT0GW1tbQDZv0R1dHRkHA0RERUlISEBFhYW4u/vL8XrABFR+VOcawETIUREn8mZBq2jo8M/gImIypHSuo2F1wEiovJLmmsBF0slIiIiIiIiIrnBRAgRERERERERyQ0mQoiIiIiIiIhIbjARQkRERERERERyg4kQIiIiIiIiIpIbTIQQERERERERkdxgIoSIiIiIiIiI5AYTIUREREREREQkN5gIISIiIiIiIiK5oSzrAIiIyirzg1FQ0EyRdRhERN+9+MHGsg4hX4N6R0BF+aOswyAikgvHzlX6Zn1xRggRERERERERyQ0mQoiIiIiIiIhIbjARQkRERERERERyg4kQIiIiIiIiIpIbTIQQERERERERkdxgIoQKde/ePUycOFHWYZSqbdu2Ydu2bbIOI19ZWVmYPn06QkJCpD7mzz//xK5duwrc/hq+RR+lZd26dbh06ZKswyAiIiIiojKCiRAqVEhICPbt2yfrMEqVj48PfHx8ZB1GvrZv34579+6hSpUqUh/j7e2NGzduFLj9NXyLPkpL7dq1MXbsWKSmpso6FCIiIiIiKgOYCCEqIzIzM7Fs2TLMmDFD1qF8V1xcXKCiooL9+/fLOhQiIiIiIioDlGUdABXO29sbZ8+exZgxY3Dw4EGEhYWhdevWGDhwoFjn2bNnWLlyJQBAQ0MDNWrUwOjRo6Grq5unnfHjx8PDwwNhYWFwdHTE0KFDoaCgINYLDw/Hli1bEBsbi4YNG0JbW1siHmn6EgQBnp6euHLlCpSUlNCxY0e0b9++0PMbO3YsduzYgYiICKxfvx5aWloIDQ3F7t27ERoaCmtra4wYMQKmpqbisXv27IGXlxcAwMDAAI6OjhgwYIDE+QCAr68vDhw4AC0tLXTq1KlY4w8Ab9++xb///ouQkBDY2dlhzJgxUFdXBwAkJiZi165d8Pf3R6VKlTBo0CBUrVoVZ8+ehZeXF1atWiXR1r59+/Dq1SssWLAgTz/nz59HTEwMunXrVuxzLM1zAoBHjx5h//79iImJgZ2dHUaPHo0KFSoU2N7ixYvh4OCArl27imV///03FBUV4ebmBgBYvXo1zM3NUaFCBfj4+CAlJQVDhw6Fg4MDPD09cf78eejo6MDNzQ1Vq1YV28k5zsjICBcuXEBGRgb69esHR0dHsU5Br0OOgQMHYuvWrRg5cuQXjRsREREREZV/nBFSxj1//hx//vkn2rVrB0EQUKlSJYwfPx5z584V6+jr68PZ2RnOzs6ws7PDhQsXULduXSQkJORpp0uXLgAACwsLzJgxAwsXLhTrREZGwsHBAb6+vrCxscGxY8cwefJkiXik6WvJkiWYOHEijI2NUblyZaxatQobN24s9Pw6deoETU1NtGrVCioqKrh58yYaNGiA0NBQ1KtXD4GBgahTpw4CAwPFY6tXry7GYmpqivnz52P06NES7Xt6esLZ2RmCIMDQ0BDjx4/HmTNnpB5/X19f1K5dGzdv3oStrS0CAgLQt29fANkJH2dnZxw6dAh16tRBeno6+vTpg8DAQFhaWmL16tV49uyZRHuLFi0qsC8vLy80adIEKioqxTrH4irsnADg7NmzcHBwwIcPH2Bra4sdO3agadOmhd5acuLECTx+/Fii7OrVqxK3IF24cAGTJk3CH3/8AXNzc0RGRqJ58+YYMGAA/vrrL9SqVQuBgYFwdHREfHy8xHEzZszAkiVLUKlSJSQnJ8PJyQm3bt0CUPjrkKNly5a4deuWRLtERERERCSfOCOkHPj06RMOHz4szmZo0KAB+vbti4kTJ8LCwgJmZmYS33RPnDgRDg4O+OeffzBt2jSxPCkpCUePHkWtWrUAADo6Oli6dCmWLl0KAPj9999hZGSEc+fOQUlJCVOnTkWbNm1w//59sQ1p+jp69Cjmz58vLrI6ffp0hIeHF3p+Hh4eaNSokVg2ZswYzJ07Fz/++KNE3V9++QUeHh4AAEdHR4lZAa6urrC2toa7uzusrKwgCAJmzZqF2bNni+c4ePBgWFtbFzHi/2f06NHo168fduzYIZblnEtISAju3buHsLAwVKpUCQDw888/Iy0tDXp6emjQoAH27t2LZcuWAQBu376NFy9eYMiQIfn2FRAQkCe2os6xJAo7JwCYOnUqJk+ejDVr1gDIfi2sra3x119/ffFtO2ZmZjh//jwUFRUxceJEXLt2Da9evcKdO3egoKCA8ePHw9TUFKdOncLgwYPF4wwMDODl5QUlJSUAQFBQEHbv3g1HR8dCX4cc1tbWyMzMRFBQEBwcHPLElZqaKpHoyZ3YIyKi7x+vA0RE8oWJkHKgQoUKErd0dO3aFcrKyrh58yYsLCwAAA8fPsTRo0cRHh6OtLQ0xMfHIyAgQKKdSpUqiUkQAKhZsyYiIiKQlZUFRUVFXLlyBb169RI/bAJA//79JRIh0vRlZ2eH7du3w8bGBi1btoSGhob4ATU/FStWlEiCvH79Gs+ePYOPjw/8/f0hCAIEQYC/vz9iY2PFellZWTh+/Dhu3ryJqKgoZGVlQVlZGc+fP4eVlRUiIiIQFBQkMdvB1NQULVu2LHLMAeDVq1cIDAzEP//8k2ccAcDExASGhoZYuHAhpk2bBjs7O2hqakJTUxMAMHToUGzYsAFLly6FgoIC9u7di+bNmxeYiElMTMxz+0lR51hcRZ1TzpjlvvVKW1sb3bp1w5UrV744EdK6dWsoKmZPRFNUVIS1tTWaNGki3uqjrKyMKlWqICwsTOI4Z2dnifdlzZo18ebNGwBFvw4AoKWlBSB7jPOzfPlyLF68+IvOjYiIyi9eB4iI5AtvjSkH9PX1JbYVFBSgr6+PqKgoAMDBgwfRtGlTfPjwAfXq1YOzszNMTEzyfOjLvQYEkP1BVBAEZGVlAQCioqJgYGAgUefzbWn62rRpE7p3747Zs2fDwMAAXbt2zZOUyU1PT09i+8OHDwCyZ0M4OTmhZcuWaNWqFcaNG4clS5aI9YYNG4Yff/wRGhoaaNq0qfhhOSeWnPEp6pwKkpN0MTY2zne/uro6rl69CkEQ0LVrV5iYmGD69OlITk4GAAwaNAghISG4fv06MjMzceDAAQwdOrTA/ipWrCiR6JHmHIurqHPKGbOKFStKlBsaGiIyMrJEfeaW33swv7LMzMwij8upU9TrAABxcXEA8p5Xjrlz5yI+Pl78CQ0NLdH5ERFR+cTrABGRfOGMkHIgMjISqampUFNTAwCkpKQgMjJSnA2yZcsWzJw5U7wFAwB27txZ7H6qVKmCkJAQibKcb91zSNOXlpYWFi1ahEWLFiEqKgpjx47F8OHDcfv2baniMDc3B5D9rX/v3r3zrZOQkID9+/fj1q1baNy4MYDsD/mjRo2SOB8g+xYWS0tLiXOqWbOm1HG8ePECNjY2+dbJWUMDAB48eIBu3brB2NgY8+bNg5mZGdq2bYu9e/fi48ePiI2NxYABAwrsr379+jhx4kSxzrG4ijqnnHF6/fo1qlWrJpYHBwdLjOHn1NTUJG5FyYnVyMioxLEWR2GvAwA8efIEmpqaEjOiclNTUxP/fRERkfzhdYCISL5wRkg5kJaWhj///FPcXr9+PXR0dODs7Awge4ZI7pkEV69exbVr14rdT58+fbB3717xm//4+Hhs27ZNoo40fR04cED8tt7IyAiOjo7iN/LSMDExQYcOHfDrr79K9BUdHY3z58+LcQCQ2J87OQNkz6Rp06YN1q1bJ8568fX1FRfZlCaO9u3b43//+x+SkpLE8nPnzgHITqj4+vqK5fXr14e5ubnEuQ4ZMgQHDx7Ezp070blz50Jno3Tt2hUPHz4Uz0macyyuos5JV1cX7du3xx9//CG+hv7+/jh58iT69etXYLvVq1eXWBj1xYsXuHz58hfFKi1pXofLly/DxcUFqqqq3yQmIiIiIiIquzgjpBwwNDTErl27cPr0aQiCAB8fH+zcuVN8tO1PP/2E3r17IzAwEGpqarh9+zZsbW2L3c/48eNx9OhR1K1bF82bN8e9e/dgbW0t3i4hbV83btzA7Nmz0aBBA6SkpOD69evYsmVLsWLZuXMn+vTpg5o1a6JFixaIj4/HmzdvxMfRamtrY9q0aejXrx86dOiAN2/eID09HRoaGhLtrFu3Du3atUODBg1gZWWFBw8ewN7eXuo4/vnnH3Tr1g01atSAo6MjgoKC0K5dO3Ts2BFqamqYM2cOEhMTUatWLQQFBSEmJkZcJBaAuKith4cHDh06VGhfDg4OaNSoEfbu3YvJkydLfY7FVdg5AcCff/4JFxcX1K1bF9WrV4e3tzcGDx4ssdbK52bNmoXWrVujadOmMDMzw7NnzyRmlHxNRb0OGRkZ8PDwwL///vtN4iEiIiIiorJNQRAEQdZBUME2b96MVatW4enTp7h9+zbCw8PRtGnTPLcpvH37Fjdv3oSqqiqcnJzw7NkzCIIgLgz6/PlzPH36FH369BGPeffuHc6ePYsRI0aIsw+ysrLg7e2N+Ph41KtXDyoqKvDx8ZF4gkdRfQHZTyG5ffs2VFRU4OjoCENDw3zPL7+4cvPz80NQUBDMzMzQuHHjPIuJ3r17F4GBgTAzM4OTkxMOHjyIFi1aSIxPXFwcLl68CC0tLTRp0gRPnz4FADg5ORU5/jljcv36dYSFhaFu3bp5Ej8PHz7E8+fPYWxsDCcnJygrS+YXT58+jcjISAwaNKjIabeXL1/G6NGjERAQIM5eKOocL1++DDU1NTRr1izf7ZKcU0pKCq5cuYKYmBjY2dnlSR7l10d0dDR8fX2hqakJR0dHPHz4EAoKCuI4X7hwAfr6+hJPbTl79ixMTEzQoEEDsezkyZOwtLQU+8zvuNu3byMxMRHt2rUTywp6Hf7++294eHjg0qVLhY59bgkJCdDV1YX21hdQ0NSW+jgiIiqZ+MH5r10lrZzf2/Hx8dDR0fnieHLa69I2ACrKvA4QEX0Lx84V/IANaRTnWsBESBmXkwh58eKFrEOhb+To0aNo1qwZTE1NZR3Kd+HMmTOoVatWsR6bzEQIEdG3xUQIERF9y0QIb40huRUdHY1Zs2YVuH/58uUwMzP7hhFlK2iBWCqZzp07yzoEIiIiIiIqQ5gIKePatGlT4KNO6cuoq6uLC87mR1NT89sFQ0RERERERN8EEyFlXM2aNaV61CsVn5aWFkaOHCnrMIiIiIiIiOgb4uNziYiIiIiIiEhuMBFCRERERERERHKDt8YQERXg7QCjUnn6ABERlU/7j5rxOkBE9B3ijBAiIiIiIiIikhtMhBARERERERGR3GAihIiIiIiIiIjkBhMhRERERERERCQ3mAghIiIiIiIiIrnBRAgRERERERERyQ0mQoiIiIiIiIhIbjARQkRERERERERyg4kQIiIiIiIiIpIbTIQQERERERERkdxgIoSIiIiIiIiI5AYTIUREREREREQkN5gIISIiIiIiIiK5wUQIEREREREREckNJkKIiIiIiIiISG4wEUJEREREREREcoOJECIiIiIiIiKSG0yEEBEREREREZHcYCKEiIiIiIiIiOQGEyFEREREREREJDeYCCEiIiIiIiIiucFECBERERERERHJDSZCiIiIiIiIiEhuKMs6ACKissr8YBQUNFNkHQYRSSl+sLGsQ6DvzKDeEVBR/ijrML6aY+cqyToEIiKZ4IwQIiIiIiIiIpIbTIQQERERERERkdxgIoSIiIiIiIiI5AYTIUREREREREQkN5gIISIiIiIiIiK5wUSIjJ0/fx5169aVdRilatasWZg1a5asw8hXYmIiGjVqhDdv3sg0jokTJ2L+/PkyjeF79fn778cff8SGDRtkGBEREREREZUlTITIWFJSEkJCQmQdRqmKjo5GdHS0rMPI17Jly1CnTh1YWloCAFJSUrBu3Tq0adMGdnZ26N27N7y9vb96HJGRkfjw4cNX7+dbGDt2LNzd3WUdhujz998PP/yAhQsXIioqSoZRERERERFRWcFECMmNxMREbNq0CRMmTBDLFi5ciMjISCxZsgQeHh6wt7dHhw4d4OPjI8NIy5eyntSpVasW6tevj7///lvWoRARERERURkg94mQvXv3wtnZGfv370f79u1ha2uL8ePHIz4+Xqzj6+sLKysrWFlZwdbWFj179sTNmzfzbefIkSPo2LEj7OzsMGbMmDwfEM+fP4/WrVujbt26GDlyJCIiIiT2S9NXfHw8ZsyYgYYNG6Jx48ZYsGABkpOTCz2/gwcPokWLFqhatSpiY2MBAMeOHYOLiwtq1aqFzp0749y5cxLHuru7i7E0bNgQEyZMQGRkpEQdQRCwfPly1K1bF82bN8eSJUuQkZEhxcj/n+PHj6Njx46wtbVF//79ERgYKO57+PAh+vXrhzp16sDFxQUeHh7ieTk4OCArK0uirTlz5mDo0KEF9lOhQgU0a9ZMLPvtt9+wbNkyODk5wc7ODr/++ivMzc1x/vz5EscMAJs2bUKzZs1Qs2ZN9O3bF48fPy60vR49emDTpk0SZTNmzJC4xWPEiBGYN28eZs2ahaZNm6J+/fpYt24dEhMTMXPmTNjZ2aF58+Y4cuSIRDsjRoyAu7s73N3d0axZMzRu3BirVq2CIAhinYLGuShz5szB+fPnsXv3bvG98scff6BRo0bIzMyUqPvTTz9hxIgRYky//PIL5s6dC0dHRzRs2BAbN27M035R71Fp3399+vTB7t27pTonIiIiIiL6vsl9IiQxMRHXrl3D77//Dnd3d/z999+4efOmxIfpRo0a4fLly7h8+TIOHz6MZs2aoU2bNnj16lWedjZt2oRFixZh+/btuHfvHiZNmiTWefz4Mbp37462bdtiz549qFevHqZPny4RjzR9TZ06FX5+fvjrr7+wdetWaGlpYe3atYWe38aNG7Fy5UpcunQJurq62L59OyZNmoQxY8bg6NGjGDx4MPr374+zZ8+Kx06bNk2MZfPmzYiIiEC3bt0kPkCvWLECa9asgbu7OzZu3IinT59i//79Uo//li1bMGjQIHTq1AkHDhzAwIEDMWfOHABAWloa2rdvDysrKxw6dAjz58/HyZMn4efnBxcXFzx8+BBXr14V20pPT8e2bdvQunXrfPu6cuUKmjRpIlGmpKQkse3j44OIiAi0bNmyRDEDwJo1azBv3jxMmTIFBw4cgIGBAZycnPDu3bsC2wwPD5dIvgFAVFSUxC0e79+/x4oVK6CtrY1t27ZhwoQJmD59Oho2bAgjIyN4eHigf//+GDhwoMT75f3791i6dCmysrLw999/Y/bs2fjll19w4MABAIWPc1F++uknODk5oXfv3uJ7ZciQIfD395dIWiQnJ2Pr1q1wcXERY1q2bBliY2OxZcsWzJgxA3PnzsX27dvFY6R5j0r7/mvatCkCAwMRHh5e5DkREREREdH3TUHI/alWDm3evBkTJkzAkydPUKdOHQDZCYu6deviwYMHqFevXr7Hde7cGc2bN8fChQvFdqZMmYLIyEjo6+sDADw8PDBx4kTExMQAAIYNG4YPHz7g9OnTYjsjRozA8ePHERcXV2CMn/dVv359uLm5YfLkyWKdzMzMPB/qc+KaNGkSQkNDUalSJbGumZkZ/vrrL/Tr10+su3DhQty+fTvPt+45Pn36BF1dXdy/fx/29vbIyMiAkZERVqxYgbFjxwIAUlNTUaVKFXTu3Bk7d+4s8JwAICMjAyYmJpg1axbmzp0rlmdlZUFRURGvX7+GtbU1QkNDYW5unudcO3fuDHNzc2zduhUAcPLkSfTr1w/v3r2Dnp5env5cXFxgY2OTZ+bFy5cv0a5dO3z69AkfP37E5s2bxZkLxY05MzMThoaGWLx4MaZOnQoge9ZC7dq10b17d6xcuRIA0K9fPxgaGmLz5s0AAAcHB/Tr108ioTJ06FAoKyuL49ipUydkZGTAy8tLrFOnTh1UrVoVJ06cEMssLS3h7u6O0aNHF3jcwIEDoampiR07dhQ5zkXp1asXzM3NJWZ0DB8+HElJSTh8+DAA4N9//8WUKVMQEREBDQ0NdOrUCcHBwQgICICCggIAYMmSJdixYweCg4Oleo8W5/339u1bWFhY4Pr16xIzgnKkpqYiNTVV3E5ISICFhQW0t76AgqZ2kWNARGVD/GBjWYdAMpKQkABdXV3Ex8dDR0en2McXdB3o0jYAKsrf73Xg2LlKsg6BiKjUFOdaIPczQgDAwMBATIIAgL29PfT09PDw4UMA2R8I169fD2dnZ9SoUQNWVla4du0aXr9+LdGOubm5mAQBAFNTU8TGxoq3CDx48CDPTINWrVpJbEvTV58+ffDLL79g3rx5uHTpEtLS0gr9wGpqaiomQQAgMDAQUVFR+PHHH2FjY4Nq1aqhatWq2LRpE4KCgsR6ERERmDZtGhwdHVG1alXUqVMHWVlZYixv3rxBXFycxDmpqanlmXVRkOfPnyMmJgadOnWSKFdUzH5bVqlSBQ4ODujatSvWr18v3l6Sc65Dhw7F4cOHxT9c9u7di65du+abBAGyZ4woKyvnKa9SpQouX74MLy8vzJo1CxMmTJCYaVKcmHPGxNnZWdynoKAAZ2dn8f30Jezt7SW2jYyMYGdnl6fs84VBPz/O1NRUrFPUOJeEm5sbTpw4Ic5o2bFjBwYOHAgNDQ2xjpOTk5gEAbL/Lbx+/RoJCQlSvUeL8/5TUVEBkD37JT/Lly+Hrq6u+GNhYVHicyciovKH1wEiIvmS91OhHFJVVc1Tpq6ujpSUFADZTxrZsWMHVq9ejVq1aqFChQqYOnWqxDcHQMEfHHMm3aSmpubpS11dXWJbmr5++eUXtGrVCsePH8e0adMQFhaGHTt2oFevXvn2n/vDJwDxvLZu3YoaNWpI7MudKOjWrRsqVaqEFStWoFKlSlBVVYWtra0YS85/izqngqSnp+cbXw5FRUVcu3YNBw8exIULF7Bs2TKYmZnB09MTVapUQa9evTBu3DicOnUKHTp0gKenJ/bs2VNgf2ZmZvk+OURFRQVWVlYAgHr16uHevXtYuXJlniSVNDHnjK2amppEuZqamrjvS+T3Hsuv7POJXoXVKWqcS6JVq1awtrbGnj170LNnT1y5cgUrVqyQqFPQ+yYlJUWq92hx3n85a9vkTgjmNnfuXMycOVPczvkmkIiI5AOvA0RE8oUzQpD9ISn3B+T379/j/fv3qF69OgDg7NmzGDduHPr27Ys6derAysoqz2wQaVSvXh1PnjyRKHv06JHEtrR9OTs7Y+3atXj8+DHGjRsnsahmUWxsbKCiooJXr16JC1zm/OTcGhEZGSkmBHJmp2RmZkp8mLe2toaSklKecypqYdAc1apVg4qKCu7cuVNgHXV1dQwfPhz//vuv+Jjh9evXAwAqVKiAXr16Ye/evfjvv/+gpqaGrl27FthW06ZNce/evSLj0tLSQlJSUolirlq1KpSUlPK8rg8fPhTfT/nR1tbGx48fJcq+5WOVCxvnoigpKeVJvADAmDFj8M8//2DHjh2oU6dOnpka+f1b0NHRgbGxsVTv0eK8/+7duwcjIyPY2Njkew5qamrQ0dGR+CEiIvnB6wARkXxhIgTZ6zvMnDkTaWlpSEtLw4wZM1CrVi1x0c3KlSvj6tWrSE1NRVZWFlasWCH1h/3cxo0bBw8PD1y5cgUAcP/+/TyP9JSmr2nTpiEgIABA9poV0dHRMDAwkDoObW1tjB8/Hu7u7vD19RXbOXv2rLjOg56eHjQ1NcWnp8THx2PixIkS7WhoaGD48OFYtGgRIiMjIQgC1q1bh+fPn0sdh5ubGxYuXCjeNhIdHY0lS5YAyP5Am7OgJgB8/PgRnz59kjjXIUOG4NSpU9i8eTP69++f7+yeHL1790ZwcDBevnwplv34448SCQdPT08cP34cffv2LVHM6urqcHNzw+LFi8WFOfft24erV69KLJz7OQcHB/z333/iWjEHDx78Zo/wLWqcT5w4ASsrq3xn0wDZsyxevHiRp3zEiBHw9/fHunXrxPVKcvPx8cG+ffsAZN/m8ttvv2H8+PEApHuPFuf9d+7cOfTt21fiVhwiIiIiIpJPTIQAsLCwgIaGBkxMTKCrq4t79+5h//794roP//vf/xAaGoqKFStCV1cXJ06cQLt27YrdT48ePTBr1ix06NABBgYG6NWrV55HvUrTV4sWLdCnTx8YGBhAX18fDx48kHjahjRWr16NUaNGoUuXLtDX14e+vj42bNiANm3aAMi+3WDbtm1wd3eHkZERKlWqhOrVq0NbW3LBsFWrVsHQ0BCVKlVCxYoVcfz4cXTo0EHqONauXYvu3bujWbNmqFixIuzt7cVv7atVq4bk5GTY2NjA1NQUFhYWcHR0xIwZM8TjXVxcoKurixs3bhT42NwcVapUQe/evSXGqnHjxujYsSMqVqwILS0tTJkyBStXriw0aVFYzACwcuVKcTaPvr4+pk+fjm3btqFBgwYFtjl79mwYGRnB1NQURkZG2LVrV4neYyVR1Di/ePECCgoK0NXVzff4cePGwd/fH4aGhrCyshIfJWxsbIwePXogJSUl39emd+/eWL9+PQwMDFCtWjXUq1cPv/zyi7i/qPcoIN37LzY2Fp6enhKLCxMRERERkfziU2M2b8aqVavw4sULZGZmIjo6GiYmJvnWjY6OhoqKCnR1dcVFIA0NDQFkP6Y2ISEBlStXFuunpKTg3bt34voTOZKSkpCSkgIDAwMkJSUhOjo6z1oMhfWVIy4uDioqKqhQoUKB55dfXLllZmYiMjIShoaG4oKS+e03MDCAmpoaQkJCYGhoCE1NTYl6Hz58gKamJjQ0NAqMtzBpaWmIjY0tcOxzYshvsdP3798jOTkZlpaWRX7j/+rVKzRt2hTPnj2TiE+asSxuzJ8+fUJCQgJMTEzEpFqOqKgoKCoqomLFihLl8fHxUFNTg7q6ep5xfP/+PVRVVSUW5H337h3U1dUlFoiNiIiApqammLjI77iYmBhkZGTA2FjyCQv5jXOHDh3g5uaGAQMGFDgWgiAgOjoanz59QuXKlcX3kqurKzIyMnDkyBGJ+p06dYKdnR1WrVqFjx8/IiMjo8BFbot6jwKFv//mzJkjPqZXWjkrTvOpMUTlC58aI7++9KkxBbXHp8YQEZUfxbkWMBGSKxFC8uHdu3fQ0dHJk8yh/L19+1bisbrSCgwMhJ2dHby8vPIsPJs7EfK1hYeHw8DAQOpFfAEmQojKKyZC5BcTISXDRAgRfU+Kcy3gU2Poq3n79i2cnJwK3H/x4kVUq1btG0aUzdTU9Jv3WZ6VJAni5OQEPz8/jB49Ot+n73xLBT0phoiIiIiI5JPcJ0KGDBmC7t27yzqM75KpqSkuX75c4P6Cbteh8s/DwwOampoFLuK7a9euQhe2JSIiIiIi+lrkPhGira2dZwFQKh3Kysp51kch+VDULJKC1lUhIiIiIiL62vjUGCIiIiIiIiKSG0yEEBEREREREZHckPtbY4iICvJ2gFGpPH2AiIjKp/1HzXgdICL6DnFGCBERERERERHJDSZCiIiIiIiIiEhuMBFCRERERERERHKDiRAiIiIiIiIikhtMhBARERERERGR3GAihIiIiIiIiIjkBhMhRERERERERCQ3mAghIiIiIiIiIrnBRAgRERERERERyQ0mQoiIiIiIiIhIbjARQkRERERERERyg4kQIiIiIiIiIpIbTIQQERERERERkdxgIoSIiIiIiIiI5AYTIUREREREREQkN5gIISIiIiIiIiK5wUQIEREREREREckNJkKIiIiIiIiISG4wEUJEREREREREcoOJECIiIiIiIiKSG0yEEBEREREREZHcYCKEiIiIiIiIiOSGsqwDICIqq8wPRkFBM0XWYRCRnIgfbCzrEOgzg3pHQEX5o6zDkJlj5yrJOgQioq+CM0KIiIiIiIiISG4wEUJEREREREREcqPEt8Z8/PgRz549Q0xMTJ59nTp1+qKgiIiIiIiIiIi+hhIlQs6ePYvBgwcjNjYWSkpKefZnZGR8cWBERERERERERKWtRLfGzJgxAxMnTkRCQgIyMjLy/BARERERERERlUUlSoSEhoZizpw50NbWLu14SM4kJSUhJCRE1mGUqujoaERHR8s6jDyioqLw4cOHr9rH+/fvERsb+1X7kEVfRERERET0/ShRIsTW1havX78u5VBIHp0/fx5169aVdRilatasWZg1a5asw8hjwoQJmD9//lftY8SIEVi2bNlX7UMWfRERERER0fdD6jVCcic+xo4di2HDhmHlypWwsbGBgoKCRF0rK6vSio+ISomxsTH09fVlHQYREREREZFMSZ0Isba2zlPWoUOHfOsKglDyiOibSkxMREJCAipXrozMzExER0fDxMREok5KSgrevXsHANDQ0Miz//N2srKyEBMTA0NDw3z7TE5ORnJyMgwMDPLsk6avHAkJCVBUVISWlpZU55ecnIyoqCiYm5tDUTF7MlRaWhpiYmJgYmKSJ6EXExODhIQEAICBgQF0dHQK7CcmJgYaGhrQ0NAosE5hMjIy8OHDBxgbG+eJQxAEREdHQ19fH8rK2f9kY2NjkZKSAjMzM4m68fHx+PjxIypXrpynj8WLF4vnDWTfWqKqqgp9fX0kJSUhKyur0LGUNt4cBbUZEREBTU1N6OrqimU5txLlvGdKGtu7d++QlZWFSpUqAch/7IiIiIiISL5JfWtMUFCQ1D9UfuzduxfNmjXDuHHjYGhoCCsrK9ja2uLx48dinbt378LZ2RnOzs6ws7ODnp4e1q5dm287U6ZMgbGxMapWrYpq1arhwYMHEvUWLVoEfX192NjYwMrKChcvXpTYL01fT58+RYMGDVCpUiVUqVIFTk5OCAgIKPT8Jk6cCBMTE7Rq1Qrx8fFIT0/HjBkzULFiRdSrVw86OjqYO3cusrKyxGPXrVsnxlKpUiXY2trC19dXov24uDh07NgRxsbGqFy5Mtq1a4eIiAipxz8tLQ0zZ86Enp4ebG1tYWpqin379on7d+3aBWNjY9SpUweGhoYYP348kpKScP78edSsWRPJyckS7Y0ePRozZszIt6/Pb40ZMWIEfvjhB7Ru3RqWlpYwMDBAt27dkJSUVOJ4gezZY4W12b17d2zatEnimOnTp0vcTlSS2LZt24aaNWviyZMnhY4dERERERHJN6kTITY2NlL/UPkSGhqKpKQkvH//HvHx8WjQoAEGDhwoJgVatGiB169f4/Xr14iKisLZs2fh7u6Oq1ev5mlHU1MT79+/R0xMDBo2bIgpU6aI+0+cOIEVK1bg3LlziImJwX///Yc9e/ZItCFNXzNnzkSDBg0QFxeHmJgYrFq1Crdv3y70/JSUlBATE4PXr19DX18fP/30E27fvo3nz5/j/fv3ePLkCQ4dOoQNGzaIxy1evFiMJSEhASNGjMCAAQMkkg8//vgjIiMjER4ejg8fPqBHjx44f/681GM/c+ZMHDlyBNeuXUNMTAyePHmCly9fAsiezeLm5oYtW7YgMjISkZGRcHBwwKtXr9CjRw8AgKenp9hWfHw8Tp06haFDh0rd/4kTJ/DLL78gKioKb968wb179/IkKaSNt6RtlkZsv/32G+bMmYPz58+jQ4cOhY5dflJTU5GQkCDxQ0RE8oPXASIi+VKixVLj4+PzfIAFgD179iA+Pv6Lg6JvS1FREWvWrIGqqipUVVWxdu1a+Pv748qVKxL1MjMzER4eDlNTU7Ro0QJnz56V2K+pqYlly5ZBSUkJysrKGDFiBO7evSvu37x5MwYOHIjWrVsDABo2bIixY8fmG1NhfcXExKBKlSrirQ5NmzbF8OHDCzw/VVVVrFy5UqyflJSEzZs3Y+rUqcjKykJoaCgUFBQwYMAAHDp0KM/xycnJePv2Lfr374+oqChxtkxycjJ2796NRYsWibeITJs2DTVr1iwwltw+fvyIv//+G0uWLEGDBg0AAEZGRli4cCEAiI+nrlq1qngebm5usLOzg4aGBvr06YO9e/eK7R0+fBhaWlro3LmzVP0DQK9evdCuXTsAgJmZGbp06SLxmhUn3pK0+aWxCYKAWbNmYePGjbhy5QocHR0BFD52+Vm+fDl0dXXFHwsLi2LHS0RE5RevA0RE8qVEiZCZM2ciLS0tT3laWlqZfFoGFc7Y2BhGRkbitomJCUxMTMTbnD58+IA+ffpAQ0MD9erVg7OzM65du4awsDCJdszMzCTWYdDS0kJycjIyMjIAZN9e9fkH0c+fGCNNXwsWLMDq1avRvHlz/PLLL7hz506h52dmZiaxdkdQUBBSU1Mxa9YsODk5oWXLlmjVqhX27dsncWuMj48P6tWrB319fTRt2hTt2rVDRkaGGEtwcDAyMjLynJO9vX2h8eSOIz09HU2aNMl3f+XKlTFp0iQ0b94cffr0wYYNGxAeHi7uHzJkCM6ePSs+Enfv3r0YMGAAVFRUpOofQJ4/9LS0tAr8FqyoeEvS5pe2s337dmzYsAGXLl1CnTp1xPKixu5zc+fORXx8vPgTGhpa7HiJiKj84nWAiEi+lCgRcvToUfTu3TtPee/evXHs2LEvjYm+sfySWikpKVBXVwcAzJ49G9HR0QgLC0NUVBRev36NTp06ITMzs1j9qKmp5ekrJSVFYluavnr27ImIiAjMnj0bsbGxaNeuXZ5ZCbl9vkimkpISgOz3cc6tLzk/169fB5A906BPnz7o1asXEhISEB4ejlevXkFFRUWMRU1NDUDe8fv8nAqiqqoKAHnW+cht48aNCAgIQNeuXXHx4kVUr14d165dAwC0a9cOhoaGOHToEMLCwnDlypVi3RZTXNLEK438Flct7nspR/fu3VGtWjX8/PPPSE9Pl9hX2Nh9Tk1NDTo6OhI/REQkP3gdICKSLyVKhAD/95SH3KKiosRv/6n8iImJwdOnT8Xtx48fIy4uTpyt8eTJE3Tt2lWcNZKamoqbN28Wu5969erl+SD6+Toj0vSVmZkJLS0t9OzZExs2bMCSJUuwa9cuqeOoVasWDAwM8N9//+XZl/OB/P3794iKisLgwYPFBMD169clkh6WlpbQ09OTOKfU1NRC1yvJrWbNmjAwMMC5c+ckynNmpeTEUqVKFYwZMwbHjh1DkyZNxNt3FBUVMXDgQOzduxf79++HlZUVmjdvLu0wFFtR8UrL0NBQfDIQkJ10yr04b3GYmJjA29sbgYGBGDBggJgMKWrsiIiIiIhIfpUoEeLi4oKZM2dKTFOPj4/HjBkz4OLiUmrB0behqKiIYcOGwcfHB9euXcOwYcPQpUsX1K9fHwDQuHFj7Ny5Ez4+Prhz5w5cXV2L9WSUHD///DO8vLywePFiPHr0CGvXroWHh4dEHWn6at26NTZv3oyHDx/i1q1bOHLkCBwcHKSOQ1lZGStWrMDq1auxbNkyPHz4EFevXsWcOXMwd+5cANkfsKtUqYKlS5fiyZMnOHnyJEaMGCExm0FZWRk//fQT5s+fjyNHjuD+/fsYMWJEvknCguJYsmQJlixZgnXr1uHx48c4cuQI+vfvDwC4f/8+XFxccOzYMQQEBMDT0xMPHz6UONehQ4fC19cXGzduxJAhQ6Qeg5IoKl5pdezYEXv27MGZM2fw8OFDTJ48Gf7+/iWOy8TEBJcuXUJAQICYDJFm7IiIiIiISD4pF10lr99//118rGW9evUgCAIePnwIfX39PAtsUtlnbW2Nn376Ce7u7ggPD0erVq2wYsUKcf///vc/CIKA8ePHQ1VVFV26dEGVKlXEW0wAQFtbG+bm5hLtqqurw9LSUkwe1K1bF56enli2bBmOHDmCBg0aYMuWLVizZk2x+tq7dy9WrFiBzZs3Q0VFBc7OzhKPhc0tv7gAwM3NDZUrV8aff/6J3bt3w8zMDF27dhWfcqOgoICTJ09i3rx56Nu3L8zMzLBixQr8+uuvqFChgtjOnDlzIAgCFi1aJC5UOmXKlDy34xQk57G+mzZtwubNm1G3bl0sX74cAODg4IA5c+Zg48aNeP78OYyNjbFy5UqJhWEbNmyIVq1a4fXr10XeFmNsbAx9fX1x28TEBAYGBhJ1DAwMYGJiUqJ4pW1z0qRJiI2Nxbx586CpqYlevXph0qRJ4swbadvJXcfU1BTe3t7o3bs3Zs+ejTVr1hQ5dkREREREJJ8UBEEQSnJgYmIi9uzZg3v37kFBQQENGjTA0KFDoa2tXdox0le0efNmrFq1Ci9evJB1KERlRkJCAnR1daG99QUUNPk7jYi+jfjBxrIOodzK+b0dHx9fKut75LTXpW0AVJTl9zpw7FwlWYdARCS14lwLSjQjBMj+pn3ChAklPZzou5eRkYG3b98WuL9y5crFesILERERERERfbkSJ0Lo+1DQrSP05d69ewdnZ+cC91+8eBHVqlX7dgERERERERFRyRIhGRkZ2LBhAw4dOoSQkJA8T4rJ/UQIKtuGDBny1RfZlFfm5uZ4/fq1rMMgIiIiIiKiXEr01JglS5bgzz//xODBgxEWFoalS5eiR48eiImJ4WKERERERERERFRmlSgRsnv3bnh4eGDy5MkAgDFjxuDvv//Gli1bcPfu3VINkIiIiIiIiIiotJToqTFKSkpITU2FsrIyKlSogPDwcOjq6iIxMRGmpqb49OnT14iViOibKO2nDxAR0df1tZ4aw+sAEVH5UZzf3SWaEZKVlQVl5ezlRaytrXHt2jUAgL+/PzQ1NUvSJBERERERERHRV/fFT42ZOHEiBg0aBEdHR/j5+WHUqFGlERcRERERERERUakrUSIk9900EydOhKWlJW7cuIGRI0di8ODBpRYcEREREREREVFp+uIZIQDQtWtXdO3atTSaIiIiIiIiIiL6ar4oERIQEAB/f38IgoDatWujVq1apRUXEREREREREVGpK1EiJDY2FiNHjoSnpyeUlJSgoKCAjIwM9OjRA7t27YKenl4ph0lERERERERE9OVK9NSY6dOnIzQ0FDdu3EBqaipSUlJw48YNhIaGYtq0aaUdIxERERERERFRqSjRjBBPT0/cvHkTNWvWFMuaNm2Kffv2oXnz5qUWHBERERERERFRaSrRjJCUlBQYGBjkKTcwMEBycvIXB0VERERERERE9DWUKBHSvHlzzJ07FykpKWJZSkoK5syZgxYtWpRacEREREREREREpalEt8b88ccf6NixI8zNzVG3bl0AwKNHj6Cqqopz586VaoBERERERERERKWlRIkQe3t7BAUF4d9//8XTp0+hoKCAAQMGYNiwYahQoUJpx0hEREREREREVCpKlAgZOHAgPDw8MH78+NKOh4iIiIiIiIjoqynRGiEnTpzgoqhEREREREREVO6UKBHi7OyMkydPlnYsRERERERERERfVYlujalVqxaGDRuGU6dOoXbt2lBVVZXYP3369NKIjYiIiIiIiIioVCkIgiAU9yA7O7tC9z958qTEARERyVpCQgJ0dXURHx8PHR0dWYdDRERFKO3f27wOEBGVP8X53V2iGSFMdBARERERERFReVSiNUKIiIiIiIiIiMqjEidC9uzZg0aNGqFChQqoUKECHBwcsG/fvtKMjYiIiIiIiIioVJXo1phly5Zh+fLlGD9+PH7++WcAwJ07dzB27FiEhIRgzpw5pRokEREREREREVFpKNFiqcbGxtixYwe6desmUX7ixAm4ubnh/fv3pRYgEdG3xkXyiIjKFy6WSkRExfndXaJbYzIyMtCqVas85a1bt0Z6enpJmiQiIiIiIiIi+upKdGtM8+bNcfDgQbi5uUmUHzx4EM2bNy+VwIiIZM38YBQUNFNKfHz8YONSjIaIiL61Qb0joKL8UdZhEImOnask6xCIvgslSoTUqFED48aNg6enJxo3bgxBEODn54dTp05h2rRp+OOPP8S606dPL6VQiYiIiIiIiIi+TIkSIefPn4etrS1evXqFV69eieW2trY4f/68RF0mQoiIiIiIiIiorChRIuTJkyelHQcRERERERER0VdXosVSiYiIiIiIiIjKoxInQvbs2YNGjRqhQoUKqFChAhwcHLBv377SjI2IiIiIiIiIqFSVKBGybNkyjB8/Hm3atMGOHTuwY8cOODs7Y+zYsfjtt99KO8av5uzZs3jw4IGswyg1aWlp2LlzJ2JiYmQdSpl19+5d3L59W9ZhfLGQkJCvnnjMysqCt7c3du/ejbt37+LZs2c4fvy4uP/Vq1c4cODAV42hJHx8fODj4yNunzlzBsHBwTKMiIiIiIiIypISJULWrVsHDw8PrFq1Cq6urnB1dcWqVauwf/9+rF27trRj/Gp+++03HDt2TNZhlJqkpCSMGjUKISEhsg7lm3rx4gUOHTpUZL1Pnz6hV69eUFBQ+AZRfV337t3DxIkTv2ofAwYMwIQJE3Dx4kW8evUK58+fx/z588X9169fx7Rp08RtaV+Hr23btm3Ytm2buB0SEoIxY8bIMCIiIiIiIipLSpQIycjIQKtWrfKUt27dGunp6V8cFFFx+Pj4YMaMGUXW27hxI2xtbdG4ceNvEFXZ8uLFC5w5c0bq+vHx8Thy5Ag8PDywa9cu9O/fH7Vr10avXr0KPEba1+FbGz16NJ48eYILFy7IOhQiIiIiIioDSvTUmObNm+PgwYNwc3OTKD948CCaN28udTtHjx6Fra0ttLS0cO/ePaiqqqJt27ZQVVUV61y8eBGhoaFQUFCAsbExHBwcYGRklG87urq68PPzg5qaGlq2bAkNDQ2Jeu/fv8fVq1dhaGiY74dhafoCgDt37iA4OBjW1tZwcHAocIZBTlyampq4desWTE1N0bJlSwBAdHQ0fHx8oKioiEaNGqFy5cricYIgYNeuXQAAZWVlWFpawtHRUWJccjx8+BDPnz+HjY0NqlatWtBQ5xEfH4+jR49iwIABCAwMRGBgIGxsbNCwYcNixZG7nSdPniAoKAhOTk6wtLREVlYWrly5gtjYWNSvXx/Kysrw8fHB4MGDAQCPHj1CaGgounbtKrb35s0b+Pr6inVyBAUF4d69ezAwMICjoyN0dHQAAG/fvoWPjw+SkpKwc+dOAEDDhg1Rt25dieMFQcBff/2F5cuXi2XPnz/H06dP0bVrV9y5cwdhYWFo1qwZqlSpkqdOt27dcOXKFURERMDV1RXPnz8vMvacY7t37w4/Pz+EhYWhYcOG+b5OBZ1fjs/Hsrh8fHywbds2dO7cuci6wcHB4swOLy8vidvHCkoiFfU6FHZ+BY2xmpoasrKycOPGDYSGhsLa2hqNGzeGoqJk/jY+Ph4XL16ElpZWvvGpqKhg4MCB2LRpE1xcXIo8fyIiIiIi+r6VKBFSo0YNjBs3Dp6enmjcuDEEQYCfnx9OnTqFadOm4Y8//hDrTp8+vcB25s6dCyMjI7x+/RqNGzfG3bt3YWBgAG9vb+jp6QEAHj9+LH4QCwkJwd27d7Fr1y6Jb6bnzp0LExMThIeHo379+rh//z7U1NRw+/ZtVKhQAQDg7e2NHj16wN7eHrq6unj58iUyMzMl4imqr4yMDPTo0QNPnjxBs2bN8PbtW6iqquL06dN5ki45cRkbG+Pt27do3Lgx2rVrh5YtW2Lnzp2YPn06mjVrBjU1NQwfPhxLly7F5MmTAWR/aL98+TIAID09Hffv34cgCPDy8pJImMyfPx9r165Fu3bt8PbtWxgYGBTxyv2fiIgIjBo1Cv/++y+ioqJgZWWFS5cuwc3NTXz9pIkjp529e/fiw4cPsLOzQ82aNWFqaorOnTvj2bNnaN68OWbOnAlra2vcv39fTBR4enri5MmTEsmEO3fuYOLEiWIdQRAwadIkHDp0CK1atUJMTAz8/f1x+PBhODk54cOHDwgICEBqaqoYa8WKFfMkQh4/foyQkBA4OzuLZd7e3li4cCEqV66MihUrAgBGjBiBXbt2wdXVVaJOlSpVYGBggMqVK6NPnz5SxZ5z7G+//QZ9fX0oKipiyJAh+PfffzFgwACpzg8AUlNT8x3LryUyMhK3bt0CANy8eRNaWloAgGfPniEpKQk9e/bMc0xBr4O9vX2R51fQGMfExKBbt25ITk6Gvb09Hjx4AENDQ5w+fRq6uroAgKdPn6Jt27YwNTWFpaUlHj58CD09PTRo0EAivjZt2mDYsGFIS0vLN6FIRERERETyo0SJkPPnz8PW1havXr3Cq1evxHJbW1ucP39eom5hiRAg+9vgR48ewdTUFAkJCWjSpAmWL1+OFStW5Hv8P//8gwkTJqBHjx4S3wxHR0fjwYMHqFChApKSklC1alXs2bMH48aNQ2ZmJsaPHw83NzdxDZN9+/ZhyJAhhcb6eV9+fn7w8vJCZGSkmKjx9fVFenp6vokQAAgLC8P9+/fFD27Pnj3DpEmTcPXqVTRq1AgA4OfnBycnJ3Tq1Ak2NjZQVFQUv1UHsj8s9+nTB4sXL8bff/8NIHsmyG+//YZLly6hdevWEAQBgwYNKnSs86OtrY0LFy5AUVERt27dQrNmzeDq6opmzZpJFUcOS0tLiVsP1q9fj2fPnuHRo0cwNjZGfHw8HBwcih3ftm3bcP78eQQGBkJfXx8AsHbtWgwfPhwvX75EvXr14ObmhgULFkjE+rmc16BSpUoS5dHR0Zg7dy5mzpwJAFi5ciUmT56MLl26QFtbW6yzZs0aDBs2rNjxR0dHY+vWrWIy7ZdffoG7u7uYCCnq/BQUFLBly5ZSGUtpOTo6YsWKFfjvv/+watUqWFlZAQD++OMPibU3civoddi6dWuR5wfkP8aurq5o2LAh/v77bygoKCAjIwMdO3bEokWLxH/HU6ZMQcuWLXHw4EEoKirC19cXTk5OeRIhdnZ2+PTpE54/fw57e/s88aempiI1NVXcTkhIKP7AERFRucXrABGRfClRIuTJkyelFsCgQYNgamoKANDR0cEPP/yATZs2iYkQIPuWg0ePHiEmJgaJiYl49+4dQkNDYWlpKdFOzuwPTU1NNGjQAIGBgQCyb8EIDAyEl5eXRP1Zs2bliaewvrS0tJCZmYlbt26hY8eOAIAWLVoUen4DBw4UkyBAdgLG1NQU/v7+ePbsGQRBAJCdkPD19YWNjY1Y18/PD8HBwfj06RN0dXVx584dcd9///2H+vXro3Xr1gAABQUFzJgxo9hP8Zg2bZqYUHJ0dESLFi1w5MgRNGvWTKo4cowbN05i+8iRIxgyZAiMjY0BALq6unBzc5O4NUUau3btQq1atXDq1CkIgiD+BAcHIyQkROI9UJgPHz6IyavcVFRUMGnSJHF76tSpWLBgAa5cuYJu3boBALS0tDB06NBixZ2jYsWKErOXWrRogeXLl0MQBCgoKEh1fiUZy5iYGHh6eorbPj4+iIyMlEhS2NrawtHRsUTnJS1pX7/Pxzg6OhqnT5+Gu7s79u7dKx5namoqzjiJjY2Ft7c3rl69Kr6HW7RogaZNm+aJI+e1j46OzjfO5cuXY/HixaV45kREVJ7wOkBEJF9KlAgpTbnXYwCyZxbkfurJ7Nmz8ddff6FZs2YwNjZGVlYWgOwPNLk/BH/+IVdNTU3M7IeEhEBRURHm5ubifgUFhTx9F9WXnZ0dVq9ejVGjRkFBQQFt2rSBm5ubxO0Wn8tJ8uQICQlBenq6RFIGALp27QoTExMAQGJiItq3b4+3b9+iSZMm0NHRwcuXLyU+xIWEhOQ7dsVV2PhLE0dh59mnT58vji8kJARpaWl5xmvEiBFiEkka2tra+PjxY55yExMTqKmpidvq6uowNjaWeA8aGxuX+Ekz+b0vMzIykJmZCWVlZanOryRj+fHjRzFhAGQvlhofHy9Rpqys/NUTIdK+fp+Pcc74P3nyBK9fvxbLVVRU0KZNG4k60vw7yHntc2b5fC73rCAg+5tACwuLIs+PiIi+D7wOEBHJly9KhERGRiIkJAQZGRkS5fl9I1uQmJiYPNs5C5S+fv0aK1euxL1798Sp7kFBQdi/f3+xPgQbGRkhKysLcXFx4vT8z/uWtq/p06dj2rRpePbsGQ4fPoz27dvj4sWL4syMz33+AVpXVxcGBgaF3saxfft2xMXF4dWrV+KH9N9//x3r1q2TOKfctyV9fj7SiomJQbVq1SS2c9afkCaOHJ+fp5GRUb6vbW5KSkpisilHcnKyxLauri6aN28use5MSdSsWRMxMTGIj4+XmKETGxsrUU8QBMTGxkoskptfEkSa2KUhzflJM5afq1KlisR7bOfOndi2bVuh77uvQdrXL79/JwAwefLkAhONOa9RTEyMRPIjJiYmzy1QwcHBUFJSQvXq1fNtS01NTSIhRkRE8oXXASIi+VKix+eGhYWhdevWMDExQePGjdGsWTOJn+I4duyYxKKlOYsqAkBUVBQASCwM6eHhUex47e3toaenhyNHjohlDx48QFBQkLgtTV/v3r1DamoqFBQUUKdOHbi7u6Nq1aq4d++e1LF06dIFDx8+xLVr1yTKP3z4gMTERDEWMzMz8YKclZUlPsUjR6tWrXDz5k2EhYWJZQcPHpQ6jhy5xyQyMhJXrlyRGP+i4ihIq1at8n1tc6tcuTKCg4MlHrn8+cyBLl26YN++fXk++OeeJaCtrV1kEsLR0REVKlSAr6+vRPmnT59w9uxZcfvUqVNIT08vMpknTezSkOb8pBnLsiC/10Ga88tPtWrVULNmTWzcuDHPvpxjzczMYGNjI/EefvfuXZ5/W0D2rUFNmjSRSIIREREREZF8KtGMkOnTp8PMzAyhoaGwsLBAVFQUbt++jalTp2LGjBnFauvDhw9wcXFBr169cPnyZdy6dQu3b98GANStWxfVq1dHr169MGDAADx69Aj//fdfsePV1tbGokWLMGXKFLx69Qp6enr466+/JGaHSNPX8+fPMXbsWPTq1QtVq1bF7du38f79e4knhxSlc+fOGDt2LDp37ozx48fD2toa/v7+OHfuHK5cuQJtbW306tULK1aswOTJk1GzZk0cOXIEr169grq6uthOt27d0KJFC7Rp0wbjxo1DaGgoDh8+XOyx2b17NxITE2FjY4OtW7fC3t4e/fv3BwCp4ijIzz//jD179qBjx47o0aMHLl26hICAAIk6PXr0wI8//oi+ffuiU6dOuHHjBi5duiRRZ/78+bh06RIaNWqEMWPGoEKFCrh9+zbCwsJw9epVANmPaU1ISMDcuXNRs2bNfB+fm/N0nv3796NLly5ieYUKFfDDDz9g3LhxEAQBq1evxvTp04ucDitN7NKQ5vykGcuyIL/XQZrzK8g///yDzp07w8XFBd26dUN8fDzOnDmDXr16Yfbs2VBQUMDvv/+O/v37Iz4+HlWrVsXWrVvzPHoYyE5q5rcmEBERERERyZ8SzQi5cuUKVq1aJa65oa+vjy5dumDv3r3YsGFDsdpasGABxo8fj+DgYFSvXh13795FrVq1AGR/ePX19UW7du3w4MEDWFpa4tatWxgxYgQMDQ3FNnr37g1bW1uJdtu2bSvxrf60adNw6NAhxMXF4ePHjzh69Ch+/vln1K9fX+q+WrduDS8vL1SsWBEPHjyAjY0Nnjx5gho1auR7bvnFBQBbtmzB8ePHoaCggGfPnqFOnTq4e/euOJ2/cePGuHHjBtTV1eHv749Ro0bh8OHD4pNGcpw+fRqTJk3CixcvYGFhgZs3b2LEiBHFeoyul5cXqlevjqCgIIwZMwYXL14UF56UJg5dXV2MGDFCXKg2h7GxMe7evYtmzZrhxYsX6NmzZ55v9/X09HD37l3Ur18fgYGB6Ny5M44ePSo+fhbIXkDX19cXS5cuxbt37/D27Vv06dMH3t7eYh1ra2tcvHgRaWlpuHLlCt68eZPvuf788884deqUxCwaU1NTXLx4EYIgIDw8PM9CvTVr1kTfvn3ztCVN7Pkda2pqihEjRohjLM355TeWnp6eEn0VxcbGRiIBVBQdHR2MGDFCfHQuANSuXVti4ddq1aqJjxkG8n8dpDm/gsa4WbNmCAgIQMeOHfHs2TNkZWVh/fr1mD17tlinV69eYluRkZHYtGkTli1bJj6aF8h+j6elpeV5ShQREREREcknBaE4i23kHKSgIK6bUbFiRTx58gRmZmZISkqCvr6+xOPHClOrVi1MnjwZkydPLm4I9IUCAgJga2uL0NBQiUVkv6Zjx45h5MiRiIuL+yb95eeff/6Bvr4+evXqhc2bN2PVqlV48eKFzOKhr2/dunWwt7dH27ZtpT4mISEBurq60N76Agqa+S+wKo34wcYlPpaIiKSX83s7Pj4+35mBJW2vS9sAqCiX/DpAVNqOnatUdCUiOVWca8EXPzWmfv362LRpE37++Wds3bo1zxMcSDZu376NZ8+e5buvcuXKcrsS+qhRo2QdAn1j06ZNk3UIRERERERUhpQoEdKzZ0/x/5cuXYquXbtiyZIlUFdXx7///it1OwXdOkJf7sWLFxKPSs3N3t4ednZ2+d7S8jVVqVKlWLdzfG0F3ZJBRERERERE368S3RrzuY8fPyIgIABWVlYSa3cQEZVHvDWGiKh84a0xJC94awxRwb7prTEAoKWlBQcHh9JoioiIiIiIiIjoqynRU2OIiIiIiIiIiMqjUpkRQkT0PXo7wKhUplgTEVH5tP+oGa8DRETfIc4IISIiIiIiIiK5wUQIEREREREREckNJkKIiIiIiIiISG4wEUJEREREREREcoOJECIiIiIiIiKSG0yEEBEREREREZHcYCKEiIiIiIiIiOQGEyFEREREREREJDeYCCEiIiIiIiIiucFECBERERERERHJDSZCiIiIiIiIiEhuMBFCRERERERERHKDiRAiIiIiIiIikhtMhBARERERERGR3GAihIiIiIiIiIjkBhMhRERERERERCQ3mAghIiIiIiIiIrnBRAgRERERERERyQ0mQoiIiIiIiIhIbjARQkRERERERERyg4kQIiIiIiIiIpIbTIQQERERERERkdxgIoSIiIiIiIiI5IayrAMgIiqrzA9GQUEzRdZhlHnxg41lHQIR0VcxqHcEVJQ/yjoM+s4dO1dJ1iEQyR3OCCEiIiIiIiIiucFECBERERERERHJDSZCiIiIiIiIiEhuMBFCRERERERERHKDiRAiIiIiIiIikhtMhBDJyJ9//oldu3bJTb9ERERERERlAR+fS1QCsbGxmDFjhritr68Pe3t7DBkyBGpqalK14e3tDUNDQ4wYMeJrhVmm+iUiIiIiIioLOCOEqAQ+ffqEXbt2wdjYGM7OzqhUqRKWLFmCZs2aITU1VdbhERERERERUQE4I4ToC3To0AHt27cHAHTu3Bn29vY4ceIE+vXrh0ePHmH//v2IiYmBnZ0dRo8ejQoVKhTY1p49e+Dl5QUAMDAwgKOjIwYMGAAFBQWxzurVq2Fubg49PT2cPn0aJiYmmDdvHhITE7Fr1y74+/ujUqVKGDRoEKpWrSrVORTV78uXL7FkyZI8x9WoUQMODg44d+4cVq9eLbFv7969eP36NebPn5/nuKJiffv2Lf7991+EhITAzs4OY8aMgbq6erHHyMjICBcuXEBGRgb69esHR0dHqcaDiIiIiIi+b5wRQlRKatWqBQUFBbx9+xZnz56Fg4MDPnz4AFtbW+zYsQNNmzYtdLZI9erV4ezsDGdnZ5iammL+/PkYPXq0RJ0LFy5g+vTpWLJkCWxsbNCgQQMIggBnZ2ccOnQIderUQXp6Ovr06YPAwECp4i6qXx0dHXG/s7MznJyccOLECdy6dQvVq1fH2rVrce/ePbG+IAiYP38+VFVV8/RVVKy+vr6oXbs2bt68CVtbWwQEBKBv377FHqMZM2ZgyZIlqFSpEpKTk+Hk5IRbt25JNR5ERERERPR944wQolJy9uxZCIIAe3t7TJgwAZMnT8aaNWsAAGPGjIG1tTX++usvibVFcnN0dJSYteDq6gpra2u4u7vDyspKLNfQ0IC3tzdUVFQAAG/evMG9e/cQFhaGSpUqAQB+/vlnpKWlSRV3Uf0aGRlh5MiR4v4pU6ZAVVUVf/31FypXrox27dphx44daNiwIQDAy8sLYWFhGD58eJ6+QkJCCo119OjR6NevH3bs2CEeEx4eXuwxMjAwgJeXF5SUlAAAQUFB2L17d4GzQlJTUyWSVAkJCUWOGxERfT94HSAiki9MhBB9gd9//x179uxBdHQ0Lly4gLFjx6J27doICgrCwIEDxXra2tro1q0brly5UmAiJCsrC8ePH8fNmzcRFRWFrKwsKCsr4/nz5xIf8tu2bSsmQQDAxMQEhoaGWLhwIaZNmwY7OztoampCU1NTqnOQtl8A2L59O7Zu3Qpvb29UrlwZAODm5oYJEyZg9erVUFNTwz///IOuXbvCxMQkT1+Fxfrq1SsEBgbin3/+kTgmJ2FSnFidnZ3FJAgA1KxZE2/evClwDJYvX47FixdLNV5ERPT94XWAiEi+8NYYoi9Qr149ODs7w83NDU+ePMGWLVsQFRUFAKhYsaJEXUNDQ0RGRhbY1rBhw/Djjz9CQ0MDTZs2FT/MJyYmStTT09OT2FZXV8fVq1chCIKYgJg+fTqSk5OlOgdp+71x4wYmTpyIzZs3o1mzZmJ57969oaSkhGPHjiEuLg5Hjx7Nc7uKNLHGxsYCAIyNjb841pw1RXIoKioiMzOzwHbnzp2L+Ph48Sc0NLTAukRE9P3hdYCISL5wRgjRF8i9WGoOS0tLAMDr169RrVo1sTw4OFjc97mEhATs378ft27dQuPGjQFkP6J31KhRUsWRsw4JADx48ADdunWDsbEx5s2bV+hx0vYbFhaGPn36YOLEiRK3yQCAqqoqhg0bhh07duDDhw/Q1dVFly5dih3rmDFjAAAvXryAjY1NiWMtCTU1Nakfe0xERN8fXgeIiOQLZ4QQlTJdXV20b98ef/zxhzgLwd/fHydPnkS/fv3yPSbnqSc5syIAYNmyZVL19+bNG/j6+orb9evXh7m5OeLi4oo8Vpp+U1JS0KtXL9jZ2WHVqlX5tuPm5gYvLy+sWbMGw4cPh7Jy/jnWwmI1MTFB+/bt8b///Q9JSUlinXPnzkkdKxERERERUVE4I4ToK/jzzz/h4uKCunXronr16vD29sbgwYMlnoCSm7a2NqZNm4Z+/fqhQ4cOePPmDdLT06GhoVFkX2pqapgzZw4SExNRq1YtBAUFISYmBhMnTizyWGn6PXz4MPz8/NCjRw9x1gaQ/fjcnBkntWvXRtOmTXH9+vUCb4uRJtZ//vkH3bp1Q40aNeDo6IigoCC0a9cOHTt2/KIxIiIiIiIiyqEgCIIg6yCIypukpCQcPHgQHTt2hJmZWb51UlJScOXKFcTExMDOzg729vYS+y9fvgw1NTWJ9Tbu3r2LwMBAmJmZwcnJCQcPHkSLFi3EW2ouXLgAfX19ODg45Onv4cOHeP78OYyNjeHk5FTgrIzi9vvixQv4+PjkacfExASdO3cWtydOnIiHDx9KzPgoSGGxZmVl4fr16wgLC0PdunVha2srcWxJxuj27dtITExEu3btiowNyL4NR1dXF9pbX0BBU1uqY+RZ/OCC13UhIvoWcn5vx8fHQ0dHp9Ta69I2ACrKvA7Q13XsXKWiKxFRkYpzLWAihIi+WGJiIiwtLbF+/XoMHTpU1uF8MSZCioeJECKSNSZCqDxjIoSodBTnWsA1Qojoi8yaNQtOTk6wtraWeGQwERERERFRWcQ1QojoizRt2hTNmzdHx44dC7wdh4iIiIiIqKzgpxYi+iIFPQmHiIiIiIioLOKtMUREREREREQkN5gIISIiIiIiIiK5wVtjiIgK8HaAUak8fYCIiMqn/UfNeB0gIvoOcUYIEREREREREckNJkKIiIiIiIiISG4wEUJEREREREREcoOJECIiIiIiIiKSG0yEEBEREREREZHcYCKEiIiIiIiIiOQGEyFEREREREREJDeYCCEiIiIiIiIiucFECBERERERERHJDSZCiIiIiIiIiEhuMBFCRERERERERHKDiRAiIiIiIiIikhtMhBARERERERGR3GAihIiIiIiIiIjkBhMhRERERERERCQ3mAghIiIiIiIiIrnBRAgRERERERERyQ0mQoiIiIiIiIhIbjARQkRERERERERyg4kQIiIiIiIiIpIbTIQQERERERERkdxgIoSIiIiIiIiI5AYTIUREREREREQkN5RlHQARUVllfjAKCpoppdZe/GDjUmuLiIi+vkG9I6Ci/FHWYVAux85VknUIRPQd4IwQIiIiIiIiIpIbTIQQERERERERkdxgIoSIiIiIiIiI5AYTIUREREREREQkN5gIISIiIiIiIiK5wUQIkZwbMWIE/ve//8k6DCIiIiIiom+CiRCiMmbOnDmwsrKClZUVqlevjiZNmmDUqFE4ffr0V+nv/fv3iImJKZV4cte1trZGgwYNMHr0aLx8+bLQGNLT07Fx40a0adMGtWvXRteuXbFr1y5kZGRI1Pv06RNWrFiBli1bokaNGmjRogWWLl2KxMTEPHH079+/GKNARERERETyQlnWARCRpOjoaBgZGeHQoUPIyspCVFQULl68iMGDB6NDhw7Yv38/lJSUymQ8uesC2UkWd3d3ODs749WrV1BRUcm3j0mTJuHMmTNYu3Yt6tSpg5CQEBw8eBBv377F/PnzAQBxcXFo3bo1AGDp0qWoWbMmXr16BXd3d+zbtw9Xr16FoaGhGEdERMTXHhoiIiIiIiqHOCOEqAxSU1ODlZUVqlatCkdHR8ybNw+XL1/G0aNHsXnzZrFejx49sGnTJoljZ8yYgVmzZkmUHT9+HB07doStrS369++PwMDAAvu+d+8e6tSpg3/++afY8eSua2VlBUdHR8yfPx9v374tcFZIZmYm9uzZgwULFqBfv36wtbVFx44dsX37dsyePVus99NPP+Ht27e4fPkyunfvjho1aqBTp064dOkSEhISMH369CLHlYiIiIiIiIkQonKifv366NixI/bu3SuWhYeHIz4+XqJeVFQUoqOjxe0tW7Zg0KBB6NSpEw4cOICBAwdizpw5+fZx+fJltG3bFhMmTMCoUaOKHc/nsrKycOrUKRgYGKBKlSr51lFUVISSkhICAgLy7FNWzp60lpaWhr1792L8+PHQ19eXqFOhQgVMnToVBw4cwMePHwuNmYiIiIiIiLfGEJUjdevWxdatW6Wun5GRgXnz5mHhwoWYMWOG2Ebv3r3z1D127BiGDx+OzZs3Y/DgwSWOx8/PD1ZWVgCAmJgYaGhowNPTE5qamvm2oaCggKVLl2LmzJnw8vJC27Zt0axZM3To0AEGBgYAgFevXiE5ORl2dnb5tmFnZ4eMjAw8f/4cjRo1kir23FJTU5GamipuJyQkFLsNIiIqv3gdICKSL5wRQlSOKCsrIz09Xer6z58/R0xMDDp16iRRrqgo+U//7NmzGDx4MDw8PKROghQUj729PS5fvozLly/j6tWr6NWrF1xdXREeHl5gO9OmTcPTp08xePBgvHr1CuPHj4eVlRWOHDkCAOKiqRoaGvken1MuCILUsee2fPly6Orqij8WFhYlaoeIiMonXgeIiOQLEyFE5UhwcHCx/jjLSVIUlEDIYWRkBGVlZTx58uSL48m9Rkj9+vWxadMmJCcn4++//y60rVq1amHu3Lk4ceIEwsPD0aZNG4wZMwZZWVmwsLCAoqIiXrx4ke+xOeWWlpbFij/H3LlzER8fL/6EhoaWqB0iIiqfeB0gIpIvTIQQlRMxMTE4ceIEunbtKpZpa2vnWRcjJCRE/P9q1apBRUUFd+7cKbTtRo0a4ezZs1i6dCl+//33EseTH0VFRWhqaiIuLk6qdgFAU1MTffv2Ff8g1dXVRbt27bB9+3ZkZmbmqb9161a0bdsWRkZGUveRm5qaGnR0dCR+iIhIfvA6QEQkX5gIISrjBEHAzZs30alTJxgaGuLnn38W9zk4OOC///4TkwwHDx6Ej4+PuF9bWxtubm5YuHAhHj58CCD70bJLlizJ00/z5s1x5swZLFmypNBkSGHx5GfPnj14+/Yt2rdvn+/+zMxMDBs2DHfv3hWTHBEREdi6dSvq1asnLo66du1ahIWFwc3NDbGxsQCAxMRETJkyBf7+/vjjjz8KjYOIiIiIiAjgYqlEZVLOgqNZWVmIiYlBxYoV0a9fP8yfP19cQBQAZs+ejdu3b8PU1BTa2tpo0qQJ2rVrJ9HW2rVroaSkhGbNmkFDQwOqqqpYs2ZNvv22aNECp0+fRufOnaGgoCA+hlfaeHLXBYC4uDioqalh3bp16NatW759KikpoVOnTpg0aRIeP34MbW1txMXFoWPHjti9e7dYr06dOrhx4wZ+/PFHVKpUCQYGBoiKioKtrS1u3LiRZyHV3HHkcHJywp49ewoeeCIiIiIi+u4pCCVdXZCIvooPHz4gMTERQPZipHp6etDS0ir0mPj4eKipqUFdXV18dK6hoaFEnbS0NMTGxsLExESi/P3791BVVZV4LG10dDQ+ffoES0vLYsWTuy4A6Orq5nncbWFyYjQyMsqzoGtuycnJiI6Ohru7O06ePIn79++jcuXKBcaRQ0NDI8/55ychIQG6urrQ3voCCpraUsdflPjBxqXWFhER/Z+c39vx8fGlcltLTntd2gZARbn0rgP05Y6dqyTrEIiojCrOtYCJECIqt5KTk9GkSRNoa2vjypUrUFFRKZV2mQghIipfmAiRH0yEEFFBinMt4K0xRFRuaWhowNfXFzExMcjKypJ1OEREREREVA4wEUJE5RpX9yciIiIiouLgU2OIiIiIiIiISG4wEUJEREREREREcoOJECIiIiIiIiKSG1wjhIioAG8HGHH9ESIiObb/qBmvA0RE3yHOCCEiIiIiIiIiucFECBERERERERHJDSZCiIiIiIiIiEhuMBFCRERERERERHKDiRAiIiIiIiIikhtMhBARERERERGR3GAihIiIiIiIiIjkBhMhRERERERERCQ3mAghIiIiIiIiIrmhLOsAiIjKGkEQAAAJCQkyjoSIiKSR8/s65/f3l+J1gIio/CnOtYCJECKiz3z48AEAYGFhIeNIiIioOBITE6Grq/vF7fA6QERUfklzLWAihIjoMwYGBgCAkJCQUvmDWp4lJCTAwsICoaGh0NHRkXU45RrHsvRwLEtPWRlLQRCQmJiISpUqlUp7vA4UT1l5H5QnHLPi4XgVj7yOV3GuBUyEEBF9RlExe/kkXV1dubp4fE06Ojocy1LCsSw9HMvSUxbGsjQTFrwOlExZeB+UNxyz4uF4FY88jpe01wIulkpEREREREREcoOJECIiIiIiIiKSG0yEEBF9Rk1NDe7u7lBTU5N1KOUex7L0cCxLD8ey9HyvY/m9ntfXwvEqPo5Z8XC8iofjVTQFobSeM0ZEREREREREVMZxRggRERERERERyQ0mQoiIiIiIiIhIbjARQkRERERERERyQ1nWARARlSWJiYl4/vw5jIyMYGlpKetwyrWsrCzcvXsX6urqsLe3l3U45drHjx8RFBQEU1NTmJmZyTqcci0zMxPPnz+HkpISrK2toaqqKuuQyr2YmBg8e/YMVapUQZUqVWQdTql49eoVYmNjUatWLVSoUEHW4ZQZPj4+ecpsbGxgamoqUZaRkYEnT55AVVUVtra2UFBQ+FYhytzHjx/x8OFDVKlSBRYWFvnWiYmJwcuXL2Fubl7g73Rp6nwPMjIy4OfnB319fdSsWVNi36dPn3D//v08x9StWxc6Ojp56gYEBEBfXx9Vq1b9qjHLUlxcHIKDg2Fubg4jI6N866Snp+Pp06dQU1NDrVq18v33J02d755ARESCIAjC5s2bBU1NTaFWrVqCpqam0K1bN+HTp0+yDqvcSUtLE5YvXy5UrVpV0NXVFVq3bi3rkMqtkJAQYdCgQYKurq5Qv359QVdXV2jTpo0QFhYm69DKpTVr1ghmZmaCvb29YGVlJRgZGQl79+6VdVjlWmZmpuDs7CwoKioK8+fPl3U4XywuLk5o06aNoK2tLdSoUUPQ1tbme+T/S09PFwAIdnZ2QosWLcSfY8eOSdTz8fERzMzMhCpVqghGRkZC7dq1hRcvXsgo6m8nLCxMmDRpkmBmZiaoqqoKS5Ysybfe4sWLBTU1NaF27dqCmpqaMHLkSCEjI6PYdcq7jx8/CgsXLhQsLCwEbW1twdXVNU+d+/fvCwAER0dHiffco0ePJOrt3btX4t9smzZthLi4uG91Kt9EQECA0L17d0FfX1+oX7++oKWlJfTo0SPPeV65ckUwNTUVLC0tBUNDQ8HOzk549epVsevIA94aQ0QE4MGDB5gwYQJ27twJf39/BAcH49GjR1iwYIGsQyt3kpKSEBcXhwsXLqBfv36yDqdcCw4ORo8ePRATE4P79+/jzZs3+PjxI9zc3GQdWrmUkZEBf39/PHr0CMHBwfjpp58wfPhwREdHyzq0cmvp0qWoWLEirK2tZR1KqZg+fTrev3+P0NBQPH/+HCtXrsTIkSPx8uVLWYdWZmzYsAE+Pj7iT8+ePcV9nz59Qr9+/dC/f3+8efMGERERsLCwwKBBg2QY8bfx8uVL1KxZE/7+/gXO4Dh16hSWLFmCCxcu4OnTp3j8+DGOHTuGdevWFavO9yAqKgpKSkq4efMmWrVqVWjdkydPSrzncs8yffHiBUaOHInVq1fj+fPnCAkJQXh4OGbMmPG1T+GbCgoKwg8//CD+PfDy5Uv4+/tLnOfHjx/Rr18/DB48GK9fv0ZERARMTU0xZMiQYtWRG7LOxBARlQVTp04VatWqJVG2dOlSQV9fX8jMzJRRVOXfmDFjOCOklP3xxx+Ctra2rMP4Lty+fVsAIDx58kTWoZRLV69eFapUqSJ8+PBBqFatWrmfEZKUlCSoq6sLmzdvFssyMzMFExMTwd3dXXaBlRE5M0L27t0r+Pn5CR8+fMhT5+DBg4KioqLw/v17sezy5csCAOHx48ffMlyZsrS0zHdGSJ8+fYT27dtLlI0fP16oU6dOsep8b7p27VrojJCbN28K9+/fFxITE/PU+eWXXwQzMzMhKytLLNu4caOgrq4uJCUlfdW4ZW3OnDlC9erVxe19+/YJSkpKQnR0tFjm5eUlABD8/f2lriMvOCOEiAjA/fv30ahRI4myJk2aIDY2Fm/evJFRVER53blzBzY2NrIOo9wKDQ2Fj48PDh8+jEmTJmHIkCGoU6eOrMMqd2JiYjB06FBs374dBgYGsg6nVPj7+yMlJUXiWqCoqIhGjRrlu06BvJo6dSpGjRoFMzMz9O3bFx8+fBD33b9/HxYWFjA2NhbLmjRpIu6TdwX9reHv74/U1FSp68ibfv36YdCgQTAwMMCUKVOQnp4u7rt//z4aNmwoscZFkyZNkJKSgoCAAFmE+834+flJ/D1w//59WFlZoWLFimLZ5//+pKkjL5gIISJC9h/1uS8KAMTtmJgYWYRElIenpyf27duHhQsXyjqUcsvb2xuzZ8/GzJkzERsbizFjxsg6pHJpzJgx6N+/P9q3by/rUEpNzu/6/K4FvA4ACgoK2L59O6KiovDo0SMEBgbi8ePHGDt2rFgnv2uphoYGNDQ0OIYo+G+NrKwsxMXFSV1HXujq6uLcuXMIDQ2Fv78/bty4gd27d2Pp0qViHXn9+23btm3w9vbGvHnzxLL8xkJbWxsqKiriWEhTR14wEUJEBEBFRQUpKSkSZcnJyQDAp0pQmXD16lUMGjQIv/76K3r37i3rcMqt4cOHw9fXFyEhIZg2bRo6dOjw3X9rWNr27dsHX19fdO7cWbxnPyUlBaGhobh+/bqswysxFRUVAMj3WsDrAKCkpITRo0eL37xbWlpi3rx5OH78uDhm+V1LBUFAWloaxxDS/a3Bv0f+j7W1NTp06CBuN2rUCD/88AM8PDzEMnkcr2PHjmHixIn466+/4OTkJJbnNxYZGRnIyMgo9P31eR15wUQIERGy/6ALCwuTKMvZLujxd0Tfio+PD7p27YqffvqJC/iWokmTJkFVVRVeXl6yDqVcEQQBNWrUgLu7O+bMmYM5c+bgw4cP8Pb2LtezlXIemZ7fteB7eSxwaTMxMUFmZibevXsHIHsMIyIiIAiCWCciIgKZmZkcQxT8t4a2tjb09fWlriPPTExMJMansL/fvsf3nKenJwYOHIh169ZJzMYCssciPDxc4t9fznbOWEhTR14wEUJEBMDFxQWXL19GYmKiWHb8+HE0btwYenp6sguM5F7ON+8zZ87EokWLZB1OuZWUlCTxhx+Q/cdyUlJSnmnCVLghQ4ZIPMHBx8cHlStXxvDhw3Hx4kVZh1diVlZWsLGxgaenp1gWERGB27dvw8XFRYaRlQ2fPn3KU3b+/Hno6enB3NwcQPa1NDY2FteuXRPrHD9+HOrq6mjZsuU3i7WscnFxwZkzZ5CRkSGWHT9+XOIWM2nqyIv83nMXLlyAnZ2duO3i4oJbt24hMjJSLDt+/DiqV68uJje/FydPnsSAAQOwZs0aTJgwIc9+FxcXREdH48aNG2LZ8ePHoampiRYtWkhdR14oyzoAIqKyYNSoUdiwYQN69uyJadOmwc/PDwcOHMCZM2dkHVq55Ofnh5SUFLx//x7x8fHw8fEBAIkpnFS0Bw8eoHPnzmjVqhVcXFzEcQSA5s2bQ1GR32dI6+nTp5gxYwZGjBgBa2trhISEYPXq1ahXrx5vNSLRb7/9hoEDB8LU1BS2trb47bffUK9ePQwYMEDWocnc3r17ce7cOfTu3RsVK1bEuXPn8Oeff2Ljxo1QVs7+SFGvXj0MGjQII0aMwP/+9z98/PgRs2fPxrx586CjoyPjM/i6UlJS4OfnBwBITU1FSEgIfHx8oKenJ35wnz59Onbu3AlXV1eMHDkSZ86cwZ07dyQ+lEpT53tx/fp1ZGVlITY2FikpKfDx8YGqqqq4eOeCBQuQnJyM9u3bQ1VVFfv27cPVq1dx+vRpsQ1XV1esWbMGPXv2xM8//wx/f39s2bIFBw4ckNVpfRXe3t7o168fBgwYgLp164p/DygpKaFZs2YAsm8d6t+/P4YNG4Zly5YhPj4e8+bNw4IFC6ClpSV1HXmhIHz+9QgRkZyKiorCb7/9hgcPHsDIyAgTJkxA69atZR1WudS/f39ERETkKc/9QZ6K9t9//2HNmjX57rtw4QI0NDS+cUTl26NHj/D3338jMDAQhoaGaN26NUaOHAk1NTVZh1buDRw4EO3atcMPP/wg61C+2Llz57Bt2zbExsaicePGmD17NmcG/n9nzpzBgQMH8O7dO1StWhVjxozJ84STtLQ0bNiwAefPn4eqqir69u2LkSNHyibgbygsLAyurq55yhs3boy1a9eK22/evMGKFSvw/PlzmJubY/r06WjQoIHEMdLU+R60a9cuz5NwKlasiOPHjwMAsrKysGfPHpw6dQqfPn1CrVq1MHnyZFhZWUkcExcXhxUrVuDOnTvQ19eHm5sbOnbs+K1O45vYunUrdu3aladcU1MT58+fF7fT0tKwbt06eHl5QU1NTUx65CZNHXnARAgRERERERERyQ3OqSUiIiIiIiIiucFECBERERERERHJDSZCiIiIiIiIiEhuMBFCRERERERERHKDiRAiIiIiIiIikhtMhBARERERERGR3GAihIiIiIiIiIjkBhMhREREZVRKSgo8PDwQHx8vsxgiIyNx+vTpIuvFxcXh2LFjXz8gIiI5FBsbCw8PD6Snp8s6lHLl7du38PLy+qp9SHudpLKFiRAiIqIyIDk5GR4eHkhMTBTL4uLiMGjQIISGhsosrmnTpuHevXtF1tPR0cGCBQvg6en5DaIiIipafHw8PDw8kJKSIutQvtjLly8xaNAgfPr0SdahlCtjx45FQEAAgP/7cuHAgQPIzMzMU/fEiRPw8PBAZGSkRP2cnyNHjuDWrVt5jtXT08OkSZNw9erVr39CVGoUBEEQZB0EERGRvHv79i0sLCzg7++PWrVqAcj+I37cuHFYtWoVzM3Nv3lMDx48QMuWLREeHg5tbe0i6+/evRsrVqzA06dPv0F0RESFe/LkCezt7REREQFTU1NZh/NF/Pz80LhxY8TGxkJPT0/W4ZQLly9fRv/+/REWFgZVVVW8e/cOZmZmAIBTp06hS5cuYt2goCDUrFkTgiDA29sbzs7OYv3WrVvD1NQUaWlpuHfvHpSVlXH+/HlUrVpVPH79+vU4cOAAfH19v/l5UslwRggREZGMZWVliTMpTp8+DQ8PD1y6dAlqamro1auXmIT4+PEjPDw88OnTJwQFBeHEiRN49OiR2M7Dhw9x/PhxvHz5Mt9+4uPjce7cOZw7dw7v378vMq6NGzeiT58+EkmQ9PR0XLt2DadOnUJYWJhE/b59++L169e4fPlycYeAiChfqampuHz5Mk6ePIng4GCJfT4+Prhw4YJE2atXr3DgwAHExcXh7NmzAIDjx4/Dw8MDPj4+EreY3L9/H0eOHMG7d++Qnp4u8c3/3bt38501kFvutp49ewZPT088f/4837oxMTE4c+YMLly4gOjoaIl9JekbAIKDg+Hh4ZGnvc9j8/f3h6enJ/z9/cX9d+/ehaenJ968eVPseL+07ZCQEBw7dgxeXl5ITk4usO3cr8+hQ4cQEhIiUVcQBBw8eLDAWZMbN27E4MGDoaqqKlHeqlUrbN++XaJs+/btaNWqVb7tzJkzBx4eHvjvv/8QGBgIJSUl/PbbbxJ1Bg8ejFu3buHBgwf5tkFlkEBEREQylZaWJvTo0UMAIHTp0kVwdXUVFi9eLERERAgAhMePHwuCIAhBQUECAKF169ZC3bp1hc6dOwsqKirCTz/9JPTv31+oX7++0LFjR0FNTU3Ys2ePRB8HDhwQ9PT0hNatWwsdO3YUdHV1hU2bNhUal6mpqbBz505x+82bN4KVlZVgb28vdO/eXahataqwdOlSiWOcnZ2FH3/8sZRGhojk2Z07dwRzc3OhUaNGQrdu3QQDAwNh6tSp4n5fX19BRUVFOHv2rCAIgpCUlCTUrl1bGDVqlBAXFyd06tRJACD07NlTcHV1FVatWiXcuXNHACB07dpVsLe3FwYMGCA8ePBA+PTpk+Dq6iq4uroKvXv3FiwtLQUHBwchOjq60PgACJ07dxZq164ttG/fXlBTUxN+/fVXiXrbt28XdHV1hbZt2wouLi6Crq6u8O+//4r7pek7p6/Y2FhBEATh3r17grGxsbBw4cJCY2vdurXQqFEjoUOHDoKSkpKwePFioXPnzkLjxo3FeI8fP16seL+k7V9//VXQ0NAQ2rVrJ9SpU0cwNTUV/Pz88rT9+evTsWNH4YcffpBoy8vLS1BSUhLCw8PznH96erqgpaUlHDt2TCzLuab+888/gpqamhAZGSnWNTU1FXbv3i0AELy9vSXqnzlzRqLt1q1bC4MGDcrTZ/369YUlS5bk+3pQ2cNECBERURkQGhoqABD8/f3FsoISIePHjxfrbNiwQQAgTJ8+XSxbvny5YG1tLW4HBQUJFSpUEK5duyaW3bx5U1BXVxcCAwPzjScsLEwAINy+fVssW7BggdCqVStxOzMzU+KPTEEQhClTpkjUISIqieTkZKFy5coSCdvw8HDByMhIOHr0qFjm7u4umJqaCpGRkcLEiRMFGxsbITExURAEQXj8+LEAQIiIiBDr53zQzv17ND/p6elC27ZthVmzZhVYJ6et/v37C5mZmYIgCMKZM2cERUVF4cmTJ4IgCML9+/cFLS0t4d69e+JxXl5egqamphAWFiZ137kTIZcvXxb09PSEdevWFRnb7NmzxbIlS5YIAAR3d3exbPbs2UKDBg3EbWniLWnbd+7cERQUFIQLFy4IgiAIWVlZwuDBg4V69eqJ41fQ63P48GFBR0dH+PTpk1g2ePBgoXv37vme/9OnTwUAEte4nGuqt7e30L59e2H16tWCIAjC0aNHhapVqwohISH5JkLmzJkj7N+/X9i9e7cwadIkwdjYWLh161aePocNGyb06NEj33io7FH+BpNOiIiIqBSNHTtW/P9mzZrlWzZv3jykpaVBVVUV+/btg7GxsTi9WPj/y4Pp6urCx8cH1atXz9NHzlRofX19sUxDQwPR0dEIDw9HpUqVoKioiJ49e0ocp6+vn+80bSKi4jh//jzev38PfX19HD58GEL2F7ioWrUqvL290atXLwDAwoULceHCBbRr1w4BAQG4du0atLS0imx/ypQp+ZY/evQIr1+/RlJSEszMzHD79u0i2/rxxx+hqJi94kCnTp1gZ2eHw4cPo06dOti9ezesrKzw8uVLvHjxQvz9q6SkhFu3bqF3797F6tvT0xOTJ0/Gpk2bMGTIkCJjGzdunPj/OdeLz8s2btwobhcn3uK2feDAATRu3Bjt27cHACgoKGDevHmws7NDQEAAateuLdb9/PXp0aMH1NXVcfjwYQwfPhzx8fE4evQo9u3bl+9553cNy23MmDFYsmQJZs6ciW3btmH06NFQUFDIt+6NGzcQHByMjIwMPHv2DPXq1YOhoWGeevr6+gXemkplDxMhRERE5UzuP+zU1NTyLRMEQUyEvH79GqmpqTh8+LBEO87OzqhYsWK+feR8kMj9hILJkyfj0aNHsLGxQZ06deDi4oIpU6aIi8/l1JdmYVUiosK8fv0aKioqOHr0qES5lZUVqlWrJm4rKSlh0aJF6NChA4YPHw5HR0ep2s/9ewvIXkOpQ4cOePPmDRo2bAgdHR0EBQUhKSmpyLasrKwktq2trcX1MV6/fo3ExMQ8v3+7dOki/q4sTt/jx4+Hq6urVEkQQLrrRe6n6kgTb0nbfvPmjcQCowDE1/LNmzcSiZDPXx8VFRUMHz4cO3bswPDhw7Fv3z5oa2uja9eu+Z537mtYfkmL3r17Y9KkSfjvv/9w4cIFbN68Od92gOw1Qjp16gQge12SoUOHomfPnnj8+LFEPV7/yhcmQoiIiL5zOjo6qFixIjw8PKQ+xsLCAhoaGggODka9evXEdjw8PPDx40f4+vpi/fr1cHBwwIsXL6ChoQEge/G+mjVrfpXzICL5oaOjg/T0dOzatUv8kJ2f9PR0zJkzB7Vr18ahQ4cwZ84c2NraFtn+59/+//nnn0hKSkJISIi4uOaiRYuk+r0ZGxsLExMTie0aNWqI51GlSpVC2ylO3wcOHMDgwYNhb2+PmTNnFhlbcUkTb0kZGhrmmTERExMj7sstv9kZbm5uWL16NV6+fIkdO3Zg2LBhUFFRybevatWqQVFREcHBwbC0tMyzX01NDUOGDMHo0aPRvn17mJub4+3bt0Weg4KCArp27Yp9+/bh/fv3Eq97cHAw7OzsimyDygY+NYaIiKgMyPn2Kve3Z6WlU6dOePz4cZ7H+iUkJODjx4/5HqOiooKWLVtKHJPzlBgtLS107NgRf/zxB8LDw8UV+wVBwPXr18Vpz0REJdWuXTsAyPN0j4yMDERGRorbCxYsQHR0NHx9fdG9e3cMHjwYaWlpAIr3e/Xdu3ewtrYWExGZmZk4duyYVLHmrhceHo6bN2+iRYsWALJ//16/fl3iCV9AdgIgJ67i9N2yZUucPn0a7u7uWLt2rVTxFYc08ZaUk5MTrl27hqioKLHs0KFD0NfXl5gNUpCaNWvCyckJs2bNgp+fH0aPHl1gXV1dXTg4OBT6ONvx48ejU6dOmDVrVrHO48WLF1BRUZGY/ZKeno7bt2/z+leOcEYIERFRGaCnp4dq1aph+fLl6NmzJ0xNTaX6w1AanTp1wujRo9G5c2dMmTIFVatWRUBAAI4dO4aLFy8WeD/9Dz/8gB9//BErVqyAoqIiNm7cCD8/P3Tq1Al6enr4999/0aBBA9jY2AAArl69itTUVPTt27dU4iYi+WVhYYFVq1Zh+vTpePz4MRwcHBAaGorDhw9j7dq1cHFxweXLl7FmzRp4eXlBT08PW7ZsQd26dbFgwQKsXLkS5ubmMDU1xeLFi9GhQwdYWFhAXV093/569uyJjh07Yt68ebCyssL+/fvx5s0biW/8C7J+/XrEx8fD3Nwcf/75J5o0aYIePXoAAAYMGIAjR47A2dkZU6dOhbm5OZ4+fYoTJ07g1q1bUFdXL3bfLVu2xKlTp9ClSxcoKChg+vTpJR7nz0kTb0kNHDgQmzZtQps2bTBhwgRERERg1apVWL9+PSpUqCBVGz/88AOGDx+Opk2bFnmN/OGHH/DHH39g/vz5+e6vXbu2VDNfrly5gri4OGRkZODp06dYt24d5syZI/FY3tOnT0NXVxedO3eW6jxI9jgjhIiIqIw4c+YMLC0tcerUKfj4+EBDQwOurq7Q09MDAGhra8PV1VXiD0Y9PT24urqKt6YA2VOMXV1dJaYMb9++HQcOHEBiYiKuX78OMzMz3Lx5E1WqVCkwnt69e0NHR0e8R3/58uWYM2eO+I1nv379cOXKFXGRwA0bNmDGjBnQ1NQszWEhIjk1bdo0XL9+Hdra2rh27RqUlJRw/PhxuLi4AMj+8LlmzRq0bt0aQPbvw3379iEsLAzv37+HsrIyLly4AAMDA5w8eRK3bt2CgYEBXF1dJT7EAtkzUM6fP4+4uDjcunULQ4cOxZ49ewpcgyK3y5cvQ0tLC35+fhgxYgTOnj0r3tqhqKiIgwcPYseOHYiKisLNmzdhbW2Nu3fvims0SdP353G3atUKp06dwq1bt/KsVZFffQAwMjKCq6srlJSUxDIzMzO4urqK29LE+yVtX7hwARMmTMCdO3cQFxeHM2fOSCz2XdDrk6Nnz55QUFAodDZIjmHDhiEpKQkXL14EAPGaamxsnG99TU1Nif059YODg3Hs2DGcOXMGGRkZOHnyJH799VeJYzds2IDZs2dDWZnzDMoLBSFnKWAiIiKiz/j6+sLLywvu7u6F1vvw4QNmzZqFTZs2fdE3hkRE5YWfnx8aN26M2NhYMWFNX9fBgwfh5uaGsLAwqRYmvXDhAh48eICffvrpq8X09u1b/Prrr9i0aZNEIojKNiZCiIiIiIiIiomJkG/nzZs38Pb2xuLFizFkyBAsXbpU1iFROce5O0RERERERMVU1G0cVHpCQ0Nx4cIFjBs3Dj/++KOsw6HvAGeEEBEREREREZHc4GKpRERERERERCQ3mAghIiIiIiIiIrnBRAgRERERERERyQ0mQoiIiIiIiIhIbjARQkRERERERERyg4kQIiIiIiIiIpIbTIQQERERERERkdxgIoSIiIiIiIiI5AYTIUREREREREQkN/4fnGfIe8vW3V8AAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import matplotlib.pyplot as plt\n", "fig, ax = plt.subplots(1, 2, figsize=(11, 3.6), sharey=True)\n", "bench[\"seconds\"].plot.barh(ax=ax[0], color=\"#0ea5e9\"); ax[0].set_xlabel(\"time (s)\")\n", "bench[\"peak_mb\"].plot.barh(ax=ax[1], color=\"#4f46e5\"); ax[1].set_xlabel(\"extra peak memory (MB)\")\n", "ax[0].invert_yaxis(); plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "id": "c3835745", "metadata": {}, "source": [ "**What the numbers say.** Two different goals, two different winners:\n", "- **Speed** comes from the file format and the engine. Reading a columnar Parquet file with column pruning\n", " and a filter, or letting DuckDB or Polars run the query, is roughly 6 to 20 times faster than parsing the\n", " CSV, because nothing irrelevant is ever read or converted from text.\n", "- **Memory** comes from holding less. Loading the whole CSV with default types is by far the most\n", " memory-hungry option. Asking pandas for only three columns with compact types, or streaming in chunks,\n", " cuts the peak about five- to six-fold, but parsing text stays slow.\n", "- The extra memory shown for Parquet, Polars and DuckDB is mostly the libraries themselves being loaded\n", " (the Arrow and DuckDB engines are tens of MB each); at this data size that fixed cost dominates. It stays\n", " roughly constant as data grows, while a full pandas load grows with the file.\n", "\n", "In practice: store data as Parquet, select columns and types up front, and let a query engine (DuckDB or\n", "Polars) do the filtering and aggregation; stream when even the selected data does not fit." ] }, { "cell_type": "markdown", "id": "0706f4be", "metadata": {}, "source": [ "## 7. Beyond one machine: PySpark\n", "When data no longer fits on one machine (hundreds of GB, many TB), or arrives continuously from many\n", "sources, the standard tool is **Apache Spark**, used from Python through **PySpark**. Spark splits a\n", "dataset into **partitions** and processes them in parallel on many **executors** (worker processes, often on\n", "many servers), coordinated by a **driver** (your Python program). Like Polars' lazy mode, nothing runs until\n", "you ask for a result: Spark first builds a plan, and its optimiser (*Catalyst*) rewrites it, pushing filters\n", "and column selection down into the file scan.\n", "\n", "| Concept | Meaning |\n", "|---|---|\n", "| `SparkSession` | the entry point; connects to a cluster, or runs everything inside one machine with `local[*]` |\n", "| DataFrame | a distributed table with a schema, split into partitions |\n", "| transformation | `filter`, `select`, `groupBy`, `join`: lazy, only adds to the plan |\n", "| action | `show`, `count`, `collect`, `write`: runs the plan |\n", "| cluster manager | where executors come from: Kubernetes, YARN, Databricks, AWS EMR, Google Dataproc |\n", "\n", "The same code runs on a laptop in local mode and on a 100-node cluster; only the session configuration\n", "changes. Spark runs on the Java virtual machine, so it needs a Java runtime (17 or newer)." ] }, { "cell_type": "code", "execution_count": 11, "id": "6a9db8ec", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:01:00.535388Z", "iopub.status.busy": "2026-10-18T01:01:00.533543Z", "iopub.status.idle": "2026-10-18T01:01:47.893421Z", "shell.execute_reply": "2026-10-18T01:01:47.891297Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING: Using incubator modules: jdk.incubator.vector\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties\n", "26/10/18 03:01:06 WARN Utils: Your hostname, patito-HP-Laptop-15-db0xxx, resolves to a loopback address: 127.0.1.1; using 192.168.0.16 instead (on interface wlo1)\n", "26/10/18 03:01:06 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties\n", "Setting default log level to \"WARN\".\n", "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "26/10/18 03:01:19 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Spark 4.2.0\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 0:> (0 + 0) / 1]\r", "\r", "[Stage 0:> (0 + 1) / 1]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "root\n", " |-- sensor_id: short (nullable = true)\n", " |-- temperature: float (nullable = true)\n", " |-- status: string (nullable = true)\n", " |-- site: string (nullable = true)\n", " |-- note: string (nullable = true)\n", "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 1:> (0 + 4) / 4]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "partitions: 4 | rows: 3,000,000\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] } ], "source": [ "from pyspark.sql import SparkSession, functions as F, Window\n", "\n", "spark = (SparkSession.builder\n", " .master(\"local[4]\") # 4 worker threads on this machine\n", " .appName(\"python1001-large-data\")\n", " .config(\"spark.driver.memory\", \"2g\")\n", " .config(\"spark.sql.shuffle.partitions\", \"8\") # small data: few partitions after a shuffle\n", " .config(\"spark.ui.enabled\", \"false\")\n", " .getOrCreate())\n", "spark.sparkContext.setLogLevel(\"ERROR\")\n", "print(\"Spark\", spark.version)\n", "\n", "readings = spark.read.parquet(PARQUET) # lazy: only the schema is read\n", "readings.printSchema()\n", "print(\"partitions:\", readings.rdd.getNumPartitions(), \"| rows:\", f\"{readings.count():,}\")" ] }, { "cell_type": "markdown", "id": "5f09e2ee", "metadata": {}, "source": [ "### Two ways to write the same query\n", "Spark offers two equivalent interfaces, and teams mix them freely:\n", "\n", "1. **The DataFrame API**: method calls on the DataFrame, with column expressions from `pyspark.sql.functions`\n", " (`F.col`, `F.avg`, `F.when`, ...). Composable, checked by your editor, easy to build programmatically.\n", "2. **SQL**: register the DataFrame as a view and run a query string with `spark.sql(\"...\")`. Familiar to\n", " anyone who knows SQL, and convenient for long analytical queries.\n", "\n", "**Query 1:** mean temperature and count of `fault` readings per site." ] }, { "cell_type": "code", "execution_count": 12, "id": "83f4f0ae", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:01:47.901996Z", "iopub.status.busy": "2026-10-18T01:01:47.901235Z", "iopub.status.idle": "2026-10-18T01:01:53.578590Z", "shell.execute_reply": "2026-10-18T01:01:53.575712Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 4:> (0 + 4) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "+-------+---------+--------+\n", "| site|mean_temp|n_faults|\n", "+-------+---------+--------+\n", "| site_0| 21.058| 1510|\n", "| site_1| 21.014| 1453|\n", "|site_10| 20.882| 1456|\n", "|site_11| 20.892| 1484|\n", "|site_12| 21.072| 1512|\n", "+-------+---------+--------+\n", "only showing top 5 rows\n" ] } ], "source": [ "# 1) DataFrame API\n", "api_result = (readings\n", " .filter(F.col(\"status\") == \"fault\")\n", " .groupBy(\"site\")\n", " .agg(F.round(F.avg(\"temperature\"), 3).alias(\"mean_temp\"), F.count(\"*\").alias(\"n_faults\"))\n", " .orderBy(\"site\"))\n", "api_result.show(5)" ] }, { "cell_type": "code", "execution_count": 13, "id": "537cdf84", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:01:53.594035Z", "iopub.status.busy": "2026-10-18T01:01:53.593027Z", "iopub.status.idle": "2026-10-18T01:02:00.697562Z", "shell.execute_reply": "2026-10-18T01:02:00.694073Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 7:> (0 + 4) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 7:============================================> (3 + 1) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "+-------+---------+--------+\n", "| site|mean_temp|n_faults|\n", "+-------+---------+--------+\n", "| site_0| 21.058| 1510|\n", "| site_1| 21.014| 1453|\n", "|site_10| 20.882| 1456|\n", "|site_11| 20.892| 1484|\n", "|site_12| 21.072| 1512|\n", "+-------+---------+--------+\n", "only showing top 5 rows\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 18:===========================================> (3 + 1) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "identical results: True\n" ] } ], "source": [ "# 2) SQL: the DataFrame becomes a temporary view that queries can refer to by name\n", "readings.createOrReplaceTempView(\"readings\")\n", "sql_result = spark.sql('''\n", " SELECT site,\n", " ROUND(AVG(temperature), 3) AS mean_temp,\n", " COUNT(*) AS n_faults\n", " FROM readings\n", " WHERE status = 'fault'\n", " GROUP BY site\n", " ORDER BY site\n", "''')\n", "sql_result.show(5)\n", "print(\"identical results:\", api_result.collect() == sql_result.collect())" ] }, { "cell_type": "markdown", "id": "7a017527", "metadata": {}, "source": [ "Both are compiled by the same optimiser into the same kind of physical plan, so neither is faster. The plan\n", "shows the optimisation at work: the filter on `status` and the three needed columns are pushed into the\n", "Parquet scan (`PushedFilters`, `ReadSchema`), so the other columns are never read." ] }, { "cell_type": "code", "execution_count": 14, "id": "0f688a0b", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:02:00.706532Z", "iopub.status.busy": "2026-10-18T01:02:00.703991Z", "iopub.status.idle": "2026-10-18T01:02:00.782535Z", "shell.execute_reply": "2026-10-18T01:02:00.779614Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "== Physical Plan ==\n", "AdaptiveSparkPlan isFinalPlan=true\n", "+- == Final Plan ==\n", " ResultQueryStage 2\n", " +- *(3) Sort [site#3 ASC NULLS FIRST], true, 0\n", " +- AQEShuffleRead coalesced\n", " +- ShuffleQueryStage 1\n", " +- Exchange rangepartitioning(site#3 ASC NULLS FIRST, 8), ENSURE_REQUIREMENTS, [plan_id=260]\n", " +- *(2) HashAggregate(keys=[site#3], functions=[avg(temperature#1), count(1)])\n", " +- AQEShuffleRead coalesced\n", " +- ShuffleQueryStage 0\n", " +- Exchange hashpartitioning(site#3, 8), ENSURE_REQUIREMENTS, [plan_id=228]\n", " +- *(1) HashAggregate(keys=[site#3], functions=[partial_avg(temperature#1), partial_count(1)])\n", " +- *(1) Project [temperature#1, site#3]\n", " +- *(1) Filter (isnotnull(status#2) AND (status#2 = fault))\n", " +- *(1) ColumnarToRow\n", " +- FileScan parquet [temperature#1,status#2,site#3] Batched: true, DataFilters: [isnotnull(status#2), (status#2 = fault)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/tmpwz1in30w/big.parquet], PartitionFilters: [], PushedFilters: [IsNotNull(status), EqualTo(status,fault)], ReadSchema: struct\n", "+- == Initial Plan ==\n", " Sort [site#3 ASC NULLS FIRST], true, 0\n", " +- Exchange rangepartitioning(site#3 ASC NULLS FIRST, 8), ENSURE_REQUIREMENTS, [plan_id=202]\n", " +- HashAggregate(keys=[site#3], functions=[avg(temperature#1), count(1)])\n", " +- Exchange hashpartitioning(site#3, 8), ENSURE_REQUIREMENTS, [plan_id=199]\n", " +- HashAggregate(keys=[site#3], functions=[partial_avg(temperature#1), partial_count(1)])\n", " +- Project [temperature#1, site#3]\n", " +- Filter (isnotnull(status#2) AND (status#2 = fault))\n", " +- FileScan parquet [temperature#1,status#2,site#3] Batched: true, DataFilters: [isnotnull(status#2), (status#2 = fault)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/tmpwz1in30w/big.parquet], PartitionFilters: [], PushedFilters: [IsNotNull(status), EqualTo(status,fault)], ReadSchema: struct\n", "\n", "\n", "== Physical Plan ==\n", "AdaptiveSparkPlan isFinalPlan=true\n", "+- == Final Plan ==\n", " ResultQueryStage 2\n", " +- *(3) Sort [site#3 ASC NULLS FIRST], true, 0\n", " +- AQEShuffleRead coalesced\n", " +- ShuffleQueryStage 1\n", " +- Exchange rangepartitioning(site#3 ASC NULLS FIRST, 8), ENSURE_REQUIREMENTS, [plan_id=360]\n", " +- *(2) HashAggregate(keys=[site#3], functions=[avg(temperature#1), count(1)])\n", " +- AQEShuffleRead coalesced\n", " +- ShuffleQueryStage 0\n", " +- Exchange hashpartitioning(site#3, 8), ENSURE_REQUIREMENTS, [plan_id=328]\n", " +- *(1) HashAggregate(keys=[site#3], functions=[partial_avg(temperature#1), partial_count(1)])\n", " +- *(1) Project [temperature#1, site#3]\n", " +- *(1) Filter (isnotnull(status#2) AND (status#2 = fault))\n", " +- *(1) ColumnarToRow\n", " +- FileScan parquet [temperature#1,status#2,site#3] Batched: true, DataFilters: [isnotnull(status#2), (status#2 = fault)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/tmpwz1in30w/big.parquet], PartitionFilters: [], PushedFilters: [IsNotNull(status), EqualTo(status,fault)], ReadSchema: struct\n", "+- == Initial Plan ==\n", " Sort [site#3 ASC NULLS FIRST], true, 0\n", " +- Exchange rangepartitioning(site#3 ASC NULLS FIRST, 8), ENSURE_REQUIREMENTS, [plan_id=302]\n", " +- HashAggregate(keys=[site#3], functions=[avg(temperature#1), count(1)])\n", " +- Exchange hashpartitioning(site#3, 8), ENSURE_REQUIREMENTS, [plan_id=299]\n", " +- HashAggregate(keys=[site#3], functions=[partial_avg(temperature#1), partial_count(1)])\n", " +- Project [temperature#1, site#3]\n", " +- Filter (isnotnull(status#2) AND (status#2 = fault))\n", " +- FileScan parquet [temperature#1,status#2,site#3] Batched: true, DataFilters: [isnotnull(status#2), (status#2 = fault)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/tmpwz1in30w/big.parquet], PartitionFilters: [], PushedFilters: [IsNotNull(status), EqualTo(status,fault)], ReadSchema: struct\n", "\n", "\n" ] } ], "source": [ "api_result.explain() # the DataFrame API version\n", "sql_result.explain() # the SQL version: the same operators" ] }, { "cell_type": "markdown", "id": "96f306c4", "metadata": {}, "source": [ "**Query 2:** something richer. Per site, the share of readings in each status and the three sensors with the\n", "highest average temperature, using a derived column, a pivot, a join and a **window function**." ] }, { "cell_type": "code", "execution_count": 15, "id": "8811c4a7", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:02:00.793118Z", "iopub.status.busy": "2026-10-18T01:02:00.790492Z", "iopub.status.idle": "2026-10-18T01:02:12.280891Z", "shell.execute_reply": "2026-10-18T01:02:12.279043Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 26:> (0 + 4) / 4][Stage 27:> (0 + 0) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 26:====> (1 + 3) / 4][Stage 27:> (0 + 1) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 26:====> (1 + 3) / 4][Stage 27:> (0 + 2) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 26:=============> (3 + 1) / 4][Stage 27:> (0 + 3) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 27:> (0 + 4) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 27:> (0 + 4) / 4][Stage 29:> (0 + 0) / 1]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 27:====> (1 + 3) / 4][Stage 29:> (0 + 1) / 1]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 27:=============> (3 + 1) / 4][Stage 29:> (0 + 1) / 1]\r", "\r", "[Stage 27:===========================================> (3 + 1) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 34:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "+------+---------+------------------+------------+----+------+-----------+---------+\n", "| site|sensor_id| avg_t|hot_readings|rank|pct_ok|pct_warning|pct_fault|\n", "+------+---------+------------------+------------+----+------+-----------+---------+\n", "|site_0| 195|22.334422131518384| 14| 1| 89.92| 8.06| 2.02|\n", "|site_0| 191|22.060088488459588| 11| 2| 89.92| 8.06| 2.02|\n", "|site_0| 217|21.775642860226515| 8| 3| 89.92| 8.06| 2.02|\n", "|site_1| 313| 22.034557895806| 12| 1| 90.11| 7.96| 1.94|\n", "|site_1| 228|21.965595207214356| 8| 2| 90.11| 7.96| 1.94|\n", "|site_1| 107|21.913695235925218| 12| 3| 90.11| 7.96| 1.94|\n", "+------+---------+------------------+------------+----+------+-----------+---------+\n", "only showing top 6 rows\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] } ], "source": [ "# DataFrame API\n", "flagged = readings.withColumn(\"is_hot\", F.when(F.col(\"temperature\") > 28, 1).otherwise(0))\n", "status_share = (flagged.groupBy(\"site\").pivot(\"status\", [\"ok\", \"warning\", \"fault\"]).count()\n", " .withColumn(\"total\", F.col(\"ok\") + F.col(\"warning\") + F.col(\"fault\"))\n", " .select(\"site\", *[F.round(F.col(s) / F.col(\"total\") * 100, 2).alias(f\"pct_{s}\") for s in [\"ok\", \"warning\", \"fault\"]]))\n", "by_sensor = flagged.groupBy(\"site\", \"sensor_id\").agg(F.avg(\"temperature\").alias(\"avg_t\"), F.sum(\"is_hot\").alias(\"hot_readings\"))\n", "rank_w = Window.partitionBy(\"site\").orderBy(F.col(\"avg_t\").desc())\n", "top3_api = (by_sensor.withColumn(\"rank\", F.row_number().over(rank_w)).filter(F.col(\"rank\") <= 3)\n", " .join(F.broadcast(status_share), \"site\") # small table: send it to every executor\n", " .orderBy(\"site\", \"rank\"))\n", "top3_api.show(6)" ] }, { "cell_type": "code", "execution_count": 16, "id": "0f15a04e", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:02:12.292115Z", "iopub.status.busy": "2026-10-18T01:02:12.290067Z", "iopub.status.idle": "2026-10-18T01:02:32.676463Z", "shell.execute_reply": "2026-10-18T01:02:32.669526Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 38:> (0 + 4) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 38:> (0 + 4) / 4][Stage 39:> (0 + 0) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 38:====> (1 + 3) / 4][Stage 39:> (0 + 1) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 39:> (0 + 4) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 39:> (0 + 4) / 4][Stage 41:> (0 + 0) / 1]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 39:====> (1 + 3) / 4][Stage 41:> (0 + 1) / 1]\r", "\r", "[Stage 39:=========> (2 + 2) / 4][Stage 41:> (0 + 1) / 1]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 39:=============> (3 + 1) / 4][Stage 41:> (0 + 1) / 1]\r", "\r", "[Stage 41:> (0 + 1) / 1]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "+------+---------+------------------+------------+----+------+-----------+---------+\n", "| site|sensor_id| avg_t|hot_readings|rank|pct_ok|pct_warning|pct_fault|\n", "+------+---------+------------------+------------+----+------+-----------+---------+\n", "|site_0| 195|22.334422131518384| 14| 1| 89.92| 8.06| 2.02|\n", "|site_0| 191|22.060088488459588| 11| 2| 89.92| 8.06| 2.02|\n", "|site_0| 217|21.775642860226515| 8| 3| 89.92| 8.06| 2.02|\n", "|site_1| 313| 22.034557895806| 12| 1| 90.11| 7.96| 1.94|\n", "|site_1| 228|21.965595207214356| 8| 2| 90.11| 7.96| 1.94|\n", "|site_1| 107|21.913695235925218| 12| 3| 90.11| 7.96| 1.94|\n", "+------+---------+------------------+------------+----+------+-----------+---------+\n", "only showing top 6 rows\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 47:> (0 + 4) / 4]\r", "\r", "[Stage 47:> (0 + 4) / 4][Stage 48:> (0 + 0) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 47:====> (1 + 3) / 4][Stage 48:> (0 + 1) / 4]\r", "\r", "[Stage 47:=========> (2 + 2) / 4][Stage 48:> (0 + 2) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 48:> (0 + 4) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 48:> (0 + 4) / 4][Stage 50:> (0 + 0) / 1]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 48:====> (1 + 3) / 4][Stage 50:> (0 + 1) / 1]\r", "\r", "[Stage 48:=============================> (2 + 2) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 48:===========================================> (3 + 1) / 4]\r", "\r", " \r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 55:> (0 + 1) / 1]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 66:> (0 + 4) / 4][Stage 67:> (0 + 0) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 66:====> (1 + 3) / 4][Stage 67:> (0 + 1) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 66:=========> (2 + 2) / 4][Stage 67:> (0 + 2) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 66:=============> (3 + 1) / 4][Stage 67:> (0 + 3) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 66:=============> (3 + 1) / 4][Stage 67:====> (1 + 3) / 4]\r", "\r", "[Stage 67:==============> (1 + 3) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "same rows: True\n" ] } ], "source": [ "# SQL: the same logic, with CTEs, CASE WHEN, conditional aggregation and ROW_NUMBER() OVER (...)\n", "top3_sql = spark.sql('''\n", " WITH flagged AS (\n", " SELECT *, CASE WHEN temperature > 28 THEN 1 ELSE 0 END AS is_hot FROM readings\n", " ),\n", " status_share AS (\n", " SELECT site,\n", " ROUND(100 * AVG(CASE WHEN status = 'ok' THEN 1 ELSE 0 END), 2) AS pct_ok,\n", " ROUND(100 * AVG(CASE WHEN status = 'warning' THEN 1 ELSE 0 END), 2) AS pct_warning,\n", " ROUND(100 * AVG(CASE WHEN status = 'fault' THEN 1 ELSE 0 END), 2) AS pct_fault\n", " FROM flagged GROUP BY site\n", " ),\n", " ranked AS (\n", " SELECT site, sensor_id, AVG(temperature) AS avg_t, SUM(is_hot) AS hot_readings,\n", " ROW_NUMBER() OVER (PARTITION BY site ORDER BY AVG(temperature) DESC) AS rank\n", " FROM flagged GROUP BY site, sensor_id\n", " )\n", " SELECT r.*, s.pct_ok, s.pct_warning, s.pct_fault\n", " FROM ranked r JOIN status_share s USING (site)\n", " WHERE rank <= 3\n", " ORDER BY site, rank\n", "''')\n", "top3_sql.show(6)\n", "cols = [\"site\", \"sensor_id\", \"rank\", \"pct_fault\"]\n", "print(\"same rows:\", top3_api.select(cols).collect() == top3_sql.select(cols).collect())" ] }, { "cell_type": "markdown", "id": "e1725b0d", "metadata": {}, "source": [ "### Getting results out, and writing data\n", "`toPandas()` brings a result to the driver as a pandas DataFrame: do it only with **small** results, never\n", "with the raw data. Large outputs are written back to storage, usually Parquet partitioned by a column so\n", "later queries can skip whole folders." ] }, { "cell_type": "code", "execution_count": 17, "id": "c3f510d5", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:02:32.688781Z", "iopub.status.busy": "2026-10-18T01:02:32.686026Z", "iopub.status.idle": "2026-10-18T01:02:48.876267Z", "shell.execute_reply": "2026-10-18T01:02:48.874232Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 82:> (0 + 4) / 4][Stage 83:> (0 + 0) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 82:====> (1 + 3) / 4][Stage 83:> (0 + 1) / 4]\r", "\r", "[Stage 82:=============> (3 + 1) / 4][Stage 83:> (0 + 3) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 83:> (0 + 4) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 83:> (0 + 4) / 4][Stage 85:> (0 + 0) / 1]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 83:====> (1 + 3) / 4][Stage 85:> (0 + 1) / 1]\r", "\r", " \r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 83:=============================> (2 + 2) / 4]\r", "\r", "[Stage 83:===========================================> (3 + 1) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 87:> (0 + 1) / 1]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 97:> (0 + 1) / 1]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "DataFrame (120, 8)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 98:> (0 + 4) / 4]\r" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "[Stage 98:===========================================> (3 + 1) / 4]\r", "\r", " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "['status=fault', 'status=warning']\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "+-------+------+\n", "| status| n|\n", "+-------+------+\n", "| fault| 59835|\n", "|warning|240089|\n", "+-------+------+\n", "\n" ] } ], "source": [ "small = top3_sql.toPandas() # fine: a few hundred rows\n", "print(type(small).__name__, small.shape)\n", "\n", "out = os.path.join(tmp, \"spark_out\")\n", "(readings.filter(F.col(\"status\") != \"ok\")\n", " .write.mode(\"overwrite\").partitionBy(\"status\").parquet(out))\n", "print(sorted(d for d in os.listdir(out) if not d.startswith((\"_\", \".\"))))\n", "# SQL can also query files directly, without registering a view first\n", "spark.sql(f\"SELECT status, COUNT(*) AS n FROM parquet.`{out}` GROUP BY status ORDER BY status\").show()" ] }, { "cell_type": "markdown", "id": "1371e9e8", "metadata": {}, "source": [ "### Is Spark faster here?" ] }, { "cell_type": "code", "execution_count": 18, "id": "d969e0d9", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T01:02:48.898366Z", "iopub.status.busy": "2026-10-18T01:02:48.897610Z", "iopub.status.idle": "2026-10-18T01:02:49.604075Z", "shell.execute_reply": "2026-10-18T01:02:49.601458Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "query 1 with both engines already running: Spark 0.19 s, DuckDB 0.13 s\n" ] } ], "source": [ "import time\n", "t0 = time.perf_counter(); api_result.collect(); t_spark = time.perf_counter() - t0\n", "t0 = time.perf_counter()\n", "duckdb.sql(f\"select site, avg(temperature), count(*) from '{PARQUET}' where status = 'fault' group by site\").df()\n", "t_duck = time.perf_counter() - t0\n", "print(f\"query 1 with both engines already running: Spark {t_spark:.2f} s, DuckDB {t_duck:.2f} s\")\n", "spark.stop()" ] }, { "cell_type": "markdown", "id": "cb0c9b0d", "metadata": {}, "source": [ "On one machine and a few million rows, no: Spark plans the query, schedules tasks and moves data between\n", "them in a separate Java process, overheads that single-node engines like DuckDB and Polars avoid. Spark pays\n", "off when the data or the work is too large for one machine, when the data already lives in a data lake or\n", "warehouse that Spark reads natively, or for streaming pipelines. A sensible path: pandas, then Polars or\n", "DuckDB, then Spark.\n", "\n", "| Tool | Function-style API | SQL style |\n", "|---|---|---|\n", "| **PySpark** | `df.filter(F.col(\"x\") > 1).groupBy(\"k\").agg(F.avg(\"v\"))` | `df.createOrReplaceTempView(\"t\")`, then `spark.sql(\"SELECT k, AVG(v) FROM t WHERE x > 1 GROUP BY k\")` |\n", "| **Polars** | `df.filter(pl.col(\"x\") > 1).group_by(\"k\").agg(pl.col(\"v\").mean())` | `df.sql(\"SELECT k, AVG(v) FROM self WHERE x > 1 GROUP BY k\")` |\n", "| **DuckDB** | relational API: `duckdb.table(\"t\").filter(\"x > 1\").aggregate(\"k, avg(v)\")` | `duckdb.sql(\"SELECT ...\")`, which can also query pandas DataFrames by name |\n", "| **pandas** | `df[df.x > 1].groupby(\"k\")[\"v\"].mean()` | `df.query(\"x > 1\")` takes a filter expression, not full SQL |\n", "\n", "Spark also offers a pandas-like API (`import pyspark.pandas as ps`) for teams moving pandas code to a cluster." ] }, { "cell_type": "markdown", "id": "4b81f70c", "metadata": {}, "source": [ "## Checklist for memory-constrained processing\n", "1. **Measure first**: `memory_usage(deep=True)`, `psutil`, or the peak of a test run.\n", "2. **Store as Parquet**, not CSV, and read only the needed columns and rows.\n", "3. **Use the right types**: `category` for repeated strings, `float32`/small integers where precision allows.\n", "4. **Stream** when the result is small: chunks, generators, or a lazy engine.\n", "5. **Delete what you no longer need** (`del df`) and avoid accidental copies.\n", "6. **Let a query engine do the work** (DuckDB, Polars) when pandas would load everything.\n", "7. Only then think about more RAM or a cluster (Spark, section 7).\n", "\n", "## Exercises\n", "1. Repeat the benchmark with `sensor_id` as the group key. Does the ranking change?\n", "2. Add an approach using `pyarrow.dataset` with a filter expression and compare it.\n", "3. Use `pl.scan_csv` instead of `scan_parquet`: how much of Polars' advantage comes from the file format?\n", "4. In PySpark, find the hottest single reading per site twice: with `Window` and `F.max`, and in SQL with `MAX(...) OVER (PARTITION BY site)`. Check both give the same rows." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 5 }