melissa-uploader/README.md

# Melissa Uploader

Automated upload tool for practice exports. Uploads insurance PDFs and patient files to the correct FTP/SFTP servers for each practice, then archives what was sent.

---

## First-time setup

1. Double-click **setup.bat** — this downloads WinSCP (needed for uploading)
2. Set up your entity folders (see below)
3. Double-click **run-all.bat** each day after running the PT STMT rename script

---

## Daily workflow

1. Run the **PT STMT rename script** manually (as usual)
2. Double-click **run-all.bat** to upload everything
3. Check the window for any failures — failed files stay in place for retry

---

## Folder layout

```
run-all.bat                        <- double-click to run everything
engine.ps1
setup.bat
WinSCP\                            <- created by setup.bat automatically
Entities\
  CAMBS\
    entity.json                    <- server details for this entity
    run.bat                        <- double-click to run just CAMBS
    Export1\
      AJMAT\
        practice.json              <- only needed if AJMAT has its own FTP login
        20260629\                  <- today's insurance PDFs go here
        Archive sent to FTP\       <- uploaded date folders move here
        Patient\
          Export\
            PT STMT_20260629\      <- created by rename script
            Archive Sent to FTP\   <- uploaded patient folders move here
      CIIR\
        ...
  CONSensio\
    entity.json
    ...
```

---

## The two config files

There are two JSON files, and they work together — one sets the defaults for
the whole entity, the other overrides just what's different for one practice.

- **`entity.json`** — lives in the entity folder (e.g. `Entities\CAMBS\entity.json`).
  Required. This is the default server info used by every practice under that
  entity: which server, which login, which folder paths.
- **`practice.json`** — lives inside one practice's folder (e.g.
  `Entities\CAMBS\Export1\AJMAT\practice.json`). Optional — only add it if a
  practice needs something *different* from `entity.json`. Whatever fields you
  put in here replace the matching field from `entity.json` for that practice
  only; anything you leave out still comes from `entity.json`.

Common reasons to add a `practice.json`:
- The practice has its own separate FTP/SFTP login instead of the shared one
- The practice's files need to go to a **different folder path** on the
  server than the standard `{practice}` pattern (e.g. the imaging vendor set
  it up with a different folder name, or a nonstandard structure) — override
  `tiff_path`, `pdf_path`, and/or `patient_path` to whatever that
  practice actually needs

You can mix and match — e.g. override just `pdf_path` for one practice and
leave its login and every other path alone.

### `entity.json`

1. Create a folder under `Entities\` with the entity name (e.g. `Entities\CAMBS\`)
2. Copy `run.bat` into it
3. Create `entity.json` in that folder — copy from `Entities\EXAMPLE\entity.json`
   and fill in your real values. A complete file looks like this:

```json
{
  "workflow": "insurance+patient",
  "insurance_file_type": "pdf",

  "ftps": {
    "host": "ftp.example.com",
    "port": 21,
    "tls": true,
    "username": "shared-username",
    "password": "shared-password",
    "tiff_path": "/{practice}/Insurance",
    "pdf_path": "/{practice}/PDF"
  },

  "sftp": {
    "host": "sftp.example.com",
    "port": 22,
    "username": "shared-username",
    "password": "shared-password",
    "patient_path": "/{practice}/Patient"
  }
}
```

4. Create the `Export1\` folder and a subfolder for each practice inside the entity folder

---

## Every setting, explained

All of these can appear in `entity.json` (entity-wide default) — and everything
except `workflow` and `insurance_file_type` can also appear in a
`practice.json` to override or add a value for just that practice.

### Top level (entity.json only)

| Setting | Values | What it does |
|---------|--------|--------------|
| `workflow` | `"insurance"` or `"insurance+patient"` | `insurance` = only the daily date-folder upload. `insurance+patient` = also look for `Patient\Export\PT STMT_<date>\` folders and send their `.txt`/`.tif` files via SFTP |
| `insurance_file_type` | `"pdf"` (default if null) or `"tif"` | Which file extension to pick up from the daily date folder. TIF entities: CAMBS, RMI, CONSENSIO, INLAND. The PT STMT split works either way (it goes by file *name*, not type) |

### `ftps` section — the daily date-folder upload

| Setting | Values | What it does |
|---------|--------|--------------|
| `host` | server name or URL | Where the daily files go. Three forms: plain name (`ftp.example.com`) = FTP or FTPS depending on `tls`; `sftp://server.com` = send over **SFTP** instead (some partners receive everything by SFTP — Pollux is one); `ftps://server.com` = force secure FTP regardless of `tls` |
| `port` | number, no quotes | Only needed if nonstandard. Null = 21 for FTP/FTPS, 22 for `sftp://` hosts |
| `tls` | `true` / `false`, no quotes | `true` = secure FTPS, `false`/null = plain FTP. Ignored when `host` starts with `sftp://` (SFTP is always encrypted) |
| `username` | text | Login for this server |
| `password` | text | Password for this server |
| `tiff_path` | remote folder path | Where the regular daily files (insurance TIFs/PDFs) land on the server, e.g. `"/home/clients/CAMBS/{practice}"`. The older name `insurance_path` still works |
| `pdf_path` | remote folder path | Where PT STMT-named files from the date folder land. Null = this entity has no PT STMTs; if one shows up anyway it's left in place with a notice |

### `sftp` section — the patient-files upload (only used when `workflow` is `insurance+patient`)

| Setting | Values | What it does |
|---------|--------|--------------|
| `host` | server name or URL | Patient-files SFTP server (a `sftp://` prefix is fine and stripped automatically) |
| `port` | number, no quotes | Null = 22 |
| `username` | text | Login |
| `password` | text | Password |
| `patient_path` | remote folder path | Where the `.txt`/`.tif` patient files land, e.g. `"/upload/{practice}/Patient"` |

### Special values that work inside any path

| Token | Becomes |
|-------|---------|
| `{practice}` | The practice's local folder name (e.g. `AJMATS`, `CIIR`) — lets one path line serve every practice under the entity |

### Special values that work anywhere

| Value | Meaning |
|-------|---------|
| `null` (no quotes) | "We don't use this — ignore it." Files that would need the setting are left in place with a yellow notice, never uploaded, never archived, never counted as failures |
| any key starting with `_` (like `"_comment"`) | Ignored by the tool — use it for notes to yourself inside the file |

**Missing keys fix themselves:** on every run, the tool checks `entity.json`
for any keys it knows about that aren't in the file, adds them with a value
of `null`, and saves the file — so you never have to type key names by hand.
A `null` value means **"we don't use this — ignore it"**: any files that
would need that setting are left in place untouched (with a yellow notice in
the output), not treated as failures. To turn a feature on, just open
`entity.json` and replace the `null` with a real value in quotes.

**Editing JSON — the rules that bite:**
- Key names must match **exactly** (all lowercase, underscores): `host`, not
  `hostname`; `username`, not `user`; `tiff_path`, not `tiffpath`
- Every `"key": "value"` pair ends with a comma **except the last one in its
  block** — a missing or extra comma breaks the whole file
- Keep values inside straight double quotes (`"`). Avoid editing in Word or
  anything that turns quotes curly — use Notepad
- `port` numbers and `tls` true/false do **not** get quotes; everything else does

---

## `practice.json` — overriding one practice

If one practice needs a different login, a different folder path, or both:

1. Create a `practice.json` file inside that practice's folder (next to its
   date folders, e.g. `Entities\CAMBS\Export1\AJMAT\practice.json`)
2. Copy from `Entities\EXAMPLE\Export1\PRACTICE_NAME\practice.json`
3. Delete every line you *don't* need to change — keep only the fields that
   differ for this practice. Everything you delete falls back to `entity.json`.

The field names are the same ones as `entity.json`, wrapped in the same
`ftps` / `sftp` sections. A `practice.json` can also **add** a field that
`entity.json` doesn't have at all — e.g. if `entity.json` has no
`tiff_path` because every practice's path is different, each practice's
`practice.json` can carry its own.

Example — this practice only has its own PDF folder name, nothing else differs:

```json
{
  "ftps": {
    "pdf_path": "/{practice}/PDF_Statements_Custom"
  }
}
```

Example — this practice has its own SFTP login but uses the same paths as everyone else:

```json
{
  "sftp": {
    "username": "ajmat-sftp-user",
    "password": "ajmat-sftp-pass"
  }
}
```

Example — this practice has completely different remote folders for everything:

```json
{
  "ftps": {
    "tiff_path": "/SpecialFolder/Claims",
    "pdf_path": "/SpecialFolder/Statements"
  },
  "sftp": {
    "patient_path": "/SpecialFolder/Patients"
  }
}
```

Note: a `practice.json` never needs `workflow` or `insurance_file_type` —
those are entity-wide only.

---

## Something went wrong?

- **Files failed to upload** — they stay in the date folder. Fix the issue and run again.
- **A date folder is still there after running** — one or more files failed. Check the output window.
- **WinSCP error on startup** — run `setup.bat` again to re-download WinSCP.
- **`... upload not configured (ftps.xyz is null/blank) - N file(s) ignored`** —
  there are files to send, but the listed settings are still `null` (or blank)
  in that entity's `entity.json`. If those files *should* upload, open
  `entity.json` and fill in real values for the listed keys. If that upload
  type genuinely doesn't apply to this entity, the message is normal and the
  files just stay where they are.
- **`Added missing key(s) to entity.json ...`** — informational, not an error.
  The tool added keys that were absent (as `null`) so they're ready to fill in.
- **`No .pdf files in <date>\` but the folder isn't empty** — the files in the
  date folder are probably TIFs. Set `"insurance_file_type": "tif",` at the top
  level of that entity's `entity.json` (right under `workflow`).
- **The startup banner shows the wrong file type** — the banner line
  (`Entity: ... | File type: .pdf | ...`) shows what was actually read from
  `entity.json`, so it's the quickest way to confirm your edit took effect.