melissa-uploader/README.md

# Melissa Uploader

Automated upload tool for practice exports. Uploads insurance PDFs and patient files to the correct FTP/SFTP servers for each practice, then archives what was sent.

---

## First-time setup

1. Double-click **setup.bat** — this downloads WinSCP (needed for uploading)
2. Set up your entity folders (see below)
3. Double-click **run-all.bat** each day after running the PT STMT rename script

---

## Daily workflow

1. Run the **PT STMT rename script** manually (as usual)
2. Double-click **run-all.bat** to upload everything
3. Check the window for any failures — failed files stay in place for retry

---

## Folder layout

```
run-all.bat                        <- double-click to run everything
engine.ps1
setup.bat
WinSCP\                            <- created by setup.bat automatically
Entities\
  CAMBS\
    entity.json                    <- server details for this entity
    run.bat                        <- double-click to run just CAMBS
    Export1\
      AJMAT\
        practice.json              <- only needed if AJMAT has its own FTP login
        20260629\                  <- today's insurance PDFs go here
        Archive sent to FTP\       <- uploaded date folders move here
        Patient\
          Export\
            PT STMT_20260629\      <- created by rename script
            Archive Sent to FTP\   <- uploaded patient folders move here
      CIIR\
        ...
  CONSensio\
    entity.json
    ...
```

---

## The two config files

There are two JSON files, and they work together — one sets the defaults for
the whole entity, the other overrides just what's different for one practice.

- **`entity.json`** — lives in the entity folder (e.g. `Entities\CAMBS\entity.json`).
  Required. This is the default server info used by every practice under that
  entity: which server, which login, which folder paths.
- **`practice.json`** — lives inside one practice's folder (e.g.
  `Entities\CAMBS\Export1\AJMAT\practice.json`). Optional — only add it if a
  practice needs something *different* from `entity.json`. Whatever fields you
  put in here replace the matching field from `entity.json` for that practice
  only; anything you leave out still comes from `entity.json`.

Common reasons to add a `practice.json`:
- The practice has its own separate FTP/SFTP login instead of the shared one
- The practice's files need to go to a **different folder path** on the
  server than the standard `{practice}` pattern (e.g. the imaging vendor set
  it up with a different folder name, or a nonstandard structure) — override
  `tiff_path`, `pdf_path`, and/or `patient_path` to whatever that
  practice actually needs

You can mix and match — e.g. override just `pdf_path` for one practice and
leave its login and every other path alone.

### `entity.json`

1. Create a folder under `Entities\` with the entity name (e.g. `Entities\CAMBS\`)
2. Copy `run.bat` into it
3. Create `entity.json` in that folder — copy from `Entities\EXAMPLE\entity.json`
   and fill in your real values. A complete file looks like this:

```json
{
  "workflow": "insurance+patient",
  "insurance_file_type": "pdf",

  "ftps": {
    "host": "ftp.example.com",
    "port": 21,
    "tls": true,
    "username": "shared-username",
    "password": "shared-password",
    "tiff_path": "/{practice}/Insurance",
    "pdf_path": "/{practice}/PDF"
  },

  "sftp": {
    "host": "sftp.example.com",
    "port": 22,
    "username": "shared-username",
    "password": "shared-password",
    "patient_path": "/{practice}/Patient"
  }
}
```

| Setting | Required? | What to put |
|---------|-----------|------------|
| `workflow` | Yes | `insurance` if no patient files, `insurance+patient` if both |
| `insurance_file_type` | No (defaults to `pdf`) | Set to `tif` for an entity whose imaging system exports the daily files as TIFs instead of PDFs (CAMBS, RMI, CONSENSIO, INLAND). Goes at the top level, next to `workflow` — NOT inside `ftps` |
| `ftps.host` | Yes | FTP server address |
| `ftps.port` | No (defaults to 21) | FTP port, only if not 21 |
| `ftps.tls` | Yes | `true` for FTPS (secure), `false` for plain FTP |
| `ftps.username` | Yes | FTP username (shared across all practices) |
| `ftps.password` | Yes | FTP password |
| `ftps.tiff_path` | Yes | Remote folder for insurance files — keep `{practice}` in it. (The older name `insurance_path` still works too) |
| `ftps.pdf_path` | Yes | Remote folder for PT STMT files — keep `{practice}` in it |
| `sftp.host` | Only if `insurance+patient` | SFTP server address |
| `sftp.port` | No (defaults to 22) | SFTP port, only if not 22 |
| `sftp.username` | Only if `insurance+patient` | SFTP username |
| `sftp.password` | Only if `insurance+patient` | SFTP password |
| `sftp.patient_path` | Only if `insurance+patient` | Remote folder for patient files — keep `{practice}` in it |

4. Create the `Export1\` folder and a subfolder for each practice inside the entity folder

`{practice}` in any path gets automatically replaced with that practice's
folder name — so one `entity.json` can serve every practice as long as they
all follow the same folder-naming pattern on the server.

**Missing keys fix themselves:** on every run, the tool checks `entity.json`
for any keys it knows about that aren't in the file, adds them with a value
of `null`, and saves the file — so you never have to type key names by hand.
A `null` value means **"we don't use this — ignore it"**: any files that
would need that setting are left in place untouched (with a yellow notice in
the output), not treated as failures. To turn a feature on, just open
`entity.json` and replace the `null` with a real value in quotes.

**Editing JSON — the rules that bite:**
- Key names must match **exactly** (all lowercase, underscores): `host`, not
  `hostname`; `username`, not `user`; `tiff_path`, not `tiffpath`
- Every `"key": "value"` pair ends with a comma **except the last one in its
  block** — a missing or extra comma breaks the whole file
- Keep values inside straight double quotes (`"`). Avoid editing in Word or
  anything that turns quotes curly — use Notepad
- `port` numbers and `tls` true/false do **not** get quotes; everything else does

---

## `practice.json` — overriding one practice

If one practice needs a different login, a different folder path, or both:

1. Create a `practice.json` file inside that practice's folder (next to its
   date folders, e.g. `Entities\CAMBS\Export1\AJMAT\practice.json`)
2. Copy from `Entities\EXAMPLE\Export1\PRACTICE_NAME\practice.json`
3. Delete every line you *don't* need to change — keep only the fields that
   differ for this practice. Everything you delete falls back to `entity.json`.

The field names are the same ones as `entity.json`, wrapped in the same
`ftps` / `sftp` sections. A `practice.json` can also **add** a field that
`entity.json` doesn't have at all — e.g. if `entity.json` has no
`tiff_path` because every practice's path is different, each practice's
`practice.json` can carry its own.

Example — this practice only has its own PDF folder name, nothing else differs:

```json
{
  "ftps": {
    "pdf_path": "/{practice}/PDF_Statements_Custom"
  }
}
```

Example — this practice has its own SFTP login but uses the same paths as everyone else:

```json
{
  "sftp": {
    "username": "ajmat-sftp-user",
    "password": "ajmat-sftp-pass"
  }
}
```

Example — this practice has completely different remote folders for everything:

```json
{
  "ftps": {
    "tiff_path": "/SpecialFolder/Claims",
    "pdf_path": "/SpecialFolder/Statements"
  },
  "sftp": {
    "patient_path": "/SpecialFolder/Patients"
  }
}
```

Note: a `practice.json` never needs `workflow` or `insurance_file_type` —
those are entity-wide only.

---

## Something went wrong?

- **Files failed to upload** — they stay in the date folder. Fix the issue and run again.
- **A date folder is still there after running** — one or more files failed. Check the output window.
- **WinSCP error on startup** — run `setup.bat` again to re-download WinSCP.
- **`... upload not configured (ftps.xyz is null/blank) - N file(s) ignored`** —
  there are files to send, but the listed settings are still `null` (or blank)
  in that entity's `entity.json`. If those files *should* upload, open
  `entity.json` and fill in real values for the listed keys. If that upload
  type genuinely doesn't apply to this entity, the message is normal and the
  files just stay where they are.
- **`Added missing key(s) to entity.json ...`** — informational, not an error.
  The tool added keys that were absent (as `null`) so they're ready to fill in.
- **`No .pdf files in <date>\` but the folder isn't empty** — the files in the
  date folder are probably TIFs. Set `"insurance_file_type": "tif",` at the top
  level of that entity's `entity.json` (right under `workflow`).
- **The startup banner shows the wrong file type** — the banner line
  (`Entity: ... | File type: .pdf | ...`) shows what was actually read from
  `entity.json`, so it's the quickest way to confirm your edit took effect.